Recently, our web server was brought down because it was hit a great deal from Yahoo's Ad crawler all at once. It seems that the crawler hit close to 200 landing pages, simultaneously, several hundred times, in response to a very large upload of new keywords and ad groups to our Yahoo SEM account. For obvious reasons, the crawler was hitting the landing pages to acquire data about the keyword landing pages and to assign a quality score for those keywords. What’s unclear is why the heck the crawler hit us all once!?
I have never seen this happen with Google and I have done far larger uploads with Google AdWords than what was done in this particular instance with Yahoo. Yahoo's crawler seemed to be acting strange so I sent in a support ticket about this issue. The response I got back was that the Ad crawler was working as normal and that we should limit how much we should upload at a time. When questioning Yahoo support as to what would be a recommended or a suggested number of keywords to upload? Yahoo replied "there is not an estimate that we can safely suggest, as it largely depends on each servers bandwidth and capacity" Really...thanks for your help.
Well first let’s look at what I have learned so far? Not to upload thousands of keywords at one time with Yahoo. Yahoo's Ad crawler will take down our websites. When the crawler does take down our website or websites Yahoo is of little or no help in troubleshooting the problem. One more thing that I was able to discover, with Yahoo's assistance (kudos to them), was how to block the crawler from hitting our website. This is definitely a solution but definitely not a recommended one. I am positive that my quality scores for my keywords would be fairly high. I would like the minimum bids for those keywords to reflect that by having a low minimum bid. So blocking the crawler keeps my web server up, but will impact "Yahoo's ability to evaluate the relevance of your landing pages to your keywords and ads," this was taken straight out of yahoo's help page.
I am not really fond of taking the crawler out of the issue. Yahoo's crawler may have its problems, but it does seem to act faster in determining a decent quality score for the keywords. From my experience, Google's quality score is based by an overtime matrix. Yahoo seems to take it a step further and tries to get an immediate deserving quality score by using their crawler.
For now I have to find the balance between optimizing the number of keywords uploaded to Yahoo search marketing and keeping our server from being owned by the crawler. There are pros to having an instant quality score but the consistency seen in Google’s method is also commendable, if only these two companies were working together…