So Google now lets you control the max crawl rate through a nifty slider. While most sites start at a max-allowed rate of 0.5 pages/sec, I’ve been able to get a few sites to 10 pages/sec.
I had some large sites that had pretty low max rates (eg 1 page/sec), and it took me some figuring out how to get Google to go faster quicker - sitemap. By submitting a sitemap with a sizeable # of pages, we were able to boost from 1 page/sec to 2.5 pages/sec in roughly 96 hours.
Across all my sites Googlebot hits me at least a million times every day.
source From :http://www.techsoapbox.com/speeding-up-googles-crawl-rate/
0 comments:
Post a Comment