00 Votes

Googlebot Crawler comes too often: How can I limit the frequency?

Question by Chematik | 2012-08-11 at 19:27

I think, it's really good that Google has taken my website in the index, since I get most of my visitors through the Google search.

I think, it's also good, that Google is trying to keep its index up-to-date and therefore sent its Googlebot Crawler often through the Web.

However, for me, the visits of the crawler are a little too often. I have a web project with quite a lot pages and now, the Google crawler is responsible for 80 percent of my traffic, what is to me a little too much, as my pages change only little over the time.

How can I tell the crawler, that he need not come so often?

ReplyPositiveNegativeDateVotes
-2Best Answer2 Votes

You can do two things, that came to my mind spontaneously:

Robots-File

If you do not already have a robots.txt in the root directory of your site, attach a file and write in it a value for "Crawl-delay". If your site is named example.com, your file should be available on example.com/robots.txt. Your robots.txt might look like this:

User-agent: *
Crawl-delay: 30

The value indicates in seconds, how much time should there be between two visits of the crawler. The value of 30 means, that Google should only look every 30 seconds. Normally, Google visits your site within one second for several times, so that should be an improvement. However, it is rumored that not many crawlers follow these values.

Google Webmaster Tools

Another way is to use the Google Webmaster Tools. Here, in the settings there is the opportunity to customize the "crawl rate" for each website individually. I think this is the best solution for Google.
2012-08-13 at 10:48

ReplyPositive Negative
00 Votes

I really do not know, if it helps, but maybe there is also a possibility to work with the "Expires Header" and the caching times of your website.

If you set here, that a page will be valid for a whole year and can be loaded from the cache until then, perhaps also the search engines could follow it.

Does anyone know more about this? I would like to know more about this topic and whether it can work.
2012-08-12 at 23:09

ReplyPositive Negative
Reply

Related Topics

Firebird: SELECT with LIMIT

Question | 1 Answer

Important Note

Please note: The contributions published on askingbox.com are contributions of users and should not substitute professional advice. They are not verified by independents and do not necessarily reflect the opinion of askingbox.com. Learn more.

Participate

Ask your own question or write your own article on askingbox.com. That’s how it’s done.