00 Votes

I can not see a webpage, Google can

Question by Guest | 2018-03-26 at 12:10

I can hardly believe it: There is a forum on the Internet, to which I have no access from my browser. Nevertheless, Google seems to be able to visit this forum and include the contents of it in its index. Because when I use Google, I see individual pages from this forum in Google's cache.

But how can that be? Google should not really have permission to look into the forum? What happens here? I would also like to browse through the forum without the detour via Google.

ReplyPositiveNegative
0Best Answer0 Votes

No fear, that is no witchcraft. If you visit a website with your browser, your browser will send some information. Among other things, it contains the so-called user agent string. This can look like this, for example:

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) 
Gecko/20100101 Firefox/23.0

This can be used by a website to recognize which browser you're using (here Mozilla Firefox version 23.0) or which operating system you use (here Windows 7 64-bit). If you want to know which User Agent String your browser currently sends, you can go to the page useragentstring.com and check it.

But now to your question with the forum. When Google visits a page, Google also sends such a user agent string. This can be for example something like this:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

This makes it possible for the operator of the forum to distinguish between Google is visiting the page or there is a normal person sitting in front of his computer and wants to access the site. If the operator now has in mind that all people who want to look at the forum, should also register in the forum, but Google should include the forum in its index, so that the forum is also found on search engines, the operator only has to do the following: If somebody visits the forum, the user agent is checked and it is displayed different content or nothing, depending on which user agent string is given.

You are welcome to do the test. With Firefox addons like "User Agent Quick Switch" or "User Agent Switcher" you can easily and quickly exchange the user agent string that you send with you. So you can also go on the Internet as a Google Bot and see your forum through Google's eyes. 

Incidentally, on the above page there is also a list of all possible user agent strings for crawlers, browsers, feed readers and more. But as you now know for sure: You must never trust a user agent string.
2018-03-27 at 11:18

ReplyPositive Negative
Reply

Related Topics

Google Tips

Tip | 0 Comments

Important Note

Please note: The contributions published on askingbox.com are contributions of users and should not substitute professional advice. They are not verified by independents and do not necessarily reflect the opinion of askingbox.com. Learn more.

Participate

Ask your own question or write your own article on askingbox.com. That’s how it’s done.