Finding Google’s Stealth IPs (Part 1)
|
| |
 | |
Anyone that runs a cloaker site has thought, at least once “Why doesn’t Google just use IPs that don’t announce themselves?”
Well, they do. It’s my personal belief that they’re the IPs used to review spam complaints, as I’ve yet to block ALL of them, and my cloaker sites do fine. However, if given the chance to isolate these IPs, I’d be a fool not to.
So here I will cover how to find search engines TRUE ips, and how to isolate them and find which ones have been examining your site.
- Step 1: Getting your base IP
- Get the IP of a bot to start. Barring that, ping the search engine itself, and get the IP like that.
- We’ll start with crawl-66-249-70-41.googlebot.com which resolves to 66.249.70.41
- Step 2: Get the Responsible Internet Registry
- Go to dnsstuff.com, and do a whois lookup on the IP.
- The Admin handle will tell you who is in charge of assigning those IP blocks. In this case, the admin handle is ZG39-ARIN. So we can conclude it’s ARIN. Most Search Engines will be.
- Step 3: Get the Net Name
- Go to (in this case) ARIN.net, and search for the IP. It will give you the Net Name. In this case, it’s “GOOGLE”
- Step 4: Get Your IP List!
- Click on the Net Name. This will take you to a page with a bunch of IP ranges. Congrats, you have found their IPs!
- Google Inc. GOOGLE (NET-216-239-32-0-1) 216.239.32.0 - 216.239.63.255
Google Inc. GOOGLE (NET-64-233-160-0-1) 64.233.160.0 - 64.233.191.255
Google Inc. GOOGLE (NET-66-249-64-0-1) 66.249.64.0 - 66.249.95.255
Google Inc. GOOGLE (NET-72-14-192-0-1) 72.14.192.0 - 72.14.255.255
Google Inc. GOOGLE (NET-209-85-128-0-1) 209.85.128.0 - 209.85.255.255
Google Inc. GOOGLE (NET-74-125-0-0-1) 74.125.0.0 - 74.125.255.255
- Step 5: Repeat the Process with new IPs
- Using a function I’ll give you later, you can take a list of associated IPs(hint: find their website IPs, and the mail servers. In the case of Google, the datacenters are known, so use all those.
- Alternatively: Just search the ARIN base for the company’s name. A search for “Google” reveals a lot more results. Ours, for example, revealed:
Google Inc. GOOGLE (NET-216-239-32-0-1) 216.239.32.0 - 216.239.63.255
Google Inc. GOOGLE (NET-64-233-160-0-1) 64.233.160.0 - 64.233.191.255
Google Inc. GOOGLE (NET-66-249-64-0-1) 66.249.64.0 - 66.249.95.255
Google Inc. GOOGLE (NET-72-14-192-0-1) 72.14.192.0 - 72.14.255.255
Google Inc. GOOGLE (NET-209-85-128-0-1) 209.85.128.0 - 209.85.255.255
Google Inc. GOOGLE (NET-74-125-0-0-1) 74.125.0.0 - 74.125.255.255
Google Inc. GOOGLE (NET-216-239-32-0-1) 216.239.32.0 - 216.239.63.255
Google Inc. EC12-1-GOOGLE (NET-64-68-80-0-1) 64.68.80.0 - 64.68.87.255
Google Inc. GOOGLE-2 (NET-66-102-0-0-1) 66.102.0.0 - 66.102.15.255
Google Inc. GOOGLE (NET-64-233-160-0-1) 64.233.160.0 - 64.233.191.255
Google Inc. GOOGLE (NET-66-249-64-0-1) 66.249.64.0 - 66.249.95.255
Google Inc. GOOGLE (NET-72-14-192-0-1) 72.14.192.0 - 72.14.255.255
Google Inc. GOOGLE-IPV6 (NET6-2001-4860-1) 2001:4860:0000:0000:0000:0000:0000:0000 - 2001:4860:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
Google Inc. GOOGLE (NET-209-85-128-0-1) 209.85.128.0 - 209.85.255.255
Google Inc. SAVV-S232078-1 (NET-216-33-229-144-1) 216.33.229.144 - 216.33.229.151
Google Inc. SAVV-S232078-2 (NET-216-33-229-160-1) 216.33.229.160 - 216.33.229.167
Google Inc. SAVV-S232078-3 (NET-209-185-108-128-1) 209.185.108.128 - 209.185.108.255
Google Inc. SAVV-S232078-11 (NET-216-109-75-80-1) 216.109.75.80 - 216.109.75.95
Google Inc. SAVV-S232078-26 (NET-64-68-88-0-1) 64.68.88.0 - 64.68.95.255
Google Inc. SAVV-S232078-24 (NET-64-68-64-64-1) 64.68.64.64 - 64.68.64.127
Google Inc. SAVV-S232078-21 (NET-64-41-221-192-1) 64.41.221.192 - 64.41.221.207
Google Inc. GOOGLE (NET-74-125-0-0-1) 74.125.0.0 - 74.125.255.255
Google Incorporated LVLT-GOOGL-1-209-247-159-144 (NET-209-247-159-144-1) 209.247.159.144 - 209.247.159.159
Google Incorporated LVLT-GOOGL-1-63-211-200-72 (NET-63-211-200-72-1) 63.211.200.72 - 63.211.200.79
Google Incorporated LVLT-GOOGL-1-209-245-184-136 (NET-209-245-184-136-1) 209.245.184.136 - 209.245.184.143
Google Incorporated LVLT-GOOGL-1-166-90-148-64 (NET-166-90-148-64-1) 166.90.148.64 - 166.90.148.79
Google Incorporated LVLT-GOOGL-1-8-6-48 (NET-8-6-48-0-1) 8.6.48.0 - 8.6.55.255
Google Incorporated LVLT-GOOGL-1-64-154-178-208 (NET-64-154-178-208-1) 64.154.178.208 - 64.154.178.223
GOOGLE ABOV-T324-64-124-112-24-29 (NET-64-124-112-24-1) 64.124.112.24 - 64.124.112.31
GOOGLE ABOV-T324-209-249-73-64-29 (NET-209-249-73-64-1) 209.249.73.64 - 209.249.73.71
GOOGLE ABOV-T324-64-124-229-168-29 (NET-64-124-229-168-1) 64.124.229.168 - 64.124.229.175
GOOGLE UU-65-214-255-96 (NET-65-214-255-96-1) 65.214.255.96 - 65.214.255.111
Google UU-65-245-24-8 (NET-65-245-24-8-1) 65.245.24.8 - 65.245.24.15
GOOGLE UU-65-211-194-96-D8 (NET-65-211-194-96-1) 65.211.194.96 - 65.211.194.111
GOOGLE UU-65-223-8-48-D6 (NET-65-223-8-48-1) 65.223.8.48 - 65.223.8.63
Google UU-65-221-133-176-D6 (NET-65-221-133-176-1) 65.221.133.176 - 65.221.133.191
GOOGLE UU-63-84-190-224-D4 (NET-63-84-190-224-1) 63.84.190.224 - 63.84.190.255
Google TWTC-GOOGLE-01 (NET-64-128-207-160-1) 64.128.207.160 - 64.128.207.175
GOOGLE UU-65-196-235-32-D4 (NET-65-196-235-32-1) 65.196.235.32 - 65.196.235.47
Google TWTC-ATLA-C-GOOGLE-0 (NET-66-192-134-32-1) 66.192.134.32 - 66.192.134.47
GOOGLE UU-65-214-112-96-D21 (NET-65-214-112-96-1) 65.214.112.96 - 65.214.112.127
GOOGLE UU-65-210-56-208-D5 (NET-65-210-56-208-1) 65.210.56.208 - 65.210.56.223
GOOGLE UU-65-204-68-160-D8 (NET-65-204-68-160-1) 65.204.68.160 - 65.204.68.175
GOOGLE CHILDREN CARE-050119015545 SBC06923603306429050119015554 (NET-69-236-33-64-1) 69.236.33.64 - 69.236.33.71
GOOGLE CHILDREN CARE-050119015545.568301 SBC-76-242-87-248-29-0709182611 (NET-76-242-87-248-1) 76.242.87.248 - 76.242.87.255
Google Corporate IT GOOGLE-CORP-REMOTES (NET-72-14-224-0-1) 72.14.224.0 - 72.14.231.255
Google Inc SBC067126100008030728 (NET-67-126-100-8-1) 67.126.100.8 - 67.126.100.15
Google Inc GOO20050928-CA (NET-67-69-26-16-1) 67.69.26.16 - 67.69.26.23
GOOGLE INC 216-235-136-72-29 (NET-216-235-136-72-1) 216.235.136.72 - 216.235.136.79
Google Inc (Hybrid) NET-GOOGINC004 (NET-206-186-136-192-1) 206.186.136.192 - 206.186.136.255
GOOGLE INC-040731031303 SBC06922402120829040731031306 (NET-69-224-21-208-1) 69.224.21.208 - 69.224.21.215
GOOGLE INC-040731032731 SBC06922403108829040731032734 (NET-69-224-31-88-1) 69.224.31.88 - 69.224.31.95
GOOGLE INC-040731032750 SBC06922403110429040731032753 (NET-69-224-31-104-1) 69.224.31.104 - 69.224.31.111
GOOGLE INC-041208041250 SBC06922807021629041208041255 (NET-69-228-70-216-1) 69.228.70.216 - 69.228.70.223
GOOGLE INC-041208041841 SBC06922807022429041208041844 (NET-69-228-70-224-1) 69.228.70.224 - 69.228.70.231
GOOGLE INC-041208042600 SBC06922807023229041208042603 (NET-69-228-70-232-1) 69.228.70.232 - 69.228.70.239
GOOGLE INC-041209044712 SBC06922807024829041209044715 (NET-69-228-70-248-1) 69.228.70.248 - 69.228.70.255
GOOGLE INC-041210150104 SBC06922807610429041210150108 (NET-69-228-76-104-1) 69.228.76.104 - 69.228.76.111
GOOGLE INC-050209021649 SBC06923712022429050209021705 (NET-69-237-120-224-1) 69.237.120.224 - 69.237.120.231
GOOGLE INC-050527011628 SBC07113003422429050527011636 (NET-71-130-34-224-1) 71.130.34.224 - 71.130.34.231
GOOGLE INC-050527011810 SBC07113003423229050527011818 (NET-71-130-34-232-1) 71.130.34.232 - 71.130.34.239
GOOGLE INC-050527011817 SBC07113003424029050527011827 (NET-71-130-34-240-1) 71.130.34.240 - 71.130.34.247
GOOGLE INC-050527013743 SBC07113010300829050527013746 (NET-71-130-103-8-1) 71.130.103.8 - 71.130.103.15
GOOGLE INC-050527013913 SBC07113010301629050527013916 (NET-71-130-103-16-1) 71.130.103.16 - 71.130.103.23
GOOGLE INC-050527013926 SBC07113010302429050527013929 (NET-71-130-103-24-1) 71.130.103.24 - 71.130.103.31
GOOGLE INC-050527014231 SBC07113010303229050527014236 (NET-71-130-103-32-1) 71.130.103.32 - 71.130.103.39
GOOGLE INC-050527014403 SBC07113010304029050527014409 (NET-71-130-103-40-1) 71.130.103.40 - 71.130.103.47
GOOGLE INC-060616081951 SBC07501704820029060616082030 (NET-75-17-48-200-1) 75.17.48.200 - 75.17.48.207
GOOGLE INC-060711235135 SBC07502305718429060711235200 (NET-75-23-57-184-1) 75.23.57.184 - 75.23.57.191
GOOGLE INC-060829070127 SBC07503725318429060829070142 (NET-75-37-253-184-1) 75.37.253.184 - 75.37.253.191
GOOGLE INC-060914232236 SBC07503018920029060914232252 (NET-75-30-189-200-1) 75.30.189.200 - 75.30.189.207
GOOGLE INC-060919152859 SBC07504214214429060919152927 (NET-75-42-142-144-1) 75.42.142.144 - 75.42.142.151
GOOGLE INC-061122060029 SBC07505224803229061122060126 (NET-75-52-248-32-1) 75.52.248.32 - 75.52.248.39
GOOGLE INC-061122061203 SBC07505214212829061122061243 (NET-75-52-142-128-1) 75.52.142.128 - 75.52.142.135
GOOGLE INC-061203180838 SBC07505224420829061203180849 (NET-75-52-244-208-1) 75.52.244.208 - 75.52.244.215
GOOGLE INC-061207060345 SBC07505224605629061207060425 (NET-75-52-246-56-1) 75.52.246.56 - 75.52.246.63
GOOGLE INC-061207060713 SBC07505224606429061207060755 (NET-75-52-246-64-1) 75.52.246.64 - 75.52.246.71
GOOGLE INC-061207060811 SBC07505224607229061207060842 (NET-75-52-246-72-1) 75.52.246.72 - 75.52.246.79
GOOGLE INC-061207061127 SBC07505224608029061207061207 (NET-75-52-246-80-1) 75.52.246.80 - 75.52.246.87
GOOGLE INC-070327203037 SBC-76-200-97-96-29-0703273044 (NET-76-200-97-96-1) 76.200.97.96 - 76.200.97.103
GOOGLE INC-070509182733 SBC-76-220-105-184-29-0705092742 (NET-76-220-105-184-1) 76.220.105.184 - 76.220.105.191
Google Inc-070816111718 SBC-76-246-222-104-29-0708161729 (NET-76-246-222-104-1) 76.246.222.104 - 76.246.222.111
GOOGLE INC-071011143940 SBC-99-163-5-80-29-0710113951 (NET-99-163-5-80-1) 99.163.5.80 - 99.163.5.87
Google Inc10988888 SBC06911114115229040325120125 (NET-69-111-141-152-1) 69.111.141.152 - 69.111.141.159
Google Inc10988957 SBC06911114116029040325120238 (NET-69-111-141-160-1) 69.111.141.160 - 69.111.141.167 - Most places with have more than one ID for Arin, so and a dump of “Google” reveals several new ones. Including ones for their child care. OH YEAH. And their IT Blocks ;). If the list is too large, chances are, ARIN truncated the results, and you may need to investigate further.
Ok, awesome. So now we have our IP ranges. Dump all of your visitor IPs (from every site)that were NOT marked as bots to a text file. Insert all IPs we got from our ARIN lookup into a database in the following format:
(NOTE: use ip2long() in php, and store that value)
ID(auto increment), start_ip, end_ip. I believe start_ip and end_ip should be big ints, or what is mysql’s long equivalent? I’ll look it up for part2.
If you haven’t figured out how to use this yet, stay tuned for Part2 (Coming as soon as I shower, and get my booty to class)
January 28th, 2008 at 5:28 pm
Corrected version of the above post..
It is possible to find all google IP adresses because to be reachable they need to declare it trough the BGP routing protcol under their Autonomous System Number which is 15169 , and tadaam, you have their ranges ….
If they are really spooky, they are maybe using the IP of their acquisitions:
Youtube ranges , Postini ranges, etc…
January 29th, 2008 at 4:58 am
A late post on this topic but I just assumed Google would be regularly setting up and rotating external VPS servers on 3rd party ISPs running Squid Web Proxy Cache or something similar (they probably have their own software).
The web cache would be accessible to Google staff/bots with an anonymous external facing IP.
So its technically very easy for them to hide their origin IP when they choose to. Just a case of avoiding drawing attention to your site as much as you can so they dont feel the need to look in the first place.
June 24th, 2008 at 1:56 pm
I’m a bit late to this post but still this is a very good post. Thanks shady
July 3rd, 2008 at 4:25 am
@Jabber: some solutions are (if you are really paranoid):
1) invest in a maxmind ISP IP ranges database (15$ + 3$/month for updates)
2) the free DUHL DNSBL from SORBS @ http://www.us.sorbs.net/faq/dul.shtml
If the visitor do not come from one of those ranges, consider it as spider.
I would recommend to log the IP/AS number of the ranges you disallow and then do a list by ASN sorted by count of disallowed access so that you can white-list them after some verifications that it is a consumer range.