fangpsh's blog

部分搜索引擎爬虫IP

有个小项目套了一个雷池WAF，担心会影响到搜索引擎的爬虫，需要单独维护一个IP组，社区关联issue：[建议] 内置各大搜索引擎蜘蛛爬虫的 IP 库 #374。

360蜘蛛IP段

来源：https://www.so.com/help/spider_ip.html

180.153.232.0/24
180.153.234.0/24
180.153.236.0/24
180.163.220.0/24
42.236.101.0/24
42.236.102.0/24
42.236.103.0/24
42.236.10.0/24
42.236.12.0/24
42.236.13.0/24
42.236.14.0/24
42.236.15.0/24
42.236.16.0/24
42.236.17.0/24
42.236.46.0/24
42.236.48.0/24
42.236.49.0/24
42.236.50.0/24
42.236.51.0/24
42.236.52.0/24
42.236.53.0/24
42.236.54.0/24
42.236.55.0/24
42.236.99.0/24

Google段

来源：

  1. https://developers.google.com/search/apis/ipranges/googlebot.json
  2. https://www.gstatic.com/ipranges/goog.json
104.154.0.0/15
104.196.0.0/14
104.237.160.0/19
107.167.160.0/19
107.178.192.0/18
108.170.192.0/18
108.177.0.0/17
108.59.80.0/20
130.211.0.0/16
142.250.0.0/15
146.148.0.0/17
162.216.148.0/22
162.222.176.0/21
172.110.32.0/21
172.217.0.0/16
172.253.0.0/16
173.194.0.0/16
173.255.112.0/20
192.158.28.0/22
192.178.0.0/15
192.178.5.0/27
193.186.4.0/24
199.192.112.0/22
199.223.232.0/21
199.36.154.0/23
199.36.156.0/24
207.223.160.0/20
208.117.224.0/19
208.65.152.0/22
208.68.108.0/22
208.81.188.0/22
209.85.128.0/17
216.239.32.0/19
216.58.192.0/19
216.73.80.0/20
23.236.48.0/20
23.251.128.0/19
34.0.0.0/15
34.100.182.96/28
34.101.50.144/28
34.118.254.0/28
34.118.66.0/28
34.126.178.96/28
34.128.0.0/10
34.146.150.144/28
34.147.110.144/28
34.151.74.144/28
34.152.50.64/28
34.154.114.144/28
34.155.98.32/28
34.16.0.0/12
34.165.18.176/28
34.175.160.64/28
34.176.130.16/28
34.2.0.0/16
34.22.85.0/27
34.3.0.0/23
34.3.128.0/17
34.3.16.0/20
34.3.3.0/24
34.3.32.0/19
34.3.4.0/24
34.3.64.0/18
34.3.8.0/21
34.32.0.0/11
34.4.0.0/14
34.64.0.0/10
34.64.82.64/28
34.65.242.112/28
34.8.0.0/13
34.80.50.80/28
34.88.194.0/28
34.89.10.80/28
34.89.198.80/28
34.96.162.48/28
35.184.0.0/13
35.192.0.0/14
35.196.0.0/15
35.198.0.0/16
35.199.0.0/17
35.199.128.0/18
35.200.0.0/13
35.208.0.0/12
35.224.0.0/12
35.240.0.0/13
35.247.243.240/28
64.15.112.0/20
64.233.160.0/19
66.102.0.0/20
66.22.228.0/23
66.249.64.0/19
66.249.64.0/27
66.249.64.128/27
66.249.64.160/27
66.249.64.192/27
66.249.64.224/27
66.249.64.32/27
66.249.64.64/27
66.249.64.96/27
66.249.65.0/27
66.249.65.160/27
66.249.65.192/27
66.249.65.224/27
66.249.65.32/27
66.249.65.64/27
66.249.65.96/27
66.249.66.0/27
66.249.66.128/27
66.249.66.160/27
66.249.66.192/27
66.249.66.32/27
66.249.66.64/27
66.249.66.96/27
66.249.68.0/27
66.249.68.32/27
66.249.68.64/27
66.249.69.0/27
66.249.69.128/27
66.249.69.160/27
66.249.69.192/27
66.249.69.224/27
66.249.69.32/27
66.249.69.64/27
66.249.69.96/27
66.249.70.0/27
66.249.70.128/27
66.249.70.160/27
66.249.70.192/27
66.249.70.224/27
66.249.70.32/27
66.249.70.64/27
66.249.70.96/27
66.249.71.0/27
66.249.71.128/27
66.249.71.160/27
66.249.71.192/27
66.249.71.224/27
66.249.71.32/27
66.249.71.64/27
66.249.71.96/27
66.249.72.0/27
66.249.72.128/27
66.249.72.160/27
66.249.72.192/27
66.249.72.224/27
66.249.72.32/27
66.249.72.64/27
66.249.72.96/27
66.249.73.0/27
66.249.73.128/27
66.249.73.160/27
66.249.73.192/27
66.249.73.224/27
66.249.73.32/27
66.249.73.64/27
66.249.73.96/27
66.249.74.0/27
66.249.74.128/27
66.249.74.32/27
66.249.74.64/27
66.249.74.96/27
66.249.75.0/27
66.249.75.128/27
66.249.75.160/27
66.249.75.192/27
66.249.75.224/27
66.249.75.32/27
66.249.75.64/27
66.249.75.96/27
66.249.76.0/27
66.249.76.128/27
66.249.76.160/27
66.249.76.192/27
66.249.76.224/27
66.249.76.32/27
66.249.76.64/27
66.249.76.96/27
66.249.77.0/27
66.249.77.128/27
66.249.77.160/27
66.249.77.192/27
66.249.77.224/27
66.249.77.32/27
66.249.77.64/27
66.249.77.96/27
66.249.78.0/27
66.249.78.32/27
66.249.79.0/27
66.249.79.128/27
66.249.79.160/27
66.249.79.192/27
66.249.79.224/27
66.249.79.32/27
66.249.79.64/27
66.249.79.96/27
70.32.128.0/19
72.14.192.0/18
74.125.0.0/16
8.34.208.0/20
8.35.192.0/20
8.8.4.0/24
8.8.8.0/24

百度

从各个网站帖子中，找到传说中的百度蜘蛛爬虫段，整理去重后，使用host {ip} 反向查询PTR记录，留下有明确指明是百度的。

host 220.181.108.75
75.108.181.220.IN-ADDR.ARPA domain name pointer baiduspider-220-181-108-75.crawl.baidu.com.

116.179.32.103
116.179.32.107
116.179.32.12
116.179.32.134
116.179.32.149
116.179.32.152
116.179.32.162
116.179.32.17
116.179.32.18
116.179.32.218
116.179.32.241
116.179.32.39
116.179.32.99
116.179.37.122
116.179.37.173
116.179.37.202
116.179.37.225
116.179.37.242
116.179.37.36
116.179.37.44
116.179.37.79
116.179.37.84
116.179.37.99
119.63.198.104
119.63.198.69
119.63.198.85
123.125.71.102
123.125.71.106
123.125.71.111
123.125.71.21
123.125.71.24
123.125.71.26
123.125.71.30
123.125.71.33
123.125.71.36
123.125.71.38
123.125.71.40
123.125.71.45
123.125.71.48
123.125.71.52
123.125.71.73
123.125.71.74
123.125.71.75
123.125.71.77
123.125.71.90
123.125.71.96
123.125.71.97
123.125.71.98
220.181.108.100
220.181.108.102
220.181.108.107
220.181.108.108
220.181.108.109
220.181.108.110
220.181.108.111
220.181.108.112
220.181.108.142
220.181.108.144
220.181.108.146
220.181.108.147
220.181.108.156
220.181.108.162
220.181.108.168
220.181.108.178
220.181.108.179
220.181.108.183
220.181.108.184
220.181.108.81
220.181.108.93

微软Bing

13.66.139.0
13.66.139.102
13.66.139.128
13.66.139.43
13.66.139.79
13.66.139.83
13.66.139.98
131.253.24.10
131.253.24.2
131.253.24.7
157.55.116.65
157.55.116.71
157.55.116.79
157.55.116.82
157.55.36.139
157.55.39.111
157.55.39.125
157.55.39.129
157.55.39.136
157.55.39.149
157.55.39.150
157.55.39.166
157.55.39.172
157.55.39.176
157.55.39.198
157.55.39.201
157.55.39.202
157.55.39.205
157.55.39.219
157.55.39.227
157.55.39.237
157.55.39.245
157.55.39.28
157.55.39.32
157.55.39.39
157.55.39.43
157.55.39.5
157.55.39.51
157.55.39.58
157.55.39.60
157.55.39.7
157.55.39.81
157.55.39.82
157.55.39.86
157.55.39.9
157.55.39.98
157.56.0.190
157.56.0.207
157.56.1.64
157.56.2.60
157.56.2.63
157.56.4.187
157.56.5.147
157.56.92.147
199.30.16.12
199.30.16.16
199.30.16.189
199.30.16.42
199.30.20.128
199.30.20.201
199.30.20.44
199.30.20.67
199.30.26.160
199.30.28.137
207.46.13.102
207.46.13.104
207.46.13.107
207.46.13.111
207.46.13.114
207.46.13.115
207.46.13.12
207.46.13.129
207.46.13.134
207.46.13.141
207.46.13.147
207.46.13.149
207.46.13.154
207.46.13.168
207.46.13.177
207.46.13.215
207.46.13.217
207.46.13.218
207.46.13.228
207.46.13.236
207.46.13.237
207.46.13.249
207.46.13.250
207.46.13.3
207.46.13.36
207.46.13.4
207.46.13.40
207.46.13.51
207.46.13.54
207.46.13.62
207.46.13.64
207.46.13.69
207.46.13.77
207.46.13.79
207.46.13.83
207.46.13.91
207.46.13.93
207.46.193.69
207.46.193.73
207.46.193.75
207.46.193.77
207.46.193.89
207.46.193.94
207.46.195.116
207.46.195.117
207.46.195.118
40.77.167.116
40.77.167.126
40.77.167.20
40.77.167.211
40.77.167.241
40.77.167.247
40.77.167.26
40.77.167.30
40.77.167.45
40.77.167.64
40.77.188.101
40.77.188.241
40.77.189.16
40.77.189.178
40.77.189.234
40.77.189.249
40.77.190.104
40.77.190.158
40.77.190.55
40.77.190.70
65.52.104.213
65.52.104.214
65.52.104.215
65.52.104.216
65.55.208.114
65.55.208.118
65.55.208.147
65.55.208.16
65.55.208.196
65.55.208.210
65.55.208.88
65.55.209.1
65.55.209.197
65.55.209.198
65.55.209.199
65.55.209.200
65.55.209.201
65.55.209.202
65.55.209.203
65.55.209.204
65.55.209.205
65.55.210.247
65.55.210.45
65.55.211.194
65.55.212.97
65.55.213.193
65.55.213.241
65.55.213.243
65.55.213.244
65.55.213.245
65.55.213.7
65.55.214.149
65.55.215.111
65.55.215.212
65.55.215.213
65.55.215.67
65.55.219.178
65.55.219.91
65.55.3.170

其他更多搜索引擎（雅虎、搜狗等）可以参考：https://www.pdflibr.com/crawler/list/search-engine

自己再调用host 二次确认下即可，这个网站还是挺准的。上线一个月，运营人员说SEO没异常。