采集器有时候会抽风显示url无效 about liuli HOT 2 CLOSED

liuli_schedule | [2022:05:26 21:32:03] INFO Request <GET: https://weixin.sogou.com/weixin?type=1&query=新华社&ie=utf8&s_from=input&_sug_=n&_sug_type_=>
liuli_schedule | [2022:05:26 21:32:08] INFO Request <GET: https://weixin.sogou.com//link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS6qAisvrRisduaJeh6-pRKyl42nw8XwabFqXa8Fplpd9gheHHxrpT61ezPwhwlejmZxKfEP_2awl3HThwt3yfRXjYZHrwCAHalF7sIBt0tRA7mXvgNQYPtE0JJ45JE3n0AmPf2mzOFCd0oFXx9F-pgrgX97Fedt7FA6HpWKPjCk1hCO6gCPh6mu5UipNdBr4OekpPAUsErw7Qmfv0Csw0YEHpHkoPMgL3A..&type=1&query=%E6%96%B0%E5%8D%8E%E7%A4%BE&token=44447970DAF4217F1510C9FC220D7C1B1674E316628F8153>
liuli_schedule | [2022:05:26 21:32:13] INFO Request <GET: >
liuli_schedule | [2022:05:26 21:32:13] ERROR Request , Retry times: 1, Retry message: >
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule | [2022:05:26 21:32:13] INFO Request <GET: >
liuli_schedule | [2022:05:26 21:32:13] ERROR Request , Retry times: 2, Retry message: >
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule |
liuli_schedule | During handling of the above exception, another exception occurred:
liuli_schedule |
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule | [2022:05:26 21:32:13] INFO Request <GET: >
liuli_schedule | [2022:05:26 21:32:13] ERROR Request , Retry times: 3, Retry message: >
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule |
liuli_schedule | During handling of the above exception, another exception occurred:
liuli_schedule |
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule |
liuli_schedule | During handling of the above exception, another exception occurred:
liuli_schedule |
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 106, in fetch
liuli_schedule | resp = await self._make_request()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 186, in _make_request
liuli_schedule | resp = await request_func
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client.py", line 507, in _request
liuli_schedule | req = self._request_class(
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 305, in init
liuli_schedule | self.update_host(url)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 366, in update_host
liuli_schedule | raise InvalidURL(url)
liuli_schedule | aiohttp.client_exceptions.InvalidURL
liuli_schedule | [2022:05:26 21:32:13] INFO Request <GET: >
liuli_schedule | [2022:05:26 21:32:13] ERROR SGWechatSpider <Callback[parse_wechat]: 'NoneType' object is not callable
liuli_schedule | Traceback (most recent call last):
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/spider.py", line 403, in handle_request
liuli_schedule | callback_result, response = await request.fetch_callback(self.sem)
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/request.py", line 161, in fetch_callback
liuli_schedule | callback_result = await self.callback(response)
liuli_schedule | File "/data/code/src/collector/wechat/sg_ruia_start.py", line 84, in parse_wechat
liuli_schedule | html = await response.text()
liuli_schedule | File "/root/.local/share/virtualenvs/code-nY5aaahP/lib/python3.9/site-packages/ruia/response.py", line 151, in text
liuli_schedule | self._html = await self._aws_text(encoding=encoding, errors=errors)
liuli_schedule | TypeError: 'NoneType' object is not callable
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Stopping spider: SGWechatSpider
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Total requests: 2
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Time usage: 0:00:15.186402
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Spider finished!
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Spider started!
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Worker started: 139879009762416
liuli_schedule | [2022:05:26 21:32:13] INFO SGWechatSpider Worker started: 139879009762624
liuli_schedule | [2022:05:26 21:32:18] INFO Request <GET: https://weixin.sogou.com/weixin?type=1&query=新闻联播&ie=utf8&s_from=input&_sug_=n&_sug_type_=>
liuli_schedule | [2022:05:26 21:32:23] INFO Request <GET: https://weixin.sogou.com//link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS6qAisvrRisd3D2wW57UcOml42nw8XwabFqXa8Fplpd9vyp1djKdpuxsjfxycE3mbuk0jakC0IR-Pr3hw6HnDJM7TOg1rdkgBGuOBB3sFSIvT4Z_j1f_CyaDu9ZtSc9E4TWrG7NURYAPTPvRM0iLp5ikK83kERYvIMK-jXYf7TzTHC8G1TtvBydk3C9ghXUf7GUYEBtTUqp1c8hovyyv1pWjotuuiAmOIg..&type=1&query=%E6%96%B0%E9%97%BB%E8%81%94%E6%92%AD&token=4444ADA61639EDB1DBDF0430E3D800FDDBAC5AF8628F8162>
liuli_schedule | [2022:05:26 21:32:28] INFO Request <GET: http://mp.weixin.qq.com/s?src=11&timestamp=1653571938&ver=3822&signature=*E6CUv8jnG7h-i4tfEATUjtMhQExSNZ2bs11Ln5jkE80P8mHiAXiC10F9UtINomG77DCiHJYQQ804RptgHCoSuuNpw6XostD7Z5CVK7sPjNNEj2uvhNoHTMeskfjwcq6&new=1>
liuli_schedule | [2022:05:26 21:32:29] INFO Liuli 来自 liuli_wechat 的文章持久化成功! 👉 新闻联播: 退休人员基本养老金，上调4%

from liuli.

@hackdoors 这是因为你测试访问比较频繁，被搜狗流量监控了，需要验证，你控制频率即可

from liuli.