liushuaikobe / gitarchiveutils Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 1.0 1.38 MB

Utils for handling and parsing the data from http://www.githubarchive.org/

License: GNU General Public License v2.0

Python 10.37% Shell 0.10% CSS 41.02% JavaScript 48.51%

gitarchiveutils's People

Contributors

Watchers

Forkers

pokerg

gitarchiveutils's Issues

Event的Payload的计数

有的一个Event中是用户在本地提交了好几次之后，才push到远程。
因此应该考虑Payload中的数量。

将项目中print替换为logging模块

调用Geoname service时，次数有限制

目前采用的处理方法是，如果出异常，就更换一个账号，sleep一分钟之后重新尝试连接。
但是出现异常的那一批协程所处理的数据可能会丢失。可能跟requests库的内部实现有关系，之后肯定要在完善一下。

更换Geonames的用户名时，config中的未更换

处理下一个文件时会重新从config里面读取，可暂时忽略这个缺陷。

将读取数据的模块也单独提出来

有以下几种情况：

每日正常的执行任务时读取
异常
- 下载未成功，人为指定一个记录文件的目录
- 规范化时出异常，人为执行从数据库中读取记录

地名缓存需要判断缓存的大小，自动处理，防止内存占用过多

Geo转换时，同一批协程中，相同的地名只请求一次

完善地名缓存，可以保存到DB
ConnectionError: HTTPConnectionPool(host='api.geonames.org', port=80): Max retries exceeded with url: /searchJSON?username=liushuaikobe&q=K%C3%B6ln%2C+Germany&maxRows=5 (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)

V2.0 Requests异常捕获错误

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/liushuai/git/PythonProject/GitArchiveUtils/daily-task/normalize.py", line 153, in search
    r = requests.get('http://api.geonames.org/searchJSON', params=params)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 354, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='api.geonames.org', port=80): Max retries exceeded with url: /searchJSON?username=liushuaikobe&q=The+Moon.+Also+London.&maxRows=1 (Caused by <class 'socket.error'>: [Errno 60] Operation timed out)
<Greenlet at 0x101adb730: <bound method Normalizer.search of <normalize.Normalizer object at 0x101afc390>>('The Moon. Also London.')> failed with ConnectionError