aio-libs / aioelasticsearch Goto Github PK

We can create a more understandable README. Take a look at this beautiful template from Billie Thompson https://gist.github.com/PurpleBooth/109311bb0361f32d87a2

https://github.com/elastic/elasticsearch-py/pull/628/files

https://github.com/elastic/elasticsearch-py/pull/411/files

elastic/elasticsearch-py#629

elastic/elasticsearch-py#924
"Since scan search-type is deprecated since 2.1, there's simpler implementation for scan logic."

https://github.com/elastic/elasticsearch-py/pull/321/files

https://github.com/elastic/elasticsearch-py/pull/794/files

elastic/elasticsearch-py#569

https://github.com/elastic/elasticsearch-py/pull/726/files

https://github.com/elastic/elasticsearch-py-async/pull/29/files

elastic/elasticsearch-py#635

elastic/elasticsearch-py@920218d

from elasticsearch.exceptions import *
place correct __ALL__
use in __init__.py and all internal modules import from .exceptions

Since it is not supported since aiohttp 3.0 and the minimum supported version is going to be 3.5.0, unnecessary python <= 3.5.3 compatibility code should be removed

https://github.com/elastic/elasticsearch-py/pull/506/files

aiohttp 2.3.0 introduced ClientConnectorSSLError and ClientConnectorCertificateError

they should be used instead of str(exc) hack

https://github.com/elastic/elasticsearch-py/pull/635/files

aioelasticsearch.AIOHttpConnection shouldn't call aiohttp.ClientSession.close in case of custom supplied session, otherwise it breaks all connections, cuz session is shared for all connections.

elastic/elasticsearch-py@d6fb953

elastic/elasticsearch-py@2d4bc06

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html

elastic/elasticsearch-py@a023bdd

elastic/elasticsearch-py#704 (comment)

doc_type was removed in elasticsearch 7.0.0.
tests requires related changes

https://github.com/elastic/elasticsearch-py-async

I am using this library in a concurrent environment but I frequently get timeout errors when there shouldn't be any.

I am instantiating the Elasticsearch object with maxsize=200 since it could be called by around 200 coroutines at any given time. The default value for the timeout is 10 seconds and should be plenty, but I increased it to 30 seconds and even if the timeouts diminish in frequency I still get a lot of them.

Do like this:

REPO_NAME={{ YOUR_REPO_NAME }}
travis encrypt -r "aio-libs/${REPO_NAME}" --api-endpoint 'https://api.travis-ci.com/'

Ref: https://github.com/orgs/aio-libs/teams/admins/discussions/9

elastic/elasticsearch-py#652

Information about what internal IO operation was interrupted by timeout expiring is important in some cases, it allows to figure out what exact IO takes too long time.

raise TimeoutError from None should be replaced with raise TimeoutError from exc

Hi,
from what i've testet this async es client is way slower than the sync version. So, did i make a mistake or is this behaviour expected?

Kind regrads

Now AIOHttpConnectionPool accepts and silently ignores all unknown parameters.
It leads to mysterious bugs when type spelling in argument name doesn't raise an exception.

If 'max_retries=0' is set and the request fails, the client goes into an infinite loop, resulting in a ddos elasticsearch.

In the synchronous version of the client, this case is handled correctly.

elastic/elasticsearch-py@d336c57

elastic/elasticsearch-py@966f689

In my app which uses aioelasticsearch, I sometimes get these errors but I can’t figure out if there is something wrong with my app or the lib. Specifically, how do I interpret what is causing it / what it means?

python:   File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 338, in perform_request
python:     ignore=ignore, timeout=timeout, headers=headers,
python:   File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 252, in _perform_request
python:     connection = await self.get_connection()
python:   File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 236, in get_connection
python:     return self.connection_pool.get_connection()
python:   File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/pool.py", line 96, in get_connection
python:     self.resurrect()
python:   File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/pool.py", line 82, in resurrect
python:     self.dead.put_nowait((timestamp, connection))
python:   File "/usr/lib/python3.6/asyncio/queues.py", line 150, in put_nowait
python:     self._put(item)
python:   File "/usr/lib/python3.6/asyncio/queues.py", line 231, in _put
python:     heappush(self._queue, item)
python: TypeError: '<' not supported between instances of 'AIOHttpConnection' and 'AIOHttpConnection'

elasticsearch-py requests based realization
http://elasticsearch-py.readthedocs.io/en/master/index.html?#running-on-aws-with-iam

Amazon docs:
https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

Currently there is no way to use custom separate session for each host.

We can solve this by allowing users to pass their own session factory class which will be called on each AIOHttpConnection initialisation if there is no explicit session provided.

https://github.com/aio-libs/aioelasticsearch/pull/211/files

Does this library support the ES Bulk helpers referenced here: http://elasticsearch-py.readthedocs.io/en/5.5.1/helpers.html#elasticsearch.helpers.bulk — if so, how would I go about importing it?

0.5.2 is the only release that require elasticsearch>=6

File "...../lib/python3.9/site-packages/aioelasticsearch/connection.py", line 61, in init
if StrictVersion(aiohttp.version).version < (3, 0):
File "/usr/lib/python3.9/distutils/version.py", line 40, in init
self.parse(vstring)
File "/usr/lib/python3.9/distutils/version.py", line 137, in parse
raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '3.7.4.post0'

aioelasticsearch==0.5.2
elasticsearch==2.4.1

AIOHttpConnection takes use_ssl argument and passes one to the parent's __init__.

Then self.use_ssl gets used instead of use_ssl here
https://github.com/aio-libs/aioelasticsearch/blob/master/aioelasticsearch/connection.py#L52

It works for the latest elasticsearch-py versions, however, it doesn't work with >2.0.0, b/c there's no self.use_ssl = use_ssl there.

The fix is straightforward - use use_ssl directly instead of self.use_ssl.