Comments (9)
Issue-Label Bot is automatically applying the label bug
to this issue, with a confidence of 0.66. Please mark this comment with π or π to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
from etcd3-py.
Since this lib is using grpc-gateway to make request, It's not likely able to set the grpc option.
But maybe you can paginate by using the key
& range_end
parameter.
For example:
range('/foo/', prefix=True)
# Is equal to
range('/foo/a', range_end='/foo/b')
range('/foo/b', range_end='/foo/c')
range('/foo/d, range_end='/foo/e')
...
range('/foo/y', range_end='/foo/z')
from etcd3-py.
Apparently the suggested way to do paging is with limit
and range_end
. Something like:
limit = 1000
range_end = incr_last_byte(prefix)
prefix_start = prefix
done = False
result = []
while not done:
subrange = range(prefix_start, range_end=range_end, limit=limit)
result.append(subrange.kvs[:-1])
done = prefix_start == subrange.kvs[-1].key
prefix_start = subrange.kvs[-1].key
return result
which grabs 1000 entries at a time until range_end. It should also use the same revision as the first result, probably. The more I think about this, the more I think hat range()
should wrap all this up.
I'll see if I can work up a PR.
from etcd3-py.
Yeah, maybe we can add a new stateful util, named like "ranger util"
Thank you so much for contribution.
from etcd3-py.
Here's what I ended up with. I can't quite figure out where to put it, so feel free to put it where you think it makes the most sense. I tried to keep as much of the normal range()
functionality as I could:
async def big_range(
kvapi,
key=None,
range_end=None,
limit=1000,
revision=None,
keys_only=False,
serializable=False,
min_mod_revision=None,
max_mod_revision=None,
min_create_revision=None,
max_create_revision=None,
sort_order=models.RangeRequestSortOrder.NONE,
sort_target=models.RangeRequestSortTarget.KEY,
prefix=False,
all=False
):
method = '/kv/range'
if all:
key = range_end = '\0'
if prefix:
range_end = incr_last_byte(key)
data = {
"key": key,
"range_end": range_end,
"limit": limit,
"revision": revision,
"sort_order": sort_order,
"sort_target": sort_target,
"serializable": serializable,
"keys_only": keys_only,
"count_only": None,
"min_mod_revision": min_mod_revision,
"max_mod_revision": max_mod_revision,
"min_create_revision": min_create_revision,
"max_create_revision": max_create_revision
}
data = {k: v for k, v in data.items() if v is not None}
result = subrange = await kvapi.call_rpc(method, data=data)
if not result.kvs: return result
subrange = result
while len(subrange.kvs) == limit:
data['key'] = subrange.kvs[-1].key
subrange = await kvapi.call_rpc(method, data=data)
result.kvs.extend(subrange.kvs[1:])
if 'revision' not in data:
data['revision'] = subrange.header.revision
return result
from etcd3-py.
I prefer adding a new stateful util.
Cause you have to make multiple request in during the process which means you must have states to tell what page you are at. (stateful)
And result.kvs.extend(subrange.kvs[1:])
is very not recommended.
Having to do a "big range" means the response must be very big, maybe too big to put them all in memory.
I suggest this:
class Ranger:
def __init__(...):
self.start
self.current = start
self.buf
self.end
self.limit
self.finished
def range() -> returns a iterator
while current < end:
buf = client.range(current, end, limit)
if not buf.kvs:
break
for pair in buf.kvs:
self.current = pair.key
yield pair
current += 1
self.finished = true
from etcd3-py.
I was planning to write some higher level apis too, like get, get_prefix, delete_prefix, get_keys.
You can, if you like, just stick to writing a function and put them in a file like /etcd3/apis/kv_extended.py
Don't for get to test them on both py2 and py3
from etcd3-py.
One issue you're going to run into is writing them to work with both sync and async methods.
from etcd3-py.
yeah, it's impossible to make a function both sync and async while it makes multiple requests brefore return.
so i prefer to write two stateful util (async and sync)
and you can make them accessible from a same method
Client.Ranger
and AioClient.Ranger
from etcd3-py.
Related Issues (20)
- setting a watch fails with AttributeError: 'ModelizedStreamResponse' object has no attribute 'raw' HOT 10
- AttributeError: 'Client' object has no attribute 'server_version_sem' HOT 2
- ErrUserEmpty error:'etcdserver: user name is empty, code:3' HOT 6
- self cancelling watcher HOT 8
- Are you aware of https://github.com/dims/etcd3-gateway ? HOT 9
- the AioClient has an exceptiion with call range() function HOT 2
- Unclosed client session HOT 3
- Package name collision when installing it with pypi HOT 3
- Does long running lead to high loads with βWatcher() APIβ HOT 4
- Timeout HOT 2
- lease instance can not be awaited HOT 4
- use Lock in python 2.6 will cause "error: release unlocked lock" problem HOT 2
- when connect etcd-3.4.7, etcd3-py dont work HOT 2
- Install fails using pip>=20.1 HOT 3
- Swagger cache fills itself up without boundaries HOT 3
- Implement `get` function in KVAPI class
- I've got no time to maintain this lib HOT 1
- AttributeError: 'AioClient' object has no attribute 'server_version_sem' My Network Problems? Why?
- 0.3s is too short to get etcd version for my scenario and always timeout.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from etcd3-py.