btrdb / btrdb-python Goto Github PK

https://packaging.python.org/tutorials/distributing-packages/

The colls parameter is apparently used for pagination.

def listCollections(self, prefix):
        colls = (prefix,)
        maximum = 10
        got = maximum
        while got == maximum:
            startingAt = colls[-1]
            colls = self.ep.listCollections(prefix, colls, maximum)
            for coll in colls:
                yield coll
            got = len(colls)

However, Python always throws a Type error

TypeError: ('blahblahblah',) has type <class 'tuple'>, but expected one of: (<class 'bytes'>, <class 'str'>) for field ListCollectionsParams.startWith

This issue should be debated.

The method is refreshMeta() whereas the GoLang version is RefreshMeta(). In Python, this should be refresh_meta(). It would be "more pythonic" to follow the formatting standards. Should the python library conform to python formatting standards or try to match the Go Lang version?

There are some other issues with spacing that are probably just being a bit nitpicky.

In general, handling nanoseconds in Python is not terribly pleasant. It would make sense to create some tooling or helper classes around this issue. As time stamps in nanoseconds are incredibly large, it is easy to screw up the order of magnitude?

Do we use datetime? Time? Arrow? Pandas?

Some possible helper functions are below.

def ConvertDateTimeToEpoch_ns(dt):
    """
    converts the datetime object to epoch time in nanoseconds
    """
    
    return (int(dt.strftime('%s'))*1e6 + dt.microsecond)*1e3

def ConvertEpoch_nsToDateTime(ns):
    """
    converts the Epoch time in nanoseconds into a datetime object
    """
    
    return (datetime.fromtimestamp(ns/1e9).strftime('%c'))

It would be nice to be able to decrement by an "hour" or "minute" or "day"

This method is not fully implemented

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-22-c0d61ae0f3fe> in <module>()
----> 1 for x in s.changes(v-1, v, 1):
      2     print(x)

~/anaconda/lib/python3.6/site-packages/btrdb4-4.4.5-py3.6.egg/btrdb4/__init__.py in changes(self, fromVersion, toVersion, resolution)
    428         for crlist, version in crs:
    429             for cr in crlist:
--> 430                 yield ChangeRange.fromProto(cr), version
    431 
    432     def flush(self):

NameError: name 'ChangeRange' is not defined

Variable types is a bit unforgiving in the library.

Nanoseconds

For example - nanoseconds must be in integers. We could simply cast nanosecond parameters as integers within the api methods instead of throwing errors. There could be a performance penalty for doing this.

String encoding as Bytes

There are also some instances where the user has to take a string and turn it into a sequence of bytes in a particular character encoding. This can easily be done inside the API using either of the following to generate b'bytes':

u'something'.encode('utf-8')
bytes(u'something', 'utf-8')

If you pick a point and then go the wrong way such that there is no data in that direction, BTrDB returns

BTrDBError [401] no such point

This should not be the case. It would be much smarter to return a None and have the API method catch the error and return none.

In cases where HOME is not set (e.g. in our case, when btrdb is imported in a webapp controlled by supervisorctl), importing btrdb fails at utils/credentials.py:30

CREDENTIALS_PATH = os.path.join(os.environ["HOME"], CONFIG_DIR, CREDENTIALS_FILENAME)

This can be reproduced as follows:

# bash
unset HOME

then

# in python
import btrdb

Which produces:

In [1]: import btrdb                                                                                                                                                   
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-36165869e3ee> in <module>
----> 1 import btrdb

~/Preteckt/tools/preteckt_dash/env/lib/python3.6/site-packages/btrdb/__init__.py in <module>
     20 from btrdb.exceptions import ConnectionError
     21 from btrdb.version import get_version
---> 22 from btrdb.utils.credentials import credentials_by_profile, credentials
     23 from btrdb.stream import MINIMUM_TIME, MAXIMUM_TIME
     24 

~/Preteckt/tools/preteckt_dash/env/lib/python3.6/site-packages/btrdb/utils/credentials.py in <module>
     28 CONFIG_DIR = ".predictivegrid"
     29 CREDENTIALS_FILENAME = "credentials.yaml"
---> 30 CREDENTIALS_PATH = os.path.join(os.environ["HOME"], CONFIG_DIR, CREDENTIALS_FILENAME)
     31 
     32 ##########################################################################

/usr/lib/python3.6/os.py in __getitem__(self, key)
    667         except KeyError:
    668             # raise KeyError with the original key value
--> 669             raise KeyError(key) from None
    670         return self.decodevalue(value)
    671 

KeyError: 'HOME'

There is of course a workaround to set the HOME var by hand, but it seems like this shouldn't be a requirement.

https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt

Search functionality for streams is currently less than ideal.

It would be good to have wildcard or more general purpose string matching available. This could be built into the Python library or could be built into BTrDB. It is probably better to put it into BTrDB.

It may make sense to have some basic utility oriented functionality built into the API or as an additional library.

The 4 in the current name is misleading. We should drop it from the name of the project repo.