pachyderm / python-pachyderm Goto Github PK

View Code? Open in Web Editor NEW

89.0 25.0 30.0 17.83 MB

Python client for Pachyderm

License: Apache License 2.0

Python 98.72% Shell 1.02% Makefile 0.19% Dockerfile 0.07%

pachyderm python python-bindings

python-pachyderm's People

Contributors

Stargazers

Watchers

python-pachyderm's Issues

Build fails with error: could not create 'build': File exists

OS: macOS High Sierra
Python 3.6.4 (Anaconda)

Trying to build from source:

~/Repos/python-pachyderm $ python setup.py install
/usr/local/anaconda3/lib/python3.6/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
  normalized_version,
running install
running bdist_egg
running egg_info
writing src/python_pachyderm.egg-info/PKG-INFO
writing dependency_links to src/python_pachyderm.egg-info/dependency_links.txt
writing requirements to src/python_pachyderm.egg-info/requires.txt
writing top-level names to src/python_pachyderm.egg-info/top_level.txt
reading manifest file 'src/python_pachyderm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.dylib' found anywhere in distribution
writing manifest file 'src/python_pachyderm.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
creating build
error: could not create 'build': File exists

Trying to install via pip:

~ $ pip install python-pachyderm
Collecting python-pachyderm
  Using cached https://files.pythonhosted.org/packages/09/11/8e362821d7a752e45753e58ecf700b8f4ca84d740f8aa64a053ddef8b24c/python-pachyderm-1.6.8.post9.tar.gz
Requirement already satisfied: protobuf in /usr/local/anaconda3/lib/python3.6/site-packages (from python-pachyderm) (3.5.2.post1)
Requirement already satisfied: grpcio in /usr/local/anaconda3/lib/python3.6/site-packages (from python-pachyderm) (1.11.0)
Requirement already satisfied: future>=0.14 in /usr/local/anaconda3/lib/python3.6/site-packages (from python-pachyderm) (0.16.0)
Requirement already satisfied: setuptools in /usr/local/anaconda3/lib/python3.6/site-packages (from protobuf->python-pachyderm) (38.4.0)
Requirement already satisfied: six>=1.9 in /usr/local/anaconda3/lib/python3.6/site-packages (from protobuf->python-pachyderm) (1.11.0)
Building wheels for collected packages: python-pachyderm
  Running setup.py bdist_wheel for python-pachyderm ... error
  Complete output from command /usr/local/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-install-bwc3fnkb/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-wheel-7oe8a4bu --python-tag cp36:
  /usr/local/anaconda3/lib/python3.6/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  error: could not create 'build': File exists
  
  ----------------------------------------
  Failed building wheel for python-pachyderm
  Running setup.py clean for python-pachyderm
Failed to build python-pachyderm
awscli 1.14.68 has requirement botocore==1.9.21, but you'll have botocore 1.9.22 which is incompatible.
Installing collected packages: python-pachyderm
  Running setup.py install for python-pachyderm ... error
    Complete output from command /usr/local/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-install-bwc3fnkb/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-record-3uk4hqtv/install-record.txt --single-version-externally-managed --compile:
    /usr/local/anaconda3/lib/python3.6/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    error: could not create 'build': File exists
    
    ----------------------------------------
Command "/usr/local/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-install-bwc3fnkb/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-record-3uk4hqtv/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/gn/nbkfw4214tq4bfhjpcsr5wrr0000gn/T/pip-install-bwc3fnkb/python-pachyderm/

Release on PyPi

This client is useful enough for GA. Can we release a first version of it?

Pachyderm 1.9

The first (non-rc) release of pachyderm 1.9 will be out shortly. Once it is, sync this library up.

Implement PFS `inspect_branch`

Unable to install python-pachyderm

I tried to install python-pachyderm into my Python3 as well as Python2 environment, however I got this error: could not create 'build': File exists. I can reproduce the issue with pip 9.0.1 and 10.0.1.

Here's the error log:

(repro-tut) mbp-tvlk:github arinto$ pip install python-pachyderm
Collecting python-pachyderm
  Using cached https://files.pythonhosted.org/packages/09/11/8e362821d7a752e45753e58ecf700b8f4ca84d740f8aa64a053ddef8b24c/python-pachyderm-1.6.8.post9.tar.gz
Requirement already satisfied: protobuf in /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages (from python-pachyderm) (3.5.2.post1)
Requirement already satisfied: grpcio in /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages (from python-pachyderm) (1.12.0)
Requirement already satisfied: future>=0.14 in /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages (from python-pachyderm) (0.16.0)
Requirement already satisfied: six>=1.9 in /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages (from protobuf->python-pachyderm) (1.11.0)
Requirement already satisfied: setuptools in /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages (from protobuf->python-pachyderm) (38.4.0)
Building wheels for collected packages: python-pachyderm
  Running setup.py bdist_wheel for python-pachyderm ... error
  Complete output from command /Users/arinto/anaconda3/envs/repro-tut/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-install-ik6qb_40/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-wheel-peujp6sm --python-tag cp36:
  /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  error: could not create 'build': File exists

  ----------------------------------------
  Failed building wheel for python-pachyderm
  Running setup.py clean for python-pachyderm
Failed to build python-pachyderm
Installing collected packages: python-pachyderm
  Running setup.py install for python-pachyderm ... error
    Complete output from command /Users/arinto/anaconda3/envs/repro-tut/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-install-ik6qb_40/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-record-_8qp1h1p/install-record.txt --single-version-externally-managed --compile:
    /Users/arinto/anaconda3/envs/repro-tut/lib/python3.6/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    error: could not create 'build': File exists

    ----------------------------------------
Command "/Users/arinto/anaconda3/envs/repro-tut/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-install-ik6qb_40/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-record-_8qp1h1p/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/fd/h7tg23rd2p3cx7mnp7j47_gw0000gn/T/pip-install-ik6qb_40/python-pachyderm/

AttributeError: 'CronInput' object has no attribute 'overwrite'

The pb2 object pps.CronInput should have the 'overwrite' attribute for Pachyderm >=1.8.6

How to update Pipeline

Hi Folks,
I'm wondering how I can update a pipeline just by giving a dict/json, the way it works with the pachctl client.

Implement authentication

Hi there.
Am I correct that no authentication method is implemented here?

Failed building wheel for python-pachyderm

Platform: Darwin-16.6.0-x86_64-i386-64bit
Python version: 3.6.4

Running pip install python-pachyderm

(.venv) ❱ pip install python-pachyderm
Collecting python-pachyderm
  Using cached https://files.pythonhosted.org/packages/09/11/8e362821d7a752e45753e58ecf700b8f4ca84d740f8aa64a053ddef8b24c/python-pachyderm-1.6.8.post9.tar.gz
Requirement already satisfied: protobuf in ./.venv/lib/python3.6/site-packages (from python-pachyderm)
Requirement already satisfied: grpcio in ./.venv/lib/python3.6/site-packages (from python-pachyderm)
Requirement already satisfied: future>=0.14 in ./.venv/lib/python3.6/site-packages (from python-pachyderm)
Requirement already satisfied: setuptools in ./.venv/lib/python3.6/site-packages (from protobuf->python-pachyderm)
Requirement already satisfied: six>=1.9 in ./.venv/lib/python3.6/site-packages (from protobuf->python-pachyderm)
Building wheels for collected packages: python-pachyderm
  Running setup.py bdist_wheel for python-pachyderm ... error
  Complete output from command /Users/ryan.wheeler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-build-ux7161v9/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/tmp_o4pe5uppip-wheel- --python-tag cp36:
  /Users/ryan.wheeler/.venv/lib/python3.6/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  error: could not create 'build': File exists

  ----------------------------------------
  Failed building wheel for python-pachyderm
  Running setup.py clean for python-pachyderm
Failed to build python-pachyderm
Installing collected packages: python-pachyderm
  Running setup.py install for python-pachyderm ... error
    Complete output from command /Users/ryan.wheeler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-build-ux7161v9/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-4ib4860i-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/ryan.wheeler/.venv/include/site/python3.6/python-pachyderm:
    /Users/ryan.wheeler/.venv/lib/python3.6/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '1.6.8-9' to '1.6.8.post9'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    error: could not create 'build': File exists

    ----------------------------------------
Command "/Users/ryan.wheeler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/private/var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-build-ux7161v9/python-pachyderm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-4ib4860i-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/ryan.wheeler/.venv/include/site/python3.6/python-pachyderm" failed with error code 1 in /private/var/folders/xh/6jrlj6dn58ggnt8x8m56d2d5p0t72f/T/pip-build-ux7161v9/python-pachyderm/
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Remove PPS `rerun_pipeline`

This function is no longer supported

python 2.7 import error

We're unable to use the pip installation w python2.7

18-08-06[17:35:50]:~:0$python
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import python_pachyderm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/python_pachyderm/__init__.py", line 5, in <module>
    from .pfs_client import PfsClient, DIR, FILE, NONE, JSON, LINE
  File "/usr/local/lib/python2.7/dist-packages/python_pachyderm/pfs_client.py", line 8, in <module>
    from .client.pfs.pfs_pb2 import *
ImportError: No module named pfs.pfs_pb2
>>>

The same worked for python3. This looks like a relative import path error.

Use `ListJobStream` in `list_job`

Our current implementation of list_job relies on the deprecated ListJob function. Change this to use ListJobStream.

Update to 1.4 API

Pachyderm version 1.4 is being release tomorrow officially. We will need to update the client to the 1.4 API as there are a few differences. We would be happy to discuss the changes on Slack or here, and happy to help where we are able. Thanks!

`input` in pipeline info empty

When using list_pipeline or inspect_pipeline, the input attribute is always empty.

pps_client.list_pipeline()
pps_client.inspect_pipeline(pipeline.pipeline.name)

Output (single input repo in pipeline spec):

...
input {
}
...

Output (two input repos in pipeline spec (using cross)):

...
input {
   cross {
   }
   cross {
   }
}
...

I would like to be able to get some information about the input repos for the pipelines.

Improve PPS test coverage

The PPS client is entirely untested. Add tests for the following functions:

Abillity to specify host in PfsClient()

Hi,

PfsClient() presumes you're only connecting to localhost. I've added host="localhost" as a kwarg in PfsClient() so you can connect, e.g., from within the same k8s cluster that pachd is running in.

Thanks!

Add support for transactions

Pachyderm 1.9 introduces a new notion of transactions - see transactions.proto. Add support for this.

Add way to convert CommitInfo object to dict

Currently there appears to be no way to convert a CommitInfo object obtained via client.list_commit to something readily usable in Python such as a dictionary. It seems natural for there to be such a conversion.

Missing Commit Description

The python client currently does not expose the description string for a commit, which is defined as such in the pachyderm docs.

message StartCommitRequest {
  reserved 2;
  // Parent.ID may be empty in which case the commit that Branch points to will be used as the parent.
  // If branch is empty, or if branch does not exist, the commit will have no parent.
  Commit parent = 1;
  // description is a user-provided string describing this commit
  string description = 4;
  string branch = 3;
  repeated CommitProvenance provenance = 5;
}

GetLogsRequest has no "input_file_id" field

Trying to extract logs for a given pipeline (using pps_client.get_logs(...)) fails with the following exception:

pps_client = python_pachyderm.PpsClient()
pipelines = pps_client.list_pipeline() # get all pipelines
for pipeline in pipelines.pipeline_info: # iterate over pipelines
   print(pps_client.get_logs(pipeline_name=pipeline.pipeline.name)) # get logs for pipeline

File "/usr/local/lib/python3.7/site-packages/python_pachyderm/pps_client.py", line 93, in get_logs
    input_file_id=input_file_id, master=master)))
ValueError: Protocol message GetLogsRequest has no "input_file_id" field.

Use `ListCommitStream` in `list_commit`

Our current implementation of list_commit relies on the deprecated ListCommit function. Change this to use ListCommitStream.

Feature request: Add get_version()

It would be great to have an equivalent for pachctl version in this package, so we can retrieve the version of pachd we're talking to.

delete_all() raises an exception

    def delete_all(self):
>       self.stub.DeleteAll()
E       TypeError: __call__() missing 1 required positional argument: 'request'

I'm not sure what request is unfortunately.

Implement PPS `flush_job`

create_repo hangs

import python_pachyderm
client = python_pachyderm.PfsClient()
client.create_repo('voc2012')

Pachyderm is installed and running on microk8s.
The execution simply hang at the last line above.
How can i print the logs of the statement?

Remove non-standard PFS functionality

The PFS client contains some extensions to standard PFS. Some of these should probably be kept for the sake of having a more idiomatic python interface, but some seem to be too specific to be in the library (e.g. provenances_for_repo.)

missing history option

Motivation
The default method for listing files in a repo "merges" all commits together (shown below). The true commit history is only available with the --history option, which is not exposed in the python client.

Goal
Expose the --history option from the python client.

Use `ListFileStream` in `list_file`

Our current implementation of list_file relies on the deprecated ListFile function. Change this to use ListFileStream.

Flush_commit not accepting any commit

I am writing some pytests for each pipeline I create, and one requirement is to verify that the pipeline has the correct output in the output repo after the test runs. I would like to do this verification by using flush-commit to block the test until the input commit produces all output commits in the output repo.

Here is some of my code:

# when
with pytest.pfs_client.commit(input_repo, 'master') as c:
    pytest.pfs_client.put_file_bytes(c, 'input.json', input_file.encode())

pytest.pfs_client.flush_commit([f'{c.repo.name}/{c.id}'])

# then
files = pytest.pfs_client.get_files(f'output-repo/master', '/', recursive=True)
assert len(files) == 66

The problem I'm having is at the flush_commit call.

The call says it can take a Commit object, that doesn't work.

It also shows in the utils.py file that from_commit takes strings (like the one I'm trying to create from the original commit) but this does not work either.
For this case, I get this error: ValueError: Protocol message FlushCommitRequest has no "commit" field.

Am I using this the correct way?

Implement PFS `copy_file`

Add `provenance` field for `start_commit`

Remove python 2.x support

Python 2.x is going to be deprecated next year. At this point, I think it's safe to remove support.

Protobufs seem to be out of date

for example create_pipeline() seems to be expecting (actually, requiring -- see #21) a resource_spec field, which no longer exists (replaced by requests and limits )

Remove `pod_spec` from `create_pipeline`

This field is deprecated

repos with ACLs are not supported

Looks like python client does not support repositories with access control. Is that right? or I missed anything?

>>> client = pachy.PfsClient(host='api.mycluster.com')
>>> e=client.list_repo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<>/lib/python3.6/site-packages/python_pachyderm-1.6.8.post9-py3.6.egg/python_pachyderm/pfs_client.py", line 93, in list_repo
    x = self.stub.ListRepo(ListRepoRequest(provenance=[Repo(name=p) for p in provenance]))
  File "<>/lib/python3.6/site-packages/grpc/_channel.py", line 514, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "<>/lib/python3.6/site-packages/grpc/_channel.py", line 448, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "error getting access level for "repo1": no authentication token (try logging in)"
	debug_error_string = "{"created":"@1531122639.454592000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1083,"grpc_message":"error getting access level for "label-samples": no authentication token (try logging in)","grpc_status":2}"

Is it supported in go client?
If acl is not supported in python only, are you planning to support it in python client in near future?

Missing offset in put_file_* methods

The client does not provide a parameter for specifying the offset for writing data to the repo, thus there is no way to overwrite files – only to append. put_file_* methods should expose the OverwriteIndex overwrite_index attribute of PutFileRequest.

Add `update` flag for `create_repo`

Exception while committing

One of my functional tests commits a file to a repo and it fails with the following error:

------------------------------------------------------------ Captured stdout call -------------------------------------------------------------
An exception occurred during an open commit. Trying to finish it (Currently a commit can't be cancelled)
------------------------------------------------------------ Captured stderr call -------------------------------------------------------------
ERROR:root:Exception iterating requests!
Traceback (most recent call last):
  File "/home/omer/Documents/Projects/drifter/drifter-collector/.tox/py36/lib/python3.6/site-packages/grpc/_channel.py", line 200, in consume_request_iterator
    request = next(request_iterator)
  File "/home/omer/Documents/Projects/drifter/drifter-collector/.tox/py36/lib/python3.6/site-packages/pypachy/client.py", line 295, in _blocks
    value=v[i:i + BUFFER_SIZE],
TypeError: unhashable type: 'slice'

It's not clear why the commit fails. Is this a bug or something that I am doing wrong?

Use `ListDatumStream` in `list_datum`

Our current implementation of list_datum relies on the deprecated ListDatum function. Change this to use ListDatumStream.

TypeError when running "stop_pipeline"

Issue: You cannot use positional arguments when creating protobuf objects.

Line:

python-pachyderm/src/python_pachyderm/pps_client.py

Line 118 in 3cfbe3a

req = proto.StopPipelineRequest(pipeline=proto.Pipeline(pipeline_name))

Error:

File "/usr/local/lib/python3.6/site-packages/python_pachyderm/pps_client.py", line 119, in stop_pipeline
    req = proto.StopPipelineRequest(pipeline=proto.Pipeline(pipeline_name))
TypeError: No positional arguments allowed

Implement PFS `build_commit`

PpsClient.create_pipeline() requires all fields of a pipeline spec

It seems this method requires all the top-level fields of a pipeline spec be passed in as individual function arguments, rather than accepting a single pipeline_spec

specific error:

Traceback (most recent call last):
  File "pipeline_creator.py", line 52, in <module>
    main()
  File "pipeline_creator.py", line 39, in main
    client.create_pipeline(spec)
TypeError: create_pipeline() missing 15 required positional arguments: 'transform', 'parallelism_spec', 'inputs', 'egress', 'update', 'output_branch', 'scale_down_threshold', 'resource_spec', 'input', 'description', 'incremental', 'cache_size', 'enable_stats', 'reprocess', and 'batch'

Implement PPS `run_pipeline`

DatumState missing enum value 3=STARTING

DatumState is missing the enum value 3 (STARTING). The current definition of DatumState look like this:

enum DatumState {
    FAILED = 0;
    SUCCESS = 1;
    SKIPPED = 2;
    STARTING = 3;
}

Perhaps pps_pb2.py needs to be regenerated.

Implement PPS `stop_job`

Improve PFS client test coverage

PFS client is not fully tested. These functions need explicit tests:

Inconsistent (?) overwrite/append behaviour

In case of a commit with no branch specified, consequent writes overwrite data files completely, no matter even whether overwrite_index is passed or not. However, for commits going to a specific branch the default behaviour is append. A quick example to illustrate this:

pfs_client.create_repo('test-repo-1')

    with pfs_client.commit('test-repo-1', 'mybranch') as c:
        pfs_client.put_file_bytes(c, 'file.dat', b'DATA')

    with pfs_client.commit('test-repo-1', 'mybranch') as c:
        pfs_client.put_file_bytes(c, 'file.dat', b'FOO')

    file = list(pfs_client.get_file('test-repo-1/{}'.format(c.id), 'file.dat'))
    print(file)

This outputs [b'DATA', b'FOO']. Now the same code but without the branch:

pfs_client.create_repo('test-repo-1')

    with pfs_client.commit('test-repo-1') as c:
        pfs_client.put_file_bytes(c, 'file.dat', b'DATA')

    with pfs_client.commit('test-repo-1') as c:
        pfs_client.put_file_bytes(c, 'file.dat', b'FOO')

    file = list(pfs_client.get_file('test-repo-1/{}'.format(c.id), 'file.dat'))
    print(file)

This outputs just [b'FOO'].

This looks rather inconsistent.

python2.7 depends on `future` library

pfs_client.py imports builtins, which is not present in a minimal python2.7 installation. This can be fixed by installing the future library via pip.

example:

root@0fb4804911b6:/opt# python
Python 2.7.12 (default, Nov 20 2017, 18:23:56)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypachy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pypachy/__init__.py", line 1, in <module>
    from .pfs_client import PfsClient, DIR, FILE, NONE, JSON, LINE
  File "/usr/local/lib/python2.7/dist-packages/pypachy/pfs_client.py", line 3, in <module>
    from builtins import object
ImportError: No module named builtins

and to fix:

root@0fb4804911b6:/opt# pip install future
Collecting future
  Downloading future-0.16.0.tar.gz (824kB)
    100% |################################| 829kB 1.4MB/s
Building wheels for collected packages: future
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /root/.cache/pip/wheels/c2/50/7c/0d83b4baac4f63ff7a765bd16390d2ab43c93587fac9d6017a
Successfully built future
Installing collected packages: future
Successfully installed future-0.16.0
root@0fb4804911b6:/opt# python
Python 2.7.12 (default, Nov 20 2017, 18:23:56)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypachy
>>> exit

pachyderm / python-pachyderm Goto Github PK

python-pachyderm's People

Contributors

Stargazers

Watchers

Forkers

python-pachyderm's Issues

Recommend Projects

Recommend Topics

Recommend Org