mlbazaar / piex Goto Github PK
View Code? Open in Web Editor NEWPipeline Explorer - Explore and analyze millions of pipelines learned using MLBlocks and MLPrimitives.
Home Page: https://mlbazaar.github.io/piex/
License: MIT License
Pipeline Explorer - Explore and analyze millions of pipelines learned using MLBlocks and MLPrimitives.
Home Page: https://mlbazaar.github.io/piex/
License: MIT License
piex fails to download publicly-available pipelines if the user has not populated an AWS credentials files. This should succeed without any configuration.
I try to run the following in an environment without AWS credentials
import piex.explorer
ex = piex.explorer.S3PipelineExplorer('ml-pipelines-2018')
ex.get_pipelines()
But I get the following credential-related error.
Traceback (most recent call last):
File "./main.py", line 238, in <module>
main()
File "./main.py", line 226, in main
make_table_1()
File "./main.py", line 117, in make_table_1
df = load_pipelines_df(ex=ex)
File "/usr/local/lib/python3.6/site-packages/funcy/calc.py", line 44, in wrapper
value = memory[key] = func(*args, **kwargs)
File "./main.py", line 40, in load_pipelines_df
df = ex.get_pipelines()
File "/usr/local/lib/python3.6/site-packages/piex/explorer.py", line 280, in get_pipelines
df = self._load_table('pipelines')
File "/usr/local/lib/python3.6/site-packages/piex/explorer.py", line 252, in _load_table
df = self._get_table(table_name)
File "/usr/local/lib/python3.6/site-packages/piex/explorer.py", line 234, in _get_table
obj = s3.get_object(Bucket=self.bucket, Key=key)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 611, in _make_api_call
operation_model, request_dict)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python3.6/site-packages/botocore/signers.py", line 157, in sign
auth.add_auth(request)
File "/usr/local/lib/python3.6/site-packages/botocore/auth.py", line 425, in add_auth
super(S3SigV4Auth, self).add_auth(request)
File "/usr/local/lib/python3.6/site-packages/botocore/auth.py", line 357, in add_auth
raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
make: *** [run] Error 1
I expect ex.get_templates(data_type='image')
to return only template for images, but it returns all templates. Looking at the source, **filters
is unused.
Furthermore, it is confusing that ex.get_templates
returns data_type
as a column whereas ex.get_datasets
and ex.get_pipelines
return data_modality
as a column. There are probably other name mismatches.
After some of our dependencies made new releases, the scores have started not being reproducible.
We need to pin the setup.py dependencies to exact versions to ensure reproducibility.
MLBlocks is a pretty heavy-weight dependency as it requires tensorflow, xgboost, etc. to be installed. To just "explore" pipelines from S3, users shoudn't need MLBlocks.
Desired functionality:
PipelineExplorer.score_pipeline
and Pipeline.score_template
, MLBlocks should not need to be installedPipelineExplorer.score_pipeline
and Pipeline.score_template
, they can enable this by pip install piex[mlblocks]
(or piex[scoring]
or piex[all]
or some other variant)Note that this would require a workaround for using MLPipeline to load pipelines/templates for the express purpose of extracting metadata from them
Most methods of PipelineExplorer and subclasses accept **filters
kwargs, but some do not:
These methods should be modified to support filters.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.