oceanprotocol / aquarius Goto Github PK

We are going to need to implement the following preconfigured queries:

• Price from/to
• License(none, Public Domain, CC BY, CC BY-SA, CC BY-ND, CC BY-NC, CC BY-NC-SA, CC BY-NC-ND)
• Category
• Type(dataset, algorithm, container, workflow, other)
• Update frequency(seldom, annually, quarterly, never)
• Time added(Last week, last month)
• Filter by the condition of have sample.

After calling /validate, the return is True.

        validate_endpoint = ocn.services._ocean_assets._aquarius_url + "api/v1/aquarius/assets/ddo/validate"
        r = requests.post(validate_endpoint, json=metadata)
        assert r.json()

However, this same metadata fails when calling ocn.assets.create(metadata, publisher_acct).

  File "/python3.6/site-packages/squid_py/ocean/ocean_assets.py", line 64, in create
    raise OceanInvalidMetadata('Metadata seems invalid. Please make sure'
squid_py.exceptions.OceanInvalidMetadata: Metadata seems invalid. Please make sure the required metadata values are filled in.

There is no log message in Aquarius to show how to fix the error. Let's call plecos.list_errors(), log all errors in the pod, and send just the first error back?

Unify terms and use only asset_id.

@MarcusJones said he connected to a remote instance of Aquarius and wasn't sure what version of Aquarius was running at the other end.

Proposal: Return the software name and version in the response when you visit the root/docroot/server endpoint. For example, if you make a GET request to:

https://node532.aquarius-hosts.net:5000/

with no api/v1/... on the end, the HTTP response might look something like:

{
  "software": "Aquarius",
  "version": "3.14.1"
}

More information could be included in the response but this would be a good starting point.

This issue was noted by an external party.

Aquarius gives the users ability to use advanced MongoDB queries right from the REST
interface. This is highlighted as an advantage in the documentation of squid-py library in the following
manner;

Here, OceanDB instance of mongodb can leverage power of mongo queries in ‘query’
attribute. For more info - https://docs.mongodb.com/manual/reference/method/
db.collection.find

Link

Aquarius and all the related libraries (including oceandb_driver_interface and oceandb-mongodb-
driver) pass the incoming query object directly to MongoDB without sanitization. However this could
prove to be an attack surface, as MongoDB injection attacks are known to exist that could be viable to
execute in this query functionality. For further info and investigation, please check the related
documentation and guidelines on OWASP Foundation web page.

Make this work:

sudo apt install ocean-aquarius

It would be necessary to udpate the business logic to support the new DID/DDO's entities. The changes to apply are:

• Support internally the DID/DDO models see DDO example

• GET /api/v1/aquarius/assets/metadata -- Update the method to return only the metadata entry

• GET /api/v1/aquarius/assets/ddo/{did} -- New method returning the complete DDO

• Delete method POST /api/v1/aquarius/assets/metadata

• New method - POST /api/v1/aquarius/assets/ddo -- Create the DDO and the associated DID

• Delete method PUT /api/v1/aquarius/assets/metadata -- Delete, we well use the

• New method PUT /api/v1/aquarius/assets/ddo -- Actualiza datos del DDO

• New method DELETE /api/v1/aquarius/assets/ddo -- Delete the DDO

• Update method - GET /api/v1/provider/assets -- Devuelve todos los DID's

• Delete method - GET /api/v1/provider/assets/metadata/query

• New method - /api/v1/provider/assets/ddo/query -- DDO's Search

In the current implementation of the provider, we have a custom implementation that allows us to use bigchaindb as a database. We should move these methods to the oceandb-bigchaindb-driver.

Failure Information (for bugs)

The system is vulnerable to reflected cross-site scripting (also known as XSS) attacks. This vulnerability stems from missing validation of the input incoming from the client-side and missing encoding of the output derived from the received input.

Possible Solution

• Always encode untrusted data (all parameters which are submitted to the server and then rendered as part of the web page) prior to including it in web pages. The type of encoding should be selected according to the context in which the data is inserted, e.g. HTML element, HTML attribute, JavaScript code, etc;

• HTML encoding should at minimum target the following characters: ' (single quote), " (double quote), > (greater than), < (lesser than), & (ampersand), / (forward slash) and \ (backslash). Note that this will depend on the context.

Specifications

More info here: https://app.cobalt.io/bigchaindb/ocean-protocol-api-201902/findings/1

Support to query both text and preconfigured options #140 at same time.

Aquarius is failing running in barge using the last version when you send a search query:

curl -X GET "http://0.0.0.0:5000/api/v1/aquarius/assets/ddo/query?text=Weather&sort=%5Bobject%20Object%5D&offset=100" -H "accept: application/json"

Error:

500Undocumented | Error: INTERNAL SERVER ERRORResponse body<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <title>500 Internal Server Error</title> <h1>Internal Server Error</h1> <p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>
-- | --

In the docker logs:

aquarius_1                   | 2019-03-05 12:04:59 7a3379aed6bd flask.app[152] ERROR Exception on /api/v1/aquarius/assets/ddo/query [GET]
aquarius_1                   | Traceback (most recent call last):
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
aquarius_1                   |     response = self.full_dispatch_request()
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
aquarius_1                   |     rv = self.handle_user_exception(e)
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
aquarius_1                   |     return cors_after_request(app.make_response(f(*args, **kwargs)))
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
aquarius_1                   |     reraise(exc_type, exc_value, tb)
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
aquarius_1                   |     raise value
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
aquarius_1                   |     rv = self.dispatch_request()
aquarius_1                   |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
aquarius_1                   |     return self.view_functions[rule.endpoint](**req.view_args)
aquarius_1                   |   File "/aquarius/aquarius/app/assets.py", line 543, in query_text
aquarius_1                   |     data.get('sort', None)),
aquarius_1                   |   File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
aquarius_1                   |     return _default_decoder.decode(s)
aquarius_1                   |   File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
aquarius_1                   |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
aquarius_1                   |   File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
aquarius_1                   |     raise JSONDecodeError("Expecting value", s, err.value) from None
aquarius_1                   | json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)

Looks that the swagger doc was break and the methods are returning :
TypeError: Failed to fetch

• Add comments/doc strings
• Rename OceanContractsWrapper to OceanKeeper
• OceanContractsWrapper/OceanKeeper should encapsulate all functionality offered by the keeper

Description

In last weeks squid-py has introduced new functionalities (new API, different way to manage the configuration, renamings...) and it is needed to update the provider to take care of the upgrade of the squid-py dependency.

Aitor asked me to make an issue summarizing some of the things we could do to make Aquarius more usable for non-developers (Aquarius "operators").

Must-Have for Trilobite

• 1 - pip install ocean-aquarius (from PyPI) should install all its requirements and not packages only needed by Aquarius developers. (Aquarius developers can have their own installation process.) Don't worry about BigchainDB as a requirement, for now. This task should be resolved once:
  • PR #96 is merged and
  • a new version of ocean-aquarius is released on PyPI.
• 2 - Minimal docs for Aquarius operators about how to install, configure and run Aquarius somehow (probably using apt and pip). This includes running MongoDB or Elasticsearch in a more production-grade way. This will be done, for now, once PR #105 is merged.
• 3 - Answers to questions about Keeper nodes, e.g. "As an Aquarius node operator, what Keeper nodes can I connect it to? Is a local one the only option? Are there public ones I can connect to for testing?"

Nice-to-Have for Trilobite

I moved all of these to new issues of their own.

Description

Rename the project and all the modules to Aquarius.

The idea is to make this work to start Aquarius:

$ aquarius

rather than relying on Flask or Gunicorn to run Aquarius indirectly.

See how BigchainDB Server does this.

Right now, the oceandb-bigchaindb-driver Python package isn't part of the installation requirements of ocean-aquarius (i.e. the Aquarius Python package).

It's not clear if Aquarius would work with BigchainDB 2.0. See https://github.com/oceanprotocol/oceandb-bigchaindb-driver/issues/22

Expected Behavior

In i.e. squid-py, the config file can be specified by file path, or by environment variable.

Current Behavior

In provider, it fails if path not specified (must be env variable).

Possible Solution

Create a single common utilities package?

• Including logging settings and parser
• Include config file parsing
• etc...

Failed to install environment from requirements.

pip install -r requirements_dev.txt

  Running setup.py install for python-rapidjson ... error
    Complete output from command /home/batman/anaconda3/envs/flask/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-k6zqy1zm/python-rapidjson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-m148gs3z-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'rapidjson' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/python-rapidjson
    gcc -pthread -B /home/batman/anaconda3/envs/flask/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./rapidjson/include -I/home/batman/anaconda3/envs/flask/include/python3.7m -c ./python-rapidjson/rapidjson.cpp -o build/temp.linux-x86_64-3.7/./python-rapidjson/rapidjson.o -pedantic
    In file included from /home/batman/anaconda3/envs/flask/include/python3.7m/Python.h:81:0,
                     from ./python-rapidjson/rapidjson.cpp:1:
    /home/batman/anaconda3/envs/flask/include/python3.7m/longobject.h:88:49: warning: ISO C++ 1998 does not support ‘long long’ [-Wlong-long]
     PyAPI_FUNC(PyObject *) PyLong_FromLongLong(long long);
                                                     ^

Ref #142

Merge in https://github.com/oceanprotocol/plecos_service functionality, to have this service included in Aqua.

Two types of validation can be exposed:

• "Local" metadata, with unencrypted URLs in the 'files' attrib
• "Remote" metadata with 'encryptedFiles' attribe

Two schemas should be prepared.

Also consider the security of sending local metadata.

Implementetion of part of this issue oceanprotocol/ocean#103

We have to move acl and blockchain modules to ocean-web3 project. Later we will use this project as a dependency to iteract with all the web3.py methods.

We will need to delist or block assets from being resolved, if they are identified as unsuitable.

Check if flag=True before the resolve asset, and search for assets. If True, hide the asset from the end users.

Using as reference the following page:
https://app.codacy.com/app/ocean-protocol/provider-backend/settings/coverage

The idea is to list the things to refactor in the Provider side (sorted by priority):

• Unify terms and use only asset_id. (#40)
• Encapsulate Azure logic into osmosis class, before starting the independent osmosis project
• Move acl and blockchain modules from provider-backend to ocean-web3 as a dependency.(#28 )
• Move dao methods in provider-backend that are made for bigchaindb to the oceandb-bigchaindb-driver
• Integrate with coverage & codacy tools (#17 )
• Add more tests improving the code coverage
• Resolve some of the main issues reported by Codacy: https://app.codacy.com/project/ocean-protocol/provider/dashboard
• Ocean-web3: (#44 )
  • Resolve some of the main issues reported by Codacy: (#30 ) https://app.codacy.com/project/ocean-protocol/provider/dashboard
  • Add comments/doc strings
  • Rename OceanContractsWrapper to OceanKeeper
  • OceanContractsWrapper/OceanKeeper should encapsulate all functionality offered by the keeper
• Use environment variables to fill the config of provider backend and oceandb-bigchaindb-driver and oceandb-mongo-driver to make it easy to run dockers and travis.
• Pass addresses of contracts as environment variable to the provider.
• Integrate bumpversion in all the projects to help in new realeases.
  • provider
  • oceandb-driver-interface
  • oceandb-mongodb-driver (https://github.com/oceanprotocol/oceandb-mongodb-driver/pull/8)
  • oceandb-bigchaindb-driver (https://github.com/oceanprotocol/oceandb-bigchaindb-driver/pull/13)
  • ocean-web3
• Use markdown in project of oceandb instead of rst.(#31)
  • oceandb-driver-interface
  • oceandb-mongodb-driver
  • oceandb-bigchaindb-driver
• Add pypi and Codacy badges.
• Improve handling of keeper events. (#45 ) Options:
  • Save a record of previous access requests in oceandb
  • Build the state of previous access requests by querying the keeper
• Move travis secure passwords in each project to a travis environment variable.(#43)
  • provider
  • keeper-contracts
  • oceandb-driver-interface
  • oceandb-mongodb-driver
  • oceandb-bigchaindb-driver
  • ocean-web3

After reviewing this, each entry will be an independent user story to be implemented in the following days.

Expected Behavior

New config class to handle configuration settings via a config file and/or environment variables, for the server provider.

To define and setup the following settings from a config class using config file and or environment variables:

1. squid-py connections/addresses
2. azure setup
3. ocean db setup
4. provider server setup

Move the settings

1. provider.scheme
2. provider.host
3. provider.port

To a new section '[provider]' in the config file

Current Behavior

The config parser from squid-py is used to manage this server's configuration data.

Update current documentation in the Oceandb projects to follow the style of the rest of the projects:

• oceandb-driver-interface
• oceandb-bigchaindb-driver
• oceandb-mongodb-driver

https://github.com/oceanprotocol/Plecos

In PyPI: https://pypi.org/project/plecos/

Validate the metadata previous to be write in OceanDB.

Wait for latest version (0.5.0 incoming)

Options:

• Save a record of previous access requests in oceandb
• Build the state of previous access requests by querying the keeper

http://localhost:5000/api/v1/aquarius/assets

Returns

{"ids": ["did:op:0x7a0506136c064d3d9cf731d7c7c17e363d32f70ed20249dfbb14fa6f1287f049", "did:op:0xaf69d79be6b04e0085fb53935c86d37807d5ca6f4a1442ef862538c34d6d370e"]}

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X GET http://localhost:5000/api/v1/aquarius/assets/did:op:0x7a0506136c064d3d9cf731d7c7c17e363d32f70ed20249dfbb14fa6f1287f049

Returns 404

It is necessary to update the requirements after the changes introduced in the OEP 8.

Expected Behavior

Package should work on Conda as well as PIP

Current Behavior

test_commit_access_requested fails due to not finding the contract abi files. The files are located for PIP but not Conda.

Failure Information (for bugs)

E           ValueError: {'message': 'Attempting to run transaction which calls a contract function, but recipient address 0x35735202cEB9C97b1A823D458C347A993bD49d87 is not a contract address', 'code': -32000, 'data': {'stack': 'r: Attempting to run transaction which calls a contract function, but recipient address 0x35735202cEB9C97b1A823D458C347A993bD49d87 is not a contract address\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:77083\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:67771\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:30:126629\n    at e.exports.i.getCode (/usr/local/lib/node_modules/ganache-cli/build/cli.node.js:2:123698)\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:30:126579\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:50739\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:54500\n    at Object.return (/usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:54077)\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:52329\n    at e (/usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:54448)\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:54459\n    at /usr/local/lib/node_modules/ganache-cli/build/cli.node.js:149:51541\n    at _combinedTickCallback (internal/process/next_tick.js:131:7)\n    at process._tickCallback (internal/process/next_tick.js:180:9)', 'name': 'r'}}

Possible Solution

Generally the handling of contracts is not as well supported in web3.py (vs. ie. web3 for javascript!). Ideally, the contracts should be returned directly from on-chain (from i.e. the running docker image, or the test/mainnets), and no hard-coding should be necessary.

Expected Behavior

I want to read in the json spec file generated by Swagger and get a consistent data structure without any errors.

Current Behavior

The json file uses a field example which seems to be non-standard, not displayed in Swagger UI, and which produces errors when importing the spec file into https://editor.swagger.io. This example field returns different data structures, sometimes an integer, sometimes an object, making it almost impossible to use and iterate over that data.

Failure Information (for bugs)

Error from https://editor.swagger.io, mainly:

Semantic error at paths./api/v1/aquarius/assets/ddo/query.get.parameters.1.example
'example' field is not allowed in parameter

Step to Reproduce the problem

1. run Aquarius locally, go to http://0.0.0.0:5000/spec
2. save that blob to some file and import to https://editor.swagger.io
3. watch some cool errors

Possible Solution

Remove the example field?

Failure Information (for bugs)

Cross-Origin Resource Sharing (CORS) is a technique to punch holes into the Same-Origin Policy (SOP) - on purpose. It enables web servers to explicitly allow cross-site access to a certain resource by returning an Access-Control-Allow-Origin (ACAO) header. Sometimes, the value is even dynamically generated based on user-input such as the Origin header send by the browser. If misconfigured, an unintended website can access the resource. In this particular situation developers made a misconfiguration in ACAO CORS header which allowing everyone to force server reflect any domain that is passing thru Origin header.

Possible Solution

Use the Access-Control-Allow-Origin header only on chosen domain that need to be accessed cross-domain. Any of values that is passing via Origin header and not match to origin domain have to be blocked.

More info here: https://app.cobalt.io/bigchaindb/ocean-protocol-api-201902/findings/2

The current JWT payload could be reconstructed because the content includes only public data as follows:

// payload
{
  "iss": "resourceowner.com",
  "sub": "WorldCupDatasetForAnalysis",
  "iat": 1516239022,
  "exp": 1526790800,
  "consumer_pubkey": "Consumer Public Key",
  "temp_pubkey": "Temp. Public Key for Encryption",
  "request_id":"Request Identifier",
  "consent_hash":"Consent Hash",
  "resource_id": "Resource Identifier",
  "timeout": "Timeout comming from AUTH contract",
  "response_type": "Signed_URL",
  "Resource_Server_plugin": "Azure",
},

So, I suggest to add nonce field to the payload in order to avoid this in the future using secure random number generator

It is needed to add a required field created and check that follow the format 2012-02-01T10:55:11Z

For whatever reason all API paths are documented with the base path prepended to them. Additionally, the name of the software is part of the path too. While the latter seems like an architectural problem, showing the full paths in the documentation shouldn't be like that.

Current behavior

This is the currently documented path for getting asset IDs:

/api/v1/aquarius/assets

Expected behavior

This is what the actual path should look like in the Swagger spec:

/assets

where the base path, ideally without aquarius, should be put into the Swagger spec as basePath

Description

• Currently, for search/query function we are exposing 2 similar endloints with only difference being the method (GET and POST).
• For GET -
  /api/v1/aquarius/assets/ddo/query (description - Get a list of DDOs that match with the given text)
  As per descriptin, this endpoint seaches based on text input (simple text search). So, ideally it shuould be renamed as /api/v1/aquarius/assets/ddo/search or /api/v1/aquarius/assets/ddo/text.
• All advanced model based queries will be handled using other endpoint (POST) /api/v1/aquarius/assets/ddo/query

In solving #42 , it was determined that the dependency error derives from installing oceandb-bigchaindb-driver. The work-around was to install oceandb-mongodb-driver instead. Therefore, @eruizgar91 and I propose to make this the default set-up, and have this reflected in the oceandb.ini file and the requirements.txt file. Optionally, 2 separate files could be created for using the oceandb-bigchaindb-driver, or this could be presented as an option in some other way.

Expected Behavior

As a user I want to be able to configure provider when running inside Docker.

Current Behavior

Some configuration parameters are hard-coded in oceandb.ini and docker-entrypoint.sh and cannot be modified with docker-entrypoint.sh

Possible Solution

Use environment variables to configure pleuston

After some discussion we have to update the ontology following this OEP: https://github.com/oceanprotocol/OEPs/tree/master/8

For proper and maintainable API docs we need to include the always up-to-date swagger.json spec file in this repo. This file should be regenerated and committed back to the repo before any new release, so might make sense having Travis handle that.

• oceanprotocol/docs#74

Expected Behavior

Return header of GET http://localhost:5000/api/v1/aquarius/assets/ddo/query?text=test
is proper to it's content: Content-Type: "application/json"

Current Behavior

Return header of GET http://localhost:5000/api/v1/aquarius/assets/ddo/query?text=test
is: Content-Type: "text/html; charset=utf-8"

Step to Reproduce the problem

Run Aquarius and query it with GET http://localhost:5000/api/v1/aquarius/assets/ddo/query?text=test

Description

It would be interesting specialliy to facilitate testing to create and endpoint that delete_all the assets already registered in oceanDB.

Expected Behavior

Aquarius not too crash if a data key field is missing in

File "/aquarius/aquarius/app/dao.py", line 72, in is_listed

Current Behavior

Aquarius crashes when it does not check the existence of a key 'isListed' the data key field.

Step to Reproduce the problem

Please provide detailed steps for reproducing the issue.

1. Start barge
2. ./start_ocean.sh --no-pleuston --no-brizo --local-spree-node --latest
3. Run squid v0.4.2 tests
4. Run squid-py tests

the older version of squid-py 0.4.2 will write an asset without the 'IsListed' field, and so all subsequent tests for squid-py v0.4.4 will fail.

Specifications

keeper_version = 0.6.12
squid-py = 0.4.4

Failure Logs

aquarius_1 | 2019-03-04 06:38:39 719d67ae7a56 flask.app[9] ERROR Exception on /api/v1/aquarius/assets/ddo/query [GET]
aquarius_1 | Traceback (most recent call last):
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
aquarius_1 | response = self.full_dispatch_request()
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
aquarius_1 | rv = self.handle_user_exception(e)
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
aquarius_1 | return cors_after_request(app.make_response(f(*args, **kwargs)))
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
aquarius_1 | reraise(exc_type, exc_value, tb)
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
aquarius_1 | raise value
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
aquarius_1 | rv = self.dispatch_request()
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
aquarius_1 | return self.view_functionsrule.endpoint
aquarius_1 | File "/aquarius/aquarius/app/assets.py", line 546, in query_text
aquarius_1 | query_result = dao.query(search_model)
aquarius_1 | File "/aquarius/aquarius/app/dao.py", line 64, in query
aquarius_1 | if self.is_listed(f['service']):
aquarius_1 | File "/aquarius/aquarius/app/dao.py", line 72, in is_listed
aquarius_1 | return service['metadata']['curation']['isListed']
aquarius_1 | KeyError: 'isListed'

Move travis secure passwords in each project to a travis environment variable in the next repos:

• provider
• keeper-contracts
• oceandb-driver-interface
• oceandb-mongodb-driver
• oceandb-bigchaindb-driver
• ocean-web3

Description

When there is no index created for OceanDB plugin (I am using Elastic Search 6.4.1), And I try to get all assets provider throws 404 (bubbled from oceanDB), but in swagger docs 500 is thrown. Ideally, we should propagate the error code and error message to the docs.

Expected Behavior

If provider receives error message and error code from OceanDB, then same message and or atleast error code should be propagated to the user.

Current Behavior

In case if index doesn't exist, provider receives error message and error code from OceanDB, but same message and or atleast error code is not propagated to the user.

Failure Information (for bugs)

2018-09-27 18:13:02 mbp.lan flask.app[70355] ERROR Exception on /api/v1/provider/assets [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/manan/projects/ocean/provider/provider/app/assets.py", line 77, in get_assets
    asset_with_id = dao.get_assets()
  File "/Users/manan/projects/ocean/provider/provider/app/dao.py", line 11, in get_assets
    assets = self.oceandb.list()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/oceandb_elasticsearch_driver/plugin.py", line 119, in list
    body = body
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/__init__.py", line 660, in search
    doc_type, '_search'), params=params, body=body)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 186, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index')

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Step to Reproduce the problem

Please provide detailed steps for reproducing the issue.

1. Setup and start oceanDB-elasticsearch-driver
2. Setup and start keeper-contracts
3. Setup provider
4. Properly fill provider's oceandb.ini config
5. Start provider
6. Got to http://127.0.0.1:5000/api/v1/docs/
7. Without creating any index, click on GET /api/v1/provider/assets

Specifications

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Version: current provider version
• Python: 3.7
• Solidity: N.A.
• JavaScript: N.A.
• Node: N.A.
• OS: macOSX 10.13.6 Sierra

Expected Behavior

Aquarius listing to return a list of assets found based on a search text.

Current Behavior

Search for assets from squid-py fails, with an html error

raise Exception(f'Unable to search for DDO: {response.content}')

Failure Information (for bugs)

squid-py failure on line

    # search for assets
    asset_ddo = ocean.assets.search('Ocean protocol')[0]

Step to Reproduce the problem

I'm trying to run the demo code from the README.md on squid-py repo, here is a sample of my 'test.py'

    ConfigProvider.set_config(Config('test_config.ini'))
    # Make a new instance of Ocean
    ocean = Ocean()
    config = ConfigProvider.get_config()
    # make account instance, assuming the ethereum account and password are set 
    # in the config file `config.ini`
    account = ocean.accounts.list()[0]
    # or 
    account = Account(config.parity_address, config.parity_password)

    # PUBLISHER
    # Let's start by registering an asset in the Ocean network
    metadata = Metadata.get_example()

    # purchase and service endpoints require `brizo.url` is set in the config file
    # or passed to Ocean instance in the config_dict.
    # define the services to include in the new asset DDO

    ddo = ocean.assets.create(metadata, account)

    # Now we have an asset registered, we can verify it exists by resolving the did
    _ddo = ocean.assets.resolve(ddo.did)
    # ddo and _ddo should be identical

    # CONSUMER
    # search for assets
    asset_ddo = ocean.assets.search('Ocean protocol')[0]
    # Need some ocean tokens to be able to purchase assets

Specifications

running latest barge

./start_ocean.sh --no-pleuston --local-spree-node --latest

squid-py 0.4.4 or branch/develop, both versions get the same error

Failure Logs

aquarius_1 | 2019-02-26 05:54:12 ad54e2f71448 flask.app[9] ERROR Exception on /api/v1/aquarius/assets/ddo/query [GET]
aquarius_1 | Traceback (most recent call last):
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
aquarius_1 | response = self.full_dispatch_request()
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
aquarius_1 | rv = self.handle_user_exception(e)
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
aquarius_1 | return cors_after_request(app.make_response(f(*args, **kwargs)))
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
aquarius_1 | reraise(exc_type, exc_value, tb)
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
aquarius_1 | raise value
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
aquarius_1 | rv = self.dispatch_request()
aquarius_1 | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
aquarius_1 | return self.view_functionsrule.endpoint
aquarius_1 | File "/aquarius/aquarius/app/assets.py", line 546, in query_text
aquarius_1 | query_result = dao.query(search_model)
aquarius_1 | File "/aquarius/aquarius/app/dao.py", line 64, in query
aquarius_1 | if self.is_listed(f['service']):
aquarius_1 | File "/aquarius/aquarius/app/dao.py", line 72, in is_listed
aquarius_1 | return service['metadata']['curation']['isListed']
aquarius_1 | KeyError: 'isListed'