Giter VIP home page Giter VIP logo

yapapi's Introduction

Golem Python API

Tests - Status Docs status PyPI - Status PyPI version GitHub license GitHub issues

What's Golem and yapapi?

The Golem Network fosters a global group of creators building ambitious software solutions that will shape the technological landscape of future generations by accessing computing resources across the platform. Golem Network is an accessible, reliable, open access and censorship-resistant protocol, democratizing access to digital resources and connecting users through a flexible, open-source platform.

Yapapi is the Python high-level API that allows developers to connect to their Golem nodes and manage their distributed, computational loads through Golem Network.

Golem application development

For a detailed introduction to using Golem and yapapi to run your tasks on Golem and a guide to Golem Network application development in general, please consult our handbook.

Installation

yapapi is available as a PyPI package.

You can install it through pip:

pip install yapapi

Or if your project uses poetry you can add it to your dependencies like this:

poetry add yapapi

API Reference

For a comprehensive API reference, please refer to our official readthedocs page.

Local setup for yapapi developers

Poetry

yapapi uses poetry to manage its dependencies and provide a runner for common tasks.

If you don't have poetry available on your system then follow its installation instructions before proceeding. Verify your installation by running:

poetry --version

Project dependencies

To install the project's dependencies run:

poetry install

By default, poetry looks for the required Python version on your PATH and creates a virtual environment for the project if there's none active (or already configured by Poetry).

All of the project's dependencies will be installed to that virtual environment.

Running unit tests

yapapi uses Poe the Poet for running tasks. Declarations of project tasks can be found in pyproject.toml.

poetry run poe tests_unit

Running goth integration tests

Prerequisites

If you'd like to run the yapapi integration test suite locally then you'll need to install an additional set of dependencies separately.

First, install the dependencies required to run goth.

Next, configure goth's GitHub API token.

Make sure you have OpenSSH installed and added to path

ssh -V

Now, you can install goth and its additional python requirements:

poetry install -E integration-tests

Finally, generate goth's default assets:

poetry run poe tests_integration_assets

Running the tests

Once you have the environment set up, to run all the integration tests, use:

poetry run poe tests_integration

Contributing

It is recommended to run unit tests and static code analysis before committing changes.

poetry run poe checks

You can clean up the artifacts created during the test runs with:

poetry run poe clean

See also

Environment variables

It's possible to set various elements of yagna configuration through environment variables. yapapi currently supports the following environment variables:

  • YAGNA_ACTIVITY_URL, URL to yagna activity API, e.g. http://localhost:7500/activity-api/v1
  • YAGNA_API_URL, base URL to yagna REST API, e.g. http://localhost:7500
  • YAGNA_APPKEY, yagna app key to be used, e.g. a70facb9501d4528a77f25574ab0f12b
  • YAGNA_MARKET_URL, URL to yagna market API, e.g. http://localhost:7500/market-api/v1
  • YAGNA_PAYMENT_NETWORK, Ethereum network name for yagna to use, e.g. rinkeby
  • YAGNA_PAYMENT_DRIVER, payment driver name for yagna to use, e.g. erc20
  • YAGNA_PAYMENT_URL, URL to yagna payment API, e.g. http://localhost:7500/payment-api/v1
  • YAGNA_NET_URL, URL to yagna net APU, e.g. http://localhost:7500/net-api/v1
  • YAGNA_SUBNET, name of the yagna sub network to be used, e.g. public
  • YAPAPI_USE_GFTP_CLOSE, if set to a truthy value (e.g. "1", "Y", "True", "on") then yapapi will ask gftp to close files when there's no need to publish them any longer. This may greatly reduce the number of files kept open while yapapi is running but requires yagna 0.7.3 or newer, with older versions it will cause errors.

yapapi's People

Contributors

83tb avatar approxit avatar azawlocki avatar etam avatar filipgolem avatar jiivan avatar johny-b avatar kamirr avatar kmazurek avatar krunch3r76 avatar lucekdudek avatar maaktweluit avatar mateuszsrebrny avatar mdtanrikulu avatar mfranciszkiewicz avatar mrdarthshoe avatar nieznanysprawiciel avatar prekucki avatar pwalski avatar rad9k avatar scx1332 avatar shadeofblue avatar tworec avatar vandavv avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yapapi's Issues

Task timeout exceeded, because one provider doesn't return results.

Description

OS: Ubuntu 18.04.5 LTS
yagna daemon version: https://github.com/golemfactory/yagna/releases/tag/pre-rel-v0.4.1-fix-738
Python version: 3.6.9
yapapi library version: 0.3.1
yapapi branch: no branch. just tag 0.3.1

Description of the issue:
If one provider doesn't return results, the whole task fails with "task timeout exceeded" and a lot of stack traces are printed.

Actual result:
See above.

Screenshots:
Are logs OK? See attachment.

Steps To Reproduce

  1. Have a provider, that doesn't return results.
  2. Run blender example.

Expected behavior

After some shorter timeout for a single task, it should be sent to another provider.
If the whole activity timeouts, there should be no stack traces printed.

Logs and any additional context

2020-10-30_15:05:04.zip

yacat example exits uncleanly from Ctrl-C

Description

OS: OSX Mojave
yagna daemon version: yagna 0.5.0-pre-rel-v0.4.1-buildtest-102-g495172ea
Python version: 3.8.5
yapapi library version: 0.4.0-alpha.1
yapapi branch: b0.4

Description of the issue:
While running the yacat example, after hitting Ctrl-C while waiting for the first task to be completed (actually, waiting for any offers from providers before the first task), received the following

[2020-11-19 13:40:53,450 ERROR yapapi.summary] Computation failed, reason: CancelledError()
Traceback (most recent call last):
  File "yacat.py", line 157, in <module>
    loop.run_until_complete(task)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/selectors.py", line 558, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "yacat.py", line 161, in <module>
    loop.run_until_complete(task)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "yacat.py", line 102, in main
    keyspace = read_keyspace()
  File "yacat.py", line 30, in read_keyspace
    with open("keyspace.txt", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'keyspace.txt'

Expected behavior

Graceful handling of the issue -> whhen starting the second phase, the code should be aware that Ctrl-C had been pressed during the first one and don't complain about missing keyspace.txt et al ... and shouldn't produce a traceback

Unhandled exception after provider fails to spawn exeunit

Description

OS: Ubuntu 18.04.5 LTS
yagna daemon version: https://github.com/golemfactory/yagna/releases/tag/pre-rel-v0.4.1-fix-738
Python version: 3.6.9
yapapi library version: 0.3.1
yapapi branch: no branch. just tag 0.3.1

Description of the issue:
If provider fails to spawn ExeUnit it causes ugly exceptions in yapapi.

Actual result:
See above.

Steps To Reproduce

  1. Have a provider that fails to spawn ExeUnit. There are some on community subnet, we're trying to get logs from them to establish why it fails.
  2. Run blender example.

Expected behavior

There should be no backtraces in output. Possibly such provider should not be selected again.

Logs and any additional context

[2020-11-05 08:11:55,323 WARNING yapapi.summary] Activity failed on provider 'tintirimintiri', reason: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '257', 'Content-Type': 'application/json', 'Date': 'Thu, 05 Nov 2020 08:11:55 GMT')>
HTTP response body: {"message":"GSB error: Remote service at `/net/0x86ab72b4cb55f824f142501b9423443452c163ca/exeunit/a159f68594724ddaab78dfbe476c41ec/Exec` error: bad request: No service registered under given address '/public/exeunit/a159f68594724ddaab78dfbe476c41ec/Exec'."}

[2020-11-05 08:11:55,418 ERROR yapapi.rest] failed to destroy activity: a159f68594724ddaab78dfbe476c41ec
Traceback (most recent call last):
  File "/home/etam/code/github/golemfactory/yapapi-0.3.1-community/yapapi/rest/activity.py", line 81, in __aexit__
    self._id, yaa.ExeScriptRequest(text='[{"terminate":{}}]')
  File "/home/etam/.cache/pypoetry/virtualenvs/yapapi-zckhRvUZ-py3.6/lib/python3.6/site-packages/ya_activity/api_client.py", line 189, in __call_api
    raise e
  File "/home/etam/.cache/pypoetry/virtualenvs/yapapi-zckhRvUZ-py3.6/lib/python3.6/site-packages/ya_activity/api_client.py", line 186, in __call_api
    _request_timeout=_request_timeout)
  File "/home/etam/.cache/pypoetry/virtualenvs/yapapi-zckhRvUZ-py3.6/lib/python3.6/site-packages/ya_activity/rest.py", line 232, in POST
    body=body))
  File "/home/etam/.cache/pypoetry/virtualenvs/yapapi-zckhRvUZ-py3.6/lib/python3.6/site-packages/ya_activity/rest.py", line 183, in request
    raise ApiException(http_resp=r)
ya_activity.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '257', 'Content-Type': 'application/json', 'Date': 'Thu, 05 Nov 2020 08:11:55 GMT')>
HTTP response body: {"message":"GSB error: Remote service at `/net/0x86ab72b4cb55f824f142501b9423443452c163ca/exeunit/a159f68594724ddaab78dfbe476c41ec/Exec` error: bad request: No service registered under given address '/public/exeunit/a159f68594724ddaab78dfbe476c41ec/Exec'."}

2020-11-05_08:10:04.zip

yapapi hangs on "Demand published on the market"

in case there's some issue with the market - e.g. wrong subnet, wrong yagna version, the requestor agent hangs on "Demand published on the market"...

Suggestion: add some timeout and a followup message - e.g.:
"Please verify your version of the yagna daemon or the subnet name (specified by: --subnet-tag)"

Add an error message when a timeout happens.

Currently, a computation timeout may cause messages about CancelledError exceptions:

[2020-10-05 12:23:12,152 INFO yapapi.rest] activity 629b5646bb124dc98071b5e87bc54560 CLOSE for [CancelledError] 
Traceback (most recent call last):
  File "yapapi/yapapi/runner/__init__.py", line 390, in start_worker
    async for step in remote:
  File "yapapi/yapapi/rest/activity.py", line 140, in __aiter__
    await asyncio.sleep(10)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 648, in sleep
    return await future
asyncio.exceptions.CancelledError

Yapapi should instead tell the user that a timeout happened.

Dump DEBUG log messages to file by default

Having a log file with DEBUG-level messages could be useful for diagnosing errors.

The function yapapi.log.enable_default_logger() could:

  • enable logging to stderr, with level INFO (currently the level is an argument, the default is DEBUG but it's overriden in blender.py)
  • enable logging to a specified file (none by default), with level DEBUG

This would provide sensible defaults: print only nicely formatted summary to the console but also provide us with a more detailed information in a log file.

make pre-realease publish to test.pypi.org

update the github build_publish workflow so that it publishes the release to test.pypi.org when a pre-release is published and only update the real pypi index when an actual release is published

Handle exceptions in score_offers

score_offers() as proposed in 903cfc0 is effectively silenced. It's nice that it won't break proposal loop in find_offers():

async for proposal in events:
emit_progress(ProposalEvent.RECEIVED, proposal.id, from_=proposal.issuer)
score = await strategy.score_offer(proposal)
if score < SCORE_NEUTRAL:
proposal_id, provider_id = proposal.id, proposal.issuer
with contextlib.suppress(Exception):
await proposal.reject()
emit_progress(ProposalEvent.REJECTED, proposal_id, for_=provider_id)
continue
if proposal.is_draft:
emit_progress(ProposalEvent.BUFFERED, proposal.id)
offer_buffer[proposal.issuer] = _BufferItem(datetime.now(), score, proposal)
else:
try:
await proposal.respond(builder.props, builder.cons)
emit_progress(ProposalEvent.RESPONDED, proposal.id)
except Exception:
emit_progress(ProposalEvent.FAILED, proposal.id)

But it makes debugging nearly impossible.

Proposed solution:

  1. Catch only known exceptions
    1. A good example is unflattened proposal (example at the bottom)
  2. Print traceback (in logs or otherwise) of unexpected exceptions
{
	"golem":{
	  "com":{
	     "pricing":{
	        "model":"linear",
	        "model.linear":{
	           "coeffs":[
	              0.001,
	              0.002,
	              0.0
	           ]
	        }
	     },
	     "scheme":"payu",
	     "scheme.payu":{
	        "interval_sec":6.0
	     },
	     "usage":{
	        "vector":[
	           "golem.usage.duration_sec",
	           "golem.usage.cpu_sec"
	        ]
	     }
	  },
	  "inf":{
	     "cpu":{
	        "architecture":"x86_64",
	        "cores":4,
	        "threads":7
	     },
	     "mem":{
	        "gib":10.612468048930168
	     },
	     "storage":{
	        "gib":81.7227783203125
	     }
	  },
	  "node":{
	     "debug":{
	        "subnet":"market-devnet"
	     },
	     "id":{
	        "name":"tworec@mf-market-devnet"
	     }
	  },
	  "runtime":{
	     "name":"vm",
	     "version":"0.1.0"
	  }
	}
}

[UX] Add user friendly warning for new added apis

If anyone tries to run updated yapapi/yajsapi from source (or on upcoming release when released) with yagna < 0.5.0, they will have 404 status code for new payment api.

What I propose instead, it's better to indicate "to be able to use this version of sdk you need to update yagna to something above >= 0.5.0" warning, so user will know what needs to be done.

SSL support

  • remove references to certifi.
  • add extra dependency for ssl

Missing warning when payment init -r is missing

Running latest yapapi master ( 4029eb2 ):

When the user forgets to call yagna payment init -r there is no clear warning in the yapapi blender example output:

python blender.py --subnet-tag devnet-alpha.2
Adding /Users/mwu-gol/src/golem-project/yapapi/examples to sys.path.
progress= {'allocation': '4fe653d4-f5ec-4c0c-8a9a-563b3f11e772', 'spent_amount': Decimal('0'), 'remaining_amount': Decimal('10')}
[2020-10-01 12:36:54,927 INFO yapapi.summary] Demand published on the market
progress= {'stage': 'wait for invoices', 'agreements_to_pay': set()}
progress= {'done': True}

Expected:

WARNING payment account is set to receive only, please run `payment init -r` to turn on sending
progress= {'done': FALSE}

Tip: https://github.com/golemfactory/ya-client/blob/master/specs/payment-api.yaml#L819

Unify terminology in yapapi code & messages

Especially review the usage of Proposal vs Offer: should it be "Received proposals from 5 providers so far" or "Received offers from 5 providers so far"? ("Received offer proposals"?)

blender.py throws an exception on Python 3.6 / 3.7"

Windows 10 Pro
when running blender example got an error:

(yagna-python-tutorial) C:\Users\ederenn\Projects\yapapi\examples\blender>python blender.py --subnet-tag devnet-alpha.2
Adding C:\Users\ederenn\Projects\yapapi\examples to sys.path.
yapapi version: �[33;1m0.3.0�[0m
Using subnet: �[33;1mdevnet-alpha.2�[0m

Traceback (most recent call last):
  File "blender.py", line 102, in <module>
    asyncio.get_event_loop().run_until_complete(task)
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 587, in run_until_complete
    return future.result()
  File "blender.py", line 98, in <module>
    asyncio.get_event_loop().run_until_complete(task)
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 587, in run_until_complete
    return future.result()
  File "blender.py", line 76, in main
    async for task in engine.map(worker, [Task(data=frame) for frame in frames]):
  File "C:\Users\ederenn\.envs\yagna-python-tutorial\lib\site-packages\yapapi\runner\__init__.py", line 362, in map
    storage_manager = await self._stack.enter_async_context(gftp.provider())
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 570, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "C:\Users\ederenn\.envs\yagna-python-tutorial\lib\site-packages\yapapi\storage\gftp.py", line 211, in __aenter__
    process = await self.__get_process()
  File "C:\Users\ederenn\.envs\yagna-python-tutorial\lib\site-packages\yapapi\storage\gftp.py", line 236, in __get_process
    process = self._process or (await self.__exit_stack.enter_async_context(service(_debug)))
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\contextlib.py", line 570, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "C:\Users\ederenn\.envs\yagna-python-tutorial\lib\site-packages\yapapi\storage\gftp.py", line 93, in __aenter__
    "gftp server", stdout=asyncio.subprocess.PIPE, stdin=asyncio.subprocess.PIPE, env=env
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\asyncio\subprocess.py", line 202, in create_subprocess_shell
    stderr=stderr, **kwds)
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 1514, in subprocess_shell
    protocol, cmd, True, stdin, stdout, stderr, bufsize, **kwargs)
  File "C:\Users\ederenn\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 462, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError

(yagna-python-tutorial) C:\Users\ederenn\Projects\yapapi\examples\blender>

I was able to run blender example when upgraded to python 3.8.6

errors are not handled properly

Yagna version: v0.3.3-alpha.5
Yapapi version: 0.2.0
Context: Happens when running blender example on testnet subnet, on a provider that fails to compute.

Task exception was never retrieved
future: <Task finished coro=<Engine.map.<locals>.start_worker() done, defined at /home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py:283> exception=CommandExecutionError('stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)', 5)>
Traceback (most recent call last):
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py", line 312, in start_worker
    async for step in remote:
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/rest/activity.py", line 99, in __aiter__
    raise CommandExecutionError(result.message, last_idx)
yapapi.rest.activity.CommandExecutionError: ('stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)', 5)
Task exception was never retrieved
future: <Task finished coro=<Engine.map.<locals>.start_worker() done, defined at /home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py:283> exception=RuntimeError('readuntil() called while another coroutine is already waiting for incoming data',)>
Traceback (most recent call last):
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py", line 306, in start_worker
    await batch.prepare()
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 132, in prepare
    await step.prepare()
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 58, in prepare
    self._src = await self.do_upload(self._storage)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 76, in do_upload
    src = await storage.upload_bytes(self._data)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/__init__.py", line 69, in upload_bytes
    return await self.upload_stream(len(data), _inner())
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/gftp.py", line 244, in upload_stream
    links = await process.publish(files=[str(file_name)])
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/gftp.py", line 140, in send_message
    msg = await self._proc.stdout.readline()
  File "/usr/lib/python3.6/asyncio/streams.py", line 488, in readline
    line = yield from self.readuntil(sep)
  File "/usr/lib/python3.6/asyncio/streams.py", line 581, in readuntil
    yield from self._wait_for_data('readuntil')
  File "/usr/lib/python3.6/asyncio/streams.py", line 452, in _wait_for_data
    'already waiting for incoming data' % func_name)
RuntimeError: readuntil() called while another coroutine is already waiting for incoming data
Task exception was never retrieved
future: <Task finished coro=<Engine.map.<locals>.start_worker() done, defined at /home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py:283> exception=TypeError('list indices must be integers or slices, not str',)>
Traceback (most recent call last):
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py", line 309, in start_worker
    batch.register(cc)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 136, in register
    step.register(commands)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 116, in register
    _from=f"container:{self._src_path}", to=self._dst_slot.upload_url
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/gftp.py", line 177, in upload_url
    return self._link["url"]
TypeError: list indices must be integers or slices, not str
Task exception was never retrieved
future: <Task finished coro=<Engine.map.<locals>.start_worker() done, defined at /home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py:283> exception=AssertionError('invalid gftp publish response',)>
Traceback (most recent call last):
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/__init__.py", line 306, in start_worker
    await batch.prepare()
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 132, in prepare
    await step.prepare()
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 58, in prepare
    self._src = await self.do_upload(self._storage)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/runner/ctx.py", line 76, in do_upload
    src = await storage.upload_bytes(self._data)
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/__init__.py", line 69, in upload_bytes
    return await self.upload_stream(len(data), _inner())
  File "/home/etam/venvs/yapapi-0.2.0/lib/python3.6/site-packages/yapapi/storage/gftp.py", line 245, in upload_stream
    assert len(links) == 1, "invalid gftp publish response"
AssertionError: invalid gftp publish response

Also related: #47

Support nested JSON in market API bindings

New (decentralised) market implementation uses nested JSON (example below), but yapapi only supports "flattened" JSON.

"properties":{
   "golem":{
      "com":{
         "pricing":{
            "model":"linear",
            "model.linear":{
               "coeffs":[
                  0.001,
                  0.002,
                  0.0
               ]
            }
         },
         "scheme":"payu",
         "scheme.payu":{
            "interval_sec":6.0
         },
         "usage":{
            "vector":[
               "golem.usage.duration_sec",
               "golem.usage.cpu_sec"
            ]
         }
      },
      "inf":{
         "cpu":{
            "architecture":"x86_64",
            "cores":4,
            "threads":7
         },
         "mem":{
            "gib":10.612468048930168
         },
         "storage":{
            "gib":81.7227783203125
         }
      },
      "node":{
         "debug":{
            "subnet":"market-devnet"
         },
         "id":{
            "name":"tworec@mf-market-devnet"
         }
      },
      "runtime":{
         "name":"vm",
         "version":"0.1.0"
      }
   }
},

"Date value out of range error" when running yacat example

yacat example does not work, every execution fails with errors as follows:

[2020-11-18 23:12:17,843 INFO yapapi.summary] Demand published on the market
[2020-11-18 23:12:19,099 INFO yapapi.summary] Received proposals from 1 providers so far
[2020-11-18 23:12:19,851 INFO yapapi.summary] Agreement proposed to provider '[email protected]'
[2020-11-18 23:12:23,065 INFO yapapi.summary] Agreement confirmed by provider '[email protected]'
[2020-11-18 23:12:24,148 WARNING yapapi.summary] Activity failed on provider '[email protected]', reason: date value out of range
[2020-11-18 23:12:31,101 INFO yapapi.summary] Received proposals from 1 providers so far
[2020-11-18 23:12:33,087 INFO yapapi.summary] Agreement proposed to provider '[email protected]'
[2020-11-18 23:12:35,050 INFO yapapi.summary] Agreement confirmed by provider '[email protected]'
[2020-11-18 23:12:36,073 WARNING yapapi.summary] Activity failed on provider '[email protected]', reason: date value out of range

Standardize logging

Currently, all logging is done using direct print calls, with some formatting applied via emit_progress.
I think it would be beneficial to use a well-established solution here, such as the logging module. Here are some functionalities we'd gain from using logging (besides adhering to what's considered a standard approach):

  • log levels (INFO, DEBUG etc.),
  • consistent message formatting,
  • more extensibility, e.g. routing log entries to multiple handlers (like stdout and file).

Computations are not retried after a provider fails to compute

yapapi version: 0.3.0-alpha.1
Reproducible: Quite often. For example, when running every hour, happened 3 times over one night.

Example log from running blender example:

Adding /home/etam/code/github/golemfactory/yapapi-b0.3/examples to sys.path.
Using subnet: .[33;1mdevnet-alpha.2.[0m
[2020-10-06 18:00:06,731 INFO yapapi.summary] Demand published on the market
[2020-10-06 18:00:07,290 INFO yapapi.summary] Received proposals from 1 providers so far
[2020-10-06 18:00:07,291 INFO yapapi.summary] Received proposals from 2 providers so far
[2020-10-06 18:00:07,291 INFO yapapi.summary] Received proposals from 3 providers so far
[2020-10-06 18:00:07,291 INFO yapapi.summary] Received proposals from 4 providers so far
[2020-10-06 18:00:07,291 INFO yapapi.summary] Received proposals from 5 providers so far
[2020-10-06 18:00:07,291 INFO yapapi.summary] Received proposals from 6 providers so far
[2020-10-06 18:00:07,341 INFO yapapi.summary] Received proposals from 7 providers so far
[2020-10-06 18:00:08,672 INFO yapapi.summary] Received proposals from 8 providers so far
[2020-10-06 18:00:08,730 INFO yapapi.summary] Agreement proposed to provider 'whirlwind'
[2020-10-06 18:00:10,988 INFO yapapi.summary] Agreement confirmed by provider 'whirlwind'
[2020-10-06 18:00:12,021 INFO yapapi.summary] Task sent to provider 'whirlwind', task data: 0
[2020-10-06 18:00:12,992 INFO yapapi.summary] Agreement proposed to provider 'eniac'
[2020-10-06 18:00:14,409 INFO yapapi.summary] Agreement confirmed by provider 'eniac'
[2020-10-06 18:00:15,406 INFO yapapi.summary] Task sent to provider 'eniac', task data: 10
[2020-10-06 18:00:16,415 INFO yapapi.summary] Agreement proposed to provider 'z1'
[2020-10-06 18:00:17,538 INFO yapapi.summary] Agreement confirmed by provider 'z1'
[2020-10-06 18:00:18,513 INFO yapapi.summary] Task sent to provider 'z1', task data: 20
[2020-10-06 18:00:23,841 INFO yapapi.summary] Task computed by provider 'whirlwind', task data: 0
[2020-10-06 18:00:24,000 INFO yapapi.summary] Task sent to provider 'whirlwind', task data: 30
[2020-10-06 18:00:27,132 INFO yapapi.summary] Task computed by provider 'eniac', task data: 10
[2020-10-06 18:00:27,278 INFO yapapi.summary] Task sent to provider 'eniac', task data: 40
[2020-10-06 18:00:30,558 INFO yapapi.summary] Task computed by provider 'z1', task data: 20
[2020-10-06 18:00:30,698 INFO yapapi.summary] Task sent to provider 'z1', task data: 50
[2020-10-06 18:00:34,176 WARNING yapapi.summary] Command failed on provider 'whirlwind', command: {'transfer': {'from': 'container:/golem/output/out0030.png', 'to': 'gftp://0x4234e1f5ebaced9eea590e75624c52d0a254245c/AqxcUfoKKEIcniNckMuugx2H3TQIS76bn7lW657jP0ZV5TWKi5YcOfN
bqPK9hxBBX'}}, output: stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)
[2020-10-06 18:00:34,177 WARNING yapapi.summary] Activity failed on provider 'whirlwind', reason: CommandExecutionError('stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)', 2)
[2020-10-06 18:00:36,792 WARNING yapapi.summary] Activity failed on provider 'eniac', reason: CancelledError()
[2020-10-06 18:00:36,792 WARNING yapapi.summary] Activity failed on provider 'z1', reason: CancelledError()
[2020-10-06 18:00:37,558 INFO yapapi.summary] Agreement proposed to provider 'manchester'
.[36;1mTask computed: Task(id=1, data=0), result: output_0.png.[0m
.[36;1mTask computed: Task(id=2, data=10), result: output_10.png.[0m
.[36;1mTask computed: Task(id=3, data=20), result: output_20.png.[0m

`Task exception was never retrieved` in worker tasks

Exceptions raised by start_worker() should be retrieved and should generate computation events (WorkerFailed?).

Currently, if an activity fails, start_worker() may raise an exception which will stop the asyncio task in which it runs but the exception is never retrieved. Retrieving exceptions could be done in the main loop of Engine.map(), in a for loop that examines finished tasks:

   for task in done:
       # present code for service tasks
       if task in services ...
       # new code for worker tasks
       if task in workers and task.exception():
           emit(Events.WorkerFailed(..., reason=str(task.exception()))

The event should contain information that will allow a summary logger to identify the provider for which the worker failed.

yacat.py generates a spurious "No offers have been collected" warning after finishing the keyspace computation

Description

OS: Windows 8.1 / OS X Mojave
yagna daemon version: 0.4.0-a1a50fd9
Python version: 3.6.8 / 3.8.5
yapapi library version: 0.4.0-alpha.0 (master) / 0.3.0
yapapi branch: master / b0.3

Description of the issue:
When executing yacat.py, after the initial task (keyspace size determination) is completed - if this step took more than 20 seconds - a warning is generated.

Actual result:

[2020-10-21 23:31:28,083 INFO yapapi.summary] Demand published on the market
[2020-10-21 23:31:29,130 INFO yapapi.summary] Received proposals from 1 provider
s so far
[2020-10-21 23:31:29,130 INFO yapapi.summary] Received proposals from 2 provider
s so far
[... cut ...]
[2020-10-21 23:31:32,005 INFO yapapi.summary] Task sent to provider 'odra', task
 data: None
[... cut ...]
[2020-10-21 23:31:45,288 INFO yapapi.summary] Task computed by provider 'odra',
task data: None
[2020-10-21 23:31:45,303 INFO yapapi.summary] Computation finished in 17.5s
[2020-10-21 23:31:45,303 INFO yapapi.summary] Negotiated 3 agreements with 3 pro
viders
[2020-10-21 23:31:45,303 INFO yapapi.summary] Provider 'odra' computed 1 tasks
[2020-10-21 23:31:45,303 INFO yapapi.summary] Provider 'ada' did not compute any
 tasks
[2020-10-21 23:31:45,303 INFO yapapi.summary] Provider 'manchester' did not comp
ute any tasks
[2020-10-21 23:31:48,210 INFO yapapi.summary] Total cost: 0.021097865195
�[36;1mTask computed: keyspace size count. The keyspace size is 9025�[0m
[2020-10-21 23:31:48,601 WARNING yapapi.summary] No offers have been collected f
rom the market for 20s. Make sure you're using the latest released versions of y
agna and yapapi, and the correct subnet.

Expected behavior

No warning should be generated - the task has been completed already.

Check if requestor account is initialised

Use /accounts entrypoint in Payment API before starting the computation to check if the account is initialised for sending payments.

Otherwise yapapi can get API errors when trying to accept invoices, which will either abort the whole computation -- if the error occurs before all tasks are computed -- or raise an unhandled error in the finalization phase. Either way it will be messy.

Prerequisite for this is #83.

Unhandled exception in process_invoices()

Description

OS:
[ e.g. Ubuntu 18.04, Windows 10, etc ]

yagna daemon version:
yagna 0.4.0-3a478e5e

Python version:
Python 2.7.10

yapapi library version:
[ can be determined using: python -c "import yapapi; print(yapapi.__version__)"]

yapapi branch:
master

Description of the issue:

during testing session on master
running blender example, after the task was requested on all 5 nodes simultaneously, in logs of one node appeared an error:

yapapi version: 0.4.0a0
Using subnet: devnet-alpha.3
[2020-11-05 13:49:03,048 INFO yapapi.summary] Demand published on the market
[2020-11-05 13:49:23,028 WARNING yapapi.summary] 3 offers have been collected from the market, but no provider has responded for 20s. Make sure you're using the latest released versions of yagna and yapapi, and the correct subnet.
[2020-11-05 13:49:43,033 WARNING yapapi.summary] 4 offers have been collected from the market, but no provider has responded for 40s. Make sure you're using the latest released versions of yagna and yapapi, and the correct subnet.
[2020-11-05 13:49:46,674 INFO yapapi.summary] Received proposals from 1 providers so far
[2020-11-05 13:49:46,689 INFO yapapi.summary] Received proposals from 2 providers so far
[2020-11-05 13:49:47,065 INFO yapapi.summary] Received proposals from 3 providers so far
[2020-11-05 13:49:47,075 INFO yapapi.summary] Agreement proposed to provider '[email protected]'
[2020-11-05 13:49:50,639 INFO yapapi.summary] Agreement confirmed by provider '[email protected]'
[2020-11-05 13:49:51,776 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 0
[2020-11-05 13:49:52,650 INFO yapapi.summary] Agreement proposed to provider '[email protected]'
[2020-11-05 13:49:54,633 INFO yapapi.summary] Agreement confirmed by provider '[email protected]'
[2020-11-05 13:49:56,656 INFO yapapi.summary] Agreement proposed to provider '[email protected]'
[2020-11-05 13:49:59,975 INFO yapapi.summary] Agreement confirmed by provider '[email protected]'
[2020-11-05 13:50:00,172 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 10
[2020-11-05 13:50:01,133 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 20
[2020-11-05 13:50:31,171 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 0
Task computed: Task(id=1, data=0), result: output_0.png
[2020-11-05 13:50:31,326 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 30
[2020-11-05 13:50:32,292 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 20
Task computed: Task(id=3, data=20), result: output_20.png
[2020-11-05 13:50:32,841 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 40
[2020-11-05 13:50:32,862 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 10
Task computed: Task(id=2, data=10), result: output_10.png
[2020-11-05 13:50:33,119 INFO yapapi.summary] Task sent to provider '[email protected]', task data: 50
[2020-11-05 13:50:45,189 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 30
Task computed: Task(id=4, data=30), result: output_30.png
[2020-11-05 13:51:08,830 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 40
Task computed: Task(id=5, data=40), result: output_40.png
[2020-11-05 13:51:20,860 INFO yapapi.summary] Task computed by provider '[email protected]', task data: 50
Wworker-0:  no more frames to render
Wworker-2:  no more frames to render
Wworker-1:  no more frames to render
Task computed: Task(id=6, data=50), result: output_50.png
[2020-11-05 13:51:20,866 INFO yapapi.summary] Computation finished in 138.1s
[2020-11-05 13:51:20,866 INFO yapapi.summary] Negotiated 3 agreements with 3 providers
[2020-11-05 13:51:20,866 INFO yapapi.summary] Provider '[email protected]' computed 2 tasks
[2020-11-05 13:51:20,866 INFO yapapi.summary] Provider '[email protected]' computed 2 tasks
[2020-11-05 13:51:20,867 INFO yapapi.summary] Provider '[email protected]' computed 2 tasks
Task exception was never retrieved
future: <Task finished name='Task-9' coro=<Executor.submit.<locals>.process_invoices() done, defined at /home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/yapapi/executor/__init__.py:193> exception=ServerDisconnectedError('Server disconnected')>
Traceback (most recent call last):
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/yapapi/executor/__init__.py", line 206, in process_invoices
    await invoice.accept(amount=invoice.amount, allocation=allocation)
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/yapapi/rest/payment.py", line 17, in accept
    await self._api.accept_invoice(self.invoice_id, acceptance)
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/ya_payment/api_client.py", line 194, in __call_api
    response_data = await self.request(
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/ya_payment/rest.py", line 268, in POST
    return await self.request(
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/ya_payment/rest.py", line 170, in request
    r = await self.pool_manager.request(**args)
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/aiohttp/client.py", line 551, in _request
    await resp.start(conn)
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/aiohttp/client_reqrep.py", line 890, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/home/etam/.envs/yagna-python-tutorial/lib64/python3.8/site-packages/aiohttp/streams.py", line 605, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

Expected behavior

(What is the expected behavior and/or result in this scenario)

Logs and any additional context

(Any additional information that could help to resolve the issue, which systems were checked to reproduce the issue)
Please upload your logs if possible

Better error reporting in case a computation fails

Currently it may look as follows (in the DEBUG log):

[2020-10-21 00:10:53,497 DEBUG yapapi.runner] ComputationFailed(reason='ApiException()')
[2020-10-21 00:10:53,497 ERROR yapapi.summary] Computation failed, reason: ApiException()

This is because we use __repr__() function on the exception that causes the failure. In case of ApiException(), __str__() would give us more useful information (but for certain other exceptions,
__str__() returns just the empty string).

Moreover, we don't store the exception info with ComputationFailed event.

Compare with how worker errors are reported:

[2020-10-21 15:07:28,698 DEBUG yapapi.executor] WorkerFinished(agr_id='241d4b5f768d226c5a6a06b2650b22826053fdb43ed0e060d09664527215e311', exc_info=(<class 'yapapi.rest.activity.CommandExecutionError'>, CommandExecutionError('stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)', 5), <traceback object at 0x7f5e65c36910>))
Traceback (most recent call last):
  File "/home/azawlocki/golem/yagna-integration/yapapi/yapapi/executor/__init__.py", line 354, in start_worker
    await command_generator.athrow(*sys.exc_info())
  File "examples/blender/blender.py", line 50, in worker
    yield ctx.commit()
  File "/home/azawlocki/golem/yagna-integration/yapapi/yapapi/executor/__init__.py", line 322, in start_worker
    async for step in remote:
  File "/home/azawlocki/golem/yagna-integration/yapapi/yapapi/rest/activity.py", line 137, in __aiter__
    raise CommandExecutionError(result.message, last_idx)
yapapi.rest.activity.CommandExecutionError: ('stderr: Local service error: Transfer error: IO error: No such file or directory (os error 2)', 5)

This is possible since the WorkerFinished event stores the whole traceback. We should use similar mechanism when reporting computation failures.

Yield completed tasks from `Engine.map()

Currently Engine.map() yields some progress information, for example:

yield {"stage": "wait for invoices", "agreements_to_pay": agreements_to_pay}
(see also other yields from this function).

There's already another mechanism for reporting progress in Engine.map(): the emit_progress function. So I think yield statements, as they are now, should be replaced by calls to emit_progress for an appropriate resource type (whole computation?).

Instead of the progress information I think Event.map() should yield some objects that correspond (preferably, 1-1) to the sequence of input items passed as the data argument: Task instances or some objects representing task results or statuses of processed tasks. That would be better for a method called map(). (I know we're planning to change this name to something like run(), but still.)

gftp on Mac

Works with gftp built from release/vm-poc branch.

Make Executor.submit more general in its `tasks` argument

Ideas from @prekucki

  1. Auto-wrap elements of tasks in Task:
def submit(worker, maybe_tasks):

    tasks_for_sure = (x if isinstance(x, Task) else Task(data=x) for x in maybe_tasks)
  1. Calls with a single object or none at all:
    result = await executor.submit(worker, 43)
    result = await executor.submit(worker)

It may be better to add variants for submit() for those scenarios in order not to loose type information (first check what can we achieve with mypys function overloading https://mypy.readthedocs.io/en/stable/more_types.html#function-overloading).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.