krunch3r76 / entropythief Goto Github PK
View Code? Open in Web Editor NEWpilfer provider entropy stashes
License: GNU General Public License v3.0
pilfer provider entropy stashes
License: GNU General Public License v3.0
example: if all tasks fail they are not replaced until all have timed out...
the assembly routines called from the third party module is apparently creating a memory leak. i am developing a c extension to be used in its place.
this should be available and quickly summarized from numerous sources or by referring the reader to section 2.1 at https://software.intel.com/content/www/us/en/develop/articles/intel-digital-random-number-generator-drng-software-implementation-guide.html
during the mixing and writing of the random bytes to the pipe writer and the view, a shutdown waits until all of it has been written. short-circuiting the sending of the bytes sent to the view could solve the issue but may require adding a separate asyncio communication channel (i.e. queue) which contains only metadata such as events.
additionally, a shared variable can be set so that the controller can recognize whenever Golem is not mid-task.
when a reader such as the included pipe reader empties the named pipe completely in one read, it appears entropythief does not recognize this. this should be easily fixed soon.
https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.payload.vm
it is now possible to specify cpu capabilities for the provider's running the vm on the vm object itself. the marketstrategy code can be safely removed and replaced by this.
because heroes are cool
[2021-08-10T13:33:25.497-0700 DEBUG yapapi.rest.market] terminateAgreement(d488072e6b509da6a4cd53f367d669eee2a10175797e6cf91c405f30cf6c36e1) failed
Traceback (most recent call last):
File "./entropythief.py", line 389, in <module>
loop.run_until_complete(task)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
event_list = self._selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/krunch3r/.local/lib/python3.8/site-packages/yapapi/rest/market.py", line 85, in terminate
await self._api.terminate_agreement(self._id, request_body=reason)
File "/home/krunch3r/.local/lib/python3.8/site-packages/ya_market/api_client.py", line 205, in __call_api
raise e
File "/home/krunch3r/.local/lib/python3.8/site-packages/ya_market/api_client.py", line 193, in __call_api
response_data = await self.request(
File "/home/krunch3r/.local/lib/python3.8/site-packages/ya_market/rest.py", line 268, in POST
return await self.request(
File "/home/krunch3r/.local/lib/python3.8/site-packages/ya_market/rest.py", line 180, in request
raise ApiException(http_resp=r)
ya_market.exceptions.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: <CIMultiDictProxy('Content-Length': '446', 'Content-Type': 'application/json', 'Date': 'Tue, 10 Aug 2021 20:33:25 GMT')>
HTTP response body: {"message":"Protocol error while terminating: Terminate Agreement [R-d488072e6b509da6a4cd53f367d669eee2a10175797e6cf91c405f30cf6c36e1] GSB error: Remote service at `/net/0x69b0bd0dab9946f6f1a32eba89df43231c45413d/market/protocol/mk1/negotiation/provider/agreement/AgreementTerminated` error: Called service `/net/0x69b0bd0dab9946f6f1a32eba89df43231c45413d/market/protocol/mk1/negotiation/provider/agreement/AgreementTerminated` is unavailable.."}
when a reader has been attempting to read from the same file descriptor to which a former process was writing and that former process has exited, there appears to be a need to recreate the pipe for the new process to see it (sometimes), ie assign a new file descriptor. this is being investigated.
entropythief defies the traditional definition of randomness of having "statistical independence, uniform distribution, and unpredictability". showing if some or all of these criteria are violated may actually prove entropythief is fulfilling its design purpose.
there are at least a couple first shots to take at this: one, see if asynchronous code running ncurses helps by switching contexts on chunks. two, determine if scrolling off the screen would occur and only write the hind side.
entropythief will report this:
The model threw the following exception:
ApiException
(400)
Reason: Bad Request
HTTP response headers: <CIMultiDictProxy('Content-Length': '51', 'Content-Type': 'application/json', 'Date': 'Thu, 05 Aug 2021 04:29:01 GMT')>
HTTP response body: {"message":"Insufficient funds to make allocation"}
not sure how i want to handle this generic exception or how i might be proactive about it.
it may be useful to pause a session in order to make adjustments to worker count and buflen
enhancement in planning
the logic in pipe_writer is causing bytes to be rewritten. it has been mitigated some by removing calls to poll that does not play nice with asyncio. a redesign is in progress.
planning this enhancement
multiple named pipes brings about problems with undefined system hard limits and such when reading hundreds of megabytes of randomness. i have learned much about named pipes and am reprogramming to use a single named pipe (this time as it is meant to be used :-p -- or at least more the unix way having learned) and buffering inside an object.
currently it has been necessary to await each task download, but the asynchronous model is designed to attend to many pending io operations (as from other tasks) at once. the problem may be with the callback, and that is being investigated as of this writing (at tag alpha-v7.4)
if the named pipe is being continually read from, the status line won't reflect this real-time during provisioning of new work.
two targets to consider:
TaskResultWriter might be able to communicate this information to the controller
A task itself might have a callback in its data to refresh and inform the controller.
there is utilization on 1 core that often goes to 100% while the pipe is being actively read. this is a suspected entropythief issue that should be resolved by making PipeWriter sleep requiring an asynchronous partial interface (or hopefully not a complete asynchronous redesign) where time is yielded to the cpu without blocking. currently it is "unblocking"
the named pipe feature is not supported by Python in windows; however, it appears to be supported in powershell (reference needed). this is being researched along with alternative IPC tools.
references:
https://hackinparis.com/data/slides/2017/2017_Cohen_Gil_The_forgotten_interface_Windows_named_pipes.pdf
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-callnamedpipea
https://rkeithhill.wordpress.com/2014/11/01/windows-powershell-and-named-pipes/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.