Comments (4)
The deadlock detector returned this. It seems like TWS is not responding on some ID requests.
r-Scheduler-Serial-1 | i.d.s.s.SessionState | Internal Error 'cc58a3f9-12ab-45b1-b71b-c53912c3facc' java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'Exception'>
Value: OrderIdRequest.get() timed out after 120.0 sec.
Line: 37
Namespace: get
File: /opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py
Traceback (most recent call last):
File "<string>", line 426, in <module>
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/__init__.py", line 1080, in order_place
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/tws_client.py", line 907, in next_order_id
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/requests.py", line 27, in next_order_id
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py", line 37, in get
at org.jpy.PyLib.executeCode(PyLib.java:-2)
at org.jpy.PyObject.executeCode(PyObject.java:138)
at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:56)
at io.deephaven.engine.util.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:190)
at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:46)
at io.deephaven.engine.util.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:189)
at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:147)
at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:71)
at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:85)
at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$8(ConsoleServiceGrpcImpl.java:219)
at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1299)
at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:847)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:143)
at java.lang.Thread.run(Thread.java:829)
This is triggered by the following code in tws_client.py
.
def next_order_id(self) -> int:
"""Gets the next valid order ID."""
return self.request_id_manager.next_order_id(self.order_id_queue)
from deephaven-ib.
#31 allows different methods to get the next order id. This is a potential work around for the problem.
from deephaven-ib.
Saw this again:
r-Scheduler-Serial-1 | i.d.s.s.SessionState | Internal Error '9aaf405f-8e59-424a-8f8f-2c5e4ed13d43' java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'Exception'>
Value: OrderIdRequest.get() timed out after 60.0 sec. A possible deadlock or TWS bug was detected! You may be able to avoid this problem by using a different OrderIdStrategy. Please create an issue at https://github.com/deephaven-examples/deephaven-ib/issues containing this error message
Stack Traces:
<_MainThread(MainThread, started 140166847809280)>Thread stack not found: thread=140166847809280
<Thread(DeadlockMonitor, started daemon 140163184277248)> File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_internal/threading.py", line 34, in _run
time.sleep(self.sleep_sec)
<Thread(OrderIdEventQueueRetry, started daemon 140161534646016)> File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py", line 95, in _retry
sleep(0.01)
<_DummyThread(Dummy-2, started daemon 140163190327040)> File "<string>", line 428, in <module>
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/__init__.py", line 1101, in order_place
req_id = self._client.next_order_id()
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/tws_client.py", line 939, in next_order_id
return self.request_id_manager.next_order_id(self.order_id_queue)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/requests.py", line 27, in next_order_id
oid = request.get()
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py", line 55, in get
trace = trace_all_threads_str()
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_internal/trace.py", line 27, in trace_all_threads_str
rst += trace_thread_str(th)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_internal/trace.py", line 15, in trace_thread_str
return "".join(traceback.format_stack(sys._current_frames()[thread.ident]))
<Thread(OrderIdEventQueueRetry, started daemon 140161526253312)> File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py", line 95, in _retry
sleep(0.01)
<EReader(Thread-3, started daemon 140161517074176)> File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/reader.py", line 35, in run
data = self.conn.recvMsg()
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/connection.py", line 93, in recvMsg
buf = self._recvAllMsg()
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/connection.py", line 114, in _recvAllMsg
buf = self.socket.recv(4096)
<Thread(IbTwsClient, started daemon 140161508681472)> File "/usr/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/client.py", line 263, in run
self.decoder.interpret(fields)
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/decoder.py", line 1296, in interpret
handleInfo.processMeth(self, iter(fields))
File "/opt/deephaven-venv/lib/python3.7/site-packages/ibapi/decoder.py", line 795, in processSymbolSamplesMsg
self.wrapper.symbolSamples(reqId, contractDescriptions)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/tws_client.py", line 470, in symbolSamples
self.contract_registry.request_contract_details_nonblocking(cd.contract)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/contract_registry.py", line 142, in request_contract_details_nonblocking
self._request_contract_details(contract=contract)
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/contract_registry.py", line 201, in _request_contract_details
req_id = self._client.request_id_manager.next_id()
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/requests.py", line 19, in next_id
with self._lock:
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_internal/threading.py", line 131, in __enter__
self.acquire()
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_internal/threading.py", line 113, in acquire
ret = self.lock.acquire(blocking)
Line: 57
Namespace: get
File: /opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py
Traceback (most recent call last):
File "<string>", line 428, in <module>
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/__init__.py", line 1101, in order_place
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/tws_client.py", line 939, in next_order_id
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/requests.py", line 27, in next_order_id
File "/opt/deephaven-venv/lib/python3.7/site-packages/deephaven_ib/_tws/order_id_queue.py", line 57, in get
at org.jpy.PyLib.executeCode(PyLib.java:-2)
at org.jpy.PyObject.executeCode(PyObject.java:138)
at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:56)
at io.deephaven.engine.util.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:188)
at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:46)
at io.deephaven.engine.util.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:187)
at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:145)
at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:71)
at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:84)
at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$8(ConsoleServiceGrpcImpl.java:187)
at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1299)
at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:847)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.deephaven.server.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:157)
at java.lang.Thread.run(Thread.java:829)
from deephaven-ib.
I will note that this deadlock seems to happen when deephaven-ib
is used at the start of the day. If deephaven-ib
is bounced and then reconnects, the problem does not seem to recur during the day.
from deephaven-ib.
Related Issues (20)
- Register error code 10147
- Update README.md note when DHaaL supports more architectures
- CI is not publishing whls to pypi HOT 1
- README images do not render in pypi
- Support full Deephaven system
- dev and release docker examples are broken
- Docker image cleanup is broken
- Make IB API version easier to update
- Class cast exception when logging
- Update to Deephaven 0.16
- Support the IB Client Portal REST Web API
- `request_tick_data_realtime` column type mismatch
- Exception when canceling orders
- Possible `OrderIdRequest.get()` deadlock
- Inconsistent use of `Account` and `AcctNumber` column names
- Ubuntu22.04 - tws client connect error Python3.10 local install no docker HOT 3
- Instead of using ibkr real time data I want to use the real time data from some place else: tda-api HOT 1
- `account_positions` table can have duplicate entries HOT 1
- Test issue
- Support the deephaven `server-ui` Docker image
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deephaven-ib.