Comments (15)
GuillaumeCisco
commented on September 26, 2024
Hello @chrisalexandrepena,
Did you try with different terminals window?
Do you get the same error?
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
I tried running them in separate screen sessions but I always get this error. What is strange is that when I input all the lines specified in the repository README for the firt time, without the --detach flag:
- It displayed the logs of my first worker
- I stoped the logs with a simple ctrl+C
- I could see through celery flower that my worker was still up
- If I powered off my running worker, all of a sudden celery launched the second one
- If I powered off that new worker celery launched the 3rd one, and so on
I haven't been able to replicate that behaviour since
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
Quick note, you will maybe need to change DJANGO_SETTINGS_MODULE=backend.settings.dev to DJANGO_SETTINGS_MODULE=backend.settings.celery.dev, but this implies having a ledger running :)
So you still have the issue?
Are you on linux, mac, windows, everything else?
Maybe --pidfile is the solution for your environment as described here: https://stackoverflow.com/questions/53521959/dockercelery-error-pidfile-celerybeat-pid-already-exists
But I don't think you need to use it at tall.
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
My ledger is running so that shouldn't be a problem :)
I've just tried it, but the DJANGO_SETTINGS_MODULE=backend.settings.celery.dev still gives me the error. I had already tried adding a --pidfile="" before, and it does work. Only I wasn't sure if that would cause any problems with the rest of the app?
I'm running the system on a linux virtuabox (ubuntu server 18.04).
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
It should not cause any problems with the rest of the app, but I suggest you to check all the celeries running on your machine.
Maybe you will need to kill some: ps aux | grep celery
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
If I try to launch all the tasks in a row, using --detach and --pidfile="" flags, and after killing all previous celery worker instances I get a strange error:
Traceback (most recent call last):
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'data'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/hfc/fabric/user.py", line 257, in _restore_state
enrollment = state_dict['enrollment']
KeyError: 'enrollment'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/dev/substra-backend/.venv/bin/celery", line 8, in <module>
sys.exit(main())
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
_main()
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in handle_argv
return self.execute(command, argv)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/celery.py", line 420, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/worker.py", line 221, in run_from_argv
*self.parse_options(prog_name, argv, command))
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/base.py", line 398, in parse_options
self.parser = self.create_parser(prog_name, command)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/base.py", line 414, in create_parser
self.add_arguments(parser)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/bin/worker.py", line 277, in add_arguments
default=conf.worker_state_db,
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/utils/collections.py", line 126, in __getattr__
return self[k]
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/utils/collections.py", line 429, in __getitem__
return getitem(k)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/utils/collections.py", line 278, in __getitem__
return mapping[_key]
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/collections/__init__.py", line 987, in __getitem__
if key in self.data:
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/app/base.py", line 141, in data
return self.callback()
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/app/base.py", line 924, in _finalize_pending_conf
conf = self._conf = self._load_config()
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/app/base.py", line 934, in _load_config
self.loader.config_from_object(self._config_source)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/loaders/base.py", line 131, in config_from_object
self._conf = force_mapping(obj)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/celery/utils/collections.py", line 46, in force_mapping
if isinstance(m, (LazyObject, LazySettings)):
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/django/utils/functional.py", line 213, in inner
self._setup()
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/django/conf/__init__.py", line 44, in _setup
self._wrapped = Settings(settings_module)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/django/conf/__init__.py", line 107, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/dev/substra-backend/backend/backend/settings/celery/dev.py", line 1, in <module>
from ..deps.ledger import *
File "/home/ubuntu/dev/substra-backend/backend/backend/settings/deps/ledger.py", line 32, in <module>
cert_path=LEDGER['client']['cert_path']
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/hfc/fabric/user.py", line 348, in create_user
user = User(name, org, state_store)
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/hfc/fabric/user.py", line 58, in __init__
self._restore_state()
File "/home/ubuntu/dev/substra-backend/.venv/lib/python3.6/site-packages/hfc/fabric/user.py", line 272, in _restore_state
raise IOError("Cannot deserialize the user", e)
OSError: [Errno Cannot deserialize the user] 'enrollment'
When I check after getting the error, a random number of the workers have been successfuly spawned. If I launch it again, I get the same error but can see new workers spawning. And if I remove all active workers again and relaunch my command I end up with the same error message and a different number of successfuly spawned workers...
Using the DJANGO_SETTINGS_MODULE=backend.settings.celery.dev doesn't affect the outcome...:(
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
What tasks are you trying to launch?
Where does come from this error? From a celery worker?
Is a docker instance running a fabric ca is running?
Can you show the result of:
$> docker ps -a
Thanks,
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
The command I'm trying to launch is:
DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,scheduler,celery --hostname owkin.scheduler --pidfile="" && \
DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,owkin.worker,celery --hostname owkin.worker --pidfile="" && \
DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,scheduler,celery --hostname chu-nantes.scheduler --pidfile="" && \
DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,chu-nantes.worker,celery --hostname chu-nantes.worker --pidfile="" && \
DJANGO_SETTINGS_MODULE=backend.settings.common celery --detach --pidfile="" -A backend beat -l info
Here are all my docker containers:
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
I've just tested your commands and it seems to work:
$> DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,scheduler,celery --hostname owkin.scheduler --pidfile="" && \
> DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=owkin BACKEND_DEFAULT_PORT=8000 BACKEND_PEER_PORT_EXTERNAL=9051 celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,owkin.worker,celery --hostname owkin.worker --pidfile="" && \
> DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,scheduler,celery --hostname chu-nantes.scheduler --pidfile="" && \
> DJANGO_SETTINGS_MODULE=backend.settings.celery.dev BACKEND_ORG=chu-nantes BACKEND_DEFAULT_PORT=8001 BACKEND_PEER_PORT_EXTERNAL=7051 celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,chu-nantes.worker,celery --hostname chu-nantes.worker --pidfile="" && \
> DJANGO_SETTINGS_MODULE=backend.settings.common celery --detach --pidfile="" -A backend beat -l info
$> ps aux | grep celery
guillau+ 30691 0.0 0.2 667912 70660 pts/10 Sl 16:55 0:00 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,scheduler,celery --hostname owkin.scheduler --pidfile=
guillau+ 30698 101 0.3 629560 102384 ? S 16:55 0:08 [celeryd: [email protected]:MainProcess] -active- (worker -l info -B -Q owkin,scheduler,celery -E -A backend --hostname=owkin.scheduler)
guillau+ 30807 0.0 0.2 667916 71052 pts/10 Sl 16:55 0:00 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n owkin -Q owkin,owkin.worker,celery --hostname owkin.worker --pidfile=
guillau+ 30813 97.1 0.1 147064 53300 ? R 16:55 0:06 /home/guillaume/.venv/substrabac/bin/python -m celery worker -l info -B -Q owkin,owkin.worker,celery -E -A backend --hostname=owkin.worker
guillau+ 30847 0.0 0.2 667904 70904 pts/10 Sl 16:55 0:00 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,scheduler,celery --hostname chu-nantes.scheduler --pidfile=
guillau+ 30855 108 0.2 537088 68528 ? R 16:55 0:05 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,scheduler,celery --hostname chu-nantes.scheduler --pidfile=
guillau+ 30890 0.0 0.2 667916 70952 pts/10 Sl 16:55 0:00 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,chu-nantes.worker,celery --hostname chu-nantes.worker --pidfile=
guillau+ 30896 100 0.2 537100 69128 ? R 16:55 0:04 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery -E --detach -A backend worker -l info -B -n chunantes -Q chu-nantes,chu-nantes.worker,celery --hostname chu-nantes.worker --pidfile=
guillau+ 30928 0.0 0.1 617796 58628 pts/10 Sl 16:56 0:00 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery --detach --pidfile= -A backend beat -l info
guillau+ 30937 132 0.1 487144 55244 ? R 16:56 0:02 /home/guillaume/.venv/substrabac/bin/python /home/guillaume/.venv/substrabac/bin/celery --detach --pidfile= -A backend beat -l info
guillau+ 30989 0.0 0.2 667928 71004 ? Sl 16:56 0:00 /home/guillaume/.venv/substrabac/bin/python -m celery worker -l info -B -Q owkin,scheduler,celery -E -A backend --hostname=owkin.scheduler
guillau+ 31007 0.0 0.2 628484 87104 ? R 16:56 0:00 [celeryd: [email protected]:MainProcess] -active- (worker -l info -B -Q owkin,scheduler,celery -E -A backend --hostname=owkin.scheduler)
guillau+ 31010 0.0 0.2 628740 87244 ? S 16:56 0:00 [celeryd: [email protected]:ForkPoolWorker-2]
guillau+ 31014 0.0 0.0 14784 1000 pts/10 S+ 16:56 0:00 grep --color=auto celery
All is running correctly.
I've also not being able to reproduce your error with the pidfile.
Maybe there is something wrong with your python virtualenv.
Can you give us the output of:
$> docker logs -f run-owkin
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
Of course:
external_orgs: ['chu-nantes']
Sign update proposal on chu-nantes ...
Send update proposal with org: chu-nantes...
Wait For Peers to join channel
Join channel substrachannel with peers ['peer1-owkin', 'peer2-owkin'] ...
Peers ['peer1-owkin', 'peer2-owkin'] successfully joined channel substrachannel
Installing chaincode on ['peer1-owkin', 'peer2-owkin'] ...
Installing chaincode on ['peer1-chu-nantes', 'peer2-chu-nantes'] ...
policy: OR('owkinMSP.member', 'chu-nantesMSP.member')
Upgraded chaincode with policy: {'identities': [{'role': {'name': 'member', 'mspId': 'owkinMSP'}}, {'role': {'name': 'member', 'mspId': 'chu-nantesMSP'}}], 'policy': {'1-of': [{'signed-by': 0}, {'signed-by': 1}]}} and result: "{'name': 'substracc', 'version': '2.0', 'escc': 'escc', 'vscc': 'vscc', 'policy': {'version': 0, 'rule': {'n_out_of': {'n': 1, 'rules': [{'signed_by': 0}, {'signed_by': 1}]}}, 'identities': [{'principal_classification': 'ROLE', 'principal': {'msp_identifier': 'owkinMSP', 'role': 'MEMBER'}}, {'principal_classification': 'ROLE', 'principal': {'msp_identifier': 'chu-nantesMSP', 'role': 'MEMBER'}}]}, 'data': {'hash': b'"\x0c% \xa2R\x81*\x89#\x18\x0fl\xaf8\xd9\x95{O\x8bN\xbc\xad\x15\xe8\x8b)\xebrz=8', 'metadatahash': b'\x0fz\x15\xa7\x95\x01*\xca\xe2\x88P \xae3\xf5\x07\xba\xee\xcd\xfb\xa85g\xa6\x7f\xafqW\x0f\x0f\xad\xc8'}, 'id': b'\xec\x06\xcf\xd7{\x16 \x13\x83H\x0b8\xe2J\xfa\xee\x91\\/\x1eLg\x85\x1d\xcdj\xdb_HO\x02\xf5', 'instantiation_policy': {'version': 0, 'rule': {'n_out_of': {'n': 1, 'rules': [{'signed_by': 0}, {'signed_by': 1}]}}, 'identities': [{'principal_classification': 'ROLE', 'principal': {'msp_identifier': 'chu-nantesMSP', 'role': 'ADMIN'}}, {'principal_classification': 'ROLE', 'principal': {'msp_identifier': 'owkinMSP', 'role': 'ADMIN'}}]}}"
Removing chaincode docker containers ...
1daa3b149306
570f179f8116
Try to query chaincode from peer ['peer1-owkin', 'peer2-owkin'] on org owkin
Queried chaincode, result: []
Congratulations! Ledger has been correctly initialized.
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
Everything is running great, I don't see how you could see these errors.
Did you try with a docker setup for substra-backend?
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
I had not tried the docker setup, it works indeed very well thanks :)
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
Can we close this issue then?
from substra-backend.
chrisalexandrepena
commented on September 26, 2024
I guess you can :)
from substra-backend.
GuillaumeCisco
commented on September 26, 2024
Thanks,
from substra-backend.
Related Issues (20)
- Algo fetch: HTTP errors aren't displayed in the logs HOT 4
- Local setup (skaffold): Incorrect DNS assumptions
- Cannot launch using docker-compose HOT 1
- Error 500 when linking dataset with datasamples
- Error 400 when adding a dataset HOT 3
- single-node compute plan: worker tries to delete missing image HOT 2
- Chart not compatible with K8S >=1.16 HOT 1
- Document password requirements on the Helm chart values
- 502 when under load HOT 5
- [Edge case] Crash when composite traintuple head and trunk models are identifcal HOT 1
- Evicted training tasks pods are never deleted
- Serializers for the CompositeTraintuple are not consistent (single vs compute plan)
- Can't get a stable deployment using line 0.7.1 of the compatibility table HOT 2
- authenticate_worker in Process has a strange behavior (not systematic)
- Ease the process to add extra volumes for local data registration HOT 2
- Add an auto-cleanup of old docker-registry images
- Is this project dead? HOT 1
- installation of backend with helm does not seem to work with microk8s HOT 1
- which backend url do I use to instantiate a Client? HOT 1
- BUG: container kaniko exit 1 Susbtra-backend crash HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from substra-backend.