The luiginlp from languagemachines

luiginlp's People

Contributors

Stargazers

Watchers

luiginlp's Issues

Too many open files due to logging

If I schedule a lot of components for parallel processing, logs will be opened for each of them, but this number may exceed the number of file handles:

2016-06-28 15:33:49 |  WARNING | Will not schedule FoliaValidator(instance_name=sciluigi_workflow, startcomponent=, inputslot=, inputfile=basilex.foliadir/Data/0/63/9.xml, outputdir=output, folia_extension=xml) or any dependencies due to error in deps() method:
Traceback (most recent call last):
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/worker.py", line 608, in _add
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/task.py", line 471, in deps
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/task.py", line 443, in _requires
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/sciluigi-0.9.4b5-py3.4.egg/sciluigi/workflow.py", line 92, in requires
  File "/usr/lib/python3.4/logging/__init__.py", line 1006, in __init__
  File "/usr/lib/python3.4/logging/__init__.py", line 1030, in _open
OSError: [Errno 24] Too many open files: '/scratch/proycon/basilexvalidation/log/workflow_foliavalidator_started_20160628_133349_909895.log'

Major scaling problems processing when dynamically yielding huge amount of tasks for parallel execution

The luigi scheduler doesn't seem to cope well with a huge amount (300,000 in my test) of scheduled tasks (siblings in the dependency graph). Scheduling becomes the bottleneck and load is not distributed over available workers.

Attempting to bypass this issue by grouping tasks in batches...

Recommend Projects