Too many open files due to logging

[proycon]

I may have run into a problematic limitation of sciluigi. If I schedule a lot of tasks for parallel processing, logs will be opened for each of them, but this number may exceed the number of file handles:

2016-06-28 15:33:49 |  WARNING | Will not schedule FoliaValidator(instance_name=sciluigi_workflow, startcomponent=, inputslot=, inputfile=basilex.foliadir/Data/0/63/9.xml, outputdir=output, folia_extension=xml) or any dependencies due to error in deps() method:
Traceback (most recent call last):
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/worker.py", line 608, in _add
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/task.py", line 471, in deps
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/luigi-2.1.1-py3.4.egg/luigi/task.py", line 443, in _requires
  File "/vol/customopt/lamachine/lib/python3.4/site-packages/sciluigi-0.9.4b5-py3.4.egg/sciluigi/workflow.py", line 92, in requires
  File "/usr/lib/python3.4/logging/__init__.py", line 1006, in __init__
  File "/usr/lib/python3.4/logging/__init__.py", line 1030, in _open
OSError: [Errno 24] Too many open files: '/scratch/proycon/basilexvalidation/log/workflow_foliavalidator_started_20160628_133349_909895.log'

[proycon]

Actually, I now realize this possibly won't happen in vanilla sciluigi, in my extended implementation LuigiNLP I use multiple WorkflowTasks as a means of subworkflows (issue #7). I'll have to find a solution in my own code base then.

[samuell]

@proycon Ah, IC! Thanks for reporting! This stresses the importance of issue #7 I guess. I haven't got to that yet, but hope to look at it during the summer.