Multiprocessing error during import

[SeanZhang99]

First of all, this error occurs during the import of a brain_pipe module "brain_pipe.preprocessing.brain.link". However, the script using this module is in your another repo. I'm not sure if it is appropriate to post the issue here, but I think the two repos are all maintained by you guys so I just add this issue in the repo where an error occur.

And here is the description:

Windows 10. Conda environment list results:
$ conda list
packages in environment at C:\Users\sean\miniconda3\envs\brain-pip:
Name Version Build Channel
audioread 3.0.1 pypi_0 pypi
brain-pipe 0.0.4 pypi_0 pypi
brian2 2.7.1 pypi_0 pypi
brian2hears 0.9.2 pypi_0 pypi
bzip2 1.0.8 h2bbff1b_6
ca-certificates 2024.7.2 haa95532_0
certifi 2024.7.4 pypi_0 pypi
cffi 1.17.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
contourpy 1.2.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cython 3.0.11 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
expat 2.6.2 hd77b12b_0
fonttools 4.53.1 pypi_0 pypi
idna 3.7 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
libffi 3.4.4 hd77b12b_1
librosa 0.10.2.post1 pypi_0 pypi
llvmlite 0.43.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.9.1.post1 pypi_0 pypi
mne 1.7.1 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
msgpack 1.0.8 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numpy 2.0.1 pypi_0 pypi
openssl 3.0.14 h827c3e9_0
packaging 24.1 pypi_0 pypi
pillow 10.4.0 pypi_0 pypi
pip 24.0 py312haa95532_0
platformdirs 4.2.2 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
python 3.12.4 h14ffc60_1
python-dateutil 2.9.0.post0 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
scikit-learn 1.5.1 pypi_0 pypi
scipy 1.14.0 pypi_0 pypi
setuptools 72.1.0 py312haa95532_0
six 1.16.0 pypi_0 pypi
soundfile 0.12.1 pypi_0 pypi
soxr 0.4.0 pypi_0 pypi
sqlite 3.45.3 h2bbff1b_0
sympy 1.13.1 pypi_0 pypi
threadpoolctl 3.5.0 pypi_0 pypi
tk 8.6.14 h0416ee5_0
tqdm 4.66.5 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024a h04d1e81_0
urllib3 2.2.2 pypi_0 pypi
vc 14.2 h2eaa2aa_4
vs2015_runtime 14.29.30133 h43f2093_4
wheel 0.43.0 py312haa95532_0
xz 5.4.6 h8cc25b3_1
zlib 1.2.13 h8cc25b3_1

(VS Code)
run python files in terminal

(brain-pip) parent_folder\training>C:/Users/sean/miniconda3/envs/brain-pip/python.exe c:/Users/sean/Downloads/auditory-eeg-challenge-2024-code-main/preprocessing_code/sparrKULee.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 131, in _main
    prepare(preparation_data)
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "c:\Users\sean\Downloads\auditory-eeg-challenge-2024-code-main\preprocessing_code\sparrKULee.py", line 26, in <module>
    from brain_pipe.preprocessing.brain.link import (
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\preprocessing\brain\link.py", line 9, in <module>
    from brain_pipe.utils.multiprocess import MultiprocessingSingleton
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\utils\multiprocess.py", line 45, in <module>
    class MultiprocessingSingleton:
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\site-packages\brain_pipe\utils\multiprocess.py", line 48, in MultiprocessingSingleton
    manager = multiprocessing.Manager()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\context.py", line 57, in Manager
    m.start()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\managers.py", line 562, in start
    self._process.start()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\sean\miniconda3\envs\brain-pip\Lib\multiprocessing\spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

And here are all things I've met. The preprocessing codes are provided as in the 85 subject dataset. I just download the repo and the dataset and directly run the script, and the error occurs.

My analysis: it seems that the error occurs during the import of necessary modules, and the main code is not executed yet. The code that direcly raise the error is

class MultiprocessingSingleton:
    """Singleton class for multiprocessing."""

    manager = multiprocessing.Manager()

And the RuntimeError said the error occurs if the start mp method is not fork (but I can only use spawn in Windows, it does not support fork.) I wonder if you had complete the code writing and testing in a Unix or MacOS machine? And is there any way to skip this error to normally execute the preprocessing pipeline on a windows PC?

Let me know if you wish to find out more informations.

[berndie]

Hi,
This seems to be a Windows-related problem with the multiprocessing package. All tests and code writing were done on Unix-like machines, so I'm unsure how the package performs on Windows.

I will try to implement a quick fix and adjust our CI pipeline to include Windows machines.

[SeanZhang99]

Dear Berndie, Thanks for your rapid reply. I've tried the codes on a WSL 1 Ubuntu and they work well. It should be the 'spawn' method (the default and only supported way of generating new process on Windows 10/11) of multiprocess which causes the problem. BTW I found an unlocated memory leakage when mp on AMD 7900 and it quickly drawn out my RAM (96GB). Still I can run the code on single process mode and it does not take much time for me waiting the preprocess finished. Ag I would like to express my appreciate for your quick response. If you need further information about the issue or the memory leakage problem, plz feel free to contact me and I'm glad to provide you with additional logs or debug info. Yours, Yuanming Zhang. ***@***.*** From: berndie Date: 2024-08-29 20:15 To: exporl/brain_pipe CC: SeanZhang99; Author Subject: Re: [exporl/brain_pipe] Multiprocessing error during import (Issue #7) Hi, This seems to be a Windows-related problem with the multiprocessing package. All tests and code writing were done on Unix-like machines, so I'm unsure how the package performs on Windows. I will try to implement a quick fix and adjust our CI pipeline to include Windows machines. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>