Kilosort2 fails with "Could not access the MATLAB Runtime component cache" [workaround provided]

[ChrisCurrin]

I'm documenting this in case anyone has a similar issue.

It may be worthwhile to investigate further. Let me know how I can help.

Context

Ubuntu 20.04
Docker version 20.10.3, build 48d30b5
Spikeforest2: b18e640

Log

Full log below

~/dev/spikeforest2
❯ python examples/example_kilosort2.py
===== Hither: running [kilosort2] in container: docker://magland/sf-kilosort2:0.1.5b
#############################################################

                #!/bin/bash

                exec docker run --name WvjOyeHO_kilosort2 -i --gpus all \
                    -v /etc/localtime:/etc/localtime:ro \
                    -v /etc/passwd:/etc/passwd -u `id -u`:`id -g` \
                    -v $KACHERY_STORAGE_DIR:/kachery-storage \
                    -v /home/ccurrin/tmp/tmp/tmp_hither_run_in_container_kilosort2_kiwrhmer:/run_in_container \
                    -v /tmp:/tmp \
                    -v $HOME:$HOME \
                    -v /home/ccurrin/tmp/tmp/tmp_hither_run_in_container_kilosort2_kiwrhmer/outputs:/outputs \
                    magland/sf-kilosort2:0.1.5b \
                    bash /run_in_container/run.sh
                
#############################################################
kilosort2 2021-02-23T16:49:03.647858: Traceback (most recent call last):
kilosort2 2021-02-23T16:49:03.647968:   File "/run_in_container/run.py", line 15, in main
kilosort2 2021-02-23T16:49:03.647990:     retval = kilosort2(**kwargs)
kilosort2 2021-02-23T16:49:03.648014:   File "/run_in_container/function_src/_kilosort2.py", line 48, in kilosort2
kilosort2 2021-02-23T16:49:03.648030:     timer = sorter.run()
kilosort2 2021-02-23T16:49:03.648051:   File "/usr/local/lib/python3.6/dist-packages/spikesorters/basesorter.py", line 110, in run
kilosort2 2021-02-23T16:49:03.648065:     self._run(recording, self.output_folders[i])
kilosort2 2021-02-23T16:49:03.648085:   File "/run_in_container/function_src/_kilosort2sorter.py", line 191, in _run
kilosort2 2021-02-23T16:49:03.648100:     raise Exception('kilosort2 returned a non-zero exit code')
kilosort2 2021-02-23T16:49:03.648146: Exception: kilosort2 returned a non-zero exit code
kilosort2 2021-02-23T16:48:11.597368: ###### RUNNING: kilosort2
kilosort2 2021-02-23T16:48:13.154839: Downloading file --- (18.6 KiB): http://kachery.flatironinstitute.org/neuro1/get/sha1/3ea5c9bd992de2d27402b2e83259c679d76e9319?channel=public&signature=5f20adc2ee7b2694e3a6a30ab67dacdd7e88f8e6 -> /kachery-storage/sha1/3e/a5/c9/3ea5c9bd992de2d27402b2e83259c679d76e9319
kilosort2 2021-02-23T16:48:13.673077: Downloaded file (18.6 KiB) in 0.517941951751709 sec.
kilosort2 2021-02-23T16:48:14.043390: Downloading file --- (274.7 MiB): http://kachery.flatironinstitute.org/neuro1/get/sha1/b0b39245b90358a092d83ad58a334a988b588e82?channel=public&signature=28c337f6b4ead393132b3fa1ebccfd60c40a644d -> /kachery-storage/sha1/b0/b3/92/b0b39245b90358a092d83ad58a334a988b588e82
kilosort2 2021-02-23T16:49:00.999492: Downloaded file (274.7 MiB) in 46.955838441848755 sec.
kilosort2 2021-02-23T16:49:01.000750: Sorting...
kilosort2 2021-02-23T16:49:03.482036: Could not access the MATLAB Runtime component cache. Details: fl:filesystem:AccessDenied; component cache root:; componentname: kilosort2_binary
kilosort2 2021-02-23T16:49:03.482162: MCL:ComponentCache
kilosort2 2021-02-23T16:49:03.632600: Error: Could not access the MATLAB Runtime component cache. Details: fl:filesystem:AccessDenied; component cache root:; componentname: kilosort2_binary
===== Hither: finished [kilosort2] with status error after 52.05110788345337 sec
Traceback (most recent call last):
  File "/home/ccurrin/dev/spikeforest2/examples/example_kilosort2.py", line 12, in <module>
    result = sorters.kilosort2.run(
  File "/home/ccurrin/dev/spikeforest2/hither_sf/_core.py", line 278, in run
    raise Exception(f'Error running job: {_label}')
Exception: Error running job: kilosort2

Workaround

Using sudo seemed to solve the issue

> sudo KACHERY_STORAGE_DIR=/path/to/KACHERY_STORAGE_DIR /path/to/conda/env/python examples/example_kilosort2.py 

I had also tried changing KACHERY_STORAGE_DIR to a local directory as it was on a mounted NTFS drive, but that did not resolve the issue.