[Bug]: SWE-Bench Eval does not work

[ramsey-coding]

Is there an existing issue for the same bug?

• I have checked the troubleshooting document at https://opendevin.github.io/OpenDevin/modules/usage/troubleshooting
• I have checked the existing issues.

Describe the bug

I get the following error when I run the ./evaluation/swe_bench/scripts/run_infer.sh [model_config] [agent] [eval_limit] script

Traceback (most recent call last):
  File "/Users/ramsey/conda/anaconda3/envs/opendevin/lib/python3.11/concurrent/futures/process.py", line 308, in weakref_cb
AttributeError: 'NoneType' object has no attribute 'util'

Current OpenDevin version

latest

Installation and Configuration

./evaluation/swe_bench/scripts/run_infer.sh eval_bedrock CodeActAgent 5

Model and Agent

CodeAct agent

Operating System

Mac

Reproduction Steps

Run the ./evaluation/swe_bench/scripts/run_infer.sh script

Logs, Errors, Screenshots, and Additional Context

No response

[SmartManoj]

Could you provide the full traceback?

[li-boxuan]

Could you please check out https://github.com/OpenDevin/OpenDevin/blob/main/evaluation/swe_bench/README.md#test-if-your-environment-works?

I am pretty sure your config is wrong.

[SmartManoj]

AssertionError: Failed to backup ~/.bashrc: mv: cannot stat '/home/opendevin/.bashrc': No such file or directory

[li-boxuan]

Could you please check out https://github.com/OpenDevin/OpenDevin/blob/main/evaluation/swe_bench/README.md#test-if-your-environment-works?

I am pretty sure your config is wrong.

You need to set run_as_devin = false in config.toml

[SmartManoj]

Is this config loaded? If so, the cache dir would be /root/.cache. Any error about the config in the log?

[SmartManoj]

Do you have only one config file?

[ramsey-coding]

yes - I have already pasted the config file.

[SmartManoj]

export RUN_AS_DEVIN=false and check

[li-boxuan]

selected_ids = ['sphinx-doc__sphinx-8721', 'sympy__sympy-14774', 'scikit-learn__scikit-learn-10508']

Btw that one doesn't go to config.toml in the root folder. It goes to swe-bench folder.

[ramsey-coding]

I now get:

ERROR:root:<class 'opendevin.runtime.browser.browser_env.BrowserException'>: Failed to start browser environment.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:33<00:00, 33.10s/it]
Exception ignored in: <function _ExecutorManagerThread.__init__.<locals>.weakref_cb at 0x179e34c20>
Traceback (most recent call last):
  File "/Users/ramsey/conda/anaconda3/envs/opendevin/lib/python3.11/concurrent/futures/process.py", line 308, in weakref_cb
AttributeError: 'NoneType' object has no attribute 'util'

[SmartManoj]

u don't have playwright? #2140 (comment)

[ramsey-coding]

Did the setup, still get the following error:

opendevin.runtime.browser.browser_env.BrowserException: Failed to start browser environment.
20:18:18 - opendevin:INFO: browser_env.py:53 - Browser env started.

[SmartManoj]

#1993 (comment)

[ramsey-coding]

@SmartManoj slightly different question, during SWE bench evaluation I dont want to the agent to browser the internet though.

[SmartManoj]

Use CodeActSWEAgent #2105

[ramsey-coding]

selected_ids = ['sphinx-doc__sphinx-8721', 'sympy__sympy-14774', 'scikit-learn__scikit-learn-10508']

Btw that one doesn't go to config.toml in the root folder. It goes to swe-bench folder.

where to set it then? Which config file?

[li-boxuan]

selected_ids = ['sphinx-doc__sphinx-8721', 'sympy__sympy-14774', 'scikit-learn__scikit-learn-10508']

Btw that one doesn't go to config.toml in the root folder. It goes to swe-bench folder.

where to set it then? Which config file?

https://github.com/OpenDevin/OpenDevin/blob/main/evaluation/swe_bench/README.md

[ramsey-coding]

So there are two config.toml, one in the root folder and one in the ./evaluation/swe_bench/ folder.

• I need to create then second config.toml in ./evaluation/swe_bench/ folder with only the following content:

selected_ids = ['sphinx-doc__sphinx-8721', 'sympy__sympy-14774', 'scikit-learn__scikit-learn-10508']

[SmartManoj]

@li-boxuan sub_config.toml name would be better?

[li-boxuan]

@li-boxuan sub_config.toml name would be better?

Eventually we will refactor the whole evaluation folder.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[mamoodi]

@li-boxuan can this issue be closed? Seems like it was a config issue?