Edit tool prompt tweaking: only plain-text format is supported

[li-boxuan]

End-user friendly description of the problem this fixes or functionality that this introduces

Tweak edit-related prompts to clarify that edit tools are for plain-text format only.

• Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

---

Give a summary of what the PR does, explaining any non-trivial design decisions

Without this PR:

With this PR:

and I verified the result is indeed of MS-doc format.

As a follow-up, we could create some micro-agents to remind LLM of python libraries related to office, such as python-docx.

---

Link of any specific issues this addresses

This is motivated by a failure I observed when running TheAgentCompany benchmark. Specifically, this one: https://github.com/TheAgentCompany/TheAgentCompany/blob/d456bb0d0c528e3495b6400f156be02695d6731f/workspaces/tasks/ds-answer-numerical-data-question/task.md?plain=1#L9

---

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:a9297a0-nikolaik   --name openhands-app-a9297a0   docker.all-hands.dev/all-hands-ai/openhands:a9297a0

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[mamoodi]

The eval pipeline seems broken right now :(

[github-actions]

Running evaluation on the PR. Once eval is done, the results will be posted.

[mamoodi]

@li-boxuan got it working to a degree. Some issue with posting final comment.. Here it is:
evaluation.zip

[li-boxuan]

Here's the report - seems lower than expected? I am not sure what the baseline shall be but I thought it should be near 50%

SWE-bench Report

This folder contains the evaluation results of the SWE-bench using the official evaluation docker containerization.

Summary

• submitted instances: 30
• empty patch instances: 2
• resolved instances: 10
• unresolved instances: 20
• error instances: 0

Resolved Instances

• astropy__astropy-14995
• django__django-10914
• django__django-10924
• django__django-11049
• django__django-11099
• django__django-11133
• django__django-11583
• django__django-11620
• django__django-11815
• django__django-11999

Unresolved Instances

• astropy__astropy-12907
• astropy__astropy-14182
• astropy__astropy-14365
• astropy__astropy-6938
• astropy__astropy-7746
• django__django-11001
• django__django-11019
• django__django-11039
• django__django-11179
• django__django-11283
• django__django-11422
• django__django-11564
• django__django-11630
• django__django-11742
• django__django-11797
• django__django-11848
• django__django-11905
• django__django-11910
• django__django-11964
• django__django-12113

Error Instances

Empty Patch Instances

• django__django-11797
• django__django-11964

Incomplete Instances

[li-boxuan]

I noticed that astropy__astropy-12907 was passing at some point: https://github.com/swe-bench/experiments/blob/203cc8fa5b8322983445b314da7038b8b6653d5f/evaluation/verified/20241029_OpenHands-CodeAct-2.1-sonnet-20241022/results/results.json#L14C6-L14C28

I inspected the evaluation log and I don't think the failure was related to this PR. It's either randomness, or a regression somewhere else.

[mamoodi]

Yeah a bit lower. I think 14-15 resolved was the baseline but as you mentioned it might be randomness. Xingyao would know better for sure.

[enyst]

You may want to run compare_outputs.py:

# diff=6
----------------------------------------------------------------------------------------------------
# x resolved but y not=4
               instance_id                                           report_x                                           report_y
0   astropy__astropy-12907  {'empty_generation': False, 'resolved': True, ...  {'empty_generation': False, 'resolved': False,...
14    django__django-11001  {'empty_generation': False, 'resolved': True, ...  {'empty_generation': False, 'resolved': False,...
2     django__django-11039  {'empty_generation': False, 'resolved': True, ...  {'empty_generation': False, 'resolved': False,...
4     django__django-11179  {'empty_generation': False, 'resolved': True, ...  {'empty_generation': False, 'resolved': False,...
----------------------------------------------------------------------------------------------------
# y resolved but x not=2
             instance_id                                           report_x                                           report_y
18  django__django-10924  {'empty_generation': False, 'resolved': False,...  {'empty_generation': False, 'resolved': True, ...
19  django__django-11049  {'empty_generation': False, 'resolved': False,...  {'empty_generation': False, 'resolved': True, ...
----------------------------------------------------------------------------------------------------
Repository comparison (x resolved vs y resolved):

astropy:
Difference: 1 instances!
X resolved but Y failed: (1 instances)
  ['astropy__astropy-12907']
Y resolved but X failed: (0 instances)

django:
Difference: 1 instances!
X resolved but Y failed: (3 instances)
  ['django__django-11001', 'django__django-11039', 'django__django-11179']
Y resolved but X failed: (2 instances)
  ['django__django-10924', 'django__django-11049']

[xingyaoww]

@mamoodi
Can we also ran an eval on main to compare with this result? 🤔

If they are comparable, this is probably ok

[mamoodi]

@li-boxuan and @xingyaoww for main:
evaluation-main.zip

[xingyaoww]

Evaluation results ready for t-main: t-main_25-01-07-17-31.tar.gz

Summary

• submitted instances: 29
• empty patch instances: 1
• resolved instances: 13
• unresolved instances: 16
• error instances: 0

Looks like we do have some degradation? 13 -> 10?

[neubig]

Hard to say, maybe we can run a bigger eval?

[li-boxuan]

Seems the noise might have come from All-Hands-AI/openhands-aci#48

discussion on slack: https://openhands-ai.slack.com/archives/C080M7BBSSG/p1736346851359599

[xingyaoww]

@mamoodi do we have larger eval? e.g., 100?

[mamoodi]

@xingyaoww I can run one with the pipeline but it will have to be manually. Did you want me to go ahead with that? With Sonnet 3.5?

[neubig]

@mamoodi , yeah I think it'd be good to run that! We could click "update branch" and then run it on main and this branch.

[mamoodi]

Sorry folks it's been a busy few days. Is this still pending a 100 instance eval?
I still need to run one on main to figure out what the baseline is. I'm planning to run one on main tonight.
Did you all want to run 100 instance on this PR afterwards? Perhaps tomorrow?

[neubig]

Yeah, sounds good!

[mamoodi]

Alright. Sorry about the wait folks. Here are the results.
I will mention that the main result was from Jan 14 and some changes have gone in which may have affected this.
I believe @xingyaoww will know the real values of the current main.

Eval 100 instances for main:

Summary

• submitted instances: 100
• empty patch instances: 12
• resolved instances: 37
• unresolved instances: 63
• error instances: 0

Eval 100 instances for this PR:

Summary

• submitted instances: 100
• empty patch instances: 14
• resolved instances: 43
• unresolved instances: 57
• error instances: 0

The sizes of the evals are too large to upload here so I will post it on Slack.