Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Isolated environments for test/evals #90

Closed
ErikBjare opened this issue Aug 9, 2024 · 2 comments
Closed

Isolated environments for test/evals #90

ErikBjare opened this issue Aug 9, 2024 · 2 comments

Comments

@ErikBjare
Copy link
Owner

ErikBjare commented Aug 9, 2024

I just had an eval of Claude 3 Haiku create git repositories in ~ and ~/Programming. Not great.

Should figure out a way to constrain its impacts, either via Docker container or chroot or CI or something.

@ErikBjare ErikBjare changed the title Improve test/eval isolation Isolated environments for test/evals Aug 9, 2024
@ErikBjare
Copy link
Owner Author

I added a basic Dockerfile for running evals in 75deabc

@ErikBjare
Copy link
Owner Author

Fixed now that Dockerfile.eval exists.

Ideally, it should prob be changed so that each eval runs in its own container (spawned by gpt-eval). But that's for another time.

@ErikBjare ErikBjare mentioned this issue Sep 6, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant