Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test] Fix for JSON tests #797

Merged

Conversation

riedgar-ms
Copy link
Collaborator

@riedgar-ms riedgar-ms commented May 5, 2024

We have been getting intermittent failures in the JSON tests. These are occurring in the 'number' and 'integer' tests, and are almost certainly due to the behaviour of the model.Mock class.

Once its supplied string has been exhausted, Mock generates random characters (or to be more precise, produces randomised logits for the valid tokens). This means that every now and then, a digit can be produced, which will pass the grammar check for a number, but then fail our round-trip check. This is not a problem for any of our other tests, since in those cases, the final character is forced (e.g. " to close a string or ] to end an array), and there is no possible valid character afterwards.

The fix is to append a character to the string supplied to Mock which never appears in the one we supply from the test. This acts as a universal 'stop code.'

@riedgar-ms riedgar-ms requested a review from Harsha-Nori May 5, 2024 12:23
@codecov-commenter
Copy link

codecov-commenter commented May 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.85%. Comparing base (bbabbca) to head (0d569a9).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #797      +/-   ##
==========================================
+ Coverage   55.34%   62.85%   +7.50%     
==========================================
  Files          55       55              
  Lines        4076     4076              
==========================================
+ Hits         2256     2562     +306     
+ Misses       1820     1514     -306     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@riedgar-ms riedgar-ms merged commit 7693a64 into guidance-ai:main May 6, 2024
106 checks passed
@riedgar-ms riedgar-ms deleted the riedgar-ms/json-test-fix-01 branch May 6, 2024 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants