Add NIST's ACVP "External" Signature Tests

[atreiber94]

This PR adds vectors extracted from NIST's ACVP KATs for ML-DSA and SLH-DSA, which have recently been updated for the "external" interface of signatures. The parsing is in part taken from Markku-Juhani O. Saarinen's py-acvp-pqc repo.

Before the update, only the "internal" interface was tested in ACVP. Because no data that would be legal by the external interface was covered by the previous test data, we could not add these tests to DSAs previously. (For ML-KEM, this was already added in #3893).

Since the update of NIST's ACVP KATs also covers the "context" signature parameters of the new PQC standards, this PR also adds KATs for the "context" case being introduced in #4567.

With these tests, Botan's test coverage of ML-DSA and SLH-DSA should increase since ACVP covers many different cases of "wrong" signatures (see the test specs for ML-DSA and SLH-DSA).

Limitations

• ML-DSA SK in Botan can only be parsed as the private seed but ACVP test data is only available for the expanded format. Hence, the ML-DSA "SigGen" test cannot be performed.
• For the same reason, the expanded SK in any KeyGen test cannot be tested.
• SLH-DSA's slh_dsa_acvp_sigver.vec is 5.7M and thus the largest test file... :( we should discuss if we want to remove some test cases (in this PR I already only take one of each verification failure possibility). Hashing is not possible because it is verification data.
• SLH-DSA "SigGen" tests take 202 seconds on my machine. Also here we should discuss if we want to remove some cases (in this PR I already only take one of each random/deterministic and context combination).

Outlook

• Towards ACVP test case scraper #4500: my 2 cents are that this could turn out to be a lot of work. Just by working with the 3 PQC standards, they (even the DSAs) have large discrepancies in variable names and test combinations, which required a lot of manual "plugging". I hope that the ACVP testing framework provided here may be generic enough for the additional coming PQC standards but each to-be-tested algorithm will require at least some manually customized parsing and testing code.
• PQC Testing infrastructure: We now have a lot of different tests for the PQC algorithms, which results in a complex test infrastructure. So we could think about removing some of them (or discuss if we are very happy with testing against a lot of different sources). The old NIST competition KATs may go when the old candidate algorithms go. We also have KATs generated from other implementations because ACVP could not be used and could think about possibly dropping them:
  • ML-DSA and SLH-DSA from the python implementations in py-acvp-pqc
  • ML-DSA wycheproof tests added in Add ML-DSA-4x4 verification tests #4522
  • ML-KEM from the Kris Kwiatkowski's repo
  • Possibly others that I forgot

PR dependencies

• Context support for ML-DSA and SLH-DSA #4567 and transitively
• Add PK_Signature_Options #4318

[randombit]

It might be better to run these tests in some offline way, eg a Python script that fetches the JSON from NIST's repo and tests the various signatures. Certainly we can't run them all, and I'm not sure we want to directly ship so much additional data.

Alternately to #4500 can start thinking about an actual ACVP client but that's a pile of work and really only makes sense if we're doing a FIPS 140 validation, and I don't have 100K$ lying around that I'd like to set on fire.

[atreiber94]

It might be better to run these tests in some offline way, eg a Python script that fetches the JSON from NIST's repo and tests the various signatures. Certainly we can't run them all, and I'm not sure we want to directly ship so much additional data.

That's a possibility but judging from the last NIST repo update the script would need to be updated with any NIST repo update (e.g. that would add SK seeds). We'd like to at least have some of these tests in the CI, also because these are the only vectors with verification failures and non-empty context strings.

We can run the SigGen tests only with --run-long-tests. We can also further reduce the file size of slh_dsa_acvp_sigver.vec. What size would be appropriate? Testing for each parameter set one verification success and one failure results in 1.2M, testing only one parameter set results in sizes between 213K (largest parameter set) and 43K (smallest parameter set). For comparison, the ML-DSA wycheproof ml_dsa_verify.vec' is 1.7M`.

Alternatively we could out-source the larger test vector files to another repository that is only checked out and tested in the CI to reduce the size of shipped files.