INCIDENT 026 | Median could not be verified

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-04-08

Summary

A difference between the precision being used by the collector and that which was being used by the validator resulted in a median value which was not verified correctly.

This anomalous system behavior was first noticed in Incident 025.

Status

Under Review

Assessment

At 0300 UTC on 8 April the regular heartbeat was missed. The issue corrected itself during the next heartbeat.

Additional Notes

Orcfax introduced an additional source to collectors earlier on in the year. With differences in programming language used for collecting and validating, the Orcfax team is seeing different levels of precision causing misalignment during validation.

Technical improvements

We are investigating:

1. More affective methods of determining the level of precision in the validator node.
2. Temporarily reducing the number of sources in collectors from six to five.
3. Identifying the correct retry mechanism between the validator and coop that increases publishing reliability.

Documentation improvements

NA