INCIDENT 015 | Error submitting Tx from local Fact store

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-01-23

Summary

The datum at 18:39 & 1900 UTC were not published. A decoding error occurred within the COOP environment.

Status

Under Review

Assessment

The COOP environment failed to store two consecutive fact statements and generated the message, error submitting Tx, unable to store fact; the publications implicated:

• A 1% deviation at 18:39 UTC
• The hourly heartbeat at 19:00 UTC with the error

The issue looks to be related to Incident 15. The error occurred in the local fact store which is used as a temporary store during publishing. When the storage of the fact failed, the resulting transaction could not be built and published as anticipated.

The development team is working to identify the root cause of the error.
Heartbeat monitoring and publishing continued without issue after the missed heartbeat.

Additional Notes

For consumers, an Orcfax Fact Statement within the validity range was unavailable on-chain from 18:19 UTC until 20:00 UTC (approx. 1hr 40 mins).

The JSON format used by Orcfax remains stable and the fact statement has been tested in a preprod environment without issue, as such the root cause that affected mainnet services remains under review.

Technical improvements

We are investigating:

1. Continuing to find more effective ways of debugging the COOP environment so as to gather as much information as possible.
2. Publishing and transaction building components of COOP are currently being rewritten and issues such as these will be incorporated in further testing.
3. Continuing to investigate mechanisms for resilient workflows in the Orcfax environment to ensure reliable publishing.

Documentation improvements

NA