INCIDENT 011 | Kraken outage prevented minimum source threshold

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-01-08

Summary

One of the three primary sources currently used in the Orcfax ADA/USD feed suffered an outage. The Kraken outage resulted in an interruption in the hourly heartbeat.

Status

Resolved

Assessment

Kraken identified a connectivity issue at 10:26 UTC and had previously been performing maintenance activities in the hour prior to that.

Becasue of this connectivity issue, the collector node was unable to gather a the required minimum 3 data points within its publication window, which affected the hourly heartbeat. Further, potential publications events initiated by a one percent deviation may also have been affected.

Orcfax monitoring picked up the anomalous event on the hour after the on-chain count of fact statements was determined to be too low, i.e. the publication hadn't happened.

Developers saw that there wasn't a data point available from Kraken and the collectors were not forwarding their data as expected. The Orcfax validator continued to poll its data sources waiting for a new set of data points to arrive that would be published immediately after validation.

Eventually data became available at approximately 10:19 UTC and was put on-chain. Data continued to arrive sporadically until approximately 11:00 UTC and eventually a data-point arrived on chain at 11:03 UTC and the regular heartbeat continued from there.

Additional Notes

The Kraken collector was unable to gather a data-point from the Kraken API for the 10:00 UTC and 11:00 UTC publication events. Oracle data was put on-chain a short time after these heart beats at 10:19 and 11:03 before the regular heartbeat continued.

Orcfax is committed to preserving data authenticity. Because of this, all feeds require a minimum of 3 primary sources in order to triangulate data before it is published on chain. Orcfax will not publish on chain if there are fewer than three data-points.

Technical improvements

In order to develop increasingly robust data feeds, Orcfax will add to the total number of available primary sources to collect data from. A solution is already in testing that will greatly increase this number for its ADA/USD feed. By having more than three primary sources to collect from, future versions of the software nodes will not fail but rather adapt gracefully and continue publishing Orcfax datum as expected, as long as there is still a minimum of three unique data sources to collect from.

We are investigating:

1. Deploying the latest Orcfax collector node software upgrade to preprod.
2. After final checks and further user acceptance testing (UAT) transfer the new collector capability to mainnet.

Documentation improvements

1. Ensure that it is made clear in documentation that streaming outages can occur as a result of external data sources becoming unavailable.