INCIDENT 019 | Orcfax federated validator node dropped publishing websocket

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-02-21

Summary

The Orcfax publishing websocket was dropped by the Orcfax federated validator node resulting in a datum not arriving on-chain.

Status

Resolved

Assessment

Recent changes to improve messaging between websockets used by Orcfax components had unforeseen impacts on how connections were managed by the websocket connection component. Additional checks should have been executed to ensure connections were still available before being written to and closed.

Without these checks, the result was that the reliability of the websocket manager would deteriorate over time as it continued to hold onto connections that were no-longer present.

Additional Notes

The 10:00 PM UTC heartbeat was missed. The problem was recognized by Orcfax monitoring and a new datum published by 10:16 UTC.

Technical improvements

The validator software has since been upgraded to better manage connections.

We are investigating:

1. More advanced connection management solutions as Orcfax upgrades its number of feeds and thus the amount of data shared between components increases.

Documentation improvements

N/A