INCIDENT 031 | Unresponsive validator node causes missed heartbeat

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-04-16

Summary

A heartbeat was missed at 1400 UTC on 16 April 2024 due to an unresponsive validator node.

Status

Resolved

Assessment

While the reason is still uncertain, the missed heartbeat was caused due to the Validator machine becoming unresponsive. We believe this to have been a problem with the virtual machine hosting provider. The team will continue to monitor for related issues.

Additional Notes

Monitoring alerted the Orcfax team as to the issue with publication, and they were able to mitigate impact by manually executing a heartbeat publication at 1412 UTC.

Technical improvements

1. We are continuing to research decentralized compute solutions.

Documentation improvements

N/A