INCIDENT 013 | Network.Socket.accept: resource exhausted (Too many open files)

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ✔️ anomalous system behaviors
• ⬛ human errors
• ⬛ unauthorized access attempts

Date

2024-01-16

Summary

COOP's local JSON Fact Store (json-fs-store) GRPC service exited with too many open file handles: Network.Socket.accept: resource exhausted (Too many open files). The incident was reported at 0310 UTC and automatically resolved at 0505 UTC.

Status

Under Review

Assessment

The issue remains under investigation and is believed to be connected to the gRPC libraries utilized by the plutus applications embedded in the Cardano Open Oracle Protocol (COOP).

Additional Notes

For consumers this issue meant that a current data was not on-chain from approximately 0200 UTC until approximtely 0400 UTC.

The short-term workaround to this issue meant that COOP could recover automatically so publication was picked up at the next hour but the recovery didn't result in retrying the fact statement publication.

Technical improvements

We are investigating:

1. How to make sure COOP is resilient against such failures in the short-term
2. To make sure COOP automatically retries publications
3. Continuing investigation of the underlying Plutus/Haskell libraries involved in he gRPC implementation.

Documentation improvements

NA