INCIDENT 020 | Data missing in upgraded Orcfax collector node message output

[Christian-MK]

Trigger

• ⬛ suspected malware infections
• ⬛ access violations
• ⬛ anomalous system behaviors
• ✔️ human errors
• ⬛ unauthorized access attempts

Date

2023-02-26

Summary

HTTP response data was partially missing in upgraded Orcfax collector node message output. Orcfax node upgrades were aborted and new datum published using the original collector software.

Status

Resolved

Assessment

HTTP response data was missing in upgraded collector node software. The new software rewrites a large part of the existing CEX collection component and it was discovered that the rewrite wasn't recording the HTTP response body in the Orcfax 'message-'. Recording this data is an important part of the Orcfax contextual data concept and creates a full audit trail showing how median prices are calculated. The response body is also used to calculate median prices and this data was only being recorded in the validation record.

The issue was spotted during user-acceptance testing during the collector node upgrade and the collectors immediately taken offline and replaced with the original software.

Additional Notes

A new datum was published to replace the one created without the complete audit trail required by Orcfax. The anomalous package was published at 13:42 UTC and the network rolled back to its original configuration by 14:04 UTC. The anomalous package can be reviewed on the Orcfax explorer: here

Technical improvements

A fix was applied to the new collector node software and important testing added to ensure the consistency of its output in future.

We are investigating:

1. Improving tooling to inspect Orcfax archival packages.
2. Using pre-production environments more effectively for testing.
3. Improving QA procedures by pairing developers and analysts for metadata-walk-through.

Documentation improvements

1. Documenting Orcfax schema more thoroughly to reflect the mandated components required for Orcfax published data.