-
Notifications
You must be signed in to change notification settings - Fork 1
UQ-CMM-Mirage/CMM-data-grabber
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Paul is the "data grabber" application that prepares data and metadata files for ingestion into the CMM MIRAGE repository as follows: - It receives file create / modify / delete events for files in the directory trees that correspond to the different instruments. - It aggregates these based on filename patterns to work out sets of files that constitute Datasets. - When a Dataset stabilizes, it "grabs" the files by copying them to a separate queue area. - It uses the information recorded by Eccles to figure out who was logged into the facility. If this is known, the Dataset is tagged with the user and marked as ready to be ingested. - Ingestible Datasets are then exposed to the downstream system (e.g. MyTardis) via an Atom feed. Paul also provides a web UI that supports ACLS session control, queue management integrity checking and repair, and configuration of instruments. Eccles is a proxy for an ACLS service that tracks sessions, and will insulate the ACLS clients from server outages. Eventually, it may evolve into a replacement for ACLS. Benny is simple HTTP service for authenticating against a backend ACLS instance.
About
This application prepares data / metadata for ingestion into a CMM Mirage repo
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published