AU Core Synthetic Data

Data generated by Synthea then mapped to be valid according to AU Core IG

Pipeline

1. Generate Data: Use dockerized Synthea and a custom module to generate data in CSV format.

   • au-core-synthetic-data/synthea/compose.yml

     Line 8 in c7e43f8

     command: java -jar synthea-with-dependencies.jar --exporter.csv.export=true --exporter.years_of_history=0 -p 1 -d modules/

   • au-core-synthetic-data/synthea/modules/encounter-observation-observation-condition-medication-encounter-end.json

     Line 2 in c7e43f8

     "name": "Encounter + Observation + Observation + Condition + Medication + Encounter end",

2. Map/Post-Process CSV Files:
   • Map target patients to the generated patients (because it is not possible to assign specific patient IDs in Synthea data).

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 1 in c7e43f8

       target_patient_ids:

   • Rename column names in the generated CSV files according to the mapping configuration.

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 5 in c7e43f8

       allergies:

   • Rename the resulting CSV files based on the mapping configuration.

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 162 in c7e43f8

       rename_files:

   • Generate UUID for CSV if ID is not exist

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 172 in c7e43f8

       add_ids_column:

   • Delete unused CSV files.

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 180 in c7e43f8

       delete_files:

   • Add additional columns to the generated CSV files to maintain a structure for the Csv2Fhir.

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 192 in c7e43f8

       columns_list:

   • Perform post-processing using the configuration to enrich the data, as Synthea has a limited number of fields.

     • au-core-synthetic-data/mapping_script/config.yaml

       Line 720 in c7e43f8

       post_processing:

3. Generate FHIR Resources: Use the Csv2Fhir application to convert CSV data into FHIR resources in JSON format.

   • au-core-synthetic-data/Makefile

     Line 25 in c7e43f8

     run-csv-2-fhir-script:

4. Validate Results: Validate the generated FHIR resources using the official FHIR Validator CLI.

   • au-core-synthetic-data/Makefile

     Line 3 in c7e43f8

     validate-results:

5. Create an Archive: Bundle the generated FHIR resources in both JSON and CSV formats into an archive.

Run

./download-mapper.sh

make pipeline