Use YAML (or TOML) as the EGSnrc input file language

[ftessier]

What do you think about using the well-known and standardized markup language YAML for EGSnrc input files, especially the egs++ .egsinp files, instead of EGSnrc's own format (using :start and :stop tags and nested key-value pairs). The yaml specification is also a key-value pair format, so it is very similar to .egsinp format.

There are libraries readily available to parse yaml in all common languages, and one can write a JSON schema to automatically validate EGSnrc input files. In fact, the idea to look at yaml again was precipitated by the recent input validation discussion.

I have also considered TOML and JSON, but I don't think those fit the bill. Note that this can be implemented on the side (i.e., leaving the traditional .egsinp format available; if in time yaml input gained traction, we could also write a converter of course.

I have been toying with the idea a little, and found the following pros for yaml input:

1. simple, clean with minimal syntactic crud
2. widely recognized standard syntax, with libraries to parse and editor support for syntax highlighting
3. support for JSON schema to specify and validate EGSnrc input (I have started to write a schema)
4. key value pairs, so almost identical to .egsinp already (notably, whitespace is allowed in keys)
5. support for scientific notation
6. convenient marker and reference notation to replicate data (any input can be turned in a variable to be reused, see &pos 0.935 and *pos in the example below)
7. can easily clone entries while modifying only some elements (see <<: *tip in example below)
8. supports explicit type annotations if needed (e.g., !!str 2.0)
9. no need to balance delimiters (except in sequences, which is then a little annoying)
10. any json file is a valid yaml file, so inputs could be written in json.
11. forced indentation 🙂

Here is what a typical input might look like:

### ----------------------------------------------------------------------------
### Simulation
### ----------------------------------------------------------------------------
simulation:
  ncase:      1.e+16
  precision:  1%
  geometry:   chamber
  source:     my_source
  rng:
    seeds: [123, 1250]

### ----------------------------------------------------------------------------
### Geometries
### ----------------------------------------------------------------------------
geometries:

  # the ion chamber spherical tip
  - &tip
    name:       chamber_tip
    class:      egs_spheres
    position:   [&pos 0.931, 0, -10]
    radii:      [0.05, 0.311, 0.356, 0.36, 0.37, 1.0]
    media:
      water:  all
      pmma:   [0, 2]
      air:    [1, 3]

  # the same chamber tip, at a different z-position
  - <<: *tip
    name: chamber_tip_2
    midpoint:   [0, *pos, -23]

  # the cylindrical body of the ion chamber
  - name:       chamber_body
    class:      egs_cones
    type:       EGS_ConeStack
    origin:     [0.985, 0, -10]
    direction:  [-1, 0, 0]

    layers:

      - name:         electrode
        thickness:    2.03
        top radii:    [0.05, 0.311, 0.356]
        bottom radii: [0.05, 0.311, 0.356]
        media:        [pmma, air, pmma]

      - name:         stem
        thickness:    2.0
        top radii:    0.62
        bottom radii: 0.62
        media:        pmma

  # join the chamber tip and body to build chamber
  - name:   chamber
    class:  egs_cdgeometry
    template: cd_planes_for_chamber
    fill region 0: chamber_body
    fill region 1: chamber_tip

### ----------------------------------------------------------------------------
### Sources
### ----------------------------------------------------------------------------
sources:
  - name: my_source
    class: egs_collimated_source
    charge: 0
    source shape:
      type: point
      position: [0, 0, 100]
    target shape:
      class: egs_rectangle
      rectangle: [-5, 0, 5, 5]
    spectrum:
      type: monochromatic
      energy: 6 MeV

I am working on a JSON schema to specify the permitted and required EGSnrc input, for validation, and will post it here as soon as I have something decent.

[rtownson]

One question is how the performance will compare when the file is very large. Since this only happens once per simulation I imagine it won't be important, but worth checking - I have an egsinp that's 9.2MB, let me know if you want to give it a try.

[ftessier]

As mentioned in that article, we ought to stick with StrictYAML if trying the yaml route.

[ftessier]

Tried to load a 10 MB yaml file using the python parser. Yikes! Way too slow, I was not patient enough to let it complete. It was 108 seconds for a 5 MB file! The c++ library is perhaps much faster. We will have to think about performance then.

[crcrewso]

With the new structure does the 9 MB Egsinp file actually translate to a 9 MB yaml file?

[ftessier]

Have not calculated exactly, but give and take, yes, it would be similar.

[ftessier]

I tried parsing a large yaml input file with yaml_rust, and it can chew through MB of inputs practically instantly. We could check performance in more detail, but I am confident parsing efficiency is not an issue with libyaml, yaml-cpp or yaml_rust.

[ftessier]

I like TOML (https://toml.io/) as well, it is simpler and clearer. I will try writing an .egsinp in TOML as well, to test this idea. Looking at it again, I found that one can write a JSON schema as well to validate TOML (and a TOML-specific schema will probably arise in the future). While looking at yaml, I found that the variable definition and reuse ability in yaml (points 6 and 7 in list above) sounded great for EGSnrc geometries!

[ftessier]

Below is a similar input file example in TOML. I really like the clean and unambiguous structure, however quoting all those strings is driving me nuts! 😄 . Use the following python snip to parse:

# install toml with: pip install toml

import toml

with open("egsinp.toml") as egsinp:
    egsinput = toml.loads(egsinp.read())

str = toml.dumps(egsinput)
print(str)
print(egsinput)

### ----------------------------------------------------------------------------
### Simulation
### ----------------------------------------------------------------------------

[simulation]
ncase       = 1000
precision   = 0.01
geometry    = "chamber"
source      = "my_source"
[simulation.rng]
    type    = "xoshiro"
    seeds   = [123, 1250]


### ----------------------------------------------------------------------------
### Geometries
### ----------------------------------------------------------------------------

# chamber tip
[[geometry]]
name        = "chamber_tip"
type        = "egs_spheres"
position    = [0.931, 0.0, -10.0]
radii       = [0.05, 0.311, 0.356, 0.36, 0.37, 1.0]
[geometry.media]
    water = "all"
    pmma  = [0, 2]
    air   = [1, 3]

# chamber body
[[geometry]]
name        = "chamber_body"
type        = "EGS_ConeStack"
origin      = [0.985, 0.01, -10.0]
direction   = [-1, 0, 0]

[[geometry.layer]]
    name        = "electrode"
    thickness   = 2.03
    top         = [0.05, 0.311, 0.356]
    bottom      = [0.05, 0.311, 0.356]
    media       = ["pmma", "air", "pmma"]

[[geometry.layer]]
    name        = "stem"
    thickness   = 2.0
    top         = 0.61
    bottom      = 0.61
    media       = "pmma"

# combine chamber tip and body
[[geometry]]
name        = "chamber"
type        = "egs_cdgeometry"
template    = "cd_planes_for_chamber"
[geometry.regions]
    0 = "chamber_body"
    1 = "chamber_tip"


### ----------------------------------------------------------------------------
### Sources
### ----------------------------------------------------------------------------

[[source]]
name                    = "my_source"
type                    = "egs_collimated_source"
particle                = "photon"
spectrum.type           = "monochromatic"
spectrum.energy         = 6
shapes.source.type      = "point"
shapes.source.position  = [0, 0, 100]
shapes.target.type      = "rectangle"
shapes.target.rectangle = [-5, 0, 5, 5]

Note that the pythom toml library at this point only support the 0.5.0 toml spec, which does not allows mixed type arrays. However mixed type arrays are in the 1.0.0-rc.3 toml spec.

[ftessier]

Playing around with this a bit more, the constant quoting is really annoying, and the nested structures is a bit opaque when looking at the input for more complicated case (nesting is not really visible, unless you add non-required indentation), so I don't think TOML is best for .egsinp files. I am back to YAML. Or perhaps StrictYaml, but then we loose references, which would be useful in egsinp geometries. Still pondering.

[mchamberland]

Having not really thought or looked about it much, but having recently looked at YAML vs TOML for another project, my vote goes to YAML.

[ftessier]

Thanks @mchamberland! Having not used much of either (and then again only as a user in conf files etc.), your comment proves useful!

[ftessier]

Here is a better layout, where the name properties are turned into the dictionary keys, which makes for a more readable input:

geometries:

  # the ion chamber spherical tip
  chamber_tip:
    class:    egs_spheres
    midpoint: [&pos 0.935, 0, -10]
    radii:    [0.05, 0.305, 0.355]
    media:
      water:  all
      air:    [1, 2, 3]
      pmma:   [4, 5]

  # the cylindrical body of the ion chamber
  chamber_body:
    class:     egs_conestack
    origin:    [*pos, 0, -10]
    direction: [-1, 0, 0]
    layers:
      - electrode:
          thickness:    2.03
          top radii:    [0.05, 0.305, 0.355]
          bottom radii: [0.05, 0.305, 0.355]
          media:        [&medium pmma, air, *medium]
      - stem:
          thickness:    2.0
          top radii:    0.61
          bottom radii: 0.61
          media:        *medium

  # join the chamber tip and body to build chamber
  chamber:
    class: egs_cdgeometry
    base geometry: cd_planes_for_chamber
    fill region 0: chamber_body
    fill region 1: chamber_tip
    new indexing style: true

which parses to the following JSON:

{
    "geometries": {
        "chamber_tip": {
            "class": "egs_spheres",
            "midpoint": [
                0.935,
                0,
                -10
            ],
            "radii": [
                0.05,
                0.305,
                0.355
            ],
            "media": {
                "water": "all",
                "air": [
                    1,
                    2,
                    3
                ],
                "pmma": [
                    4,
                    5
                ]
            }
        },
        "chamber_body": {
            "class": "egs_conestack",
            "origin": [
                0.935,
                0,
                -10
            ],
            "direction": [
                -1,
                0,
                0
            ],
            "layers": [
                {
                    "electrode": {
                        "thickness": 2.03,
                        "top radii": [
                            0.05,
                            0.305,
                            0.355
                        ],
                        "bottom radii": [
                            0.05,
                            0.305,
                            0.355
                        ],
                        "media": [
                            "pmma",
                            "air",
                            "pmma"
                        ]
                    }
                },
                {
                    "stem": {
                        "thickness": 2.0,
                        "top radii": 0.61,
                        "bottom radii": 0.61,
                        "media": "pmma"
                    }
                }
            ]
        },
        "chamber": {
            "class": "egs_cdgeometry",
            "base geometry": "cd_planes_for_chamber",
            "fill region 0": "chamber_body",
            "fill region 1": "chamber_tip",
            "new indexing style": true
        }
    }
}

I used the following python snippet to convert yaml to json:

import sys
import yaml
import json

with open(sys.argv[1]) as f:
    yaml_content = yaml.safe_load(f)

print(json.dumps(yaml_content, indent=4))

[ftessier]

Having a json representation would allow validation through a json schema. A json schema also allows for automatically generating a validating web interface to egsinp content. Strict json as an input syntax is not practical since there are too many quotes to type, and more critically it does not allow comments; rjson (relaxed json does), but at that point why not use yaml? 😉

[ftessier]

Scrap the idea of turning the name properties into the dictionary keys, this precludes writing a proper schema! So we should have the following, where for example geometries is a proper array of dictionaries for each geometry, which can then be specified in a schema:

geometries:

  # the ion chamber spherical tip
  - name:     chamber_tip
    class:    spheres
    midpoint: [&pos 0.935, 0, -10]
    radii:    [0.05, 0.305, 0.355]
    media:
      water:  all
      air:    [1, 2, 3]
      pmma:   [4, 5]

  # the cylindrical body of the ion chamber
  - name:      chamber_body
    class:     conestack
    origin:    [*pos, 0, -10]
    direction: [-1, 0, 0]
    layers:
      - name: electrode
        thickness:    2.03
        top radii:    [0.05, 0.305, 0.355]
        bottom radii: [0.05, 0.305, 0.355]
        media:        [&medium pmma, air, *medium]
      - name: stem
        thickness:    2.0
        top radii:    0.61
        bottom radii: 0.61
        media:        *medium

  # join the chamber tip and body to build chamber
  - name:  chamber
    class: cdgeometry
    base geometry: cd_planes_for_chamber
    fill region 0: chamber_body
    fill region 1: chamber_tip
    new indexing style: true

[rtownson]

The built-in variable referencing is nice. Any ideas on how to do input loops?

[ftessier]

Probably keep input loops as an input object that is interpreted by EGSnrc. There are no looping constructs in YAML.

[ftessier]

Note that LLMs can help with extracting the schema from the current geometry .cpp input parsing code. Here is a first draft straight out of ChatGPT o1-preview fed with egs_box.cpp:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "EGS_Box",
  "type": "object",
  "properties": {
    "dimensions": {
      "oneOf": [
        {
          "type": "array",
          "items": { "type": "number" },
          "minItems": 1,
          "maxItems": 1,
          "description": "Uniform size for all dimensions if only one value is provided."
        },
        {
          "type": "array",
          "items": { "type": "number" },
          "minItems": 3,
          "maxItems": 3,
          "description": "Individual sizes for the x, y, and z dimensions."
        }
      ]
    },
    "affine transform": {
      "type": "object",
      "description": "Optional affine transform applied to the box.",
      "properties": {
        "rotation": {
          "type": "array",
          "items": { "type": "number" },
          "minItems": 9,
          "maxItems": 9,
          "description": "3x3 rotation matrix values in row-major order."
        },
        "translation": {
          "type": "array",
          "items": { "type": "number" },
          "minItems": 3,
          "maxItems": 3,
          "description": "Translation vector for the transform."
        }
      },
      "required": ["rotation", "translation"]
    },
    "boundary tolerance": {
      "type": "number",
      "description": "Tolerance value for the boundary of the box."
    },
    "media": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Array of media names associated with the box geometry."
    },
    "labels": {
      "type": "array",
      "items": { "type": "string" },
      "description": "Array of labels for different parts or regions of the box."
    }
  },
  "required": ["name", "dimensions"]
}

[crcrewso]

@ftessier I'm personally preferring yaml or toml to json, seems more human readable.

[mchamberland]

+1 for YAML.

[rtownson]

Since we generally expect people to type input files, I'm hesitant to use any format that has much markup at all, or special characters. That's why I like the current format. Very little fluff, easy to read, no special characters except for ":". Editing any of these formats manually sounds like a huge headache. Unless we're proposing a whole new way to design input files, by GUI I suppose. The egs_editor addition would help a bit.

[ftessier]

Agreed. I seek something that has little fluff, allows comments (JSON doesn't!!), doesn't require constant quoting, while remaining readable.

In my experience, YAML fits the bill. I agree that the current egs++ input format is great. However I want something standard so that it is portable and can work with other tools (schema, validators, one's own programs and script, etc.). Another advantage is that with a YAML parsing into a DOM, we don't need to write new parsing code for a new entity, we just need to ensure that the new entity can be described as YAML.

The new .eginp input file editor implemented by @rtownson and his students is AMAZING, and I trust it will not be too difficult to adapt it to YAML. I expect it will in fact make maintenance much easier through the schema.

So who is going to start writing schemas for egs++ entities? 😉

[rtownson]

In YAML, indentation matters. As we've learned from teaching EGSnrc courses, non-programmers struggle to no end to consider indentation at all. I think that the indentation requirement will spawn a wide range of unintuitive issues for users. So I will throw another wet blanket on this discussion and consider that a point against ;)

[rtownson]

I think from my perspective, writing an input format parser is easy, and once it's done it's done. When we design our own, we can also tailor it seamlessly, instead of making awkward exceptions to an official structure. I think the choice of input format should entirely be for improved user experience, because it's the user's main interface with EGSnrc. I'm not yet sold on how YAML improves the user experience over what we can achieve by slightly improving the current design.

[ftessier]

I agree with the points you raise, it is a question of balance between pros and cons.

I understand your concerns about indentation. On the flip side, YAML can be validated against a schema to report where there are syntax errors and explain how to fix them. Editors automatically insert indentation, as with python for example. Many people claimed at the outset that python would fail because of significant indentation, yet it has led to clearer and more consistent coding in the long run, as envisioned by Guido van Rossum!

My overarching issue is that the egs++ input format is not a standard, it is just our own. We then have access to a large body of tools! Moreover, I think that Monte Carlo input definition eventually ought to be harmonized across software, so that for example an EGSnrc geometry could be loaded in Geant4 or Penelope. I think it is much easier to promote a recognized standard than promote our own, or adopt someone else's!

At the end of the day, we should implement a YAML input to test these theoretical ideas in practice.

Irregardless, we shall write a validator for egs++ inputs.

[crcrewso]

New Repository

I've created a new repo for some ideas. Should there be a GitHub Project somewhere?
Goals:

• Define schema, with examples
• Implement bi-directional parser in GUI, such that the GUI can be extended in functionality later
• Document Schema, possibly visually (with UML?)

[ftessier]

Super. For the gui part, we could implement the parsing and conversion in egs_view. @rtownson has implemented a full egsinp editor in egs_view with his co-op students. Syntax highlighting, auto-complete, templates, etc. To be release soon!

If we write a JSON shema for YAML input, I would want to sanitize the inputs to harmonize, there are a few idiosyncrasies in the current input names and conventions...

[crcrewso]

@ftessier What is your current thinking around JSON support? I ask because I find it painful to deal with as a human, and the libraries aren't that much nicer than YAML/TOML

[ftessier]

I don't support Jason, because of two significant shortcomings

• cannot enter comments
• endless quoting everywhere

JSON is a very standard and efficient "middleware" machine digestible format, but not a great human input format.

My proposal is to use YAML, but with a JSON schema to validate YAML.

[crcrewso]

My proposal is to use YAML, but with a JSON schema to validate YAML.

I see what you're suggesting now, if we pick a library and a class design, then the JSON and YAML schemas come with them. If that's the case then the work should be reversed. Could I propose that we design the JSON from a reference class implementation, not the reverse?

[ftessier]

I don't know that it is possible to generate a schema directly from the c++ classes. Is there a tool for that (I mean, apart from LLMs)? I was suggesting to write what we want the input to look like in YAML, then express the schema as JSON (to validate any input written in YAML), and tweak the classes to match if needed. The YAML can be parsed in one go to build a DOM with libyamlcpp or other standard c++ Li brary. Le ven. 22 nov. 2024, 15 h 28, Cody Crewson ***@***.***> a écrit :

…

My proposal is to use YAML, but with a JSON schema to validate YAML. I see what you're suggesting now, if we pick a library and a class design, then the JSON and YAML schemas come with them. If that's the case then the work should be reversed. Could I propose that we design the JSON from a reference class implementation, not the reverse? — Reply to this email directly, view it on GitHub <#659 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALATEIDWESVYDPHFZS2UE32B6HWJAVCNFSM6AAAAABRKLMMXKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZVGMYTGMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[crcrewso]

A workflow I've found is to use a library to save sample data to the format you want, then make changes within the source code. Manually creating the schema and parser generally aren't required anymore. The libraries will rely on things like class names and object types, with options to label the JSON override.

[ftessier]

Yes, absolutely. There are many tool to generate the schema from YAML examples, some even online. I use AI now, because it can provide an analysis of efficiency and consistency of input, suggest improvements, etc.