More generic API for output files

[tkarna]

Currently outputs are defined by the following options:

options.output_directory = 'outputs'
options.simulation_export_time = 300.
options.fields_to_export = ['uv_2d', 'elev_2d']
options.fields_to_export_hdf5 = ['uv_2d', 'elev_2d']

We need a more general way to declare model outputs. The new API could be:

options.add_export(
    field_list, export_type, interval=None, time_list=None, initial_only=False,
    output_directory=None)

• field_list list of field to export, e.g. ['elev_2d', 'uv_2d']
• export_type specifies the output file type, currently 'vtk' or 'hdf5'
• interval is the export interval in seconds
• time_list alternatively, user can provide a list of export times e.g. [100, 110, 120]. Seconds since the beginning of the simulation.
• initial_only is equivalent to interval=None and time_list=[0]
• output_directory optionally the user can specify a different output directory

Examples:

options.add_export(['elev_2d', 'uv_2d'], 'vtk', interval=300.)
options.add_export(['viscosity_2d'], 'vtk', initial_only=True)
options.add_export(['elev_2d'], 'vtk', time_list=np.linspace(1000, 2000, 100),
                   output_directory='custom_output')

There are some corner cases that need to be checked, for example if the user speficies two exports that write to the same file, we should raise an error:

options.add_export(['elev_2d', 'uv_2d'], 'vtk', interval=100.)
options.add_export(['elev_2d'], 'vtk', interval=220.)  # prohibited

Is this sufficient? Are there other use cases that should be considered?

The above proposal does not handle:

• dates, as Thetis solver is not yet datetime aware
• adding multiple fields to the same file. Not relevant(?) for vtk but could be useful for other formats (hdf5, netcdf, and the forthcoming Firedrake XDMF)

[jhill1]

This looks like a good change from my point of view. I like the idea of the the time_list (v. useful for tidal sims with spin-up). It would be useful to add multiple fields to the same file. What if you (for some reason!) wanted multiple output types? Could you do multiple .add_exports? Or is there an implied restriction of one file type, field, output times, etc per simulation?

[tkarna]

What if you (for some reason!) wanted multiple output types? Could you do multiple .add_exports? Or is there an implied restriction of one file type, field, output times, etc per simulation?

Good point, there's no restriction on how many add_export definitions can be added, as long as they do not clash. For example:

options.add_export(['elev_2d'], 'vtk', time_list=np.linspace(...))  # for visualization
options.add_export(['elev_2d', 'uv_2d'], 'hdf5', interval=24*3600.)  # daily checkpoints
options.add_export(['elev_2d', 'uv_2d'], 'netcdf', interval=3600.)  # outputs to share

It would be useful to add multiple fields to the same file.

I tend to agree. Need to give that some thought. One solution is to add filename keyword argument which, if specified, would dump all the fields in that file. Depending on the file type there may be additional constraints though, e.g. it can be difficult to include fields from different function spaces in the same file etc.

[jhill1]

We could handle different function space by either a project under the hood, with warning, or an error and ask user to specify multiple files - one per function space. Either is fine (though latter is probably cleaner).

[tkarna]

Yes, this probably depends on the file format as well. Firedrake's VTK File seems to figure out the appropriate common function space: selecting-the output space when outputting multiple functions

[tkarna]

Also, this output API would imply deprecation of options

options.simulation_export_time
options.fields_to_export
options.fields_to_export_hdf5

I'm inclined to think that we'd still keep options.output_directory as the default. (One possibility is that add_export can only create subdirectories within the output_directory)