Skip to content

DTVF: Data Structure

Jump to bottom
Michael Hillman edited this page Nov 25, 2021 · 21 revisions

The DTVF data structure consists of an arbitrary hierarchy of directories, with each leaf (i.e. lowest level) directory containing a complete, self-contained, set of files for visualisation. This hierachy allows developers to effectively provide grouping to their data. Each level in the tree contains a meta.json file detailing the name of that group and the user-facing names/locations of its sub-groups. Each leaf directory, should contain a complete set of data for visualisation.

Controls within the visualisation will be generated allowing the user to select an option for each group level, ultimately selecting a single leaf directory; the contents of this leaf directory will then be plotted. There is now no sytem to combine data from multiple directories; whilst this may promote data duplication, these visualisation data files are normally quite small, and the simplicity of the expected data format should help new users get up and running much quicker.

Update: If wholly different data sets are needed (perhaps with different default map options), the DTVF now supports the ability to use multiple root directories, each containing a separate hierarchal structure (as outlined above).

Branch Directory

A branch directory is a directory in which no visualisation data exists; the directory only exists to act as a group for its sub-directories. Within such a directory, the meta.json files is used to provide a user-facing group name, and details for each of the sub-directories (that will become sub-groups within the framework).

Without this file, or the proper contents within it, the framework will not be able to offer the grouping or data you may expect. A example of a meta.json file for a branch directory is shown below.

{
    "label": "My Group #1",
    "groups": [
        {
            "name": "My Subgroup #1",
            "directory": "sub-group-1"
        },
        {
            "name": "My Subgroup #2",
            "directory": "sub-group-2"
        },
        {
            "name": "My Subgroup #3",
            "directory": "sub-group-3"
        }
    ]
}

Root Directory

The root directory for your data structure is essentially a branch directory, but will require some additional metadata in the form of a tree.json file and an additional global node within its meta.json file. Note that if you're plotting data with no grouping (i.e. the data files are just within the root directory), the above conditions are still applicable.

The tree.json file defines the structure of the Layer Tree that is shown to the user; this control allows them to hide/show different layers of data on the visualisation. Details about this file and its use can be seen on the Layer Tree page.

The meta.json file of the root directory will be the same as those seen in the branching directory, however that content will be nested within a local JSON node, and an additional global node is added to specify global visualisation options. An array of hyperlinks can be specified within the linkedFiles property, these will be added to the side panel automatically (this is intended for developers which to include reports, logs, further instructions etc). Note that URLs will be appended to the base URL of the visualisation unless they start with the "http" characters.

{
    "global": {
        "defaultCenter": [0.39713859558105463, 52.75354199742786],
        "defaultZoom": 15,
        "defaultBearing": -30,
        "defaultPitch": 65,
        "linkedFiles": [
            {
                "text": "Local Test File",
                "url": "linked-files/test-file-1.txt"
            },
            {
                "text": "External Location",
                "url": "http://google.com"
            }
        ]
    },
    "local": {
        "label": "My Group #1",
        "groups": [
            {
                "name": "My Subgroup #1",
                "directory": "sub-group-1"
            }
        ]
    }
}

Leaf Directory

A leaf directory is the lowest level within a group and contains the actual data to be visualised. Leaf directories will also contain a meta.json file, but in this case it will describe the data sets available within the group (rather than any sub-groups).

The meta.json file here will contain a dataSets JSON array, within which each object defines a single data set to be plotted. Each data set will be added to the visualisation as a single source, and will be displayed via a single layer (caveat: additional layers may actually be added to define interaction areas, clustering, labels etc).

Each of these datasets will comprise of a single GeoJSON file (read more about their expected contents on the GeoJSON page), one or more metadata files (in which each location's non-geographic data is added), and one or more timeseries files (in which temporal data series are defined).

An example leaf directory's meta.json file is shown below:

{
    "dataSets": [
        {
            "name": "assets-1",
            "dataLocation": "assets-1.geojson",
            "locationType": "circle",
            "metaFiles": ["assets-1-meta.json"],
            "timeseriesFiles": ["assets-1-timeseries.json"]
        },
        {
            "name": "assets-2",
            "dataLocation": "assets-2.geojson",
            "locationType": "circle",
            "metaFiles": ["assets-2-meta.json"]
        },
        {
            "name": "assets-3",
            "dataLocation": "assets-3.geojson",
            "locationType": "circle"
        }
    ]
}

Location Types

Each data set specifies a locationType property; this property defines the type of layer(s) that the framework will create to represent this data.

The following options are available for the locationType property:

  • circle or point: Only available for data sources that define point locations. This option will render colored circles around each location. If the locationType field is missing, this option will be assumed.

  • symbol: Only available for data sources that define point locations. This option will render pre-defined icons at each location.

  • fill or polygon: Only available for data sources that define polygons/regions. This option will render a colored, polygonal shape in which each vertex corresponds to a coordinate listed in the data source.

  • extrusion of building: Only available for data sources that define polygons/regions. This option will use the additional fill-extrusion-base and fill-extrusion-height properties of each location to render a 3d extruded polygon for each location.

  • line: Will generate colored lines following the set of coordinates listed for each location.

Note that a single data set uses a single locationType, this means that all locations with the set will be treated the same. If the developer has a single data source in which they want some locations to be circles, and some to be symbols, they will need to split this source into multiple data sets.

Each of these location types comes with its own set of visual properties for each location that are set in the GeoJSON file. A list of the supported properties (as not all properties MapBox offers are supported by the framework) are listed on the GeoJSON page.

Clustering

The MapBox library also supports location clustering, improving the visibility of data at high zoom levels. This options can be enabled and configured for each data set individually.

To enable and configure clustering for a data set, a few additional properties must be set for that set in the meta.json file. These are listed below:

  • cluster: this boolean property enables/disables the clustering feature for that data source.

  • clusterRadius: defines the distance between locations when clustering will occur. A value of 512 represents an entire map tile, defaults to 50.

  • clusterProperties: When locations are clustered, a new location on the map is added. By default this new location does not contain any of the metadata from its constituent locations. One can however define expressions within this properties node to aggregate/sum/etc the properties from each location and set them as properties for the clustered location.

Metadata and Timeseries

Each data set can be linked to one or more metadata and timeseries JSON files. For examples of each, see the mock data supplied with the example implementation in the example-dt-vis directory.

Metadata

Each metadata file should define non-visualisation properties for each location specified within that data set's GeoJSON file. The id field used to indentify each location in the GeoJSON (see the GeoJSON page for more on this), should be used within the metadata file to specify which location each node belongs to. When a location is selected, the corresponding node from each linked metadata file will be found and added to a single in-memory JSON structure for display in the visualisation's side panel.

Timeseries

Timeseries files provide data for a particular location that has been measured against Time. These files need to be in a specific format (again utilising the id field to link to the location); to that end, the TimeSeriesClient class within the JPS Base Library should be used to output these files; this is the format of JSON file the framework expects.

Contents

Clone this wiki locally