Changing the metadata codec of existing metadata in a tree sequence from binary to JSON

[JoshuaGensel]

Hey everyone,
I recently used the SLiM simulator to generate some tree sequences and now I want to run tsinfer on them. Unfortunately some of the metadata that SLiM writes onto its tree sequences is encoded in binary via the struct module. This binary data is currently not supported by tsinfer, so here comes my question:
Is there any way to rewrite or convert the metadata of my SLiM sequences into the JSON format to get rid of all the binary metadata?

I tried reading about this in the documentation of tskit, but since I am new to the API and not the most experienced programmer I wasn't able to solve this problem on my own. So I'm sorry if my question is silly.

Cheers and thanks for any help or advice of other sorts,
Joshua

[hyanwong]

That's a great question, and we should add the answer it to the metadata tutorial. I suspect @benjeffery will know the best answer here, but there must be a way. I guess the obvious thing is to decode the struct into python then reencode it into a json dict. I'll work up some example code.

[hyanwong]

Here's some dummy code using a schema for individuals (from here) - NB: now edited to reflect the discussion below

import json

basic_schema = tskit.MetadataSchema({'codec': 'json'})
complex_schema = tskit.MetadataSchema({
    'codec': 'json',
    'additionalProperties': False,
    'properties': {'accession': {'description': 'ENA accession number',
                                 'type': 'string'},
                   'pcr': {'description': 'Was PCR used on this sample',
                           'name': 'PCR Used',
                           'type': 'boolean'}},
    'required': ['accession', 'pcr'],
    'type': 'object',
})

# Make an example ts with struct metadata
tables = tskit.TableCollection(sequence_length=1)
tables.individuals.metadata_schema = complex_schema
row_id = tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True})
ts = tables.tree_sequence()

Convert struct to json:

def convert_to_json_metadata(ts):
    # make a new ts with json metadata
    new_tables = ts.dump_tables()
    # iterate through (nearly) all the tables
    for table_name, table in new_tables.name_map.items():
        # provenance table doesn't have metadata
        if table_name not in ["provenances"]:
            metadata = []
            for row in table:
                try:
                    row_metadata = row.metadata or {}  # convert empty byte strings to empty json
                    metadata.append(json.dumps(row_metadata).encode())
                except TypeError:
                    raise TypeError(f"Can't convert {row.metadata} to JSON")
            # packset_metadata doesn't validate, so dump json in here and switch schema after
            table.packset_metadata(metadata)
            table.metadata_schema = basic_schema
    # May also need to convert top level metadata?
    return new_tables.tree_sequence()

# test
print(convert_to_json_metadata(ts).individual(0))

[JoshuaGensel]

This way the row is not defined in the except-block. I tried this:

for table_name, table in new_tables.name_map.items():
            # provenance table doesn't have metadata
            if table_name not in ["provenances"]:
                # packset_metadata doesn't validate, so dump json in here and switch schema after
                for row in table:
                    try:
                        table.packset_metadata(json.dumps(row.metadata).encode())
                    except TypeError:
                        raise TypeError(f"Can't convert {row.metadata} to JSON")
                    table.metadata_schema = tskit.MetadataSchema({'codec': 'json'})

this gave me the following error:
TypeError: Can't convert b'' to JSON

And since I don't know what is stored in these byte objects I'm not sure if I will need any of that downstream. But I think in my case I am happy if the inferring works and sacrificing some of the SLiM metadata probably won't be a big deal later on.

[hyanwong]

Oh yes, sorry. What you did isn't quite right, though, not that it actually matters:

def convert_to_json_metadata(ts):
    # make a new ts with json metadata
    new_tables = ts.dump_tables()
    # iterate through (nearly) all the tables
    for table_name, table in new_tables.name_map.items():
        # provenance table doesn't have metadata
        if table_name not in ["provenances"]:
            metadata = []
            for row in table:
                try:
                    metadata.append(json.dumps(row.metadata).encode())
                except TypeError:
                    raise TypeError(f"Can't convert {row.metadata} to JSON")
            # packset_metadata doesn't validate, so dump json in here and switch schema after
            table.packset_metadata(metadata)
            table.metadata_schema = basic_schema
    # May also need to convert top level metadata?
    return new_tables.tree_sequence()

Looks like it can't convert the empty byte string. Doh!

[hyanwong]

You could easily replace the empty byte string with the empty string:

def convert_to_json_metadata(ts):
    # make a new ts with json metadata
    new_tables = ts.dump_tables()
    # iterate through (nearly) all the tables
    for table_name, table in new_tables.name_map.items():
        # provenance table doesn't have metadata
        if table_name not in ["provenances"]:
            metadata = []
            for row in table:
                try:
                    row_metadata = row.metadata or {}
                    metadata.append(json.dumps(row_metadata).encode())
                except TypeError:
                    raise TypeError(f"Can't convert {row.metadata} to JSON")
            # packset_metadata doesn't validate, so dump json in here and switch schema after
            table.packset_metadata(metadata)
            table.metadata_schema = basic_schema
    # May also need to convert top level metadata?
    return new_tables.tree_sequence()

It'll probably die on something else, but at least you'll be a bit further on.

[hyanwong]

Can you post the SLiM ts here (or a smaller version)?

[JoshuaGensel]

mtDNA_test_tree.zip

Here is an example sequence downsampled to 30 individuals. It is basically only a single tree because it is supposed to emulate data from the mitochdonrial DNA.

After implementing your changes to the function it seems to actually work! The only errors left are now in my code after that when trying to infer the new ts.

Thank you so much for your help!

[benjeffery]

@JoshuaGensel Which versions of sLiM and tsinfer are you using? If SlIm has the correct schema everything should work.

[hyanwong]

I'm not sure it will. I think tsinfer always outputs a tree sequence with JSON schema. The issue here is probably the tsinfer.SampleData.from_tree_sequence() function, which ports the metadata from a tree sequence, and probably only allows JSON format?

[JoshuaGensel]

I recently updated everything. the verions are:
SLiM - 3.7
tskit - 0.4.1
tsinfer - 0.2.1

The error that gets raised looks like this:
Traceback (most recent call last):
File "d:/Daten/programming_projects/AncestryProject/src/tree_parsing/haplotype_calc.py", line 130, in
haplotypeCalc(path)
File "d:/Daten/programming_projects/AncestryProject/src/tree_parsing/haplotype_calc.py", line 111, in haplotypeCalc
mtDNA_sample_data = tsinfer.SampleData.from_tree_sequence(mts_mtDNA, path = "mtDNAtest.samples")
File "C:\Users\joshu\anaconda3\lib\site-packages\tsinfer\formats.py", line 1487, in from_tree_sequence
self.individuals_metadata_schema = schema
File "C:\Users\joshu\anaconda3\lib\site-packages\tsinfer\formats.py", line 1121, in individuals_metadata_schema
self.__metadata_schema_setter(self.data["individuals"], schema)
File "C:\Users\joshu\anaconda3\lib\site-packages\tsinfer\formats.py", line 1082, in __metadata_schema_setter
raise ValueError("Only the JSON codec is currently supported")
ValueError: Only the JSON codec is currently supported

[hyanwong]

Did you try running my convert_to_json_metadata conversion script above?

[hyanwong]

Now open as an issue: tskit-dev/tskit#2129