Your_second_ASDF_converter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2ae2873f",
   "metadata": {},
   "source": [
    "Writing your second ASDF converter tutorial\n",
    "===================================\n",
    "\n",
    "This tutorial is intended to include the use of ASDF schemas\n",
    "for the purposes of validating the ASDF files generated and\n",
    "read by this converter.\n",
    "\n",
    "What are ASDF schemas for?\n",
    "----------------------------------\n",
    "\n",
    "As such, it represents a different level of sophistication and\n",
    "consequently, more work. But it has the benefit of ensuring that\n",
    "the files it tries to read are legal against the expectations\n",
    "of the converter. For example, if someone hand edited an ASDF\n",
    "file where elements of the file include your serialized object,\n",
    "and they modify that object in such a way as to invalidate the\n",
    "object, the schema mechanism provides a way of checking that\n",
    "they haven't broken it. This makes it simple to help ensure that\n",
    "a file that is causing problems isn't with the file itself.\n",
    "\n",
    "An aside: the schema helps eliminate many problems, but not all\n",
    "necessarily. For example if one of the attributes of your object\n",
    "requires that it be a prime number, there is no schema mechanism\n",
    "to ensure it is a prime number, but it can ensure that the file\n",
    "has the required attributes, that they are of the right type,\n",
    "and can place some constraints on their values (e.g., min and \n",
    "max values, or that strings match some regular expression, or \n",
    "are one of an enumerated set of permitted values).\n",
    "\n",
    "Goal of this tutorial\n",
    "-----------------------\n",
    "\n",
    "To write converters that use schemas, and to ensure that they\n",
    "be language independent (i.e., don't require installing Python)\n",
    "we will be building a framework to allow that (if you ever \n",
    "suspect that that the schemas will be used by other libraries\n",
    "that are not in Python). Doing so adds some added complexity,\n",
    "so this tutorial is aimed mostly at those that will be writing\n",
    "schemas for pipeline code that generate products that should\n",
    "not intrinsically require Python to read. (A separate tutorial\n",
    "may be written for cases where schemas are desired, but only \n",
    "for Python).\n",
    "\n",
    "It takes a bit of effort to understand how to write schemas. \n",
    "If they seem odd and a bit confusing, that is normal. They \n",
    "are their own language of a sort and it takes a little time\n",
    "to get used to them.\n",
    "\n",
    "The schemas also involve providing mechanisms to link the schemas\n",
    "to tags so that the library knows how to associate the the schemas\n",
    "and tags with each other.\n",
    "\n",
    "Finally, we will be using schema manifests, which provide a\n",
    "more convenient way of handling multiple schemas and tags,\n",
    "and to illustrate, we will use two tags this time around.\n",
    " \n",
    "Groundwork\n",
    "--------------\n",
    "\n",
    "This tutorial requires making two separate packages, one for\n",
    "the schemas, a separate one for the converters (again, so that\n",
    "the schemas are not tied directly to Python; but note there\n",
    "will be a small amount of code in the schema package for Python,\n",
    "and we expect that if other languages are used, code for those\n",
    "may appear there as well).\n",
    "\n",
    "A utility has been developed so that the schema package is \n",
    "fairly easily generated and this tutorial will use that utility.\n",
    "The structure of the converter package will be very similar to\n",
    "that of the first tutorial.\n",
    "\n",
    "Required Software\n",
    "---------------------\n",
    "\n",
    "- numpy\n",
    "- asdf v2.8 or higher\n",
    "\n",
    "The following is recommended when creating your own package but \n",
    "not directly needed for this tutorial\n",
    "\n",
    "- cookiecutter (do a `pip install cookiecutter`)\n",
    "\n",
    "Software uninstall\n",
    "---------------------\n",
    "\n",
    "If you did the first tutorial (Your_first_ASDF_converter), in a terminal window \n",
    "type `pip uninstall mfconverter` since this tutorial will reuse the same \n",
    "photoID class and we don't want confusion between the two converters.\n",
    "\n",
    "Create schema package\n",
    "----------------------------\n",
    "\n",
    "First ensure you are in the directory that you want to create both packages in, by default it will be your home directory below, but change it to what you want before executing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb0c1b6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ~/converter_tutorial\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "067eed9a",
   "metadata": {},
   "source": [
    "The following utility will build a schema package. It will\n",
    "prompt for answers to a series of questions. Here is an \n",
    "explanation of what is expected for these questions. **You must\n",
    "run this utility from a terminal window.** Type the command as\n",
    "shown in this directory and provide the answers as shown below.\n",
    "If this is rerun you may get an initial prompt mentioning the\n",
    "template has been downloaded before; just type return to accept\n",
    "the default `yes`\n",
    "\n",
    "`\n",
    "cookiecutter gh:asdf-format/schemas-package-template\n",
    "`\n",
    "\n",
    "`\n",
    "package_name [asdf-example-schemas]: myschemas\n",
    "`\n",
    "\n",
    "This will be the directory name it will go in.\n",
    "\n",
    "`\n",
    "module_name [myschemas]: \n",
    "`\n",
    "\n",
    "Accept the default (providing a different answer will create a\n",
    "different module name within the package\n",
    "\n",
    "`\n",
    "Short_description [mfconverter]: my first schemas\n",
    "`\n",
    "\n",
    "Self evident\n",
    "\n",
    "`\n",
    "author_name [Author Name]: Bullwinkle Moose\n",
    "`\n",
    "\n",
    "Provide a phony answer. You don't really want people to know who \n",
    "wrote this, do you?\n",
    "\n",
    "`\n",
    "author_email [author@example.com]: bullwinkle@stsci.edu\n",
    "`\n",
    "\n",
    "`\n",
    "github_project [github-org/myschemas]: [bullwinkle/myschemas]\n",
    "`\n",
    "\n",
    "This won't actually create a repository on github, but is useful\n",
    "if you will add it to github\n",
    "\n",
    "`\n",
    "project_url [https://github.com/bullwinkle/myschemas]]: \n",
    "`\n",
    "\n",
    "`\n",
    "uri_authority [example.com]: stsci.edu \n",
    "`\n",
    "\n",
    "`\n",
    "uri_project [example-project]:\n",
    "`\n",
    "\n",
    "But since this is jupyter-based, we will do part of what it does\n",
    "manually later, partly because editing files within jupyter is \n",
    "not simple within the notebook for a tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbf7bee",
   "metadata": {},
   "source": [
    "Setting up the converters\n",
    "-----------------------------\n",
    "\n",
    "This will be similar to the first tutorial, but with some changes\n",
    "the structure. There will be two converters to illustrate how to\n",
    "handle multiple converters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8564c41d",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir ~/converter_tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e93474e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ~/converter_tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f25df85",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir myconverters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e931f085",
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bbc846f",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir myconverters/src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1ca1d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir myconverters/src/myconverters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "978d413d",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd myconverters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e94f6849",
   "metadata": {},
   "source": [
    "Add the usual files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61da8bca",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/photo_id.py\n",
    "class PhotoID:\n",
    "    \"Holds Phot ID information\"\n",
    "\n",
    "    def __init__(self, last_name, first_name, image):\n",
    "        \"expects a monochromatic numpy array for image\"\n",
    "        self.last_name = last_name\n",
    "        self.first_name = first_name\n",
    "        self.photo = image\n",
    "    \n",
    "    def name(self):\n",
    "        return self.last_name + ', ' + self.first_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e318f5c2",
   "metadata": {},
   "source": [
    "Add a second class that references the first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff6bd411",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/traffic_citation.py\n",
    "from .photo_id import PhotoID\n",
    "\n",
    "\n",
    "class TrafficCitation:\n",
    "    \"Record of a traffic violation\"\n",
    "\n",
    "    def __init__(self, ociffer, violation, date, time, photo_id):\n",
    "        self.ociffer = ociffer\n",
    "        self.violation = violation\n",
    "        self.date = date\n",
    "        self.time = time\n",
    "        if not isinstance(photo_id, PhotoID):\n",
    "            raise ValueError(\"not a PhotoID instance\")\n",
    "        self.photo_id = photo_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "482ba07e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/converters.py\n",
    "from asdf.extension import Converter\n",
    "from .photo_id import PhotoID\n",
    "from .traffic_citation import TrafficCitation\n",
    "\n",
    "\n",
    "class PhotoIDConverter(Converter):\n",
    "    tags = [\"asdf://stsci.edu/example-project/tags/photo_id-*\"]\n",
    "    types = [\"myconverters.photo_id.PhotoID\"]\n",
    "    # The above registers the tag that the converter is used for, as well as\n",
    "    # associating the class that the converter is used for.\n",
    "\n",
    "    # This method converts from the Python object to yaml\n",
    "    def to_yaml_tree(self, obj, tags, ctx):\n",
    "        # The yaml conversion expects a dictionary returned\n",
    "        node = {}\n",
    "        node['first_name'] = obj.first_name\n",
    "        node['last_name'] = obj.last_name\n",
    "        node['photo'] = obj.photo\n",
    "        return node\n",
    "\n",
    "    # This method converts from yaml to the Python object\n",
    "    def from_yaml_tree(self, node, tag, ctx):\n",
    "        return PhotoID(node['last_name'],\n",
    "                       node['first_name'],\n",
    "                       node['photo'])\n",
    "\n",
    "\n",
    "class TrafficCitationConverter(Converter):\n",
    "    tags = [\"asdf://stsci.edu/example-project/tags/traffic_citation-1.0.0\"]\n",
    "    types = [\"myconverters.traffic_citation.TrafficCitation\"]\n",
    "\n",
    "    def to_yaml_tree(self, obj, tags, ctx):\n",
    "        node = {}\n",
    "        node['ociffer'] = obj.ociffer\n",
    "        node['violation'] = obj.violation\n",
    "        node['date'] = obj.date\n",
    "        node['time'] = obj.time\n",
    "        node['photo_id'] = obj.photo_id\n",
    "        return node\n",
    "\n",
    "    def from_yaml_tree(self, node, tag, ctx):\n",
    "        return TrafficCitation(node['ociffer'],\n",
    "                               node['violation'],\n",
    "                               node['date'],\n",
    "                               node['time'],\n",
    "                               node['photo_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d5ff931",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/extensions.py\n",
    "from asdf.extension import ManifestExtension\n",
    "from .converters import (\n",
    "    PhotoIDConverter,\n",
    "    TrafficCitationConverter,)\n",
    "\n",
    "MY_CONVERTERS = [\n",
    "    PhotoIDConverter(),\n",
    "    TrafficCitationConverter(),\n",
    "]\n",
    "\n",
    "MY_EXTENSIONS = [\n",
    "    ManifestExtension.from_uri(\n",
    "        \"asdf://stsci.edu/example-project/manifests/allmyschemas-1.0\",\n",
    "        converters=MY_CONVERTERS)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa724a37",
   "metadata": {},
   "source": [
    "Notice the changes to the last file as compared to the first tutorial.\n",
    "Now it references a manifest that is elsewhere and provides a list\n",
    "of converter instances instead. This manifest will be located in the\n",
    "schema package and is used to associate schemas with tags.\n",
    "\n",
    "In addition, an additional file is created for the purposes of this\n",
    "module's entry point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d09bd82",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/integration.py\n",
    "def get_extensions():\n",
    "    from . import extensions\n",
    "    return extensions.MY_EXTENSIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4577bcd7",
   "metadata": {},
   "source": [
    "Add the `__init__.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99753484",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myconverters/__init__.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "799981ad",
   "metadata": {},
   "source": [
    "And now to add the setup files. Note the change to integrations.py for get_extensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b82c5f58",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile setup.cfg\n",
    "[metadata]\n",
    "name = myconverters\n",
    "description = TODO\n",
    "long_description = TODO\n",
    "author = Bullwinkle\n",
    "version='0.1.0'\n",
    "license = BSD-3-Clause\n",
    "\n",
    "[options]\n",
    "zip_safe = True\n",
    "python_requires = >=3.6\n",
    "setup_requires =\n",
    "    setuptools_scm\n",
    "install_requires =\n",
    "    jsonschema>=3.0.2\n",
    "    asdf>=2.8\n",
    "    psutil>=5.7.2\n",
    "    numpy>=1.16\n",
    "package_dir =\n",
    "    =src\n",
    "packages = find:\n",
    "\n",
    "[options.entry_points]\n",
    "asdf.extensions =\n",
    "    my_extension = myconverters.integration:get_extensions\n",
    "\n",
    "[options.packages.find]\n",
    "where = src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ebbffa3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile setup.py\n",
    "#!/usr/bin/env python3\n",
    "from setuptools import setup\n",
    "\n",
    "setup()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21c2ef73",
   "metadata": {},
   "source": [
    "Dealing with the Schemas\n",
    "------------------------------\n",
    "The utility previously mentioned would have contructed a directory tree for us with many files populated\n",
    "\n",
    "Only the strictly necessary parts will be done manually, partly to minimize what has to be done, but also it isn't easy to edit files in a tutorial notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02fb0e4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28402214",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir myschemas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "308c46c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd myschemas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a36535a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd3a8d6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir src/myschemas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "886f2ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir src/myschemas/resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0009d892",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir src/myschemas/resources/manifests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "146a9a4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir src/myschemas/resources/schemas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90720a6a",
   "metadata": {},
   "source": [
    "Create the setup files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32e10f16",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile setup.py\n",
    "#!/usr/bin/env python3\n",
    "from setuptools import setup\n",
    "\n",
    "setup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b8c71d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile setup.cfg\n",
    "[metadata]\n",
    "name = myschemas\n",
    "description = Schemas for second converter tutorial\n",
    "long_description = TODO\n",
    "author = Bullwinkle\n",
    "version='0.1.0'\n",
    "license = BSD-3-Clause\n",
    "\n",
    "[options]\n",
    "zip_safe = True\n",
    "python_requires = >=3.6\n",
    "setup_requires =\n",
    "    setuptools_scm\n",
    "install_requires =\n",
    "    jsonschema>=3.0.2\n",
    "    asdf>=2.8\n",
    "    psutil>=5.7.2\n",
    "    numpy>=1.16\n",
    "package_dir =\n",
    "    =src\n",
    "packages = find:\n",
    "\n",
    "[options.entry_points]\n",
    "asdf.resource_mappings =\n",
    "    myschemas = myschemas.integration:get_resource_mappings\n",
    "\n",
    "[options.package_data]\n",
    "myschemas.resources = manifests/*.yaml, schemas/*.yaml\n",
    "\n",
    "[options.packages.find]\n",
    "where = src\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e0e88f6",
   "metadata": {},
   "source": [
    "Note that the form of the entry points is different now that schemas are involved.\n",
    "Also there is a need to add an indication that the schema and manefits files must\n",
    "installed along with the software.\n",
    "\n",
    "Now create the two needed schema files. The explanation of how the schema works\n",
    "will follow the creation of the schemas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b53eac2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/resources/schemas/photo_id-1.0.0.yaml\n",
    "%YAML 1.1\n",
    "---\n",
    "$schema: \"http://stsci.edu/schemas/yaml-schema/draft-01\"\n",
    "id: asdf://stsci.edu/example-project/schemas/photo_id-1.0.0\n",
    "\n",
    "title: Photo ID information\n",
    "type: object\n",
    "properties:\n",
    "  last_name:\n",
    "    title: Last name of the ID holder\n",
    "    type: string\n",
    "  first_name:\n",
    "    title: First name of ID holder\n",
    "    type: string\n",
    "  photo:\n",
    "    title: Monochromatic photo of ID holder\n",
    "    tag: tag:stsci.edu:asdf/core/ndarray-1.0.0\n",
    "    datatype: int8\n",
    "    ndim: 2\n",
    "\n",
    "propertyOrder: [last_name, first_name, photo]\n",
    "flowStyle: block\n",
    "required: [last_name, first_name, photo]\n",
    "...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee0ecaff",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/resources/schemas/traffic_citation-1.0.0.yaml\n",
    "%YAML 1.1\n",
    "---\n",
    "$schema: \"http://stsci.edu/schemas/yaml-schema/draft-01\"\n",
    "id: asdf://stsci.edu/example-project/schemas/traffic_citation-1.0.0\n",
    "\n",
    "title: Traffic Citation Information\n",
    "type: object\n",
    "properties:\n",
    "  ociffer:\n",
    "    title: Name of ociffer issuing citation\n",
    "    type: string\n",
    "  violation:\n",
    "    title: Type of traffic violation\n",
    "    type: string\n",
    "    enum: [\"speeding\", \"DWI\", \"Failure to Signal\", \"Driving like a jerk\"]\n",
    "  date:\n",
    "    title: Date of violation\n",
    "    type: string \n",
    "  time:\n",
    "    title: Time of day of violation\n",
    "    type: string\n",
    "  photo_id:\n",
    "    title: photo_id of the driver committing violation\n",
    "    tag: asdf://stsci.edu/example-project/tags/photo_id-1.0.0\n",
    "\n",
    "propertyOrder: [ociffer, violation, date, time, photo_id]\n",
    "flowStyle: block\n",
    "required: [ociffer, violation, date, time, photo_id]\n",
    "...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3cc78f3",
   "metadata": {},
   "source": [
    "As mentioned previously, the purpose of the schemas is to indicate what \n",
    "is required of the associated ASDF yaml content corresponding to the\n",
    "tag that identifies that content.\n",
    "\n",
    "The schemas follow the the definition of schema elements defined by\n",
    "JSON Schema, which is a standard for schemas. Nominally this standard\n",
    "expects schemas to be written in JSON, but just to confuse you, we \n",
    "write schemas in YAML to be consistent with the fact that we are\n",
    "heavily centered on YAML (essentially there is a simple one-to-one\n",
    "correspondence between the elements defined for the JSON representation\n",
    "and the YAML representation; in fact, JSON is a subset of YAML).\n",
    "\n",
    "A full and fairly accessible explanation of JSON Schema can be found\n",
    "in this online book: https://json-schema.org/understanding-json-schema/\n",
    "\n",
    "What follows is an annotated version of the photo ID schema \n",
    "```\n",
    "%YAML 1.1\n",
    "---\n",
    "# The previous is a standard YAML header with --- indicating the start\n",
    "# of the YAML content\n",
    "\n",
    "# The following line identifies the meta-schema that applies to the \n",
    "# schema itself. It is possible to customize that for more specialized\n",
    "# versions\n",
    "$schema: \"http://stsci.edu/schemas/yaml-schema/draft-01\"\n",
    "# The following line identifies the schema uniquely\n",
    "id: asdf://stsci.edu/example-project/schemas/photo_id-1.0.0\n",
    "\n",
    "# Wherever you see title as an attribute it is intended as a short\n",
    "# description of the contents\n",
    "title: Photo ID information\n",
    "\n",
    "# \"type: object\" means that the contents matching this schema should\n",
    "essentially be a dictionary at the first level (there can be more \n",
    "than one level in an object schema.\n",
    "type: object\n",
    "\n",
    "# The properties attribute lists all the attributes that have any \n",
    "# information about them, and particularly any constraints on their\n",
    "# types or valuse\n",
    "properties:\n",
    "  last_name:\n",
    "    title: Last name of the ID holder\n",
    "    # type indicates the type that the actual value in the YAML content\n",
    "    # must have. It will fail validation if the value is not the right type.\n",
    "    type: string\n",
    "  first_name:\n",
    "    title: First name of ID holder\n",
    "    type: string\n",
    "  photo:\n",
    "    title: Monochromatic photo of ID holder\n",
    "    # tag is an alternate way of specifying the type. In this case the\n",
    "    # appearance of this type means that the content of the photo\n",
    "    # attribute must be consistent with what this tag specifies (through\n",
    "    # the corresponding schema). In this example, a 2-d byte array is\n",
    "    # required, and the datatype and ndim attributes are extensions of\n",
    "    # the JSON Schema machinery for ASDF (they are not part of the basic\n",
    "    # JSON Schema system.)\n",
    "    tag: tag:stsci.edu:asdf/core/ndarray-1.0.0\n",
    "    datatype: int8\n",
    "    ndim: 2\n",
    "\n",
    "# In the following, only \"required\" is part of validation. It means these\n",
    "# attributes must be present in the content covered by the schema. Without\n",
    "# this, content with type or value restrictions is generally optional.\n",
    "# PropertyOrder and flowStyle are indications to the library as to the\n",
    "# desired way the YAML content is to be written to a file. PropertyOrder\n",
    "# specifies the order that attributes should appear in the ASDF file, and\n",
    "# flowStyle indicates which of the two ways of representing YAML should be\n",
    "# used to write the file (block means using an indented style rather than\n",
    "# one using block delimiters such as brackets of any kind).\n",
    "propertyOrder: [last_name, first_name, photo]\n",
    "flowStyle: block\n",
    "required: [last_name, first_name, photo]\n",
    "...\n",
    "```\n",
    "\n",
    "Regarding the traffic_citation schema, all the above applies as well,\n",
    "but there is one new element for the violations attribute.\n",
    "\n",
    "```\n",
    "  violation:\n",
    "    title: Type of traffic violation\n",
    "    type: string\n",
    "    # The following enum attribute indicates that the required string can\n",
    "    # only have four allowed values, those listed for enum. Any other \n",
    "    # value will cause a validation error.\n",
    "    enum: [\"speeding\", \"DWI\", \"Failure to Signal\", \"Driving like a jerk\"]\n",
    "```\n",
    "\n",
    "The rest of the content of this package involves making the necessary\n",
    "connections to ASDF and to the associated tags.\n",
    "\n",
    "We will start with the manifest file. This file makes it simpler to \n",
    "put all the connections in one place, and it allows sharing schemas\n",
    "between different types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bac28bcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/resources/manifests/allmyschemas-1.0.yaml\n",
    "%YAML 1.1\n",
    "---\n",
    "id: asdf://stsci.edu/example-project/manifests/allmyschemas-1.0\n",
    "extension_uri: asdf://stsci.edu/example-project/extensions/allmyschemas-1.0\n",
    "title: All my schemas 1.0\n",
    "description: |-\n",
    "  A set of tags for serializing all my schemas.\n",
    "asdf_standard_requirement:\n",
    "  gte: 1.1.0\n",
    "tags:\n",
    "# Object Modules\n",
    "- tag_uri: asdf://stsci.edu/example-project/tags/photo_id-1.0.0\n",
    "  schema_uri: asdf://stsci.edu/example-project/schemas/photo_id-1.0.0\n",
    "  title: Photo ID information\n",
    "  description: |-\n",
    "    Name on Photo ID and the photo from the Photo ID\n",
    "- tag_uri: asdf://stsci.edu/example-project/tags/traffic_citation-1.0.0\n",
    "  schema_uri: asdf://stsci.edu/example-project/schemas/traffic_citation-1.0.0\n",
    "  title: Information about an issued traffic citation\n",
    "  description: |-\n",
    "    Information about an issued traffic citation including ociffer name, type of violation,\n",
    "    date, time, and photo ID.\n",
    "...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81338de6",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Note that the manifest associates a tag with a schema, and ASDF will use\n",
    "that to map from one to the other.\n",
    "\n",
    "The last file to add is the Python code used for entry points to do just\n",
    "that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3c2d92",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/integration.py\n",
    "import sys\n",
    "\n",
    "from asdf.resource import DirectoryResourceMapping\n",
    "\n",
    "if sys.version_info < (3, 9):\n",
    "    import importlib_resources\n",
    "else:\n",
    "    import importlib.resources as importlib_resources\n",
    "\n",
    "\n",
    "def get_resource_mappings():\n",
    "    \"\"\"\n",
    "    Get the resource mapping instances for myschemas\n",
    "    and manifests.  This method is registered with the\n",
    "    asdf.resource_mappings entry point.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    list of collections.abc.Mapping\n",
    "    \"\"\"\n",
    "    from . import resources\n",
    "    resources_root = importlib_resources.files(resources)\n",
    "\n",
    "    return [\n",
    "        DirectoryResourceMapping(\n",
    "            resources_root / \"schemas\", \"asdf://stsci.edu/example-project/schemas/\"),\n",
    "        DirectoryResourceMapping(\n",
    "            resources_root / \"manifests\", \"asdf://stsci.edu/example-project/manifests/\"),\n",
    "    ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f70eef1",
   "metadata": {},
   "source": [
    "The key element in this file are the last few lines. What this is \n",
    "indicating is that when a schema id is encountered and the beginning\n",
    "of that schema_id matches the last string, the remainder of that\n",
    "schema id is taken from the schema ID and appended to the local\n",
    "path for the resource directory plus the \"schemas\" subdirectory\n",
    "to construct the path to the actual schema file.\n",
    "\n",
    "Likewise for the manifest, where the manifest id is located in the \n",
    "extensions.py file for the converter package. Basically, it tells the\n",
    "converter package, where the manifest file may be found on the filesystem,\n",
    "and from that, where the schemas may be found on the local filesystem.\n",
    "\n",
    "Did we say last file? Not quite. We need an empty `__init__.py` to make\n",
    "this a package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d47a1b5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/__init__.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebb6630a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/myschemas/resources/__init__.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5b7344a",
   "metadata": {},
   "source": [
    "Installing the packages\n",
    "---------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a67e777",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install --editable ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8261435e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ../myconverters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd7dd553",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install --editable ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7e5895",
   "metadata": {},
   "source": [
    "Testing the converters\n",
    "--------------------------\n",
    "\n",
    "The following code can restart the converter without having\n",
    "go through the notebook again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76aace1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display_html\n",
    "def restartkernel() :\n",
    "    display_html(\"<script>Jupyter.notebook.kernel.restart()</script>\",raw=True)\n",
    "restartkernel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d89252f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bffb3e96",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import asdf\n",
    "from myconverters.photo_id import PhotoID\n",
    "from myconverters.traffic_citation import TrafficCitation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e23b16dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "im = np.zeros((10,10), dtype=np.int8)\n",
    "p = PhotoID('man', 'invisible', im)\n",
    "tc = TrafficCitation('Dudley Doright','speeding','2021-7-1', '11:11:11', p)\n",
    "af = asdf.AsdfFile()\n",
    "#af.tree = {'id': p}\n",
    "af.tree = {'citation': tc}\n",
    "af.write_to('test.asdf', auto_inline=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e297c6b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "more test.asdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75c02aa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "af2 = asdf.open('test.asdf') # See what happens when we read it back in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c032b8ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(af2.tree['citation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6652c859",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(af2.tree['citation'].photo_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65ec0844",
   "metadata": {},
   "outputs": [],
   "source": [
    "af2.tree['citation'].ociffer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78e35c3f",
   "metadata": {},
   "source": [
    "What happens if we break the law?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b56f3ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "af2.tree['citation'].ociffer = 27"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce501b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "af2.write_to('test2.asdf')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28cd39ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "asdf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7617970",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}