Merge pull request #18 from micahjohnson150/database_initial

SnowEx Database Add
snowex-hackweek · Aug 17, 2024 · b379764 · b379764
2 parents 4a4b885 + c9dddd5
commit b379764
Show file tree

Hide file tree

Showing 12 changed files with 1,003 additions and 2 deletions.
diff --git a/book/_toc.yml b/book/_toc.yml
@@ -34,6 +34,15 @@ parts:
             - file: tutorials/Data_access/earthdata_search.md
             - file: tutorials/Data_access/earthaccess_snowex.ipynb
             - file: tutorials/Data_access/earthaccess_icesat2.ipynb
+        - file: tutorials/snowex_database/index
+          title: Introduction to the SnowEx Database
+          sections:
+            - file: tutorials/snowex_database/1_getting_started_example
+            - file: tutorials/snowex_database/2_database_structure
+            - file: tutorials/snowex_database/3_forming_queries
+            - file: tutorials/snowex_database/4_get_spiral_example
+            - file: tutorials/snowex_database/5_plot_raster_example
+            - file: tutorials/snowex_database/6_wrap_up
         - file: tutorials/swesarr/index
           title: SWESARR
           sections:

diff --git a/book/tutorials/snowex_database/1_getting_started_example.ipynb b/book/tutorials/snowex_database/1_getting_started_example.ipynb
@@ -0,0 +1,137 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SnowEx Database Introduction\n",
+    "\n",
+    " \n",
+    "__Tutorial Author Micah'__: [Micah Sandusky](https://github.com/micah-prime)\n",
+    "\n",
+    "__Tutorial Author Micah_o__: [Micah Johnson](https://github.com/micahjohnson150)\n",
+    "\n",
+    "[SnowEx](https://snow.nasa.gov/campaigns/snowex) has introduced a unique opportunity to study SWE in a way thats unprecedented, but with more data comes new challenges. \n",
+    "\n",
+    "![examples](./images/data_examples.png)\n",
+    "\n",
+    "\n",
+    "<!-- \n",
+    "<img src=\"https://snowexsql.readthedocs.io/en/latest/_images/gallery_overview_example_12_0.png\" alt=\"Grand Mesa Overview\" width=\"1000px\"> -->\n",
+    "\n",
+    "**The SnowEx database is a resource that shortcuts the time it takes to ask cross dataset questions**\n",
+    "\n",
+    "      \n",
+    "- Standardizing diverse data\n",
+    "- Cross referencing data\n",
+    "- Provenance!\n",
+    "- Added GIS functionality\n",
+    "- Connect w/ ArcGIS or QGIS!\n",
+    "- **CITABLE** \n",
+    "\n",
+    "    * [*2022- Estimating snow accumulation and ablation with L-band interferometric synthetic aperture radar (InSAR)*](https://tc.copernicus.org/articles/17/1997/2023/tc-17-1997-2023-discussion.html)\n",
+    "    * [*2024 - Thermal infrared shadow-hiding in GOES-R ABI imagery: snow and forest temperature observations from the SnowEx 2020 Grand Mesa field campaign*](https://tc.copernicus.org/articles/18/2257/2024/)\n",
+    "      \n",
+    "      \n",
+    "\n",
+    "## What's in it?\n",
+    "\n",
+    "* Snow pits - Density, hardness profiles, grain types + sizes\n",
+    "* Manual snow depths - TONS of depths (Can you say spirals?)\n",
+    "* Snow Micropenetrometer (SMP) profiles - (Subsampled to every 100th)\n",
+    "* Snow depth + SWE rasters from ASO Inc.\n",
+    "* GPR\n",
+    "* Pit site notes\n",
+    "* Camera Derived snow depths\n",
+    "* Snow off DEM from USGS 3DEP \n",
+    "* And almost all the associated metadata\n",
+    "\n",
+    "## Technically, what is it?\n",
+    "\n",
+    "* PostgreSQL database\n",
+    "* PostGIS extension\n",
+    "* Supports vector and raster data\n",
+    "* And a host of GIS operations\n",
+    "* AND NOW WITH API!\n",
+    "\n",
+    "\n",
+    "### So whats the catch?\n",
+    "New tech can create barriers...\n",
+    "\n",
+    "```{figure} ./images/pits_not_bits.jpg\n",
+    ":scale: 20 %\n",
+    ":alt: pits not bits\n",
+    "```\n",
+    "\n",
+    "### TL;DR Do less wrangling, do more crunching. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## How do I get at this magical box of data ?\n",
+    "* [SQL](https://www.postgresql.org/docs/13/tutorial-sql.html) \n",
+    "* [snowexsql](https://github.com/SnowEx/snowexsql/) <span style=\"font-size:20pt;\"> **&#8592; 😎**</span>\n",
+    "\n",
+    "\n",
+    "### Welcome to API Land"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from snowexsql.api import PointMeasurements\n",
+    "\n",
+    "df = PointMeasurements.from_filter(type=\"depth\", instrument='pit ruler', limit=100)\n",
+    "df.plot(column='value', cmap='jet', vmin=10, vmax=150)\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Old Ways / Advanced Users \n",
+    "Advanced queries can be made using SQL or SQAlchemy under the hood. \n",
+    "\n",
+    "See previous presentations\n",
+    "\n",
+    "Engine objects, session objects, and a crash course in ORM, oh my! \n",
+    "* [Hackweek 2021](https://snowex-2021.hackweek.io/tutorials/database/index.html)\n",
+    "* [Hackweek 2022](https://snowex-2022.hackweek.io/tutorials/database/index.html)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/book/tutorials/snowex_database/2_database_structure.ipynb b/book/tutorials/snowex_database/2_database_structure.ipynb
@@ -0,0 +1,150 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8d563d3c",
+   "metadata": {},
+   "source": [
+    "# How is the Database Structured?\n",
+    "\n",
+    "The goal of the database is to hold as much of the SnowEx data in one place and make it easier to \n",
+    "do research with. With that in mind follow the steps below to see how the the data base is structured.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a9f0a9c7",
+   "metadata": {},
+   "source": [
+    "## Where do datasets live (i.e. tables)?\n",
+    "\n",
+    "Data in the database lives in 1 of 4 places. \n",
+    "\n",
+    "\n",
+    "```{figure} ./images/structure.png\n",
+    ":scale: 50 %\n",
+    ":alt: Structure of the snowex db\n",
+    "\n",
+    "Layout of the database tables\n",
+    "\n",
+    "```\n",
+    "\n",
+    "The 4th table is a table detailing the site information. Lots and lots of metadata for which the API has not been written yet.\n",
+    "\n",
+    "So how does this look in python?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "979fad96",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from snowexsql.api import PointMeasurements, LayerMeasurements, RasterMeasurements"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07bf71eb",
+   "metadata": {},
+   "source": [
+    "## How are tables structured?\n",
+    "Each table consists of rows and columns. Below are the available columns!\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8fd4e693",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import the class reflecting the points table in the db\n",
+    "from snowexsql.api import PointMeasurements as measurements\n",
+    "\n",
+    "# Grab one measurement to see what attributes are available\n",
+    "df = measurements.from_filter(type=\"depth\", limit=1)\n",
+    "\n",
+    "# Print out the results nicely\n",
+    "print(\"These are the available columns in the table:\\n \\n* {}\\n\".format('\\n* '.join(df.columns)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba2df485",
+   "metadata": {},
+   "source": [
+    "**Try this:** Using what we just did, but swap out PointMeasurements for LayerMeasurements.\n",
+    "\n",
+    "\n",
+    "**Question:** Did you collect any data? What is it? What table do you think it would go in?\n",
+    "\n",
+    "For more detail, checkout the readthedocs page on [database structure](https://snowexsql.readthedocs.io/en/latest/database_structure.html) to see how data gets categorized. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6d106e6e",
+   "metadata": {},
+   "source": [
+    "## Bonus Step: Learning to help yourself\n",
+    "[snowexsql](https://github.com/SnowEx/snowexsql/) has a host of resources for you to  help your self. First when you are looking for something be sure to check the snowexsql's docs.\n",
+    "There you will find notes on the database structure. datasets, and of course our new API! \n",
+    "\n",
+    "### Database Usage/Examples\n",
+    "* [snowexsql Code](https://github.com/SnowEx/snowexsql/) \n",
+    "* [snowexsql Documentation](https://snowexsql.readthedocs.io/en/latest/) \n",
+    "\n",
+    "### Database Building/Notes\n",
+    "* [snowex_db Code](https://github.com/SnowEx/snowex_db/) \n",
+    "* [snowex_db Documentation](https://snowex_db.readthedocs.io/en/latest/) "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fbe9ae13",
+   "metadata": {},
+   "source": [
+    "## Recap \n",
+    "You just explored the database structure and discussed how they differ.\n",
+    "\n",
+    "**You should know:**\n",
+    "* Which table a dataset might live in\n",
+    "* What columns you can work with (or how to get the available columns)\n",
+    "* Some resources to begin helping yourself.\n",
+    "\n",
+    "If you don't feel comfortable with these, you are probably not alone, let's discuss it!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e260d71e-c4d8-4f4f-b170-4f64f8fa56ed",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}