Merge pull request #7 from morand-g/main

version 0.6.2

.gitignore

@@ -1,3 +1,6 @@
+# Launcher
+update.sh
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

.readthedocs.yaml

@@ -7,9 +7,9 @@ version: 2
 
 # Set the version of Python and other tools you might need
 build:
-  os: ubuntu-20.04
+  os: ubuntu-22.04
   tools:
-    python: "3.8"
+    python: "3.10"
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:

.zenodo.json

@@ -3,17 +3,22 @@
         {
             "orcid": "0000-0002-0826-4487",
             "affiliation": "UMR Marbec, IRD",
-            "name": "Gaetan Morand"
+            "name": "Morand, Gaetan"
         },
         {
             "affiliation": "UMR Marbec, IRD",
-            "name": "Sylvain Poulain"
+            "name": "Poulain, Sylvain"
+        },
+        {
+            "orcid": "0000-0002-3519-6141",
+            "affiliation": "UMR Marbec, IRD",
+            "name": "Barde, Julien"
         }
     ],
 
     "license": "GPL-3.0",
 
-    "title": "GeoEnrich v0.5.8: a new tool for scientists to painlessly enrich species occurrence data with environmental variables",
+    "title": "GeoEnrich v0.6.2: a new tool for scientists to painlessly enrich species occurrence data with environmental variables",
 
     "related_identifiers": [
         {

CHANGELOG.md

@@ -1,3 +1,22 @@
+## v0.6.2
+
+#### New functions:
+ - Accept semicolon delimited CSV files for personal variable catalogs
+ - Use closest lower depth instead of closest depth to stay in the water column
+
+
+## v0.6.1
+
+#### Bug fixes:
+ - Fixed metadata generation when downlaoding Copernicus data
+
+## v0.6
+
+#### New functions:
+ - Added support for the new Copernicus data store (using copernicusmarine API)
+ - Added Support for 'nearest' mode for data recovery at depth.
+
+
 ## v0.5.8
 
 #### New functions:

README.md

@@ -1,4 +1,4 @@
-# **geoenrich 0.5.8**
+# **geoenrich 0.6.2**
 
 [![Read the Docs](https://img.shields.io/readthedocs/geoenrich)](https://geoenrich.readthedocs.io/en/latest/)
 [![License](https://img.shields.io/github/license/morand-g/geoenrich?color=green)](https://github.com/morand-g/geoenrich/blob/main/LICENSE)
@@ -29,9 +29,10 @@ Documentation on [Read the Docs](https://geoenrich.readthedocs.io).
 
 ![Illustration of an occurrence dataset enriched with bathymetry data](https://github.com/morand-g/geoenrich/blob/main/geoenrich/data/readme_illus_1.png?raw=true "Illustration of an occurrence dataset enriched with bathymetry data")
 
+# Acknowledgment
 This project is being developed as part of the G2OI project, cofinanced by the European union, the Reunion region, and the French Republic.
 
-<a href="https://european-union.europa.eu/index_fr"><img alt='Union Européenne' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_ue.png" height="40" ></a> &nbsp; &nbsp; <a href="https://regionreunion.com/"><img alt='Région Réunion' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_reunion.png" height="40" ></a> &nbsp; &nbsp; <a href="https://www.gouvernement.fr/"><img alt='République Française' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_france.png" height="40" ></a>
+<a href="https://european-union.europa.eu/index_fr"><img alt='Union Européenne' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_ue.png" height="60" ></a> &nbsp; &nbsp; <a href="https://regionreunion.com/"><img alt='Région Réunion' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_reunion.png" height="60" ></a> &nbsp; &nbsp; <a href="https://www.gouvernement.fr/"><img alt='République Française' src="https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/app/static/assets/logo_france.png" height="60" ></a>
 
 ## Installation
 

docker/Dockerfile

@@ -1,11 +1,11 @@
-FROM tiangolo/uwsgi-nginx-flask:python3.8
+FROM tiangolo/uwsgi-nginx-flask:python3.10
 
 RUN apt-get update
-RUN apt-get install ffmpeg libsm6 libxext6 python3-mpi4py -y
+RUN apt-get install ffmpeg libsm6 libxext6 -y
 
-RUN python3 -m pip install geoenrich mpi4py
+RUN python3 -m pip install geoenrich
 
 RUN rm /app/*
 RUN curl https://raw.githubusercontent.com/morand-g/geoenrich/main/docker/initialize.py -o /home/initialize.py
 
-RUN old_path="'./'" && new_path="'/app/data/'" && sed -i "s%$old_path%$new_path%g" /usr/local/lib/python3.8/site-packages/geoenrich/credentials_example.py
+RUN old_path="'./'" && new_path="'/app/data/'" && sed -i "s%$old_path%$new_path%g" /usr/local/lib/python3.10/site-packages/geoenrich/credentials_example.py

docker/app/main.py

@@ -16,7 +16,7 @@
 app = Flask(__name__)
 
 # enable debugging mode
-app.config["DEBUG"] = True
+app.config["DEBUG"] = False
 
 # App variables
 app.config['UPLOAD_FOLDER'] =  'static/uploads/'

docker/app/templates/home.html

@@ -79,37 +79,31 @@ <h1>GeoEnrich online</h1>
                     <div class="field padding-bottom--24">
                         <input list="var_id" name = "var_id" placeholder="Start typing to see available variables">
                         <datalist id="var_id">
-                            <option value="backscattering3d">Backscattering (3D)</option>
                             <option value="bathymetry">Bathymetry (ETOPO1)</option>
-                            <!-- <option value="bathymetry-gebco">Bathymetry (GEBCO)</option> -->
                             <option value="chlorophyll">Chlorophyll (Copernicus)</option>
-                            <option value="chlorophyll3d">Chlorophyll (3D)</option>
-                            <option value="chlorophyll-occi">Chlorophyll (OCCI)</option>
                             <option value="current3d-u">Current (u)</option>
                             <option value="current3d-v">Current (v)</option>
                             <option value="diatoms">Diatoms</option>
                             <option value="dinophytes">Dinophytes</option>
+                            <option value="fsle">FSLE (strength)</option>
+                            <option value="fsle-orientation">FSLE (orientation)</option>
                             <option value="geos-current-u">Geostrophic current (u)</option>
                             <option value="geos-current-v">Geostrophic current (v)</option>
                             <option value="green-algae">Green algae</option>
                             <option value="haptophytes">Haptophytes</option>
                             <option value="microphytoplankton">Microphytoplankton</option>
                             <option value="mixed-layer-thickness">Mixed Layer Thickness</option>
                             <option value="nanophytoplankton">Nanophytoplankton</option>
-                            <option value="ocean-heat-content">Ocean Heat Content</option>
                             <option value="organic-carbon3d">Organic Carbon (3D)</option>
                             <option value="oxygen">Oxygen</option>
                             <option value="ph">PH</option>
                             <option value="picophytoplankton">Picophytoplankton</option>
                             <option value="primary-production">Primary production</option>
                             <option value="prochlorophytes">Prochlorophytes</option>
                             <option value="prokaryotes">Prokaryotes</option>
-                            <option value="salinity">Salinity (SMOS)</option>
                             <option value="salinity3d">Salinity (3D)</option>
-                            <option value="sla">Sea level anomaly</option>
+                            <option value="ssh">Sea surface height</option>
                             <option value="sst">Sea surface temperature (Copernicus)</option>
-                            <option value="sst-mur">Sea surface temperature (MUR)</option>
-                            <option value="sst-mur25">Sea surface temperature (MUR25)</option>
                             <option value="surface-current-u">Surface current (u)</option>
                             <option value="surface-current-v">Surface current (v)</option>
                             <option value="surface-wind-u">Surface wind (u)</option>

docker/docker-compose.yml

@@ -13,7 +13,6 @@ services:
       - "8080:8080"
     environment:
       - FLASK_APP=main.py
-      - FLASK_DEBUG=1
       - 'RUN=flask run --host=0.0.0.0 --port=8080'
     command: bash -c "python /home/initialize.py
              && flask run --host=0.0.0.0 --port=8080"

docker/initialize.py

@@ -53,9 +53,9 @@
         Path('/app/static/assets/' + asset).open('wb').write(r.content)
 
 if Path('/app/conf/credentials.py').exists():
-    shutil.copy(Path('/app/conf/credentials.py'), Path('/usr/local/lib/python3.8/site-packages/geoenrich/credentials.py'))
+    shutil.copy(Path('/app/conf/credentials.py'), Path('/usr/local/lib/python3.10/site-packages/geoenrich/credentials.py'))
 
 if Path('/app/conf/personal_catalog.csv').exists():
-    shutil.copy(Path('/app/conf/personal_catalog.csv'), Path('/usr/local/lib/python3.8/site-packages/geoenrich/data/personal_catalog.csv'))
+    shutil.copy(Path('/app/conf/personal_catalog.csv'), Path('/usr/local/lib/python3.10/site-packages/geoenrich/data/personal_catalog.csv'))
 
-print('Initialization complete.')
+print('Initialization complete.')

docs/requirements.txt

@@ -1,7 +1,7 @@
 numpy
 pandas
 geopandas
-netCDF4==1.5.8
+netCDF4
 python-dwca-reader
 tqdm
 opencv-python
@@ -14,3 +14,5 @@ geojson_rewind
 geomet
 nbsphinx
 sphinx_copybutton
+sphinx_rtd_theme
+copernicusmarine

docs/source/conf.py

@@ -21,11 +21,11 @@
 # -- Project information -----------------------------------------------------
 
 project = 'geoenrich'
-copyright = '2022, Gaétan Morand (UMR Marbec). Project under GNU GPL v3 license'
+copyright = '2024, Gaétan Morand (UMR Marbec). Project under GNU GPL v3 license'
 author = 'Gaétan Morand (UMR Marbec)'
 
 # The full version, including alpha/beta/rc tags
-release = '0.5.8'
+release = '0.6.2'
 
 
 # -- General configuration ---------------------------------------------------

docs/source/enrichment.rst

@@ -33,6 +33,8 @@ Other functions (for internal use)
 
 .. autofunction:: geoenrich.enrichment.enrich_compute
 
+.. autofunction:: geoenrich.enrichment.enrich_copernicus
+
 .. autofunction:: geoenrich.enrichment.enrich_download
 
 .. autofunction:: geoenrich.enrichment.get_enrichment_id

docs/source/examples.rst

@@ -19,27 +19,31 @@ You may also use a custom csv file that does not follow any standard. In this ca
 A column with a unique ID is mandatory, to be able to link downloaded data to the corresponding occurrence. Date, latitude, and longitude columns are mandatory. Here is an exemple of such a file:
 
 .. list-table:: turtles.csv
-   :widths: 20 20 20 20 20
+   :widths: 10 20 20 10 20 20
    :header-rows: 1
 
    * - ID
      - Lat
      - Lon
+     - Depth
      - Day
      - Comments
    * - turtle1
      - -28.752241
      - 154.8926541
+     - 12
      - 2018-07-29
      - bottom feeding
    * - turtle2
      - 2.5754611
      - 72.964164
+     - 4
      - 2019-02-13
      - cruising
    * - turtle3
      - -21.2871554
      - 55.316446
+     - 3
      - 2021-01-05
      - resting
 
@@ -49,7 +53,8 @@ This file can be imported the following way::
 					id_col = 'ID',
 					date_col = 'Day',
 					lat_col = 'Lat',
-					lon_col = 'Lon')
+					lon_col = 'Lon',
+          depth_col = 'Depth')
 
 The date parser should work with any common date format. If you encounter problems with a custom date format, you can try to provide an explicit format string using the *date_format* parameter. See *strptime* documentation `here <https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`_.
 

docs/source/index.rst

@@ -1,12 +1,12 @@
-geoenrich 0.5.8 documentation
+geoenrich 0.6.2 documentation
 ==============================
 
 |Read the Docs| |License| |PyPI| |Python versions| |Last commit| |DOI|
 
 
 GeoEnrich provides functionalities to enrich georeferenced events (such as species occurrences) with environmental data from satellites or models. Users can specify a geographic or temporal buffer to include data in the neighbourhood of occurrences into their analyses. Two main outputs are available: a simple summary of the variable in the requested area, or the full data (as a geotiff raster, a png image, or a numpy array).
 
-Sea surface temperature, chlorophyll, and 40 other environmental variables are available natively, and others can easily be added by the user. This package is intended for large numbers of occurrences: local storage is implemented to avoid redundant requests to remote servers.
+Sea surface temperature, chlorophyll, and 40 other environmental variables are available natively, and other sources can easily be added by the user. This package is intended for large numbers of occurrences: local storage is implemented to avoid redundant requests to remote servers.
 
 The package provides functions to retrieve occurrence data directly from GBIF, or open a custom dataset from any source. Arbitrary areas defined by the user can also be enriched.
 

docs/source/install.rst

@@ -5,14 +5,16 @@ Installation instructions for Python
 1. Work environment
 -------------------
 
-This package was tested on Ubuntu 20.04 with Python 3.8.
+This package was tested on Ubuntu 20.04 with Python 3.8 and on Ubuntu 22.04 with Python 3.10.
 It should work on other operating systems and with other versions of Python 3, but this wasn't tested yet.
 
 2. Prerequisites
 ----------------
 
 Assuming you have Python3 and pip installed. This is automatic in all recent Linux distributions. Otherwise instructions are available here: `Python <https://wiki.python.org/moin/BeginnersGuide/Download>`_ and `pip <https://pip.pypa.io/en/stable/installation/>`_.
 
+If you want to use Copernicus data, you need to install Copernicus Marine API (`instructions <https://help.marine.copernicus.eu/en/articles/7970514-copernicus-marine-toolbox-installation>`_) and set it up with your Copernicus account (`instructions <https://help.marine.copernicus.eu/en/articles/8185007-copernicus-marine-toolbox-credentials-configuration>`_).
+
 
 3. Installation
 ---------------
@@ -53,14 +55,20 @@ There is also a dictionary named *dap_creds* that is intended to store credentia
 4.2. Adding other data sources
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-At the same location, there is a *catalog.csv* file that already contains a list of available variables. If you want to use a dataset from Copernicus, you first need to register on `their website <https://resources.marine.copernicus.eu/registration-form>`_ and write your credentials in the *credentials.py* file.
+At the same location, there is a *catalog.csv* file that already contains a list of available variables. 
 
-If you need additional variables, you can update add a *personal_catalog.csv* file to the same folder (template on `GitHub <https://github.com/morand-g/geoenrich/blob/main/geoenrich/data/personal_catalog.csv>`_). Three columns are compulsory:
+If you need additional variables, you can add a *personal_catalog.csv* file to the same folder (template on `GitHub <https://github.com/morand-g/geoenrich/blob/main/geoenrich/data/personal_catalog.csv>`_). Three columns are compulsory:
 
 - *variable*: A unique name for that variable (user defined). It needs to be different from the variable names already in the built-in catalog.
 - *url*: OpenDAP URL.
 - *varname*: Name of the variable in the remote dataset.
 
+If the required variable is from a Copernicus data set, the fields are slightly different:
+
+- *variable*: A unique name for that variable (user defined). It needs to be different from the variable names already in the built-in catalog.
+- *source*: Must be set to "Copernicus"
+- *url*: Copernicus Dataset ID
+- *varname*: Name of the variable in the remote dataset.
 
 6. Using the package
 --------------------

docs/source/r-install.rst

@@ -7,6 +7,8 @@ Installation instructions for R
 
 Assuming you have a version of R installed on your computer, as well as Python3 and pip. This is automatic in all recent Linux distributions. Otherwise instructions are available here: `Python <https://wiki.python.org/moin/BeginnersGuide/Download>`_ and `pip <https://pip.pypa.io/en/stable/installation/>`_.
 
+If you want to use Copernicus data, you need to install Copernicus Marine API (`instructions <https://help.marine.copernicus.eu/en/articles/7970514-copernicus-marine-toolbox-installation>`_) and set it up with your Copernicus account (`instructions <https://help.marine.copernicus.eu/en/articles/8185007-copernicus-marine-toolbox-credentials-configuration>`_).
+
 
 
 2. Installation
@@ -64,14 +66,20 @@ There is also a dictionary named *dap_creds* that is intended to store credentia
 3.2. Adding other data sources
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-At the same location, there is a *catalog.csv* file that already contains a list of available variables. If you want to use a dataset from Copernicus, you first need to register on `their website <https://resources.marine.copernicus.eu/registration-form>`_ and write your credentials in the *credentials.py* file.
+At the same location, there is a *catalog.csv* file that already contains a list of available variables.
 
-If you need additional variables, you can update add a *personal_catalog.csv* file to the same folder (template on `GitHub <https://github.com/morand-g/geoenrich/blob/main/geoenrich/data/personal_catalog.csv>`_). Three columns are compulsory:
+If you need additional variables, you can add a *personal_catalog.csv* file to the same folder (template on `GitHub <https://github.com/morand-g/geoenrich/blob/main/geoenrich/data/personal_catalog.csv>`_). Three columns are compulsory:
 
 - *variable*: A unique name for that variable (user defined). It needs to be different from the variable names already in the built-in catalog.
 - *url*: OpenDAP URL.
 - *varname*: Name of the variable in the remote dataset.
 
+If the required variable is from a Copernicus data set, the fields are slightly different:
+
+- *variable*: A unique name for that variable (user defined). It needs to be different from the variable names already in the built-in catalog.
+- *source*: Must be set to "Copernicus"
+- *url*: Copernicus Dataset ID
+- *varname*: Name of the variable in the remote dataset.
 
 4. Using the package
 --------------------

docs/source/satellite.rst

@@ -16,10 +16,14 @@ Other functions (for internal use)
 
 .. autofunction:: geoenrich.satellite.create_nc_calculated
 
+.. autofunction:: geoenrich.satellite.create_nc_copernicus
+
 .. autofunction:: geoenrich.satellite.ellipsoid_mask
 
 .. autofunction:: geoenrich.satellite.get_metadata
 
+.. autofunction:: geoenrich.satellite.get_metadata_copernicus
+
 .. autofunction:: geoenrich.satellite.get_var_catalog
 
 .. autofunction:: geoenrich.satellite.insert_multidimensional_slice