Skip to content

Commit

Permalink
major update to migrate from CREODIAS to COPERNICUS API for ESA satel…
Browse files Browse the repository at this point in the history
…lite

migrate from CREODIAS to COPERNICUS API for ESA satellite.
- work around 2 step authentication of CREODIAS
- Add support for sentinel-3 altimeter (SRAL), and synergy products (SYN)
- add product options for OLCI and SLSTR
- Add support for Sentinel-1 for sea ice, fishing, and oil spill monitoring
- Add support for Landast 5, 7, and 8 over Europe
  • Loading branch information
guibourdin committed Jul 17, 2023
1 parent 1a13e27 commit faed5bd
Show file tree
Hide file tree
Showing 3 changed files with 572 additions and 393 deletions.
370 changes: 186 additions & 184 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,199 +1,201 @@
getOC
=====

_Bulk download Ocean Color images (ESA and NASA platforms)._
_Bulk download Ocean Color images from ESA and NASA platforms._

getOC is a python utility in command line to easily bulk download Ocean Color images from [NASA Ocean Data](https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/) and [ESA CREODIAS](https://finder.creodias.eu/resto/api2/collections/Sentinel2/search.json?) APIs. Provide a list of positions and dates in a csv file (ex: test.csv), select an instrument, a processing level, and a product type and getOC will get the images.
getOC is a python utility simplifying bulk download of Ocean Color images from [NASA Ocean Data](https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/) and [ESA COPERNICUS Dataspace](https://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel2/search.json?) APIs. getOC takes as input a csv file with the list of positions and dates of interest (ex: test.csv) and it downloads images for the selected
instrument, processing level, and product type. Depending on the options selected an [EARTHDATA](https://urs.earthdata.nasa.gov/users/new) or a [COPERNICUS dataspace] (https://identity.dataspace.copernicus.eu/auth/realms/CDSE/login-actions/registration?client_id=cdse-public&tab_id=G8kjxyKCxI8) account is required. A strong internet connection is recommended.

[EARTHDATA account](https://urs.earthdata.nasa.gov/users/new) required to download NASA satellites data
[CREODIAS account](https://auth.creodias.eu/auth/realms/dias/protocol/openid-connect/auth?scope=openid+profile+email&response_type=code&redirect_uri=https%3A%2F%2Fcreodias.eu%2Fc%2Fportal%2Flogin&state=9ff87a6f1cefa7fa0e8ec5b49bf9bc33&client_id=CLOUDFERRO_PARTNERS&response_mode=query) required to download ESA satellites data
Synthax for ESA satellite collections can be found [here](https://catalogue.dataspace.copernicus.eu/resto/api/collections/Sentinel3/describe.xml) (example for Sentinel-3 collection)

Level 3 'L3b' and 'L3m' download are only available for NASA satellites
DEPRECATED: Previous getOC versions used the copernicus API but the copernicus v2 required 2 step authentications preventing automatic downloads. ESA satellites (+Landsat over europe) are now downloaded using the new Copernicus API

### Argument description:
- **`-i`** **instrument**
- **`SeaWiFS`**
- **`MODIS-Aqua`**
- **`MODIS-Terra`**
- **`OCTS`**
- **`CZCS`**
- **`MERIS`**
- **`VIIRSN`**
- **`VIIRSJ1`**
- **`HICO`**
- **`OLCI`**
- **`SLSTR`**
- **`MSI`**
- **`GOCI`**

- **`-l`** **level**
- **MSI**:
- **`L1C`**
- **`L2A`**

- **MODIS-Aqua** **MODIS-Terra** **VIIRSN** **VIIRSJ1** **SeaWiFS** **OCTS** **CZCS**:
- **`L1A`**
- **`L2`**
- **`L3b`**
- **`L3m`**

- **OCLI** full resolution [default]:
- **`L1`** or **`L1_EFR`**
- **`L2`** or **`L2_WFR`**

- **OCLI** low resolution:
- **`L1_ERR`**
- **`L2_WRR`**

- **SLSTR** low resolution:
- **`L1`** or **`L1_RBT`**
- **`L2`** or **`L2_WST`** [default; GHRSST recommendations]
- **`L2_WCT`** [weighted combinations of brightness temperatures]

- **MERIS**:
- **`L1`**
- **`L2`**
- **`L3b`**
- **`L3m`**

- **GOCI**:
- **`L1B`**
- **`L2`**

- **`-u`** **username**
- Earthdata login for NASA satellites
- Creodias login for ESA satellites
If you only need to download images in a given time frame without specifying a position, the [File Search](https://oceandata.sci.gsfc.nasa.gov/api/file_search) utility from NASA might be a better tool. The NASA utility also provides a few wget examples on this [Ocean Color Forum Post](https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=12520).

- **`-w`** **write**
- Prints the image list on a copy of the csv file after the query is completed.
## Supported instrument

- **`-r`** **read**
- getOC loads the list previously queried and printed, to avoid querying twice the same list
| Instrument Name | Instrument Acronym | Temporal Coverage | Spatial Coverage | Level | L2 Product |
| --- | --- | --- | --- | --- | --- |
| Coastal Zone Color Scanner (CZCZ) | CZCZ | 1978-1986 | global@1km | L1A, L2, L3m | OC |
| Ocean Color Temperature Sensor (OCTS) | OCTS | 1996-1997 | global@700m | L1A, L2 | OC |
| Sea Viewing Wide Field Scanner (SeaWiFS) | SeaWiFS | 1997-2010@daily | [email protected] | L1A, L2, L3m | OC |
| Moderate Resolution Imaging Spectroradiometer (MODIS, Terra) | MODIS-Terra | 1999-now@daily | [email protected],0.5,1km | L1A, L2, L3m | OC |
| Moderate Resolution Imaging Spectroradiometer (MODIS, Aqua) | MODIS-Aqua | 2002-now@daily | [email protected],0.5,1km | L1A, L2, L3m | OC, SST |
| MEdium Resolution Imaging Spectrometer (MERIS, Full/Reduced Resolution) | MERIS-FR, MERIS-FR | 2002-2012@daily | global@300m-1km | L1, L2 | OC |
| Geostationary Ocean Color Imager (GOCI) | GOCI | 2010-now@hourly | 36ºN 130ºE@500m | L1B, L2 | OC |
| Visible Infrared Imaging Radiometer Suite (VIIRS, Suomi NPP) | VIIRSN | 2011-now@daily | global@1km | L1A, L2, L3m | OC, SST |
| Visible Infrared Imaging Radiometer Suite (VIIRS, JPSS-1) | VIIRSJ1 | 2017-now@daily | global@1km | L1A, L2, L3m | OC |
| MultiSpectral Instrument (MSI, Sentinel-2A/B) | MSI | 2015-now@5days | coastal@10,20,60m | L1C, L2A | OC |
| Ocean and Land Colour Instrument (OLCI, Sentinel-3A/B) | OLCI | 2016-now@daily | global@300m-1km | L1-EFR, L1-ERR, L2-WFR, L2-WRR, L2-LFR, L2-LRR | OC |
| Sea and Land Surface Temperature Radiometer (SLSTR, Sentinel-3A/B) | SLSTR | 2016-now@daily | global@1km | L1-RBT, L2-WST, L2-LST, L2-AOD, L2-FRP | SST |
| OLCI/SLSTR Synergy products (SYN, Sentinel-3A/B) | SYN | 2016-now@daily | global@1km | L2-SYN, L2-V10, L2-VG1, L2-VGP, L2-AOD | OC, SST |
| SAR Radar Altimeter (SRAL, Sentinel-3A/B) | SRAL | 2016-now@daily | global@300m | L1-SRA, L1-SRA-A, L1-SRA-BS, L2-WAT, L2-LAN, L2-LAN-HY, L2-LAN-LY, L2-WAT-SI | OTHER (Altimeter) |
| Synthetic Aperture Radar (SAR, Sentinel-1A/B) | SAR | 2014-now@12days | global@5m | L0-RAW, L1-GRD, L1-GRD-COG, L1-SLC, L2-OCN, L2-CARD-BS, L2-CARD-COH12 | OTHER (ice, fishing, and oil spill monitoring) |
| Thematic Mapper (TM, Landsat-5) | L5-TM | 1984-2013@16days | Europe@30m-OC 120m-SST | L1G, L1T | OC, SST |
| Enhenced Thematic Mapper Plus (ETM+, Landsat-7) | L7-ETM | 1999-2011@16days | Europe@30m-OC 120m-SST | L1G, L1GT, L1T, TC-1P | OC, SST |
| Operational Land Imager (OLI) & Thermal Infrared Sensor (TIRS) (Landsat-8) | L8-OLI-TIRS | 2013-now@16days | Europe@30m-OC 100m-SST | L1, L1GT, L1T, L1TP, L2, L2SP| OC, SST |

- **`-p`** **product**
Specify the product type to download:
- **`OC`** [default]
- **`IOP`**
- **`SST`**
Resolutions are indicative for Level 1 and 2 data, Level 3 is only available at 4 or 9 km for NASA satellites.

**`--box`** **bounding box**
- Define the size of the bounding box around the point of interest in nautical miles. Downloads all images that intersect with this box (must be > 0).

**`-q`** **quiet**
- Quiet please ! getOC does not output any information relative to the download and querying of the points of interest.
## Command Line Usage

### Usage examples:
python getOC -i MODIS-Aqua -l L2 <filename> -u <earthdata-username> -w -p OC --box 60
python getOC -i MODIS-Terra -l L2 <filename> -u <earthdata-username> -w -p SST --box 60
python getOC -i VIIRSJ1 -l L1A <filename> -u <earthdata-username> -w --box 60
python getOC -i OLCI -l L1 <filename> -u <creodias-username> -w -p OC --box 60
python getOC -i OLCI -l L2_WRR <filename> -u <creodias-username> -w -p OC --box 60
python getOC -i MSI -l L1C <filename> -u <creodias-username> -w -p OC --box 60
python getOC -i MODIS-Aqua -l L3m test.csv <earthdata-username> -p CHL -b 8D --res 4km -w
Typical call from bash:

If you only need to download images in a given time frame without specifying a position, the [File Search](https://oceandata.sci.gsfc.nasa.gov/api/file_search) utility from NASA might be a better tool. The NASA utility also provides a few wget examples on this [Ocean Color Forum Post](https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=12520).
python getOC.py -i VIIRSJ1 -l L2 <csv_filename> -u <earthdata-username> -w --box 60

### Argument description:
- `-i INSTRUMENT`, `--instrument=INSTRUMENT`: specify instrument, available options are:
- `SeaWiFS`
- `MODIS-Aqua`
- `MODIS-Terra`
- `OCTS`
- `CZCS`
- `MERIS`
- `VIIRSN`
- `VIIRSJ1`
- `HICO`
- `OLCI`
- `SLSTR`
- `SYN`
- `SRAL`
- `MSI`
- `GOCI`
- `L5-TM`
- `L7-ETM`
- `L8-OLI-TIRS`
- `-l LEVEL`, `--level=LEVEL`: specify processing level, available options are listed below (c.f. table above for level compatible with the instrument selected).
- MODIS-Aqua MODIS-Terra VIIRSN VIIRSJ1 SeaWiFS OCTS CZCS:
- `L1A`
- `L2`
- `L3b`
- `L3m`
- OLCI:
- `L1-EFR` full resolution or `L1_ERR` reduced resolution
- `L2-WFR` full resolution or `L2_WRR` reduced resolution [Water]
- `L2-LFR` full resolution or `L2_LRR` reduced resolution [Land]
- SLSTR:
- `L1-RBT`
- `L2_WST` [Water temperature]
- `L2_LST` [Land temprature]
- `L2_AOD` [Aerosol optical depth]
- `L2_FRP` [Fire radiative power]
- SRAL:
- `L1-SRA`
- `L1-SRA-A`
- `L1-SRA-BS`
- `L2-WAT` [Water altimetry]
- `L2-LAN` [Land altimetry]
- `L2-LAN-HY`
- `L2-LAN-LY`
- `L2-LAN-SI`
- SYN:
- `L2-SYN` [Reflectance and aerosol parameter over land]
- `L2-V10` [Vegetation-Like product (~VGT-S10), 10 day synthesis surface reflectance and NDVI]
- `L2-VG1` [Vegetation-Like product (~VGT-S1), 1 day synthesis surface reflectance and NDVI]
- `L2-VGP` [Vegetation-Like product (~VGT-P), TOA reflectance]
- `L2-AOD` [Aerosol optical depth over land and water 4.5x4.5 km resolution]
- MERIS:
- `L1`
- `L2`
- `L3b`
- `L3m`
- SAR:
- `L0-RAW`
- `L1-GRD`
- `L1-GRD-COG`
- `L1-SLC`
- `L2-OCN`
- `L2-CARD-BS`
- `L2-CARD-COH12`
- GOCI:
- `L1B`
- `L2`
- MSI:
- `L1C`
- `L2A`
- `-u USERNAME`, `--username=USERNAME`: specify username to login to Copernicus (for OLCI, SLSTR, SLAR, SAR, MSI, L5-TM, L7-ETM and L8-OLI-TIRS instruments) or
EarthData (for any other sensor). Password will be prompted when the script is executed. (TO DO: add option to use a credential file *credentials.ini*)
- Earthdata login for NASA satellites
- Copernicus dataspace login for ESA satellites
- `-w`, `--write-image-name`: getOC first query an api to retrieve the list of images to download. The output of that query can be written to a csv file. getOC can then be restarted directly from that file saving that query time.
- `-r`, `--read-image-list`: getOC loads the list previously queried and printed, to avoid querying twice the same list
- `-q`, `--quiet`: Quiet please ! getOC does not output any information relative to the download and querying of the points of interest.

- `-p` product
Specify the product type to download:
- `OC` [default]
- `IOP`
- `SST`

Options specific to a level:

- Level 1 & 2:
- `--box=BOUNDING_BOX_SIZE`, `--bounding-box-size=BOUNDING_BOX_SIZE`: Define the size of the bounding box around the
point of interest in nautical miles. getOC downloads all images that intersect with this box (must be > 0).
- Level 2 & 3:
- `-p PRODUCT`, `--product=PRODUCT`: Specify the product type to download (only for NASA satellites) (c.f. table above for level compatible
with the instrument selected):
- Level 2 (NASA):
- `OC` (default)
- `IOP` (deprecated)
- `SST`
- Level 3 (NASA):
- `CHL`
- `POC`
- not tested: GSM, IOP, KD, LAND, PAR, PIC, QAA, RRS, and ZLEE
- Level 3: at this level the world's ocean is downloaded, getOC ignores the latitude and longitudes in the input csv
file.
- `-b BINNING_PERIOD`, `--binning-period=BINNING_PERIOD`: specify binning period (only for L3), available options
are:
- DAY (default)
- 8D
- MO
- YR
- `--res=SRESOL`, `--spatial-resolution=SRESOL`: specify spatial resolution (only for L3), available options are:
- 4km (default)
- 9km

Options specific to a downloading platform:

- NASA Ocean Color Level 1&2 Browser:
- `-d QUERY_DELAY`, `--delay=QUERY_DELAY`:Delay between queries only needed to query L1L2_browser. (default=1
second)

Instruments specificity:

- VIIRS: GEO files required to process L1A files from that sensor are downloaded automatically when the level L1A is
selected for this instrument.
- Level synthax for ESA satellites are specific to sensors providing different options: Use L1-EFR for OLCI full resolution data and L1-ERR for low resolution (cf table and level paragraph)

### Examples:

Level 1:

python getOC.py -i MODIS-Aqua -l L1A test.csv -u <earthdata-username> -w --box 60
python getOC.py -i GOCI -l L1B test.csv -u <copernicus-username> -w --box 60
python getOC.py -i VIIRSN -l L1A test.csv -u <earthdata-username> -w --box 60
python getOC.py -i VIIRSJ1 -l L1A test.csv -u <earthdata-username> -w --box 60
python getOC.py -i MSI -l L1C test.csv -u <copernicus-username> -w --box 60
python getOC.py -i OLCI -l L1-EFR test.csv -u <copernicus-username> -w --box 60
python getOC.py -i SLSTR -l L1-WST test.csv -u <copernicus-username> -w --box 60
python getOC.py -i SRAL -l L1-RBT test.csv -u <copernicus-username> -w --box 60
python getOC.py -i L5-TM -l L1T test.csv -u <copernicus-username> -w --box 60

Level 2:

python getOC.py -i MODIS-Aqua -l L2 test.csv -u <earthdata-username> -p OC -w --box 60
python getOC.py -i GOCI -l L2 test.csv -u <copernicus-username> -w --box 60
python getOC.py -i VIIRSN -l L2 test.csv -u <earthdata-username> -p OC -w --box 60
python getOC.py -i VIIRSJ1 -l L2 test.csv -u <earthdata-username> -p OC -w --box 60
python getOC.py -i MSI -l L2A test.csv -u <copernicus-username> -w --box 60
python getOC.py -i OLCI -l L2-WRR test.csv -u <copernicus-username> -w --box 60
python getOC.py -i SLSTR -l L2-WST test.csv -u <copernicus-username> -w --box 60
python getOC.py -i SYN -l L2-AOD test.csv -u <copernicus-username> -w --box 60
python getOC.py -i L8-OLI-TIRS -l L2 test.csv -u <copernicus-username> -w --box 60

Level 3:

python getOC.py -i MODIS-Aqua -l L3b test.csv <earthdata-username> -p CHL -b 8D --res 4km -w
python getOC.py -i MODIS-Aqua -l L3m test.csv <earthdata-username> -p CHL -b YR --res 9km -w

A strong internet connection is recommended.

### List of samples
For level 1 and level 2 downloads only (not needed for level 3), a comma separated value (csv) file must be prepared before using getOC. Each line of the csv file must contain the following information (ex: `test.csv`):
- variable name: sample_id,date&time,latitude,longitude
- variable type/units: string,yyyy/mm/dd HH:MM:SS (UTC),degN,degE

Images will be downloaded if they are in the same day UTC as the one specified by the date of the point of interest. For times close to the beginning or the end of the day it can be worth adding a line with the previous or following day.

### Level 1
For **MODIS-Aqua**:

./getOC -i MODIS-Aqua -l L1A test.csv -u <earthdata-username> -w --box 60

Note GEO file are included in SeaDAS for MODIS-Aqua (no need to download them).

For **VIIRSN**:

./getOC -i VIIRSN -l L1A test.csv -u <earthdata-username> -w --box 60

Note getOC will automatically download GEO files in the same time as L1A files for VIIRSN L1A.

For **VIIRSJ1**:

./getOC -i VIIRSJ1 -l L1A test.csv -u <earthdata-username> -w --box 60

Note getOC will automatically download GEO files in the same time as L1A files for VIIRSJ1 L1A.

For **OLCI**:
Use either level L1 or L1_EFR for full resolution
Use either level L1_ERR for low resolution

./getOC -i OLCI -l L1 test.csv -u <creodias-username> -w --box 60

Note no need for GEO files (images are already geo referenced).

For **SLSTR**:
Use either level L1 or L1_RBT

./getOC -i SLSTR -l L1 test.csv -u <creodias-username> -w --box 60

Note no need for GEO files (images are already geo referenced).

For **MSI**:

./getOC -i MSI -l L1C test.csv -u <creodias-username> -w --box 60

Note no need for GEO files (images are already geo referenced).

For **GOCI**:

./getOC -i GOCI -l L1B test.csv -u <creodias-username> -w --box 60

SeaWiFS is not supported by getOC at level 1.


### Level 2
For **MODIS-Aqua**:
Choose either OC or SST

./getOC -i MODIS-Aqua -l L2 test.csv -u <earthdata-username> -p OC -w --box 60

For **VIIRSN**:
Choose either OC or SST

./getOC -i VIIRSN -l L2 test.csv -u <earthdata-username> -p OC -w --box 60

For **VIIRSJ1**:
Only OC available

./getOC -i VIIRSJ1 -l L2 test.csv -u <earthdata-username> -p OC -w --box 60

For **OLCI**:
Only OC available
Use either level L2 or L2_WFR for full resolution
Use either level L1_ERR for low resolution

./getOC -i OLCI -l L1 test.csv -u <creodias-username> -w --box 60

For **SLSTR**:
Only SST available
Use either level L2 or L2_WST (default; GHRSST recommendations) or L2_WCT for weighted combinations of brightness temperatures

./getOC -i SLSTR -l L2 test.csv -u <creodias-username> -w --box 60

For **MSI**:

./getOC -i MSI -l L2A test.csv -u <creodias-username> -w --box 60

For **GOCI**:

./getOC -i GOCI -l L2 test.csv -u <creodias-username> -w --box 60


### Level 3
At level 3 the worlds ocean is downloaded. getOC ignores the latitude and longitude in the input csv file. Usage:

./getOC -i MODIS-Aqua -l <L3b|L3m> test.csv -u <earthdata-username> -p <product> -b <binning-period> --res 4km -w

Examples:

python getOC -i MODIS-Aqua -l L3b test.csv -u <earthdata-username> -p POC -b DAY --res 4km -w
python getOC -i MODIS-Aqua -l L3m test.csv -u <earthdata-username> -p CHL -b 8D --res 9km -w
python getOC -i MODIS-Aqua -l L3m test.csv -u <earthdata-username> -p CHL -b MO --res 9km -w
Loading

0 comments on commit faed5bd

Please sign in to comment.