Should pvlib have a get_solaranywhere function for fetching weather data?

[kandersolar]

SolarAnywhere is a commercial weather data provider popular in the industry system modeling community. I only have a little experience with it, but my understanding is that they provide dataset downloads through both a point-and-click website and an API. I am aware of a few industry analysts that have rolled their own python wrappers for the SolarAnywhere weather API for fetching weather data in bulk, so I speculate a function in pvlib.iotools would get some use.

However the API does require authentication to use. Maybe CPR could give us credentials suitable for testing purposes? Maybe CPR would be interested in contributing the wrapper function itself? cc @cpr-chas

Note that their API docs already provide example python snippets, so you could argue that we're not bringing much value by duplicating it in pvlib: https://developers.cleanpower.com/irradiance-and-weather-data/irradiance-and-weather-requests/

Note also that I'm talking specifically about fetching weather data, not their recent pvlib-based power offering.

[mikofski]

Also, not sure if you heard, but SolarAnywhere Public just launched. You raise a good question about the proliferation of code. Usually less is more, and the zen of python says there should be one - and only one - way to do things. If CPR already has a client, why should we reproduce it so now there's two? What does ours add? Maybe consistent output?

[kandersolar]

To me, ease of integration w/ pvlib is the compelling point in favor. Because these providers have paid employees that could do this, and I myself have no use for commercial data, I am unlikely to add functions along these lines. But I wouldn't be opposed if others contribute them. Someone would need to coordinate with the provider for authentication and such for the tests.

Just another note: commercial weather data access is not the only potential pvlib feature for proprietary/non-public data or formats. For example PVsyst PAN files (#1002 (comment)). This is a case where some kind of steering committee and better knowledge of our users could be helpful.

[williamhobbs]

I support this idea, and not just for SolarAnywhere. Adding @KWagnerCPR.

[KWagnerCPR]

@AdamRJensen SolarAnywhere recently enabled the use of JSON. This page of our documentation contains a python sample that should return data in JSON rather than XML: [https://developers.cleanpower.com/irradiance-and-weather-data/irradiance-and-weather-requests/].

We are happy to collaborate on this function. API access is now self-serve, so anyone who registers for a SolarAnywhere Public account is able to generate their own API keys to use in accessing the SolarAnywhere Public dataset. This page of our support center provides instructions on that process: [https://www.solaranywhere.com/support/using-solaranywhere/api/]

[AdamRJensen]

@AdamRJensen SolarAnywhere recently enabled the use of JSON. This page of our documentation contains a python sample that should return data in JSON rather than XML: [https://developers.cleanpower.com/irradiance-and-weather-data/irradiance-and-weather-requests/].

@KWagnerCPR good to hear about the json option. I'll draft up a pull request in the next few days!

[williamhobbs]

Would a read_solaranywhere function for reading .csv files in the standard SolarAnywhere format also be appropriate to add?

[mikofski]

I believe the SATGY format is similar to TMY3 format except it only has about 7 columns and the header has some extra info after the last field, so hoping to repurpose or reuse some of the existing read_tmy3() code maybe?

[williamhobbs]

The SolarAnywhere and TMY3 formats are fairly similar.

Note that SolarAnywhere can be downloaded in "TMY3" format or "SolarAnywhere" format.

The "TMY3" appears to deviate from the "official" TMY3 format: I think SolarAnywhere in TMY3 format uses hours 00 through 23, while official TMY3 uses 01:00 through 24:00.

SolarAnywhere files can also start and end on any day of the year, and can span multiple years. I think they always include leap day, if that matters.

Sample files for SolarAnywhere v3.4 are here, NREL/SAM#491 (comment), although as of May 2022 version 3.6 is available (https://www.solaranywhere.com/support/historical-data/release-notes/).

Finally, https://github.com/NREL/SAM/blob/develop/deploy/runtime/macros/Solar%20Resource%20File%20Converter.lk might be a useful reference.

[KWagnerCPR]

@williamhobbs Thanks for providing all this info! One small correction is that our TMY3 files do start at 01:00 and end at 24:00 (00:00 of the next day).

SolarAnywhere users specify the start and end date of the data they'd like to download. When you select the TMY3 output format option, the data will be automatically broken down into individual year files. If you specify a start or end time part of the way through a year, then you'll get a partial year file for that year. When you select the SolarAnywhere output format option, you get one file containing the full time series for the requested period.

We'd recommend using the SolarAnywhere output format if possible. This format allows our users to include the snow depth, precipitation, clear sky irradiance and particulate matter fields. The TMY3 format does not have existing columns for these fields so users can't include them in TMY3 output format downloads. I've attached a file containing the full SolarAnywhere V3.6 time series in SolarAnywhere output format for one of our public sites in case this is helpful.

El Paso, United States SolarAnywhere Time Series 19980101 to 20220102 Lat_31_775 Lon_-106_485 SA format.csv

[williamhobbs]

@KWagnerCPR I was referring to the use of "00:00" vs "24:00" in CPR's "TMY3" format vs NREL's TMY3 format. Depending on how times are handled, I think that difference can matter.

Example of CPR's TMY3-formatted file:

Example of TMY3 file included with pvlib:

[KWagnerCPR]

@williamhobbs I see! Thank you for clarifying.

[AdamRJensen]

@williamhobbs please see #1494 which proposes a change to the pvlib read_tmy3 function to handle this difference in the TMY3 format.