diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst index 55f20da51..011fe7d27 100644 --- a/RELEASE_NOTES.rst +++ b/RELEASE_NOTES.rst @@ -1,10 +1,42 @@ -Next release -============ +v3.0.0 (2020-06-05) +=================== + +ixmp v3.0.0 coincides with message_ix v3.0.0. Migration notes --------------- -**Updated dependencies.** The minimum version of JPype is 0.7.5. +Excel input/output (I/O) + The file format used by :meth:`.Scenario.to_excel` and :meth:`.read_excel` is now fully specified; see :doc:`file-io`. + + ixmp writes and reads items with more elements than the ~10⁶ row maximum of the Excel data format, by splitting these across multiple sheets. + + The I/O code now explicitly checks for situations where the index *sets* and *names* for an item are ambiguous; see :ref:`this example ` for how to initialize and read these items. + +Updated dependencies + The minimum versions of the following dependencies are increased: + + - JPype1 0.7.5 + - pandas 1.0 + - dask 2.14 (for reporting) + +Deprecations and deprecation policy + The following items, marked as deprecated in ixmp 2.0, are removed (:pull:`254`): + + - :file:`$HOME/.local/ixmp/` as a configuration location. + Configuration files are now placed in the standard :file:`$HOME/.local/share/ixmp/`. + - positional and ``dbtype=`` arguments to :class:`.Platform`/:class:`.JDBCBackend`. + - ``first_model_year=``, ``keep_sol=``, and ``scen=`` arguments to :meth:`~.Scenario.clone`. + Use `shift_first_model_year`, `keep_solution`, and `scenario`, respectively. + - ``rixmp.legacy``, an earlier version of :ref:`the R interface ` that did not use reticulate. + + Newly deprecated is: + + - `cache` keyword argument to :class:`.Scenario`. + Caching is controlled at the :class:`.Platform`/Backend level, using the same keyword argument. + + Starting with ixmp v3.0, arguments and other features marked as deprecated will follow a standard deprecation policy: they will be removed no sooner than the second major release following the release in which they are marked deprecated. + For instance, a feature marked deprecated in ixmp version "10.5" would be retained in ixmp versions "11.x", and removed only in version "12.0" or later. All changes @@ -27,16 +59,11 @@ All changes - :pull:`270`: Include all tests in the ixmp package. - :pull:`212`: Add :meth:`Model.initialize` API to help populate new Scenarios according to a model scheme. - :pull:`267`: Apply units to reported quantities. -- :pull:`254`: Remove deprecated items: - - - ~/.local/ixmp as a configuration location. - - positional and ``dbtype=`` arguments to :class:`.Platform`/:class:`.JDBCBackend`. - - ``first_model_year=``, ``keep_sol=``, and ``scen=`` arguments to :meth:`~.Scenario.clone`. - - ``rixmp.legacy``, an earlier version of :ref:`the R interface ` that did not use reticulate. - :pull:`261`: Increase minimum pandas version to 1.0; adjust for `API changes and deprecations `_. - :pull:`243`: Add :meth:`.export_timeseries_data` to write data for multiple scenarios to CSV. - :pull:`264`: Implement methods to get and create new subannual timeslices. + v2.0.0 (2020-01-14) =================== diff --git a/doc/source/api.rst b/doc/source/api.rst index 34fea6623..c3bae648b 100644 --- a/doc/source/api.rst +++ b/doc/source/api.rst @@ -128,6 +128,11 @@ Scenario GAMS code, so cannot be modified by a Scenario. See :meth:`init_equ`, :meth:`equ_list`, and :meth:`equ`. + .. deprecated:: 3.0 + The `cache` keyword argument to :class:`Scenario` has no effect and raises a warning. + Use `cache` as one of the `backend_args` to :class:`Platform` to disable/enable caching for storage backends that support it. + Use :meth:`load_scenario_data` to load all data in the Scenario into an in-memory cache. + .. autosummary:: add_par add_set diff --git a/doc/source/file-io.rst b/doc/source/file-io.rst index 98105e35c..540118574 100644 --- a/doc/source/file-io.rst +++ b/doc/source/file-io.rst @@ -50,6 +50,8 @@ Reading variables and equations Thus, while :meth:`.to_excel` will write files containing variable and equation data, :meth:`.read_excel` can not add these to a Scenario, and only emits log messages indicating that they are ignored. +.. _excel-ambiguous-dims: + Multiple dimensions indexed by the same set :meth:`.read_excel` provides the `init_items` argument to create new sets and parameters when reading a file. However, the file format does not capture information needed to reconstruct the original data in all cases. diff --git a/doc/source/install.rst b/doc/source/install.rst index 166171ce9..4946e4d01 100644 --- a/doc/source/install.rst +++ b/doc/source/install.rst @@ -1,22 +1,21 @@ Installation -============ +************ -Most users will have the |ixmp| installed automatically when `installing MESSAGEix`_. +A desktop or laptop computer is sufficient for most purposes using :mod:`ixmp`. +Most users will have :mod:`ixmp` installed automatically when `installing MESSAGEix`_. The sections below cover other use cases: -- Installing *ixmp* to be used alone (i.e., with models or frameworks other than - |MESSAGEix|): +- Installing *ixmp* to be used alone (i.e., with models or frameworks other than |MESSAGEix|): - - see the section `Install GAMS`_, + - see the sections `Install system dependencies`_, - then `Install ixmp via Anaconda`_. -- Installing *ixmp* from source, for development purposes: see - `Install ixmp from source`_. +- Installing *ixmp* from source, for development purposes: see `Install ixmp from source`_. - Installing the R API to *ixmp*: - - Start with `Install GAMS`_. + - Start with `Install system dependencies`_. - Then install *ixmp* either via Anaconda, or from source. - Finally, see `Install rixmp`_. @@ -25,171 +24,172 @@ The sections below cover other use cases: .. contents:: :local: +Install system dependencies +=========================== -Technical requirements ----------------------- +GAMS +---- -A high-quality desktop computer or laptop is sufficient for most purposes -using the |ixmp|. +:mod:`ixmp` requires `GAMS`_. +1. Download GAMS for your operating system; either the `latest version`_ or `version 29`_ (see note below). -Install GAMS ------------- +2. Run the installer. -*ixmp* requires `GAMS`_. - -1. Download the latest version of `GAMS`_ for your operating system; run the - installer. - -2. Add GAMS to the ``PATH`` environment variable: +3. Ensure that the ``PATH`` environment variable on your system includes the path to the GAMS program: - on Windows, in the GAMS installer… + - Check the box labeled “Use advanced installation mode.” - - Check the box labeled “Add GAMS directory to PATH environment variable” - on the Advanced Options page. - - on macOS or Linux, add the following line to your ``.bash_profile`` (Mac) or ``.bashrc`` (Linux):: + - Check the box labeled “Add GAMS directory to PATH environment variable” on the Advanced Options page. + + - on other platforms (macOS or Linux), add the following line to a file such as :file:`~/.bash_profile` (macOS), :file:`~/.bashrc`, or :file:`~/.profile`:: + + export PATH=$PATH:/path/to/gams-directory-with-gams-binary - export PATH=$PATH:/path/to/gams-directory-with-gams-binary +.. note:: + :mod:`message_ix` requires GAMS version 24.8; :mod:`ixmp` has no minimum requirement *per se*. + The latest version is recommended. + + GAMS is proprietary software and requires a license to solve optimization problems. + To run both the :mod:`ixmp` and :mod:`message_ix` tutorials and test suites, a “free demonstration” license is required; the free license is suitable for these small models. + Versions of GAMS up to `version 29`_ include such a license with the installer; since version 30, the free demo license is no longer included, but may be requested via the GAMS website. .. note:: - For using `GAMS`_ to solve numerical optimisation problems, you need to - install the latest version of GAMS (in particular 24.8 or higher). If you - only have a license for an older version, install both the older and the - latest version of GAMS. + If you only have a license for an older version of GAMS, install both the older and the latest versions. + + +Graphviz +-------- + +:meth:`ixmp.reporting.Reporter.visualize` uses `Graphviz`_, a program for graph visualization. +Installing ixmp causes the python :mod:`graphviz` package to be installed. +If you want to use :meth:`.visualize` or run the test suite, the Graphviz program itself must also be installed; otherwise it is **optional**. +If you `Install ixmp via Anaconda`_, Graphviz is installed automatically via `its conda-forge package`_. +For other methods of installation, see the `Graphviz download page`_ for downloads and instructions for your system. -Install *ixmp* via Anaconda ---------------------------- -After installing GAMS, we recommend that new users install Anaconda, and then -use it to install *ixmp*. Advanced users may choose to install *ixmp* from -source code (next section). +Install ``ixmp`` via Anaconda +============================= -3. Install Python via `Anaconda`_. We recommend the latest version, i.e., - Python 3.6+. +After installing GAMS, we recommend that new users install Anaconda, and then use it to install :mod:`ixmp`. +Advanced users may choose to install :mod:`ixmp` from source code (next section). -4. Open a command prompt. We recommend Windows users use the “Anaconda Prompt” - to avoid permissions issues when installing and using *ixmp*. This program - is available in the Windows Start menu after installing Anaconda. +4. Install Python via `Anaconda`_. + We recommend the latest version; currently Python 3.8. -5. Install the ``ixmp`` package:: +5. Open a command prompt. + We recommend Windows users use the “Anaconda Prompt” to avoid permissions issues when installing and using :mod:`ixmp`. + This program is available in the Windows Start menu after installing Anaconda. + +6. Install the ``ixmp`` package:: $ conda install -c conda-forge ixmp -Install *ixmp* from source --------------------------- +Install ``ixmp`` from source +============================ -3. (Optional) If you intend to contribute changes to *ixmp*, first register - a Github account, and fork the `ixmp repository `_. This will create a new repository ``/ixmp``. +4. (Optional) If you intend to contribute changes to *ixmp*, first register a Github account, and fork the `ixmp repository `_. + This will create a new repository ``/ixmp``. -4. Clone either the main repository, or your fork; using the `Github Desktop`_ - client, or the command line:: +5. Clone either the main repository, or your fork; using the `Github Desktop`_ client, or the command line:: $ git clone git@github.com:iiasa/ixmp.git # or: $ git clone git@github.com:USER/ixmp.git -5. Open a command prompt in the ``ixmp`` directory and type:: +6. Open a command prompt in the :file:`ixmp/` directory that is created, and type:: - $ pip install --editable . + $ pip install --editable .[docs,tests,tutorial] The ``--editable`` flag ensures that changes to the source code are picked up every time ``import ixmp`` is used in Python code. + The ``[docs,tests,tutorial]`` extra dependencies ensure additional dependencies are installed. -6. (Optional) Run the built-in test suite to check that *ixmp* functions - correctly on your system:: +7. (Optional) Run the built-in test suite to check that :mod:`ixmp` functions correctly on your system:: - $ pip install --editable .[tests] - $ py.test + $ pytest -Install ``rixmp`` ------------------ +Install development tools +------------------------- -See also the :ref:`rixmp documentation `. +Developers making changes to the :mod:`ixmp` source **may** need one or more of the following tools. +Users developing models using existing functionality **should not** need these tools. -1. `Install R `_. +Git + Use one of: - .. warning:: - Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java. - Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors. + - https://git-scm.com/downloads + - https://desktop.github.com + - https://www.gitkraken.com -2. Enter the directory ``rixmp/`` and use R to build and install the package and its dependencies, including reticulate_:: + In addition, set up an account at https://github.com, and familiarize yourself with forking and cloning repositories, as well as pulling, committing and pushing changes. - $ cd rixmp - $ Rscript -e "install.packages(c('knitr', 'reticulate'), repos='http://cran.rstudio.com/')" - $ R CMD build . +Java Development Kit (JDK) + - Install the Java Development Kit (JDK) for Java SE version 8 from https://www.oracle.com/technetwork/java/javase/downloads/index.html -3. Check that there is only one ``*tar.gz`` in the folder:: + .. note:: At this point, ixmp is not compatible with JAVA SE 9. - $ R CMD INSTALL rixmp_* + - Follow the `JDK website instructions `_ to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not exist, add it as a new system variable. -4. (Optional) Run the built-in test suite to check that *ixmp* and *rixmp* functions, as in *Install ixmp from source 6.* (installing - the R ``devtools`` package might be a pre-requisite). In the ``ixmp`` directory type:: + - Update your ``PATH`` environment variable to point to the JRE binaries and server installation (e.g., :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\`, :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\server`). - $ py.test --test-r + .. warning:: Do not overwrite the existing ``PATH`` environment variable, but add to the list of existing paths. -5. (Optional) For working with Jupyter notebooks using R, install the `IR kernel `_. -6. (Optional) Install `Rtools `_ and add the path to the environment variables. -.. _reticulate: https://rstudio.github.io/reticulate/ +Install ``rixmp`` +================= +See also the :ref:`rixmp documentation `. -Install development tools -------------------------- +1. `Install R `_. -Developers making changes to the *ixmp* source may need one or more of the following tools. -Users developing models using existing *ixmp* functionality **should not** need these tools. + .. warning:: + Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java. + Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors. -- **Java Development Kit (JDK).** +2. Enter the directory ``rixmp/`` and use R to build and install the package and its dependencies, including reticulate_:: - - Install the Java Development Kit (JDK) for Java SE version 8 from - https://www.oracle.com/technetwork/java/javase/downloads/index.html + $ cd rixmp + $ Rscript -e "install.packages(c('knitr', 'reticulate'), repos='http://cran.rstudio.com/')" + $ R CMD build . - .. note:: At this point, ixmp is not compatible with JAVA SE 9. +3. Check that there is only one ``*tar.gz`` in the folder, then run:: - - Follow the `JDK website instructions - `_ - to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not - exist, add as new system variable. + $ R CMD INSTALL rixmp_* - - Update your `PATH` environment variable to point to the JRE binaries and - server installation (e.g., ``C:\Program Files\Java\jdk[YOUR JDK - VERSION]\jre\bin\``, ``C:\Program Files\Java\jdk[YOUR JDK - VERSION]\jre\bin\server``). +4. (Optional) Run the built-in test suite to check that *ixmp* and *rixmp* functions, as in *Install ixmp from source 6.* (installing + the R ``devtools`` package might be a pre-requisite). In the ``ixmp`` directory type:: - .. warning:: Do not overwrite the existing `PATH` environment variable, but - add to the list of existing paths. + $ pytest -m rixmp -- **Git.** Use one of: +5. (Optional) To use rixmp in Jupyter notebooks, install the `IR kernel `_. - - https://git-scm.com/downloads - - https://desktop.github.com - - https://www.gitkraken.com +6. (Optional) Install `Rtools `_ and add the path to the environment variables. - In addition, set up an account at https://github.com, and familiarize - yourself with forking and cloning repositories, as well as pulling, - committing and pushing changes. +.. _reticulate: https://rstudio.github.io/reticulate/ Troubleshooting ---------------- +=============== + +Run ``ixmp show-versions`` on the command line to check that you have all dependencies installed, or when reporting issues. -For Anaconda users experiencing problems during installation of ixmp, -Anaconda might not have been added to the PATH system variable properly. -So, if ``install.bat`` fails, check if:: +For Anaconda users experiencing problems during installation of ixmp, check that the following paths are part of the ``PATH`` environment variable, and add them if missing:: C:\[YOUR ANACONDA LOCATION]\Anaconda3; C:\[YOUR ANACONDA LOCATION]\Anaconda3\Scripts; C:\[YOUR ANACONDA LOCATION]\Anaconda3\Library\bin; -are all part of the PATH system variable. If they are not there, add them. - -Run `ixmp show-versions` on the command line to check that you have all dependencies installed, or when reporting issues. - .. _`installing MESSAGEix`: https://message.iiasa.ac.at/en/latest/getting_started.html .. _`Anaconda`: https://www.continuum.io/downloads .. _`GAMS`: http://www.gams.com +.. _`latest version`: https://www.gams.com/download/ +.. _`version 29`: https://www.gams.com/29/ +.. _`its conda-forge package`: https://anaconda.org/conda-forge/graphviz +.. _`Graphviz download page`: https://www.graphviz.org/download/ .. _`Github Desktop`: https://desktop.github.com diff --git a/ixmp/model/base.py b/ixmp/model/base.py index 74e3c52a8..455d4c658 100644 --- a/ixmp/model/base.py +++ b/ixmp/model/base.py @@ -109,7 +109,6 @@ def initialize_items(cls, scenario, items): for key, values in item_info.items(): values = values or [] existing = getattr(scenario, key)(name) - print(name, key, values, existing) if existing != values: # The existing index sets or names do not match log.warning( diff --git a/ixmp/tests/reporting/test_exceptions.py b/ixmp/tests/reporting/test_exceptions.py index d74020a39..845b24c2c 100644 --- a/ixmp/tests/reporting/test_exceptions.py +++ b/ixmp/tests/reporting/test_exceptions.py @@ -31,6 +31,8 @@ def test_computationerror(caplog): def test_computationerror_ipython(test_data_path, tmp_path, tmp_env): + # NB this requires nbformat >= 5.0, because the output kind "evalue" was + # different pre-5.0 fname = test_data_path / 'reporting-exceptions.ipynb' nb, _ = run_notebook(fname, tmp_path, tmp_env, allow_errors=True) diff --git a/rixmp/DESCRIPTION b/rixmp/DESCRIPTION index c4da02e60..f329bf4bd 100644 --- a/rixmp/DESCRIPTION +++ b/rixmp/DESCRIPTION @@ -1,8 +1,8 @@ Package: rixmp Type: Package Title: R package for ix modeling platform (ixmp) -Version: 2.0.0.9000 -Date: 2020-01-13 +Version: 3.0.0 +Date: 2020-05-02 Authors@R: c( person("Adriano", "Vinca", email = "vinca@iiasa.ac.at", role = c("aut", "cre")), person("Matthew", "Gidden", role = "aut"), diff --git a/setup.cfg b/setup.cfg index 15abeb7f7..afb54d76c 100644 --- a/setup.cfg +++ b/setup.cfg @@ -15,7 +15,7 @@ include_package_data = True install_requires = JPype1 >= 0.7.5 click - dask [array] + dask [array] >= 2.14 graphviz pandas >= 1.0 pint @@ -32,6 +32,7 @@ tests = codecov jupyter memory_profiler + nbformat >= 5.0 pretenders >= 1.4.4 pytest >= 5 pytest-cov