Merge branch 'develop' into 8739-publisher-during-harvesting #8739

README.md

@@ -1,20 +1,81 @@
 Dataverse®  
 ===============
 
-Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Dataverse team](https://dataverse.org/about) at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][]).
+![Dataverse-logo](https://github.com/IQSS/dataverse-frontend/assets/7512607/6c4d79e4-7be5-4102-88bd-dfa167dc79d3)
 
-[dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our [project board][], our development [roadmap][], and more.
+## Table of Contents
 
-We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
+1. [❓ What is Dataverse?](#what-is-dataverse)
+2. [✔ Try Dataverse](#try-dataverse)
+3. [🌐 Features, Integrations, Roadmaps, and More](#website)
+4. [📥 Installation](#installation)
+5. [🏘 Community and Support](#community-and-support)
+6. [🧑‍💻️ Contributing](#contributing)
+7. [⚖️ Legal Information](#legal-informations)
 
-To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. Docker users should consult the [Container Guide][].
+<a name="what-is-dataverse"></a>
 
-To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
+## ❓  What is Dataverse?
 
-We love contributors! Please see our [Contributing Guide][] for ways you can help.
+Welcome to Dataverse®, the [open source][] software platform designed for sharing, finding, citing, and preserving research data. Developed by the Dataverse team at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][], our platform makes it easy for research organizations to host, manage, and share their data with the world.
+
+<a name="try-dataverse"></a>
+
+## ✔  Try Dataverse
+
+We invite you to explore our demo site at [demo.dataverse.org][]. This site is ideal for testing and evaluating Dataverse in a risk-free environment.
+
+<a name="website"></a>
+
+## 🌐 Features, Integrations, Roadmaps, and More
+
+Visit [dataverse.org][], our home on the web, for a comprehensive overview of Dataverse. Here, you will find:
+
+- An interactive map showcasing Dataverse installations worldwide.
+- A detailed list of [features][].
+- Information on [integrations][] that have been made possible through our [REST APIs][].
+- Our [project board][] and development [roadmap][].
+- News, events, and more.
+
+<a name="installation"></a>
+
+## 📥 Installation
+
+Ready to get started? Follow our [Installation Guide][] to download and install the latest release of Dataverse.
+
+If you are using Docker, please refer to our [Container Guide][] for detailed instructions.
+
+<a name="community-and-support"></a>
+
+## 🏘 Community and Support
+
+Engage with the vibrant Dataverse community through various channels:
+
+- **[Mailing List][]**: Join the conversation on our [mailing list][].
+- **[Community Calls][]**: Participate in our regular [community calls][] to discuss new features, ask questions, and share your experiences.
+- **[Chat][]**: Connect with us and other users in real-time at [dataverse.zulipchat.com][].
+- **[Dataverse Community Meeting][]**: Attend our annual [Dataverse Community Meeting][] to network, learn, and collaborate with peers and experts.
+- **[DataverseTV][]**: Watch the video content from the Dataverse community on [DataverseTV][] and on [Harvard's IQSS YouTube channel][].
+
+<a name="contributing"></a>
+## 🧑‍💻️ Contribute to Dataverse
+
+We love contributors! Whether you are a developer, researcher, or enthusiast, there are many ways you can help.
+
+Visit our [Contributing Guide][] to learn how you can get involved.
+
+Join us in building and enhancing Dataverse to make research data more accessible and impactful. Your support and participation are crucial to our success!
+
+<a name="legal-informations"></a>
+## ⚖️ Legal Information
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
+---
+For more detailed information, visit our website at [dataverse.org][].
+
+Feel free to [reach out] with any questions or feedback. Happy researching!
+
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg "Dataverse Project")](http://dataverse.org)
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
@@ -37,6 +98,11 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Contributing Guide]: CONTRIBUTING.md
 [mailing list]: https://groups.google.com/group/dataverse-community
 [community call]: https://dataverse.org/community-calls
-[chat.dataverse.org]: https://chat.dataverse.org
+[Chat]: https://dataverse.zulipchat.com
+[dataverse.zulipchat.com]: https://dataverse.zulipchat.com
 [Dataverse Community Meeting]: https://dataverse.org/events
 [open source]: LICENSE.md
+[community calls]: https://dataverse.org/community-calls
+[DataverseTV]: https://dataverse.org/dataversetv
+[Harvard's IQSS YouTube channel]: https://www.youtube.com/@iqssatharvarduniversity8672
+[reach out]: https://dataverse.org/contact

doc/release-notes/10171-exlude-metadatablocks.md

@@ -0,0 +1,4 @@
+Extension of API `{id}/versions` and `{id}/versions/{versionId}` with an optional ``excludeMetadataBlocks`` parameter,
+that specifies whether the metadataBlocks should be listed in the output. It defaults to ``false``, preserving backward
+compatibility. (Note that for a dataset with a large number of versions and/or metadataBlocks having the metadata blocks
+included can dramatically increase the volume of the output). See also [the guides](https://dataverse-guide--10778.org.readthedocs.build/en/10778/api/native-api.html#list-versions-of-a-dataset), #10778, and #10171.

doc/release-notes/10241-new-solr-client.md

@@ -0,0 +1,9 @@
+[HttpSolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html) is deprecated as of Solr 9, and which will be removed in a future major release of Solr. It's recommended to use [Http2SolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html) instead.
+
+[Solr documentation](https://solr.apache.org/guide/solr/latest/deployment-guide/solrj.html#types-of-solrclients) describe it as a _async, non-blocking and general-purpose client that leverage HTTP/2 using the Jetty Http library_.
+
+With Solr 9.4.1, the Http2SolrClient is indicate as experimental. But since the 9.6 version of Solr, this mention is no longer maintained.
+
+The ConcurrentUpdateHttp2SolrClient is now also used in some cases, which is supposed to be more efficient for indexing.
+
+For more information, see issue [#10161](https://github.com/IQSS/dataverse/issues/10161) and pull request [#10241](https://github.com/IQSS/dataverse/pull/10241)

doc/release-notes/10340-forbidden.md

@@ -0,0 +1,3 @@
+### Backward Incompatible Changes
+
+The [Show Role](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/changelog.html), #10340, and #11116.

doc/release-notes/10472-review-modify-jsonprinter-for-datasetfieldtype.md

@@ -0,0 +1,8 @@
+### Json Printer Bug fix
+
+DatasetFieldTypes in MetadataBlock response that are also a child of another DatasetFieldType were being returned twice. The child DatasetFieldType was included in the "fields" object as well as in the "childFields" of it's parent DatasetFieldType. This fix suppresses the standalone object so only one instance of the DatasetFieldType is returned (in the "childFields" of its parent).
+This fix changes the Json output of the API `/api/dataverses/{dataverseAlias}/metadatablocks`
+
+## Backward Incompatible Changes
+
+The Json response of API call `/api/dataverses/{dataverseAlias}/metadatablocks` will no longer include the DatasetFieldTypes in "fields" if they are children of another DatasetFieldType. The child DatasetFieldType will only be included in the "childFields" of it's parent DatasetFieldType.

doc/release-notes/10516_legacy_permalink_config_fix.md

@@ -0,0 +1 @@
+Support for legacy configuration of a PermaLink PID provider, e.g. using the :Protocol,:Authority, and :Shoulder settings, is fixed. 

doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md

@@ -0,0 +1,6 @@
+### Flyway Script added to Fix File Access Requests when upgrading from Dataverse 6.0
+
+Database update script added to prevent duplicate keys when upgrading from V6.0
+This script will delete access requests made after the initial request and will set the initial request to "Created"
+
+See: https://github.com/IQSS/dataverse/issues/10714

doc/release-notes/11038-unconsidered-harvesting-granularity.md

@@ -0,0 +1,2 @@
+Bug Fix:
+OAI Client harvesting now uses the correct granularity while re-run a partial harvest (using the `from` parameter). The correct granularity comes from the `Identify` verb request.

doc/release-notes/11083-mydata-npe-with-harvested-dataverses.md

@@ -0,0 +1 @@
+Fix a bug with My Data where listing dataverses for a user with only rights on harvested dataverses would result in a server error response.

doc/release-notes/11095-fix-extcvoc-indexing.md

@@ -0,0 +1,7 @@
+Some External Controlled Vocabulary scripts/configurations, when used on a metadata field that is single-valued could result 
+in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing).
+Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to /api/admin/index/solr/schema.
+Configuring the Solr schema and the update-fields.sh script as usually recommended when using custom metadata blocks will resolve the issue.
+
+The overall release notes should include a Solr update (which hopefully is required by an update to 9.7.0 anyway) and our standard instructions 
+should change to recommending use of the update-fields.sh script when using custom metadatablocks *and/or external vocabulary scripts*.

doc/release-notes/11107-fake-to-perma-demo.md

@@ -0,0 +1,3 @@
+### Demo/Eval Container Tutorial
+
+The demo/eval container tutorial has been updated to use the Permalink PID provider instead of the FAKE DOI Provider. See also #11107.

doc/release-notes/11113-avoid-orphan-perm-docs.md

@@ -0,0 +1,5 @@
+This release fixes a bug that caused Dataverse to generate unnecessary solr documents for files when a file is added/deleted from a draft dataset. These documents could accumulate and potentially impact performance.
+
+Assuming the upgrade to solr 9.7.0 also occurs in this release, there's nothing else needed for this PR. (Starting with a new solr insures the solr db is empty and that a reindex is already required.)
+
+

doc/release-notes/11130-update-dataverse-api-remove-metadatablocks.md

@@ -0,0 +1,8 @@
+### Fixes consequences for not adding some optional fields in update dataverse API
+
+Omitting optional fields inputLevels, facetIds, or metadataBlockNames caused data to be deleted.
+This fix no longer deletes data for these fields. Two new flags have been added to the ``metadataBlocks`` Json object to signal the deletion of the data.
+- ``inheritMetadataBlocksFromParent: true`` will remove ``metadataBlockNames`` and ``inputLevels`` if the Json objects are omitted.
+- ``inheritFacetsFromParent: true`` will remove ``facetIds`` if the Json object is omitted.
+
+For more information, see issue [#11130](https://github.com/IQSS/dataverse/issues/11130)

doc/release-notes/11133-search-fix.md

@@ -0,0 +1,3 @@
+### Search fix when using AVOID_EXPENSIVE_SOLR_JOIN=true
+
+Dataverse v6.5 introduced a bug which causes search to fail for non-superusers in multiple groups when the AVOID_EXPENSIVE_SOLR_JOIN feature flag is set to true. This releases fixes the bug.

doc/release-notes/8808-10575-update-global-role.md

@@ -0,0 +1,11 @@
+## Release Highlights
+
+### Update a Global Role
+
+A new API endpoint has been added that allows a global role to be updated. See [Native API Guide > Update Global Role](https://guides.dataverse.org/en/6.3/api/native-api.html#update-global-role) (#10612)
+
+## Bug fixes
+
+### Edition of custom role fixed
+
+It is now possible to edit a custom role with the same alias (reported in #8808)

doc/release-notes/9294-improvement-and-internationalization-of-harvest-status.md

@@ -0,0 +1,6 @@
+## Improvement and internationalization of harvest status
+
+Added a harvest status to differentiate a complete harvest with errors (Completed with failures) and without errors (Completed)
+Harvest status labels are now internationalized 
+
+For more information, see issue [#9294](https://github.com/IQSS/dataverse/issues/9294)

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -564,8 +564,7 @@ Using External Vocabulary Services
 
 The Dataverse software has a mechanism to associate specific fields defined in metadata blocks with a vocabulary(ies) managed by external services. The mechanism relies on trusted third-party Javascripts. The mapping from field type to external vocabulary(ies) is managed via the :ref:`:CVocConf <:CVocConf>` setting.
 
-*This functionality is considered 'experimental'. It may require significant effort to configure and is likely to evolve in subsequent Dataverse software releases.*
-
+*This functionality may require significant effort to configure and is likely to evolve in subsequent Dataverse software releases.*
 
 The effect of configuring this mechanism is similar to that of defining a field in a metadata block with 'allowControlledVocabulary=true':
 
@@ -590,6 +589,9 @@ Configuration involves specifying which fields are to be mapped, to which Solr f
 These are all defined in the :ref:`:CVocConf <:CVocConf>` setting as a JSON array. Details about the required elements as well as example JSON arrays are available at https://github.com/gdcc/dataverse-external-vocab-support, along with an example metadata block that can be used for testing.
 The scripts required can be hosted locally or retrieved dynamically from https://gdcc.github.io/ (similar to how dataverse-previewers work).
 
+Since external vocabulary scripts can change how fields are indexed (storing an identifier and name and/or values in different languages),
+updating the Solr schema as described in :ref:`update-solr-schema` should be done after adding new scripts to your configuration.
+
 Please note that in addition to the :ref:`:CVocConf` described above, an alternative is the :ref:`:ControlledVocabularyCustomJavaScript` setting.
 
 Protecting MetadataBlocks

doc/sphinx-guides/source/api/changelog.rst

@@ -11,6 +11,7 @@ v6.6
 ----
 
 - **/api/metadatablocks** is no longer returning duplicated metadata properties and does not omit metadata properties when called.
+- **/api/roles**: :ref:`show-role` now properly returns 403 Forbidden instead of 401 Unauthorized when you pass a working API token that doesn't have the right permission.
 
 v6.5
 ----