Bug?: unexpected sorting of results when sorting by "newest first" (Search + My Data)

[vera]

I noticed the following unexpected behaviour related to sorting by "newest first" (either in the search results or on the "My Data" page).

When a dataset is created and published, and then a new draft version of that dataset is created, I expected the new draft to be sorted first (because it is newer than the published version). However, this is not the case:

(The draft of dataset 10.5072/FK2/NCG6QT is sorted below the published version of the same dataset, even though we are sorting by "newest".)

Looking at the index, it seems the draft version receives a copy of the dateSort value from the published version, instead of its own timestamp:

{
      "id":"dataset_29644",
      "dateSort":"2025-01-22T15:59:40.150Z",
      ...
},
{
      "id":"dataset_29644_draft",
      "dateSort":"2025-01-22T15:59:40.150Z",
      ...
}

Is this intended?

This bug was discovered by one of our curators who used "My data" to get a list of datasets to be reviewed. A dataset which was just submitted for review today was not to be found on page 1, as expected, but on one of the latest pages.

It seems that the dataset was sorted based on the dateSort timestamp which was copied from the latest published version (which was published September 24). This means the draft was sorted as if it was created/submitted last September instead of today.

What steps does it take to reproduce the issue?

I've added a test for the behaviour that I expected here: https://github.com/vera/dataverse/blob/ef8ab7ee6202a2101d17a634c0d61e7cc87c1d5c/src/test/java/edu/harvard/iq/dataverse/api/DataRetrieverApiIT.java#L108-L260

Run with: mvn test -Dtest="DataRetrieverApiIT#testRetrieveMyDataAsJsonStringSortOrder"

When does this issue occur?

When looking at datasets on the search results or my data page.

Which page(s) does it occurs on?

• Search results page
• My data page

What happens?

New draft versions are not sorted before the published versions.

To whom does it occur (all users, curators, superusers)?

All users.

What did you expect to happen?

Dataset draft versions to be sorted according to their creation time.

Which version of Dataverse are you using?

6.5

Any related open or closed issues to this bug report?

I didn't find any.

Screenshots:

/

Are you thinking about creating a pull request for this issue?

Yes, possibly

[pdurbin]

I agree it sounds like a bug. 🐞 Good catch!

[qqmyers]

The code seems to be picking the data of the last major release (so presumably the same thing happens with v1.1 etc that you're seeing with :draft.) So - it seems intentional, but probably a choice from long ago. Sorting by version dates seems more useful to me - not sure if anyone in the community relies on it working as it does now.

[vera]

Thanks for confirming! I've opened a PR for this issue: #11180