Skip to content

Commit

Permalink
Orderby support in entity odata endpoint (#1078)
Browse files Browse the repository at this point in the history
* Orderby support in entity odata endpoint

* Add way to make orderby sort stable

* orderby for submission odata

* addressing PR comments

* added nulls first/last null ordering

* Updated docs to describe
  • Loading branch information
ktuite authored Feb 2, 2024
1 parent c74eab8 commit 9c5f6b2
Show file tree
Hide file tree
Showing 11 changed files with 565 additions and 19 deletions.
33 changes: 27 additions & 6 deletions docs/api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ info:

Here major and breaking changes to the API are listed by version.

## ODK Central v2024.1

**Added**:

- OData Data Document for requests of Submissions and Entities now allow use of `$orderby`.
- ETag headers on all Blobs.


## ODK Central v2023.5

**Added**:
Expand Down Expand Up @@ -556,7 +564,6 @@ tags:

* The actual data documents, linked from the Service Document, are a simple JSON representation of the submission data or entity, conforming to the schema we describe in our Metadata Document.

As our focus is on the bulk-export of data from ODK Central so that more advanced analysis tools can handle the data themselves, we do not support most of the features at the Intermediate and above conformance levels, like `$sort` or `$filter`.
- name: System Endpoints
description: There are some resources available for getting or setting system information
and configuration. You can set the [Usage Reporting configuration](/central-api-system-endpoints/#usage-reporting-configuration)
Expand Down Expand Up @@ -5417,6 +5424,7 @@ paths:
responses:
200:
description: OK
headers:
ETag:
schema:
type: string
Expand Down Expand Up @@ -11572,7 +11580,7 @@ paths:

The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data. While paging is possible through these parameters, it will not greatly improve the performance of exporting data. ODK Central prefers to bulk-export all of its data at once if possible.

As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by any data field in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error.
As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by certain data fields in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error.

The fields you can query against are as follows:

Expand All @@ -11599,6 +11607,8 @@ paths:

+ Child properties of repeats can't be requested using `$select`

As of ODK Central v2024.1, the [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) is now supported, and can sort on the same fields as `$filter`, noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/submitterId ASC, __system/reviewState DESC`.

As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers.
operationId: Data Document
parameters:
Expand Down Expand Up @@ -11663,6 +11673,12 @@ paths:
schema:
type: string
example: year(__system/submissionDate) lt year(now())
- name: '%24orderby'
in: query
description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together.
schema:
type: string
example: __system/submitterId asc, __system/updatedAt desc
- name: '%24expand'
in: query
description: Repetitions, which should get expanded. Currently, only `*` is
Expand Down Expand Up @@ -12067,15 +12083,12 @@ paths:

The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data.

The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter by any data field in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported.
The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter certain data fields in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported.

The fields you can query against are as follows:

| Entity Metadata | OData Field Name |
| ------------------------| -------------------- |
| Entity UUID | `__id` |
| Entity Name (same as UUID) | `name` |
| Entity Label | `label` |
| Entity Creator Actor ID | `__system/creatorId` |
| Entity Timestamp | `__system/createdAt` |
| Entity Update Timestamp | `__system/updatedAt` |
Expand All @@ -12087,6 +12100,8 @@ paths:

The [`$select` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358942) will return just the fields you specify and is supported on `__id`, `__system`, `__system/creatorId`, `__system/createdAt` and `__system/updatedAt`, as well as on user defined properties.

The [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) will return Entities sorted by different fields, which come from the same list used by `$filter`, as noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/creatorId ASC, __system/conflict DESC`.

As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers.
operationId: Data Document for Dataset
parameters:
Expand Down Expand Up @@ -12135,6 +12150,12 @@ paths:
schema:
type: string
example: year(__system/createdAt) lt year(now())
- name: '%24orderby'
in: query
description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together.
schema:
type: string
example: __system/creatorId asc, __system/updatedAt desc
- name: '%24select'
in: query
description: If provided, will return only the selected fields.
Expand Down
30 changes: 29 additions & 1 deletion lib/data/odata-filter.js
Original file line number Diff line number Diff line change
Expand Up @@ -87,5 +87,33 @@ const odataFilter = (expr, odataToColumnMap) => {
return op(ast);
};

module.exports = { odataFilter };
const odataOrderBy = (expr, odataToColumnMap, stableOrderColumn = null) => {
let initialOrder = null;
const clauses = expr.split(',').map((exp) => {
const [col, order] = exp.trim().split(/\s+/);

// validate field
if (!odataToColumnMap.has(col))
throw Problem.internal.unsupportedODataField({ text: col });

// validate order (asc or desc)
if (order && !order?.toLowerCase().match(/^(asc|desc)$/))
throw Problem.internal.unsupportedODataField({ text: order });

const sqlOrder = (order?.toLowerCase() === 'desc') ? sql`DESC NULLS LAST` : sql`ASC NULLS FIRST`;

// Save the order of the initial property to use for the stable sort column order
if (initialOrder == null)
initialOrder = sqlOrder;

return sql`${sql.identifier(odataToColumnMap.get(col).split('.'))} ${sqlOrder}`;
});

if (stableOrderColumn != null)
clauses.push(sql`${sql.identifier(stableOrderColumn.split('.'))} ${initialOrder}`);

return sql`ORDER BY ${sql.join(clauses, sql`,`)}`;
};

module.exports = { odataFilter, odataOrderBy };

2 changes: 1 addition & 1 deletion lib/http/endpoint.js
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ const isJsonType = (x) => /(^|,)(application\/json|json)($|;|,)/i.test(x);
const isXmlType = (x) => /(^|,)(application\/(atom(svc)?\+)?xml|atom|xml)($|;|,)/i.test(x);

// various supported odata constants:
const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken' ];
const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken', '$orderby' ];
const supportedFormats = {
json: [ 'application/json', 'json' ],
xml: [ 'application/xml', 'atom' ]
Expand Down
6 changes: 4 additions & 2 deletions lib/model/query/entities.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const { equals, extender, unjoiner, page, markDeleted } = require('../../util/db
const { map, mergeRight, pickAll } = require('ramda');
const { blankStringToNull, construct } = require('../../util/util');
const { QueryOptions } = require('../../util/db');
const { odataFilter } = require('../../data/odata-filter');
const { odataFilter, odataOrderBy } = require('../../data/odata-filter');
const { odataToColumnMap, parseSubmissionXml, getDiffProp, ConflictType } = require('../../data/entity');
const { isTrue } = require('../../util/http');
const Problem = require('../../util/problem');
Expand Down Expand Up @@ -450,7 +450,9 @@ WHERE
AND entities."deletedAt" IS NULL
AND entity_defs.current=true
AND ${odataFilter(options.filter, odataToColumnMap)}
ORDER BY entities."createdAt" DESC, entities.id DESC
${options.orderby ? sql`
${odataOrderBy(options.orderby, odataToColumnMap, 'entities.id')}
`: sql`ORDER BY entities."createdAt" DESC, entities.id DESC`}
${page(options)}`)
.then(stream.map(_exportUnjoiner));

Expand Down
6 changes: 4 additions & 2 deletions lib/model/query/submissions.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ const { map } = require('ramda');
const { sql } = require('slonik');
const { Frame, table } = require('../frame');
const { Actor, Form, Submission } = require('../frames');
const { odataFilter } = require('../../data/odata-filter');
const { odataFilter, odataOrderBy } = require('../../data/odata-filter');
const { odataToColumnMap, odataSubTableToColumnMap } = require('../../data/submission');
const { unjoiner, extender, equals, page, updater, QueryOptions, insertMany } = require('../../util/db');
const { blankStringToNull, construct } = require('../../util/util');
Expand Down Expand Up @@ -359,7 +359,9 @@ where
${odataFilter(options.filter, options.isSubmissionsTable ? odataToColumnMap : odataSubTableToColumnMap)} and
${equals(options.condition)}
and submission_defs.current=true and submissions."formId"=${formId} and submissions."deletedAt" is null
order by submissions."createdAt" desc, submissions.id desc
${options.orderby ? sql`
${odataOrderBy(options.orderby, odataToColumnMap, 'submissions.id')}`
: sql`order by submissions."createdAt" desc, submissions.id desc`}
${page(options)}`;
};

Expand Down
10 changes: 10 additions & 0 deletions lib/util/db.js
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,11 @@ class QueryOptions {
result.filter = query.$filter;
if ((params.table === 'Submissions') && (query.$skiptoken != null))
result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken);
if (query.$orderby != null)
result.orderby = query.$orderby;

if (result.orderby && result.skiptoken)
throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' });

return new QueryOptions(result);
}
Expand All @@ -397,6 +402,11 @@ class QueryOptions {
result.filter = query.$filter;
if (query.$skiptoken != null)
result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken);
if (query.$orderby != null)
result.orderby = query.$orderby;

if (result.orderby && result.skiptoken)
throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' });

return new QueryOptions(result);
}
Expand Down
2 changes: 1 addition & 1 deletion lib/util/problem.js
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,7 @@ const problems = {

// returned when we don't support certain kinds of odata filter expressions.
unsupportedODataExpression: problem(501.4, (({ at, type, text }) => `The given OData filter expression uses features not supported by this server: ${type} at ${at} ("${text}")`)),
unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text} at ${at}`)),
unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text}${(at != null) ? ` at ${at}` : ''}`)),
unsupportedODataExpandExpression: problem(501.6, (({ text }) => `The given OData expand expression is not supported by this server: "${text}". Currently, only "$expand=*" is supported.`)),

invalidDatabaseConfig: problem(501.7, ({ reason }) => `The server's database configuration is invalid. ${reason}`),
Expand Down
Loading

0 comments on commit 9c5f6b2

Please sign in to comment.