diff --git a/docs/api.yaml b/docs/api.yaml index 85553fedc..f15aeca61 100644 --- a/docs/api.yaml +++ b/docs/api.yaml @@ -36,6 +36,14 @@ info: Here major and breaking changes to the API are listed by version. + ## ODK Central v2024.1 + + **Added**: + + - OData Data Document for requests of Submissions and Entities now allow use of `$orderby`. + - ETag headers on all Blobs. + + ## ODK Central v2023.5 **Added**: @@ -556,7 +564,6 @@ tags: * The actual data documents, linked from the Service Document, are a simple JSON representation of the submission data or entity, conforming to the schema we describe in our Metadata Document. - As our focus is on the bulk-export of data from ODK Central so that more advanced analysis tools can handle the data themselves, we do not support most of the features at the Intermediate and above conformance levels, like `$sort` or `$filter`. - name: System Endpoints description: There are some resources available for getting or setting system information and configuration. You can set the [Usage Reporting configuration](/central-api-system-endpoints/#usage-reporting-configuration) @@ -5417,6 +5424,7 @@ paths: responses: 200: description: OK + headers: ETag: schema: type: string @@ -11572,7 +11580,7 @@ paths: The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data. While paging is possible through these parameters, it will not greatly improve the performance of exporting data. ODK Central prefers to bulk-export all of its data at once if possible. - As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by any data field in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error. + As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by certain data fields in the schema. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error. The fields you can query against are as follows: @@ -11599,6 +11607,8 @@ paths: + Child properties of repeats can't be requested using `$select` + As of ODK Central v2024.1, the [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) is now supported, and can sort on the same fields as `$filter`, noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/submitterId ASC, __system/reviewState DESC`. + As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers. operationId: Data Document parameters: @@ -11663,6 +11673,12 @@ paths: schema: type: string example: year(__system/submissionDate) lt year(now()) + - name: '%24orderby' + in: query + description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together. + schema: + type: string + example: __system/submitterId asc, __system/updatedAt desc - name: '%24expand' in: query description: Repetitions, which should get expanded. Currently, only `*` is @@ -12067,15 +12083,12 @@ paths: The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. If `$top` parameter is provided in the request then the response will include `@odata.nextLink` that you can use as is to fetch the next set of data. - The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter by any data field in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. + The [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) can be used to filter certain data fields in the system-level schema, but not the Dataset properties. The operators `lt`, `le`, `eq`, `ne`, `ge`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. The fields you can query against are as follows: | Entity Metadata | OData Field Name | | ------------------------| -------------------- | - | Entity UUID | `__id` | - | Entity Name (same as UUID) | `name` | - | Entity Label | `label` | | Entity Creator Actor ID | `__system/creatorId` | | Entity Timestamp | `__system/createdAt` | | Entity Update Timestamp | `__system/updatedAt` | @@ -12087,6 +12100,8 @@ paths: The [`$select` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358942) will return just the fields you specify and is supported on `__id`, `__system`, `__system/creatorId`, `__system/createdAt` and `__system/updatedAt`, as well as on user defined properties. + The [`$orderby` query parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358952) will return Entities sorted by different fields, which come from the same list used by `$filter`, as noted above. The order can be specified as `ASC` (ascending) or `DESC` (descending), which are case-insensitive. Multiple sort expressions can be used together, separated by commas, e.g. `$orderby=__system/creatorId ASC, __system/conflict DESC`. + As the vast majority of clients only support the JSON OData format, that is the only format ODK Central offers. operationId: Data Document for Dataset parameters: @@ -12135,6 +12150,12 @@ paths: schema: type: string example: year(__system/createdAt) lt year(now()) + - name: '%24orderby' + in: query + description: If provided, will sort responses according to specified order expression. Only the same fields as `$filter` above can be used to sort. Multiple expressions can be used together. + schema: + type: string + example: __system/creatorId asc, __system/updatedAt desc - name: '%24select' in: query description: If provided, will return only the selected fields. diff --git a/lib/data/odata-filter.js b/lib/data/odata-filter.js index 909d65ea9..69429a66c 100644 --- a/lib/data/odata-filter.js +++ b/lib/data/odata-filter.js @@ -87,5 +87,33 @@ const odataFilter = (expr, odataToColumnMap) => { return op(ast); }; -module.exports = { odataFilter }; +const odataOrderBy = (expr, odataToColumnMap, stableOrderColumn = null) => { + let initialOrder = null; + const clauses = expr.split(',').map((exp) => { + const [col, order] = exp.trim().split(/\s+/); + + // validate field + if (!odataToColumnMap.has(col)) + throw Problem.internal.unsupportedODataField({ text: col }); + + // validate order (asc or desc) + if (order && !order?.toLowerCase().match(/^(asc|desc)$/)) + throw Problem.internal.unsupportedODataField({ text: order }); + + const sqlOrder = (order?.toLowerCase() === 'desc') ? sql`DESC NULLS LAST` : sql`ASC NULLS FIRST`; + + // Save the order of the initial property to use for the stable sort column order + if (initialOrder == null) + initialOrder = sqlOrder; + + return sql`${sql.identifier(odataToColumnMap.get(col).split('.'))} ${sqlOrder}`; + }); + + if (stableOrderColumn != null) + clauses.push(sql`${sql.identifier(stableOrderColumn.split('.'))} ${initialOrder}`); + + return sql`ORDER BY ${sql.join(clauses, sql`,`)}`; +}; + +module.exports = { odataFilter, odataOrderBy }; diff --git a/lib/http/endpoint.js b/lib/http/endpoint.js index 657edf6dc..5540b1b0f 100644 --- a/lib/http/endpoint.js +++ b/lib/http/endpoint.js @@ -313,7 +313,7 @@ const isJsonType = (x) => /(^|,)(application\/json|json)($|;|,)/i.test(x); const isXmlType = (x) => /(^|,)(application\/(atom(svc)?\+)?xml|atom|xml)($|;|,)/i.test(x); // various supported odata constants: -const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken' ]; +const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand', '$select', '$skiptoken', '$orderby' ]; const supportedFormats = { json: [ 'application/json', 'json' ], xml: [ 'application/xml', 'atom' ] diff --git a/lib/model/query/entities.js b/lib/model/query/entities.js index 00e4b74eb..4cedefb44 100644 --- a/lib/model/query/entities.js +++ b/lib/model/query/entities.js @@ -13,7 +13,7 @@ const { equals, extender, unjoiner, page, markDeleted } = require('../../util/db const { map, mergeRight, pickAll } = require('ramda'); const { blankStringToNull, construct } = require('../../util/util'); const { QueryOptions } = require('../../util/db'); -const { odataFilter } = require('../../data/odata-filter'); +const { odataFilter, odataOrderBy } = require('../../data/odata-filter'); const { odataToColumnMap, parseSubmissionXml, getDiffProp, ConflictType } = require('../../data/entity'); const { isTrue } = require('../../util/http'); const Problem = require('../../util/problem'); @@ -450,7 +450,9 @@ WHERE AND entities."deletedAt" IS NULL AND entity_defs.current=true AND ${odataFilter(options.filter, odataToColumnMap)} -ORDER BY entities."createdAt" DESC, entities.id DESC +${options.orderby ? sql` + ${odataOrderBy(options.orderby, odataToColumnMap, 'entities.id')} + `: sql`ORDER BY entities."createdAt" DESC, entities.id DESC`} ${page(options)}`) .then(stream.map(_exportUnjoiner)); diff --git a/lib/model/query/submissions.js b/lib/model/query/submissions.js index 3faeba804..17d047a17 100644 --- a/lib/model/query/submissions.js +++ b/lib/model/query/submissions.js @@ -11,7 +11,7 @@ const { map } = require('ramda'); const { sql } = require('slonik'); const { Frame, table } = require('../frame'); const { Actor, Form, Submission } = require('../frames'); -const { odataFilter } = require('../../data/odata-filter'); +const { odataFilter, odataOrderBy } = require('../../data/odata-filter'); const { odataToColumnMap, odataSubTableToColumnMap } = require('../../data/submission'); const { unjoiner, extender, equals, page, updater, QueryOptions, insertMany } = require('../../util/db'); const { blankStringToNull, construct } = require('../../util/util'); @@ -359,7 +359,9 @@ where ${odataFilter(options.filter, options.isSubmissionsTable ? odataToColumnMap : odataSubTableToColumnMap)} and ${equals(options.condition)} and submission_defs.current=true and submissions."formId"=${formId} and submissions."deletedAt" is null -order by submissions."createdAt" desc, submissions.id desc +${options.orderby ? sql` + ${odataOrderBy(options.orderby, odataToColumnMap, 'submissions.id')}` + : sql`order by submissions."createdAt" desc, submissions.id desc`} ${page(options)}`; }; diff --git a/lib/util/db.js b/lib/util/db.js index a4d8235d9..f692adf6e 100644 --- a/lib/util/db.js +++ b/lib/util/db.js @@ -383,6 +383,11 @@ class QueryOptions { result.filter = query.$filter; if ((params.table === 'Submissions') && (query.$skiptoken != null)) result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken); + if (query.$orderby != null) + result.orderby = query.$orderby; + + if (result.orderby && result.skiptoken) + throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' }); return new QueryOptions(result); } @@ -397,6 +402,11 @@ class QueryOptions { result.filter = query.$filter; if (query.$skiptoken != null) result.skiptoken = QueryOptions.parseSkiptoken(query.$skiptoken); + if (query.$orderby != null) + result.orderby = query.$orderby; + + if (result.orderby && result.skiptoken) + throw Problem.internal.notImplemented({ feature: 'using $orderby and $skiptoken together' }); return new QueryOptions(result); } diff --git a/lib/util/problem.js b/lib/util/problem.js index 7a2d74f3c..86395f394 100644 --- a/lib/util/problem.js +++ b/lib/util/problem.js @@ -229,7 +229,7 @@ const problems = { // returned when we don't support certain kinds of odata filter expressions. unsupportedODataExpression: problem(501.4, (({ at, type, text }) => `The given OData filter expression uses features not supported by this server: ${type} at ${at} ("${text}")`)), - unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text} at ${at}`)), + unsupportedODataField: problem(501.5, (({ at, text }) => `The given OData filter expression references fields not supported by this server: ${text}${(at != null) ? ` at ${at}` : ''}`)), unsupportedODataExpandExpression: problem(501.6, (({ text }) => `The given OData expand expression is not supported by this server: "${text}". Currently, only "$expand=*" is supported.`)), invalidDatabaseConfig: problem(501.7, ({ reason }) => `The server's database configuration is invalid. ${reason}`), diff --git a/test/integration/api/odata-entities.js b/test/integration/api/odata-entities.js index 52e31ae5c..065c3e3a3 100644 --- a/test/integration/api/odata-entities.js +++ b/test/integration/api/odata-entities.js @@ -321,6 +321,63 @@ describe('api: /datasets/:name.svc', () => { }); })); + it('should NOT filter by label', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$filter=label eq \'Alice\'') + .expect(501) + .then(({ body }) => { + body.message.should.eql('The given OData filter expression references fields not supported by this server: label at 0'); + }); + })); + + it('should NOT filter by id', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$filter=__id eq \'1234\'') + .expect(501) + .then(({ body }) => { + body.message.should.eql('The given OData filter expression references fields not supported by this server: __id at 0'); + }); + })); + + it('should NOT filter by name/uuid', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$filter=uuid eq \'1234\'') + .expect(501) + .then(({ body }) => { + body.message.should.eql('The given OData filter expression references fields not supported by this server: uuid at 0'); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$filter=name eq \'1234\'') + .expect(501) + .then(({ body }) => { + body.message.should.eql('The given OData filter expression references fields not supported by this server: name at 0'); + }); + })); + it('should return filtered entities with pagination', testService(async (service, container) => { const asAlice = await service.login('alice'); const asBob = await service.login('bob'); @@ -379,14 +436,29 @@ describe('api: /datasets/:name.svc', () => { await createSubmissions(asAlice, container, 2); - await container.run(sql`UPDATE entities SET "createdAt" = '2020-01-01'`); + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$select=__id') + .expect(200) + .then(({ body }) => { + body.value.map((e) => Object.keys(e).should.eql(['__id'])); + body.value.length.should.be.eql(2); + }); + })); - await createSubmissions(asAlice, container, 2, 2); + it('should return selected user-defined only', testService(async (service, container) => { + const asAlice = await service.login('alice'); - await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$select=__id') + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$select=age') .expect(200) .then(({ body }) => { - body.value.length.should.be.eql(4); + body.value.map((e) => Object.keys(e).should.eql(['age'])); + body.value.length.should.be.eql(2); }); })); @@ -455,6 +527,213 @@ describe('api: /datasets/:name.svc', () => { }); })); + + it('should return entities in specified order', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 3); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/createdAt asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.age).should.eql(['1', '2', '3']); + body.value[0].__system.createdAt.should.be.lessThan(body.value[2].__system.createdAt); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/createdAt desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.age).should.eql(['3', '2', '1']); + body.value[0].__system.createdAt.should.be.greaterThan(body.value[2].__system.createdAt); + }); + })); + + it('should return entities with order specified in multiple clauses', testService(async (service, container) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + await createSubmissions(asBob, container, 2, 2); + await createSubmissions(asAlice, container, 2, 4); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId desc, __system/createdAt asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['6', '6', '5', '5', '5', '5']); + body.value.map((e) => e.age).should.eql(['3', '4', '1', '2', '5', '6']); + }); + })); + + it('should return entities in stable order', testService(async (service, container) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + await createSubmissions(asBob, container, 2, 2); + await createSubmissions(asAlice, container, 2, 4); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['6', '6', '5', '5', '5', '5']); + body.value.map((e) => e.age).should.eql(['4', '3', '6', '5', '2', '1']); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['5', '5', '5', '5', '6', '6']); + body.value.map((e) => e.age).should.eql(['1', '2', '5', '6', '3', '4']); + }); + })); + + it('should return sorted null values in the correct order', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + await createConflict(asAlice, container); + + // Default sort order is asc + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/conflict') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.conflict).should.eql([ null, null, 'hard' ]); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/conflict asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.conflict).should.eql([ null, null, 'hard' ]); + }); + + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/conflict desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.conflict).should.eql([ 'hard', null, null ]); + }); + })); + + it('should return sorted null values in the correct order in any clause', testService(async (service, container) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + await createSubmissions(asBob, container, 2, 2); + await createConflict(asAlice, container); + + // Default sort order is asc + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId asc,__system/conflict') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['5', '5', '5', '6', '6']); + body.value.map((e) => e.__system.conflict).should.eql([ null, null, 'hard', null, null ]); + }); + + // Default sort order is asc + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId asc,__system/conflict asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['5', '5', '5', '6', '6']); + body.value.map((e) => e.__system.conflict).should.eql([ null, null, 'hard', null, null ]); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId asc,__system/conflict desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['5', '5', '5', '6', '6']); + body.value.map((e) => e.__system.conflict).should.eql([ 'hard', null, null, null, null ]); + }); + + await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/creatorId desc,__system/conflict desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.creatorId).should.eql(['6', '6', '5', '5', '5']); + body.value.map((e) => e.__system.conflict).should.eql([ null, null, 'hard', null, null ]); + }); + })); + + it('should combine orderby with other filters', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + const lastEntity = await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$top=1') + .expect(200) + .then(({ body }) => body.value[0]); + + await createSubmissions(asAlice, container, 4, 2); + + await asAlice.get(`/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/createdAt desc&$filter=__system/createdAt gt ${lastEntity.__system.createdAt}`) + .expect(200) + .then(({ body }) => { + body.value.length.should.eql(4); + body.value[0].__system.createdAt.should.be.greaterThan(body.value[3].__system.createdAt); + }); + + await asAlice.get(`/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/createdAt asc&$filter=__system/createdAt gt ${lastEntity.__system.createdAt}`) + .expect(200) + .then(({ body }) => { + body.value.length.should.eql(4); + body.value[0].__system.createdAt.should.be.lessThan(body.value[3].__system.createdAt); + }); + })); + + it('should reject if orderby used with skiptoken', testService(async (service, container) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms?publish=true') + .set('Content-Type', 'application/xml') + .send(testData.forms.simpleEntity) + .expect(200); + + await createSubmissions(asAlice, container, 2); + + const token = await asAlice.get('/v1/projects/1/datasets/people.svc/Entities?$top=1') + .expect(200) + .then(({ body }) => { + const tokenData = { + uuid: body.value[0].__id, + }; + return encodeURIComponent(QueryOptions.getSkiptoken(tokenData)); + }); + + await asAlice.get(`/v1/projects/1/datasets/people.svc/Entities?$orderby=__system/createdAt desc&%24skiptoken=${token}`) + .expect(501) + .then(({ body }) => { + body.message.should.be.eql('The requested feature using $orderby and $skiptoken together is not supported by this server.'); + }); + })); }); describe('GET service document', () => { diff --git a/test/integration/api/odata.js b/test/integration/api/odata.js index 22adf5c67..ac6c2af25 100644 --- a/test/integration/api/odata.js +++ b/test/integration/api/odata.js @@ -1190,6 +1190,147 @@ describe('api: /forms/:id.svc', () => { }); })); + describe('orderby', () => { + it('should return submissions in specified order', testService(async (service) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.one) + .set('Content-Type', 'text/xml') + .expect(200); + + await asBob.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.two) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.three) + .set('Content-Type', 'text/xml') + .expect(200); + + // extra submission not in form that shouldn't be returned + await asAlice.post('/v1/projects/1/forms/simple/submissions') + .send(testData.instances.simple.one) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.get('/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/submitterId asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.submitterId).should.eql(['5', '5', '6']); + body.value.map((e) => e.age).should.eql([30, 38, 34]); + }); + + await asAlice.get('/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/submitterId desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.submitterId).should.eql(['6', '5', '5']); + body.value.map((e) => e.age).should.eql([34, 38, 30]); + }); + })); + + it('should combine orderby and other things like filtering', testService(async (service) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.one) + .set('Content-Type', 'text/xml') + .expect(200); + + await asBob.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.two) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.three) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.patch('/v1/projects/1/forms/withrepeat/submissions/rone') + .send({ reviewState: 'rejected' }) + .expect(200); + + await asAlice.patch('/v1/projects/1/forms/withrepeat/submissions/rtwo') + .send({ reviewState: 'rejected' }) + .expect(200); + + await asAlice.get("/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/submitterId asc&$filter=__system/reviewState eq 'rejected'") + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.submitterId).should.eql(['5', '6']); + body.value.map((e) => e.age).should.eql([30, 34]); + }); + })); + + it('should return null values at the correct end of list with orderby', testService(async (service) => { + const asAlice = await service.login('alice'); + const asBob = await service.login('bob'); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.one) + .set('Content-Type', 'text/xml') + .expect(200); + + await asBob.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.two) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.three) + .set('Content-Type', 'text/xml') + .expect(200); + + await asAlice.patch('/v1/projects/1/forms/withrepeat/submissions/rone') + .send({ reviewState: 'approved' }) + .expect(200); + + await asAlice.patch('/v1/projects/1/forms/withrepeat/submissions/rtwo') + .send({ reviewState: 'rejected' }) + .expect(200); + + await asAlice.get('/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/reviewState asc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.reviewState).should.eql([ null, 'approved', 'rejected' ]); + }); + + await asAlice.get('/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/reviewState desc') + .expect(200) + .then(({ body }) => { + body.value.map((e) => e.__system.reviewState).should.eql([ 'rejected', 'approved', null ]); + }); + })); + + it('should reject if both orderby and skiptoken are used together', testService(async (service) => { + const asAlice = await service.login('alice'); + + await asAlice.post('/v1/projects/1/forms/withrepeat/submissions') + .send(testData.instances.withrepeat.one) + .set('Content-Type', 'text/xml') + .expect(200); + + const token = await asAlice.get('/v1/projects/1/forms/withrepeat.svc/Submissions?$top=1') + .expect(200) + .then(({ body }) => { + const tokenData = { + instanceId: body.value[0].__id, + }; + return encodeURIComponent(QueryOptions.getSkiptoken(tokenData)); + }); + + await asAlice.get(`/v1/projects/1/forms/withrepeat.svc/Submissions?$orderby=__system/submitterId&$skiptoken=${token}`) + .expect(501) + .then(({ body }) => { + body.message.should.be.eql('The requested feature using $orderby and $skiptoken together is not supported by this server.'); + }); + })); + }); + it('should count correctly while windowing', testService((service) => service.login('alice', (asAlice) => service.login('bob', (asBob) => diff --git a/test/unit/data/odata-filter.js b/test/unit/data/odata-filter.js index de68defaf..2d5dff16f 100644 --- a/test/unit/data/odata-filter.js +++ b/test/unit/data/odata-filter.js @@ -10,10 +10,11 @@ const appRoot = require('app-root-path'); const assert = require('assert'); const { sql } = require('slonik'); -const { odataFilter: _odataFilter } = require(appRoot + '/lib/data/odata-filter'); +const { odataFilter: _odataFilter, odataOrderBy: _odataOrderBy } = require(appRoot + '/lib/data/odata-filter'); const { odataToColumnMap } = require(appRoot + '/lib/data/submission'); const odataFilter = (exp) => _odataFilter(exp, odataToColumnMap); +const odataOrderBy = (exp, stableOrderColumn = null) => _odataOrderBy(exp, odataToColumnMap, stableOrderColumn); describe('OData filter query transformer', () => { it('should transform binary expressions', () => { @@ -73,3 +74,65 @@ describe('OData filter query transformer', () => { }); }); +describe('OData orderby/sort query transformer', () => { + it('should transform order by queries', () => { + odataOrderBy('__system/updatedAt desc').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} DESC NULLS LAST`); + odataOrderBy('__system/updatedAt DESC').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} DESC NULLS LAST`); + odataOrderBy('__system/updatedAt DESC ').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} DESC NULLS LAST`); + odataOrderBy('__system/updatedAt asc').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} ASC NULLS FIRST`); + odataOrderBy(' __system/updatedAt AsC ').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} ASC NULLS FIRST`); + }); + + it('should default to ASC if no sort order provided', () => { + odataOrderBy('__system/updatedAt').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} ASC NULLS FIRST`); + }); + + it('should ignore things after sort order', () => { + odataOrderBy(' __system/updatedAt ASC DESC OTHER STUFF ').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} ASC NULLS FIRST`); + }); + + it('should combine multiple sort operators', () => { + odataOrderBy('__system/updatedAt desc, __system/submissionDate ASC').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} DESC NULLS LAST,${sql.identifier([ 'submissions', 'createdAt' ])} ASC NULLS FIRST`); + }); + + + it('should NOT handle more complex filters in an orderby clause because sort order validation fails', () => { + assert.throws(() => { odataOrderBy('__system/submitterId ne 5 desc '); }, (err) => { + err.problemCode.should.equal(501.5); + err.message.should.equal('The given OData filter expression references fields not supported by this server: ne'); + return true; + }); + }); + + it('should reject unparseable expressions', () => { + assert.throws(() => { odataOrderBy('hello my dear'); }, (err) => { + err.problemCode.should.equal(501.5); + err.message.should.equal('The given OData filter expression references fields not supported by this server: hello'); + return true; + }); + }); + + it('should reject unrecognized field names', () => { + assert.throws(() => { odataOrderBy('myfield asc'); }, (err) => { + err.problemCode.should.equal(501.5); + err.message.should.equal('The given OData filter expression references fields not supported by this server: myfield'); + return true; + }); + }); + + it('should reject unrecognized sort orders', () => { + assert.throws(() => { odataOrderBy('__system/updatedAt UP'); }, (err) => { + err.problemCode.should.equal(501.5); + err.message.should.equal('The given OData filter expression references fields not supported by this server: UP'); + return true; + }); + }); + + it('should add last sort clause to insure stable sort order', () => { + odataOrderBy('__system/updatedAt asc', 'entities.id').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} ASC NULLS FIRST,${sql.identifier([ 'entities', 'id' ])} ASC NULLS FIRST`); + }); + + it('should use first sort order for stable sort order', () => { + odataOrderBy('__system/updatedAt desc, __system/submitterId asc', 'entities.id').should.eql(sql`ORDER BY ${sql.identifier([ 'submissions', 'updatedAt' ])} DESC NULLS LAST,${sql.identifier([ 'submissions', 'submitterId' ])} ASC NULLS FIRST,${sql.identifier([ 'entities', 'id' ])} DESC NULLS LAST`); + }); +}); diff --git a/test/unit/http/endpoint.js b/test/unit/http/endpoint.js index eaa18415d..777637eb6 100644 --- a/test/unit/http/endpoint.js +++ b/test/unit/http/endpoint.js @@ -657,7 +657,7 @@ describe('endpoints', () => { }); it('should reject requests for unsupported OData features', () => { - const request = createRequest({ url: '/odata.svc?$orderby=magic' }); + const request = createRequest({ url: '/odata.svc?$inlineCount=magic' }); return odataPreprocessor('json')(null, new Context(request), request) .should.be.rejectedWith(Problem, { problemCode: 501.1 }); });