[DRAFT] fix: Use CTE in Enrollment analytics queries [DHIS-16705]

[luciano-fiandesio]

WIP

Changes

This PR addresses the generation of SQL queries for Enrollment analytics

Problem

Currently, the Enrollment queries are structured so that sub-queries are used to fetch events values from the analytics_event_* tables.
For instance:

select
    enrollment,
    ...
    ax."ou",
    (select
         "fyjPqlHE7Dn"
     from
	 analytics_event_M3xtLkYBlKI
     where
	 analytics_event_M3xtLkYBlKI.eventstatus != 'SCHEDULE'
	 and analytics_event_M3xtLkYBlKI.enrollment = ax.enrollment
	 and ps = 'CWaAcQYKVpq'
limit 1 ) as "CWaAcQYKVpq[-1].fyjPqlHE7Dn",
from
     analytics_enrollment_m3xtlkyblki as ax
...

The above query works in Postgres but does not work in Doris, because correlation with outer layers of the parent query is not supported.

Mitigation

The current approach is trying to refactor the Enrollment query so that the subqueries (both as select and as where conditions) are "moved" into CTE (Common Table Expressions).
Common Table Expressions are temporary result sets in SQL, defined within a WITH clause, that simplify complex queries by improving readability and modularizing logic. They are particularly useful for recursive queries and can be referenced multiple times within the main query.

The above query can be rewritten like so:

with ps_cwaacqykvpq_fyjpqlhe7dn as (
select
	distinct on
	(enrollment) enrollment,
	"fyjPqlHE7Dn" as value
from
	analytics_event_M3xtLkYBlKI
where
	eventstatus != 'SCHEDULE'
	and ps = 'CWaAcQYKVpq'
	
order by
	enrollment,
	occurreddate desc,
	created desc)
select
	ax.enrollment,
	...
	ps_cwaacqykvpq_fyjpqlhe7dn.value as "CWaAcQYKVpq.fyjPqlHE7Dn"
from
	analytics_enrollment_m3xtlkyblki as ax
left join ps_cwaacqykvpq_fyjpqlhe7dn on
	ax.enrollment = ps_cwaacqykvpq_fyjpqlhe7dn.enrollment
where
	(((enrollmentdate >= '2021-01-01'
	and enrollmentdate < '2022-01-01')))
	and (ax."uidlevel1" = 'ImspTQPwCqd' )
...

The above query structure is compatible with Doris and it makes the execution of the query faster.
As a comparison:

Original query with sub-select

Refactored query with CTEs

Testing strategy

I am using the e2e project and aim at having 100% green tests on both Postgres and Doris.

To fix the problem, we need to validate the offset parameter before performing any arithmetic operations. Specifically, we should ensure that the offset value falls within a safe range to prevent overflow or underflow. We can achieve this by adding a validation step in the computeRowNumberOffset method.

1. Add a validation step to check if the offset value is within a safe range.
2. If the offset value is outside the safe range, handle it appropriately (e.g., throw an exception or set it to a default value).

[larshelge]

Let's use lower-case for SQL keywords in Java, i.e. with.

[larshelge]

Remove println.

[larshelge]

Use lower-case keywords for SQL in Java.

[larshelge]

Remove outer @.

[larshelge]

Rename to contributeCte.

[sonarqubecloud]

Quality Gate failed

Failed conditions
27 New issues
27 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE