From ca488a6c12f6727a7adaf4bc148574f9ee244c17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Fri, 13 Dec 2024 12:16:38 -0800 Subject: [PATCH 01/11] Crosslink between Entities sections --- docs/central-entities.rst | 7 +++++++ docs/entities-intro.rst | 12 ++++++++---- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index 57d170c45..f0d09f36e 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -5,6 +5,11 @@ Managing Entities in Central **Entities** in ODK let you share information between Forms so you can collect longitudinal data, manage cases over time, and support other complex workflows. +.. seealso:: + +* :doc:`Entities intro `: similar information organized as a list of questions and answers +* :doc:`Community reporting tutorial ` + Without Entities, you can attach existing data during the Form creation process, often through ``choices`` sheets or ``.csv`` files. Once this data is loaded into the Form, you can use it as the source for selection choices, to prepopulate data fields in the Form, or validate new input. A list of districts, for example, can be used as choices for selection, and information about each district can then be shown to the user or checked against. If you are new to these techniques, you can learn more about them on the :doc:`Form Datasets ` page. This is wonderful if the data you want to use is already known when you are publishing your Form, and if it doesn't change over the course of your project. In many cases, however, the data being collected today for the project *is* the data you need tomorrow for the same project. Think about seeing a patient for the first and then second time, or registering and then revisiting a survey site for a follow-up. These kinds of workflows are called :ref:`multiple encounter ` or longitudinal workflows. @@ -77,6 +82,8 @@ We recommend watching the video below once or twice to get an overview of how En In Central versions prior to v2023.4, Entity Lists were called Datasets. The video below was recorded with Central v2022.3 so many small improvements have been made since. + You may also want to see the :doc:`community reporting tutorial ` for another Entity-based workflow. + .. youtube:: hbff-oaI8yg :width: 100% diff --git a/docs/entities-intro.rst b/docs/entities-intro.rst index a3d825671..67be2492b 100644 --- a/docs/entities-intro.rst +++ b/docs/entities-intro.rst @@ -102,6 +102,9 @@ For example, let's say that you have trees that you want to evaluate over time. Limitations =========== +.. seealso:: + :ref:`Entities roadmap and limitations ` + I filled out a registration form and don't immediately see my Entity in follow-up forms, why? --------------------------------------------------------------------------------------------- @@ -202,15 +205,16 @@ Add an ``entities`` sheet to your form. This sheet is used to define how data fr Currently, a single submission can only affect a single Entity in a fixed Entity List. To specify which list to create or update an Entity in, use the ``list_name`` column. -If you're creating Entities, you'll also need to specify an expression to label each Entity in the ``label`` column. This is very similar to :ref:`the instance_name column ` for naming filled forms. +If you're creating Entities, you'll also need to specify an expression that defines the label of each Entity in the ``label`` column. This is very similar to :ref:`the instance_name column ` for naming filled forms. -If you're updating Entities, you must add an ``entity_id`` column. In that column, put a reference to a form field that holds the unique id of the Entity you want to update. For example, if you have a select question named ``tree`` that lets the user select a tree from the Entity List, you would put ``${tree}`` in the ``entity_id`` column. You may also specify an expression to label each Entity in the ``label`` column if you would like the label to change, for example to show an updated status. +If you're updating Entities, you must add an ``entity_id`` column. In that column, put a reference to a form field that holds the unique id of the Entity you want to update. For example, if you have a select question named ``tree`` that lets the user select a tree from the Entity List, you would put ``${tree}`` in the ``entity_id`` column. You may also specify an expression that defines the label for each Entity in the ``label`` column if you would like the label to change, for example to show an updated status. Next, specify which form fields should be saved to Entity properties. This is done on the ``survey`` sheet by putting the desired property name in the ``save_to`` column for each form field that you want to save. .. seealso:: * :doc:`Community reporting tutorial ` - * :ref:`Central Entities documentation ` + * :ref:`Build a form that creates Entities ` + * :ref:`Build a form that updates Entities ` How do I access Entities from my forms? --------------------------------------- @@ -228,7 +232,7 @@ You can access a specific Entity's properties using a :ref:`lookup expression ` * :doc:`Community reporting tutorial ` - * :ref:`Central Entities documentation ` + * :ref:`Build a form that uses Entities ` What form fields should I save to my Entities as properties? ------------------------------------------------------------ From 0d3f26944c55226e2bedbb8f108e502c7870a22f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Fri, 13 Dec 2024 12:28:43 -0800 Subject: [PATCH 02/11] Update limitation sections --- docs/central-entities.rst | 7 ++++--- docs/entities-intro.rst | 4 +--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index f0d09f36e..db7e8eaf1 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -50,6 +50,9 @@ If you would prefer a more personalized introduction, request a demo at https:// Roadmap and limitations ----------------------- +.. seealso:: + :ref:`Details about limitations ` + Entities are a big new concept that open up a lot of new possibilities. While we think many workflows can benefit from Entities today, they have some limitations that you should be aware of. What's available now: @@ -62,11 +65,9 @@ What's available now: Important limitations: -- Entity create and update requires Internet access -- Entity delete is only available via API +- A form submission can only create or update a single Entity - Client performance (ODK Collect or Enketo web forms) suffers when managing more than 50,000 Entities - All devices will always download all Entities which may be a privacy concern -- The Form specification and API may change - Currently, Entity properties may only be strings. Submission values saved to an Entity are converted to strings. We expect all of these limitations to be addressed over time. The roadmap at `getodk.org/roadmap `_ has what we're working on now and what's coming next. diff --git a/docs/entities-intro.rst b/docs/entities-intro.rst index 67be2492b..a50c58381 100644 --- a/docs/entities-intro.rst +++ b/docs/entities-intro.rst @@ -108,12 +108,10 @@ Limitations I filled out a registration form and don't immediately see my Entity in follow-up forms, why? --------------------------------------------------------------------------------------------- -Currently, in order for a submission to create or update an Entity, that submission has to be processed by your server. That means that if you create a new Entity or update an existing one by filling out a form, you won't see that change reflected in follow-up forms until you download the latest update to your Entity List from your server. +If you are using a version of ODK Central older than 2024.3.0, Enketo web forms, or a version of Collect older than 2024.3.0, Entities are not created or updated offline. This means that in order for a submission to create or update an Entity, that submission has to be processed by your server. If you create a new Entity or update an existing one by filling out a form, you won't see that change reflected in follow-up forms until you download the latest update to your Entity List from your server. If you usually have Internet connectivity, this is unlikely to be very important. Similarly, if your registration and follow-up periods happen at very different times, this limitation is not a problem. But for workflows in which follow-up needs to happen immediately after registration or multiple follow-ups are needed while offline, this limitation is significant. -Offline Entity support is expected in late 2024, read more `on the forum `_. - I need to assign specific Entities to specific data collectors, how can I represent this? ----------------------------------------------------------------------------------------- From bb7ccf058f1d241ee17a4004dde0f8922aeecd76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Fri, 13 Dec 2024 12:57:42 -0800 Subject: [PATCH 03/11] Move the conflict section below the howto section --- docs/central-entities.rst | 57 ++++++++++++++++++++++----------------- docs/entities-intro.rst | 2 +- 2 files changed, 33 insertions(+), 26 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index db7e8eaf1..6ae651df0 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -308,32 +308,8 @@ Updating Entities from Forms You can use Forms to update Entity information. These Forms can be authored to, for example, update previous observations to new values or change the status of an Entity. Just like Entity creation through Forms, you can specify which properties on which Entity instances will be updated when the Form's Submissions are uploaded to Central. The data in Submissions uploaded by Entity-updating Forms are applied to the Entity data saved on the Central server. These updated Entity values are then distributed to data collection clients once they synchronize with Central. -.. _central-entities-update-conflicts: - -Parallel Updates in Updating Forms -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once you begin allowing updates to Entities to come in through the field, you can run into incidents where two people attempt to update the same Entity at the same time. If you have a project setup where Entities would only be updated by one person at a time (for example, if that person is the only one assigned to a particular area), you can ignore this section. - -.. _central-entities-update-conflicts-what-is: - -What is a parallel update? -"""""""""""""""""""""""""" - -What does it mean that updating an Entity "at the same time" causes a conflict? Every time an Entity is updated, whether by direct edits on the Central management panel, through the API, or by a Submission upload, a new "version" of that Entity is created that has the latest data. - -Say Alice fetches all her Tree data on her device. This update includes a brand new Tree entity called "Zach's backyard tree". Alice updates Zach's tree with new measurements. This results in version 2 of that tree once the Submission is uploaded to Central. In a "clean," not-conflicting update Bob would get the latest Tree data, see version 2 of Zach's tree in his update Form, and upload his new changes to create version 3 of the tree on Central. Version 1 led to version 2, and version 2 was updated to create version 3. - -However, if Bob doesn't update his data before filling his Form, and so he's still working off version 1 of the tree, then a conflict results. Two updates have been created based on version 1, and Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob are both looking at version 1 and trying to create version 2. - .. note:: - The same rules and behaviors apply when updating Entities through updates made directly to Entities in Central's management panel, or over the API. It's possible, for example, that Bob opened an Entity data edit window in Central before Alice's update is uploaded, but he didn't actually press Submit on his changes until much later. The same is true: Bob is looking at version 1 and thinking he's creating version 2. - -Central's behavior in these cases is basic: **it will always apply any changes it receives at the time and in the order it receives them**. This means that in this case, Bob's changes will always "win" over Alice's. Any time this happens, Central will generate a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. - -Of course, it's possible that Bob's changes and Alice's changes are totally compatible. For example, it's possible Alice only updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. How often and severely conflicts create actual consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. Central attempts to help Project managers determine the severity of conflicts by separating conflict warnings into two levels of severity. - -**All** conflicts surfaced by Central occur because two Entity updates were generated based off the same base version of an Entity. This circumstance is called a :guilabel:`Parallel Update` in Central. In the special case that a property included in an update has also been changed in the meantime, *and* the value currently stored in Central is different from the new updated value, a full :guilabel:`Conflict` error will be created instead, indicating that multiple possibly conflicting values were written into the same property at once. So if Alice and Bob have both updated the circumference of the tree at the same time to the same value, only a Parallel Update warning and not a Conflict will be flagged. Only if their inputs disagree will you see a full Conflict. + Entity updates from form submissions open up the possibility of two data collectors attempting to update the same Entity at the same time. This case is detected by Central and shown as a conflict. If your workflow involves data collectors who are offline for a long time or many updates to the same Entity, be sure to read the section on `parallel updates `. .. _central-entities-build-update: @@ -401,6 +377,37 @@ Creating AND Updating Entities with one Form You can give both a ``create_if`` and an ``update_if`` rule for the same Form. If only one of these expression evaluates to ``true`` or ``1``, then only that operation will be carried out. If both rules evaluate to ``true`` or ``1``, the Entity will be created if it does not exist (as identified by the ``entity_id`` expression), and updated if it does. Of course, if neither rule evaluates to ``true`` or ``1``, no Entity changes will occur. +.. _central-entities-update-conflicts: + +Parallel Updates +~~~~~~~~~~~~~~~~~ + +Updates from form submissions open up the possibility of two people attempting to update the same Entity at the same time. Central always applies all updates it receives in the order they are received. If it detects that two updates to the same Entity were made in parallel, it marks that Entity as possibly being in a conflict state. A Project Manager can then look at the conflict and decide how to resolve it. + +If you can, try to organize work so that conflicts are unlikely. For example, you can use :ref:`choice filters ` to only show each data collector the Entities that they are assigned to. You can filter based on Entity properties that represent regions, departments or even individual data collectors. + +If you are able to prevent or reduce the risk of conflicts, you can skip this section! If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will need to understand conflicts. + +.. _central-entities-update-conflicts-what-is: + +What is a parallel update? +"""""""""""""""""""""""""" + +What does it mean that updating an Entity "at the same time" causes a conflict? Every time an Entity is updated, whether by direct edits on the Central management panel, through the API, or by a Submission upload, a new "version" of that Entity is created that has the latest data. + +Say Alice fetches all her Tree data on her device. This update includes a brand new Tree entity called "Zach's backyard tree". Alice updates Zach's tree with new measurements. This results in version 2 of that tree once the Submission is uploaded to Central. In a "clean," not-conflicting update Bob would get the latest Tree data, see version 2 of Zach's tree in his update Form, and upload his new changes to create version 3 of the tree on Central. Version 1 led to version 2, and version 2 was updated to create version 3. + +However, if Bob doesn't update his data before filling his Form, and so he's still working off version 1 of the tree, then a conflict results. Two updates have been created based on version 1, and Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob are both looking at version 1 and trying to create version 2. + +.. note:: + The same rules and behaviors apply when updating Entities through updates made directly to Entities in Central's management panel, or over the API. It's possible, for example, that Bob opened an Entity data edit window in Central before Alice's update is uploaded, but he didn't actually press Submit on his changes until much later. The same is true: Bob is looking at version 1 and thinking he's creating version 2. + +Central's behavior in these cases is basic: **it will always apply any changes it receives at the time and in the order it receives them**. This means that in this case, Bob's changes will always "win" over Alice's. Any time this happens, Central will generate a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. + +Of course, it's possible that Bob's changes and Alice's changes are totally compatible. For example, it's possible Alice only updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. How often and severely conflicts create actual consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. Central attempts to help Project managers determine the severity of conflicts by separating conflict warnings into two levels of severity. + +**All** conflicts surfaced by Central occur because two Entity updates were generated based off the same base version of an Entity. This circumstance is called a :guilabel:`Parallel Update` in Central. In the special case that a property included in an update has also been changed in the meantime, *and* the value currently stored in Central is different from the new updated value, a full :guilabel:`Conflict` error will be created instead, indicating that multiple possibly conflicting values were written into the same property at once. So if Alice and Bob have both updated the circumference of the tree at the same time to the same value, only a Parallel Update warning and not a Conflict will be flagged. Only if their inputs disagree will you see a full Conflict. + .. _central-entities-managing: Managing Entities diff --git a/docs/entities-intro.rst b/docs/entities-intro.rst index a50c58381..725cd94f8 100644 --- a/docs/entities-intro.rst +++ b/docs/entities-intro.rst @@ -137,7 +137,7 @@ Not yet, but this is something we will eventually support. If you find yourself wanting to create or update multiple Entities of the same type in a repeat, your best option currently is to use multiple submissions of the same form instead of a repeat. You can capture base information in one form and then use a separate form to create each Entity that you currently represent by repeat instances. -If there is a parent-child relationship between the different Entities, you can save the parent's ID to each child. If you are working in an environment with Internet connectivity, you can refresh the forms to see your created parent Entities in your child Entity creation forms. If you are working in a disconnected environment, you can have data collectors copy the ID from the parent form to the child forms. +If there is a parent-child relationship between the different Entities, you can save the parent's ID to each child. If your versions of Central and Collect support offline Entities, parent Entities will be available to other forms the moment the registration form is finalized. This means the registration form for the child Entities can include a question to select the parent Entity which will establish the link between the two. Similarly, if you'd like to establish relationships between multiple Entities of different types, you can have a registration form for each type and include a field to represent a link to another Entity. From 5499912e99cc2feeac24f6aaa909e7ccf34ff67e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 09:42:30 -0800 Subject: [PATCH 04/11] Clarify conditional actions --- docs/central-entities.rst | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index 6ae651df0..ee174cace 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -316,7 +316,7 @@ You can use Forms to update Entity information. These Forms can be authored to, Building a Form that updates an Entity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Updating Entities isn't too different from creating new Entities, but you'll have to specify the ``entity_id`` of the specific Entity instance you want to update in the ``entities`` sheet. +Updating an Entity with a form submission is similar to creating a new Entity. You'll also specify the ``list_name`` in the ``entities`` sheet and will also have to specify the ``entity_id`` of the specific Entity instance you want to update. You can optionally update the ``label`` as well. .. rubric:: XLSForm @@ -359,23 +359,25 @@ To update the label of an Entity from a Form, fill in the optional ``label`` col Setting conditions under which an Entity is updated ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You can limit a Form to only update an Entity under certain conditions, based on an expression using the Entity and Submission data. +You can limit a Form to only update an Entity under certain conditions, based on an expression. The Entity is only updated when the expression evaluates to ``True``. .. rubric:: XLSForm .. csv-table:: entities :header: list_name, entity_id, update_if - trees, ${tree}, true() + trees, ${tree}, ${approval} != "" -In this case, ``true()`` is given in the optional ``update_if`` column, which means the Entity will always be updated upon submission. If an expression is given instead, the Entity is only updated when the expression evaluates to ``true`` or ``1``. +In this case, the Entity will only be updated if the value of the ``approval`` field in the form is non-blank. If the ``approval`` field has a blank value, the submission will still be sent but it will have no impact on the corresponding Entity. .. _central-entities_build-update-create: Creating AND Updating Entities with one Form ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You can give both a ``create_if`` and an ``update_if`` rule for the same Form. If only one of these expression evaluates to ``true`` or ``1``, then only that operation will be carried out. If both rules evaluate to ``true`` or ``1``, the Entity will be created if it does not exist (as identified by the ``entity_id`` expression), and updated if it does. Of course, if neither rule evaluates to ``true`` or ``1``, no Entity changes will occur. +You can give both a ``create_if`` and an ``update_if`` rule for the same Form. If only one of these expression evaluates to ``true`` or ``1``, then only that operation will be carried out. If both rules evaluate to ``true`` or ``1``, the Entity will be created if it does not exist (as identified by the ``entity_id`` expression), and updated if it does. If neither rule evaluates to ``true`` or ``1``, no Entity changes will occur. + +See a sample form `here `__. .. _central-entities-update-conflicts: From 4a3981fcb8a2dee49971ef9e59f0e17edcfd541d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 13:33:48 -0800 Subject: [PATCH 05/11] Reorganize and cross link conflicts sections --- docs/central-entities.rst | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index ee174cace..173e55035 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -309,7 +309,7 @@ Updating Entities from Forms You can use Forms to update Entity information. These Forms can be authored to, for example, update previous observations to new values or change the status of an Entity. Just like Entity creation through Forms, you can specify which properties on which Entity instances will be updated when the Form's Submissions are uploaded to Central. The data in Submissions uploaded by Entity-updating Forms are applied to the Entity data saved on the Central server. These updated Entity values are then distributed to data collection clients once they synchronize with Central. .. note:: - Entity updates from form submissions open up the possibility of two data collectors attempting to update the same Entity at the same time. This case is detected by Central and shown as a conflict. If your workflow involves data collectors who are offline for a long time or many updates to the same Entity, be sure to read the section on `parallel updates `. + Entity updates from form submissions open up the possibility of two data collectors attempting to update the same Entity at the same time. This case is detected by Central and shown as a conflict. If your workflow involves many updates to the same Entity, especially from data collectors who are offline for a long time, be sure to read the section on :ref:`parallel updates `. .. _central-entities-build-update: @@ -381,34 +381,47 @@ See a sample form `here ` to only show each data collector the Entities that they are assigned to. You can filter based on Entity properties that represent regions, departments or even individual data collectors. -If you are able to prevent or reduce the risk of conflicts, you can skip this section! If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will need to understand conflicts. +When Central v2024.3.0 or later is used with Collect v2024.3.0 or later, Collect maintains its own offline representation of Entity Lists. This means that any Entity create or update by form submission is reflected locally the moment that submission is finalized. When a chain of multiple updates to the same Entity can be created offline, these can sometimes arrive out of order which will also be marked as conflicts by Central. + +If you are able to prevent or reduce the risk of conflicts and don't expect chains of offline updates, you can skip this section! If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will need to understand conflicts. .. _central-entities-update-conflicts-what-is: What is a parallel update? """""""""""""""""""""""""" -What does it mean that updating an Entity "at the same time" causes a conflict? Every time an Entity is updated, whether by direct edits on the Central management panel, through the API, or by a Submission upload, a new "version" of that Entity is created that has the latest data. +Every time an Entity is updated, a new version of that Entity is created. This versioning is the same whether the Entity is updated by :ref:`direct edits ` from Central, through `the API `__, or by :ref:`a Submission upload `. While direct edits can only be submitted if they're based on the latest version available, updates by Submission or through the API can be based on older Entity versions. These updates based on older versions are called parallel updates. -Say Alice fetches all her Tree data on her device. This update includes a brand new Tree entity called "Zach's backyard tree". Alice updates Zach's tree with new measurements. This results in version 2 of that tree once the Submission is uploaded to Central. In a "clean," not-conflicting update Bob would get the latest Tree data, see version 2 of Zach's tree in his update Form, and upload his new changes to create version 3 of the tree on Central. Version 1 led to version 2, and version 2 was updated to create version 3. +Here's an example of how parallel updates can happen: -However, if Bob doesn't update his data before filling his Form, and so he's still working off version 1 of the tree, then a conflict results. Two updates have been created based on version 1, and Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob are both looking at version 1 and trying to create version 2. +#. Alice gets a form update to her device, including an update to a ``trees`` Entity List with a brand new tree Entity called "Zach's backyard tree". This new Entity has version 1. +#. Bob also updates his device and gets the same ``trees`` Entity List with version 1 of "Zach's backyard tree". +#. Alice fills out a Form that updates "Zach's backyard tree" with new measurements. +#. Central receives Alice's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 2 of that tree Entity. +#. Bob also visits "Zach's backyard tree" and fills out a Form to update its measurements. +#. Central receives Bob's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 3 of that tree Entity. Because Bob based his update on version 1, just as Alice did, his update is considered a parallel update. -.. note:: - The same rules and behaviors apply when updating Entities through updates made directly to Entities in Central's management panel, or over the API. It's possible, for example, that Bob opened an Entity data edit window in Central before Alice's update is uploaded, but he didn't actually press Submit on his changes until much later. The same is true: Bob is looking at version 1 and thinking he's creating version 2. +In the example above, Bob was still working off version 1 of the tree, resulting in a conflict. Two updates were created based on version 1 because Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob were both looking at version 1 and trying to create version 2. Central applied Bob's update to create version 3 and marked "Zach's backyard tree" as being in a state of conflict. + +In a "clean," not-conflicting update Bob would have received the latest ``trees`` data before visiting "Zach's backyard tree" so that he would have version 2 on his device. When he made his update, it would have been based off of version 2 and Central would have applied it on version 2 to create version 3. + +Addressing parallel updates +""""""""""""""""""""""""""""" + +Central's behavior when there are parallel updates is basic: **it always applies any changes it receives at the time and in the order it receives them**. This means that in the case above, Bob's changes will always "win" over Alice's because they are submitted last (this is sometimes called a last-write-wins strategy). Central has simple, predictable behavior that attempts to always use the latest available data. It also detects and shows conflict cases so that you can resolve them in more sophisticated ways as needed. If there are Entities in a conflict state, there will be a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. -Central's behavior in these cases is basic: **it will always apply any changes it receives at the time and in the order it receives them**. This means that in this case, Bob's changes will always "win" over Alice's. Any time this happens, Central will generate a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. +How often and how severely conflicts create real consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. When looking at a specific Entity, any versions that were created from a parallel update will be labeled as either a :guilabel:`Parallel update` or a :guilabel:`Conflict`. These two conflict levels are intended to help Project managers determine the severity of those conflicts. -Of course, it's possible that Bob's changes and Alice's changes are totally compatible. For example, it's possible Alice only updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. How often and severely conflicts create actual consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. Central attempts to help Project managers determine the severity of conflicts by separating conflict warnings into two levels of severity. +An Entity version with the :guilabel:`Parallel update` label was created from an update based on an older version but it only updated properties that were not involved in the other parallel update(s). For example, in the example above, Alice might have updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. These two changes are totally compatible and it doesn't matter that Bob based his update on an outdated version of "Zach's backyard tree" so the conflict can be quickly dismissed. In some workflows, this kind of parallel update is expected. Central surfaces it because in many contexts it can point to a process failure. For example, Alice and Bob may have been supposed to go to different regions on the day they both visited "Zach's backyard tree." -**All** conflicts surfaced by Central occur because two Entity updates were generated based off the same base version of an Entity. This circumstance is called a :guilabel:`Parallel Update` in Central. In the special case that a property included in an update has also been changed in the meantime, *and* the value currently stored in Central is different from the new updated value, a full :guilabel:`Conflict` error will be created instead, indicating that multiple possibly conflicting values were written into the same property at once. So if Alice and Bob have both updated the circumference of the tree at the same time to the same value, only a Parallel Update warning and not a Conflict will be flagged. Only if their inputs disagree will you see a full Conflict. +An Entity version with the :guilabel:`Conflict` label means that this version was created from an update based on an older version **AND** one or more of the parallel updates modified the same properties. In the example above, maybe Alice and Bob both updated the circumference to different values. This kind of conflict almost always means that something went wrong. You should carefully review the Entity history and make any edits needed to the latest Entity version before dismissing the conflict. You may need to get in contact with the individuals who made the conflicting updates to figure out which value is correct. .. _central-entities-managing: @@ -479,7 +492,7 @@ To complete the process press the :guilabel:`Update` button to save your changes Managing Entity conflicts ------------------------- -When an Entity is in a possible conflict, Central will raise the issue in many places throughout the management panel: in the homepage Entity Lists tables, in Entities tables on Project pages, in the Entity Data table, and on Entity Detail pages. +When an Entity is in a :ref:`possible conflict `, Central will raise the issue in many places throughout the management panel: in the homepage Entity Lists tables, in Entities tables on Project pages, in the Entity Data table, and on Entity Detail pages. You can review and dismiss conflicts from the :ref:`Entity table `, or from the :ref:`Entity Detail page `. When you dismiss a conflict warning, the warning goes away and whatever values are currently recorded in Central are considered correct. From 5a620f2b7f7d0da737ba116c425412e3b9e3635a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 13:42:20 -0800 Subject: [PATCH 06/11] Combine conflict management and description sections --- docs/central-entities.rst | 89 ++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 48 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index 173e55035..f3d17f2df 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -309,7 +309,7 @@ Updating Entities from Forms You can use Forms to update Entity information. These Forms can be authored to, for example, update previous observations to new values or change the status of an Entity. Just like Entity creation through Forms, you can specify which properties on which Entity instances will be updated when the Form's Submissions are uploaded to Central. The data in Submissions uploaded by Entity-updating Forms are applied to the Entity data saved on the Central server. These updated Entity values are then distributed to data collection clients once they synchronize with Central. .. note:: - Entity updates from form submissions open up the possibility of two data collectors attempting to update the same Entity at the same time. This case is detected by Central and shown as a conflict. If your workflow involves many updates to the same Entity, especially from data collectors who are offline for a long time, be sure to read the section on :ref:`parallel updates `. + Entity updates from form submissions open up the possibility of two data collectors attempting to update the same Entity at the same time. This case is detected by Central and shown as a conflict. If your workflow involves many updates to the same Entity, especially from data collectors who are offline for a long time, be sure to read the section on :ref:`Entity conflicts `. .. _central-entities-build-update: @@ -379,50 +379,6 @@ You can give both a ``create_if`` and an ``update_if`` rule for the same Form. I See a sample form `here `__. -.. _central-entities-update-conflicts: - -Parallel Updates and conflicts -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Updates from form submissions open up the possibility of two people attempting to update the same Entity at the same time. Central always applies all updates it receives in the order they are received. If it detects that two updates to the same Entity were made in parallel, it marks that Entity as possibly being in a conflict state. A Project Manager can then look at the conflict and decide how to resolve it. - -If you can, try to organize work so that conflicts are unlikely. For example, you can use :ref:`choice filters ` to only show each data collector the Entities that they are assigned to. You can filter based on Entity properties that represent regions, departments or even individual data collectors. - -When Central v2024.3.0 or later is used with Collect v2024.3.0 or later, Collect maintains its own offline representation of Entity Lists. This means that any Entity create or update by form submission is reflected locally the moment that submission is finalized. When a chain of multiple updates to the same Entity can be created offline, these can sometimes arrive out of order which will also be marked as conflicts by Central. - -If you are able to prevent or reduce the risk of conflicts and don't expect chains of offline updates, you can skip this section! If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will need to understand conflicts. - -.. _central-entities-update-conflicts-what-is: - -What is a parallel update? -"""""""""""""""""""""""""" - -Every time an Entity is updated, a new version of that Entity is created. This versioning is the same whether the Entity is updated by :ref:`direct edits ` from Central, through `the API `__, or by :ref:`a Submission upload `. While direct edits can only be submitted if they're based on the latest version available, updates by Submission or through the API can be based on older Entity versions. These updates based on older versions are called parallel updates. - -Here's an example of how parallel updates can happen: - -#. Alice gets a form update to her device, including an update to a ``trees`` Entity List with a brand new tree Entity called "Zach's backyard tree". This new Entity has version 1. -#. Bob also updates his device and gets the same ``trees`` Entity List with version 1 of "Zach's backyard tree". -#. Alice fills out a Form that updates "Zach's backyard tree" with new measurements. -#. Central receives Alice's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 2 of that tree Entity. -#. Bob also visits "Zach's backyard tree" and fills out a Form to update its measurements. -#. Central receives Bob's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 3 of that tree Entity. Because Bob based his update on version 1, just as Alice did, his update is considered a parallel update. - -In the example above, Bob was still working off version 1 of the tree, resulting in a conflict. Two updates were created based on version 1 because Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob were both looking at version 1 and trying to create version 2. Central applied Bob's update to create version 3 and marked "Zach's backyard tree" as being in a state of conflict. - -In a "clean," not-conflicting update Bob would have received the latest ``trees`` data before visiting "Zach's backyard tree" so that he would have version 2 on his device. When he made his update, it would have been based off of version 2 and Central would have applied it on version 2 to create version 3. - -Addressing parallel updates -""""""""""""""""""""""""""""" - -Central's behavior when there are parallel updates is basic: **it always applies any changes it receives at the time and in the order it receives them**. This means that in the case above, Bob's changes will always "win" over Alice's because they are submitted last (this is sometimes called a last-write-wins strategy). Central has simple, predictable behavior that attempts to always use the latest available data. It also detects and shows conflict cases so that you can resolve them in more sophisticated ways as needed. If there are Entities in a conflict state, there will be a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. - -How often and how severely conflicts create real consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. When looking at a specific Entity, any versions that were created from a parallel update will be labeled as either a :guilabel:`Parallel update` or a :guilabel:`Conflict`. These two conflict levels are intended to help Project managers determine the severity of those conflicts. - -An Entity version with the :guilabel:`Parallel update` label was created from an update based on an older version but it only updated properties that were not involved in the other parallel update(s). For example, in the example above, Alice might have updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. These two changes are totally compatible and it doesn't matter that Bob based his update on an outdated version of "Zach's backyard tree" so the conflict can be quickly dismissed. In some workflows, this kind of parallel update is expected. Central surfaces it because in many contexts it can point to a process failure. For example, Alice and Bob may have been supposed to go to different regions on the day they both visited "Zach's backyard tree." - -An Entity version with the :guilabel:`Conflict` label means that this version was created from an update based on an older version **AND** one or more of the parallel updates modified the same properties. In the example above, maybe Alice and Bob both updated the circumference to different values. This kind of conflict almost always means that something went wrong. You should carefully review the Entity history and make any edits needed to the latest Entity version before dismissing the conflict. You may need to get in contact with the individuals who made the conflicting updates to figure out which value is correct. - .. _central-entities-managing: Managing Entities @@ -487,15 +443,52 @@ As you type, Central will highlight any fields you have changed in yellow so you To complete the process press the :guilabel:`Update` button to save your changes to the Entity. You will see a confirmation that the save succeeded. If you don't like your changes, you can click on the :guilabel:`x` or the :guilabel:`Never mind` link to close the dialog. -.. _central-entities-manage-conflicts: +.. _central-entities-update-conflicts: Managing Entity conflicts -------------------------- +------------------------------- -When an Entity is in a :ref:`possible conflict `, Central will raise the issue in many places throughout the management panel: in the homepage Entity Lists tables, in Entities tables on Project pages, in the Entity Data table, and on Entity Detail pages. +It's possible for two people to attempt to update the same Entity at the same time. Central always applies all updates it receives in the order they are received and if it detects that two updates to the same Entity were made in parallel, it marks that Entity as possibly being in a conflict state. When an Entity is in a possible conflict, Central will raise the issue in many places throughout the management panel: in the homepage Entity Lists tables, in Entities tables on Project pages, in the Entity Data table, and on Entity Detail pages. You can review and dismiss conflicts from the :ref:`Entity table `, or from the :ref:`Entity Detail page `. When you dismiss a conflict warning, the warning goes away and whatever values are currently recorded in Central are considered correct. +If you can, try to organize work so that conflicts are unlikely. For example, you can use :ref:`choice filters ` to only show each data collector the Entities that they are assigned to. You can filter based on Entity properties that represent regions, departments or even individual data collectors. + +When Central v2024.3.0 or later is used with Collect v2024.3.0 or later, Collect maintains its own offline representation of Entity Lists. This means that any Entity create or update by form submission is reflected locally the moment that submission is finalized. When a chain of multiple updates to the same Entity can be created offline, these can sometimes arrive out of order which will also be marked as conflicts by Central. + +If you are able to prevent or reduce the risk of conflicts and don't expect chains of offline updates, you can skip this section. If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will likely need to understand conflicts. + +.. _central-entities-update-conflicts-what-is: + +What is a parallel update? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Every time an Entity is updated, a new version of that Entity is created. This process is the same whether the Entity is updated by :ref:`direct edits ` from Central, through `the API `__, or by :ref:`a Submission upload `. While direct edits can only be submitted if they're based on the latest version available, updates by Submission or through the API can be based on older Entity versions. These updates based on older versions are called parallel updates. + +Here's an example of how parallel updates can happen: + +#. Alice gets a form update to her device, including an update to a ``trees`` Entity List with a brand new tree Entity called "Zach's backyard tree". This new Entity has version 1. +#. Bob also updates his device and gets the same ``trees`` Entity List with version 1 of "Zach's backyard tree". +#. Alice fills out a Form that updates "Zach's backyard tree" with new measurements. +#. Central receives Alice's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 2 of that tree Entity. +#. Bob also visits "Zach's backyard tree" and fills out a Form to update its measurements. +#. Central receives Bob's Submission and applies it to "Zach's backyard tree". Central applies the update to create version 3 of that tree Entity. Because Bob based his update on version 1, just as Alice did, his update is considered a parallel update. + +In the example above, Bob was still working off version 1 of the tree, resulting in a conflict. Two updates were created based on version 1 because Bob didn't know that Alice had already made the changes resulting in version 2. Alice and Bob were both looking at version 1 and trying to create version 2. Central applied Bob's update to create version 3 and marked "Zach's backyard tree" as being in a state of conflict. + +In a "clean," not-conflicting update Bob would have received the latest ``trees`` data before visiting "Zach's backyard tree" so that he would have version 2 on his device. When he made his update, it would have been based off of version 2 and Central would have applied it on version 2 to create version 3. + +Addressing parallel updates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Central's behavior when there are parallel updates is basic: **it always applies any changes it receives at the time and in the order it receives them**. This means that in the case above, Bob's changes will always "win" over Alice's because they are submitted last (this is sometimes called a last-write-wins strategy). Central has simple, predictable behavior that attempts to always use the latest available data. It also detects and shows conflict cases so that you can resolve them in more sophisticated ways as needed. If there are Entities in a conflict state, there will be a :guilabel:`conflict` warning in several places in the Central management panel so Project managers can see that a problem might have occurred. + +How often and how severely conflicts create real consistency problems with Entity data will depend on the design and complexity of your data collection process and your Forms. When looking at a specific Entity, any versions that were created from a parallel update will be labeled as either a :guilabel:`Parallel update` or a :guilabel:`Conflict`. These two conflict levels are intended to help Project managers determine the severity of those conflicts. + +An Entity version with the :guilabel:`Parallel update` label was created from an update based on an older version but it only updated properties that were not involved in the other parallel update(s). For example, in the example above, Alice might have updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. These two changes are totally compatible and it doesn't matter that Bob based his update on an outdated version of "Zach's backyard tree" so the conflict can be quickly dismissed. In some workflows, this kind of parallel update is expected. Central surfaces it because in many contexts it can point to a process failure. For example, Alice and Bob may have been supposed to go to different regions on the day they both visited "Zach's backyard tree." + +An Entity version with the :guilabel:`Conflict` label means that this version was created from an update based on an older version **AND** one or more of the parallel updates modified the same properties. In the example above, maybe Alice and Bob both updated the circumference to different values. This kind of conflict almost always means that something went wrong. You should carefully review the Entity history and make any edits needed to the latest Entity version before dismissing the conflict. You may need to get in contact with the individuals who made the conflicting updates to figure out which value is correct. + .. _central-entities-settings: Changing Entity List Settings From 0760cecdbff007740404d5effeb7abf6e4d2e97e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 14:50:30 -0800 Subject: [PATCH 07/11] Add info about offline Entities --- docs/central-entities.rst | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index f3d17f2df..b852f82c6 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -6,9 +6,8 @@ Managing Entities in Central **Entities** in ODK let you share information between Forms so you can collect longitudinal data, manage cases over time, and support other complex workflows. .. seealso:: - -* :doc:`Entities intro `: similar information organized as a list of questions and answers -* :doc:`Community reporting tutorial ` + * :doc:`Entities intro `: similar information organized as a list of questions and answers + * :doc:`Community reporting tutorial ` Without Entities, you can attach existing data during the Form creation process, often through ``choices`` sheets or ``.csv`` files. Once this data is loaded into the Form, you can use it as the source for selection choices, to prepopulate data fields in the Form, or validate new input. A list of districts, for example, can be used as choices for selection, and information about each district can then be shown to the user or checked against. If you are new to these techniques, you can learn more about them on the :doc:`Form Datasets ` page. @@ -454,13 +453,13 @@ You can review and dismiss conflicts from the :ref:`Entity table ` to only show each data collector the Entities that they are assigned to. You can filter based on Entity properties that represent regions, departments or even individual data collectors. -When Central v2024.3.0 or later is used with Collect v2024.3.0 or later, Collect maintains its own offline representation of Entity Lists. This means that any Entity create or update by form submission is reflected locally the moment that submission is finalized. When a chain of multiple updates to the same Entity can be created offline, these can sometimes arrive out of order which will also be marked as conflicts by Central. +Some special cases that can occur when an Entity is updated multiple times while offline are also marked by Central as conflicts. If you are able to prevent or reduce the risk of conflicts and don't expect chains of offline updates, you can skip this section. If your workflow is highly dynamic with the same Entities being periodically encountered or if your data collectors spend long periods of time offline, you will likely need to understand conflicts. .. _central-entities-update-conflicts-what-is: -What is a parallel update? +What are Entity conflicts? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Every time an Entity is updated, a new version of that Entity is created. This process is the same whether the Entity is updated by :ref:`direct edits ` from Central, through `the API `__, or by :ref:`a Submission upload `. While direct edits can only be submitted if they're based on the latest version available, updates by Submission or through the API can be based on older Entity versions. These updates based on older versions are called parallel updates. @@ -487,7 +486,22 @@ How often and how severely conflicts create real consistency problems with Entit An Entity version with the :guilabel:`Parallel update` label was created from an update based on an older version but it only updated properties that were not involved in the other parallel update(s). For example, in the example above, Alice might have updated the circumference of the tree's trunk, while Bob only reported that moss is now growing. These two changes are totally compatible and it doesn't matter that Bob based his update on an outdated version of "Zach's backyard tree" so the conflict can be quickly dismissed. In some workflows, this kind of parallel update is expected. Central surfaces it because in many contexts it can point to a process failure. For example, Alice and Bob may have been supposed to go to different regions on the day they both visited "Zach's backyard tree." -An Entity version with the :guilabel:`Conflict` label means that this version was created from an update based on an older version **AND** one or more of the parallel updates modified the same properties. In the example above, maybe Alice and Bob both updated the circumference to different values. This kind of conflict almost always means that something went wrong. You should carefully review the Entity history and make any edits needed to the latest Entity version before dismissing the conflict. You may need to get in contact with the individuals who made the conflicting updates to figure out which value is correct. +An Entity version with the :guilabel:`Conflict` label means that this version was created from an update based on an older version **AND** one or more of the parallel updates modified the same properties. In the example above, maybe Alice and Bob both updated the circumference to different values. This kind of conflict almost always means that something went wrong. You should carefully review the Entity history and make any edits needed to the latest Entity version before dismissing the conflict. You may need to get in contact with the individuals who made the conflicting updates to figure out which value is correct. The :guilabel:`Conflict` label is also used in certain special cases related to offline Entity updates. + +Conflicts related to Offline Entities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When Central v2024.3.0 or later is used with Collect v2024.3.0 or later, Collect maintains its own offline representation of Entity Lists. This means that any Entity create or update by Form Submission is reflected in other follow-up forms the moment that Submission is finalized. + +Central will adds the label :guilabel:`Offline update` to any Entity version that was created from an update on a device that did not get a server update in between the latest update and the prior Entity action. For example, if Alice is working with a ``trees`` Entity list, registers "Zach's backyard tree", fills out and finalizes the "Update circumference" form, and then submits both without having received a server update in between the two, version 2 of "Zach's backyard tree" will have the label :guilabel:`Offline update`. + +Collect v2024.3.0 or later attempts to send Submissions in the order they were created. However, this may not always be possible. If Central receives a Submission that is an offline update based on an earlier Submission it has not received yet, it will hold the newer Submission for up to 5 days. If the earlier Submission(s) arrive within 5 days, all updates in the offline series will be immediately processed in order. If, after 5 days, earlier Submissions have not been received, the held Submission(s) will be force processed and the Entity will be marked as having a conflict. + +If an Entity is created and then updated offline, there is the possibility that the Form Submissions representing updates will be received by Central before the Submission representing the creation of the Entity. In that case, the update Submissions will be held for up to 5 days. If the create Submission is received any time before 5 days, the Entity will be created by Central and any held updates will immediately be processed in order. If the Submission that would have created the Entity is not received within 5 days, Central will process the first update Submission it has as a create and the Entity will be marked as having a conflict. + +Information about out-of-order submission processing is shown on each Submission's :ref:`detail page `. + +When multiple different users can go offline and each create several offline updates to the same Entity, conflicts can be hard to understand. If you see conflict situations that you are unable to understand from the information shown in Central, please post to `the forum `_. .. _central-entities-settings: From 72ccf8dcc714be18698710abd3078bc5dec7cafb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 14:56:40 -0800 Subject: [PATCH 08/11] Update section link --- docs/entities-intro.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/entities-intro.rst b/docs/entities-intro.rst index 725cd94f8..8615c3d99 100644 --- a/docs/entities-intro.rst +++ b/docs/entities-intro.rst @@ -108,7 +108,7 @@ Limitations I filled out a registration form and don't immediately see my Entity in follow-up forms, why? --------------------------------------------------------------------------------------------- -If you are using a version of ODK Central older than 2024.3.0, Enketo web forms, or a version of Collect older than 2024.3.0, Entities are not created or updated offline. This means that in order for a submission to create or update an Entity, that submission has to be processed by your server. If you create a new Entity or update an existing one by filling out a form, you won't see that change reflected in follow-up forms until you download the latest update to your Entity List from your server. +If you are using a version of Central older than 2024.3.0, Enketo web forms, or a version of Collect older than 2024.3.0, Entities are not created or updated offline. This means that in order for a submission to create or update an Entity, that submission has to be processed by your server. If you create a new Entity or update an existing one by filling out a form, you won't see that change reflected in follow-up forms until you download the latest update to your Entity List from your server. If you usually have Internet connectivity, this is unlikely to be very important. Similarly, if your registration and follow-up periods happen at very different times, this limitation is not a problem. But for workflows in which follow-up needs to happen immediately after registration or multiple follow-ups are needed while offline, this limitation is significant. @@ -253,7 +253,7 @@ One of our goals with Entities is to let field staff make as much progress as po When possible, we recommend using Entity properties and a :ref:`choice_filter ` to limit the number of Entities that a specific field worker sees. This will greatly reduce the chance of conflicts. .. seealso:: - * :ref:`Entity updates from submissions ` + * :ref:`Managing Entity conflicts ` Should I analyze Entity data, form submission data or both? ----------------------------------------------------------- From c8fc759f3528914f496051856d6eb0a2a4b6978a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 15:00:22 -0800 Subject: [PATCH 09/11] Improve wording --- docs/entities-intro.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/entities-intro.rst b/docs/entities-intro.rst index 8615c3d99..cbda1b64c 100644 --- a/docs/entities-intro.rst +++ b/docs/entities-intro.rst @@ -246,9 +246,9 @@ Currently, once a property is added to an Entity List, it can't be removed. You What are Entity conflicts and what can I do to avoid them? ---------------------------------------------------------- -A conflict happens when two form submissions that are received by the server have updates to the same Entity with the same version. If the two submissions specify different, overlapping updates to one or more properties, the server will provide an interface for resolving it. All conflicts have to be explicitly dismissed. +A conflict happens when two form Submissions both representing updates to the same Entity with the same version are received by the server. If the two Submissions specify different, overlapping updates to one or more properties, Central will provide an interface for understanding and resolving the conflict. All conflicts have to be explicitly dismissed. -One of our goals with Entities is to let field staff make as much progress as possible without interruption so the server applies conflicting updates with the latest one taking precedence. The conflict is shown on the server and office staff can look at the submitted data and work with field staff to resolve the issue. +One of our goals with Entities is to let field staff make as much progress as possible without interruption. For this reason, Central uses a last-write-wins strategy and applies all Entity updates it receives. Conflicts are shown from Central so that project administrators can look at the submitted data and work with field staff to resolve the issue. When possible, we recommend using Entity properties and a :ref:`choice_filter ` to limit the number of Entities that a specific field worker sees. This will greatly reduce the chance of conflicts. From 31c88eb2487dee2667f02f5dc2e2caca93435051 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 16:59:50 -0800 Subject: [PATCH 10/11] Add offline Entities to available now --- docs/central-entities.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index b852f82c6..1f56e9915 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -56,9 +56,10 @@ Entities are a big new concept that open up a lot of new possibilities. While we What's available now: -- Create an Entity with a registration form (automatically or after project manager approval) -- Use multiple different registration forms targeting the same Entity List (e.g., registration at school vs. registration at home) -- Use Entities in one or more follow-up form +- Create an Entity with a registration Form (automatically or after project manager approval) +- Use multiple different registration Forms targeting the same Entity List (e.g., registration at school vs. registration at home) +- Use or update Entities in one or more follow-up Forms +- Create or update Entities offline using Collect v2024.3 or later - Bulk create many Entities by uploading a .csv data file into an existing Entity List - Download Entities into Power BI, Excel, Python, and R @@ -67,7 +68,7 @@ Important limitations: - A form submission can only create or update a single Entity - Client performance (ODK Collect or Enketo web forms) suffers when managing more than 50,000 Entities - All devices will always download all Entities which may be a privacy concern -- Currently, Entity properties may only be strings. Submission values saved to an Entity are converted to strings. +- Entity properties may only be strings. Submission values saved to an Entity are converted to strings. We expect all of these limitations to be addressed over time. The roadmap at `getodk.org/roadmap `_ has what we're working on now and what's coming next. From 1c9c73736d63391be5ebc1eacf3312b339c5ff2b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?H=C3=A9l=C3=A8ne=20Martin?= Date: Mon, 16 Dec 2024 17:02:14 -0800 Subject: [PATCH 11/11] Add reasoning for processing queue --- docs/central-entities.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/central-entities.rst b/docs/central-entities.rst index 1f56e9915..12404d582 100644 --- a/docs/central-entities.rst +++ b/docs/central-entities.rst @@ -498,7 +498,7 @@ Central will adds the label :guilabel:`Offline update` to any Entity version tha Collect v2024.3.0 or later attempts to send Submissions in the order they were created. However, this may not always be possible. If Central receives a Submission that is an offline update based on an earlier Submission it has not received yet, it will hold the newer Submission for up to 5 days. If the earlier Submission(s) arrive within 5 days, all updates in the offline series will be immediately processed in order. If, after 5 days, earlier Submissions have not been received, the held Submission(s) will be force processed and the Entity will be marked as having a conflict. -If an Entity is created and then updated offline, there is the possibility that the Form Submissions representing updates will be received by Central before the Submission representing the creation of the Entity. In that case, the update Submissions will be held for up to 5 days. If the create Submission is received any time before 5 days, the Entity will be created by Central and any held updates will immediately be processed in order. If the Submission that would have created the Entity is not received within 5 days, Central will process the first update Submission it has as a create and the Entity will be marked as having a conflict. +If an Entity is created and then updated offline, there is the possibility that the Form Submissions representing updates will be received by Central before the Submission representing the creation of the Entity. In that case, the update Submissions will be held for up to 5 days. If the create Submission is received any time before 5 days, the Entity will be created by Central and any held updates will immediately be processed in order. If the Submission that would have created the Entity is not received within 5 days, Central will process the first update Submission it has as a create and the Entity will be marked as having a conflict. Our goal is to provide as much information as is available to follow-up forms while letting project administrators know that something may have gone wrong. Information about out-of-order submission processing is shown on each Submission's :ref:`detail page `.