From 042e43811c9dbf2d7db8c58633f5f0f236db89d1 Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Thu, 5 Oct 2023 15:07:55 +0100 Subject: [PATCH 1/5] Added link to master repo for LR --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 53da16f4..6295b1b7 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ This is the repo for the application that presents UKHPI open data on behalf of Land Registry (England and Wales), Registers of Scotland, Land and Property Services (Northern Ireland) and the UK Office for National Statistics (ONS). +Please see the other repositories in the [HM Land Registry Open +Data](https://github.com/epimorphics/hmlr-linked-data/) project for more +details. + Development work was carried out by [Epimorphics Ltd](http://www.epimorphics.com), funded by [HM Land Registry](https://www.gov.uk/government/organisations/land-registry). From b433370b9df2adb0e930048133b29507ea07e40e Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Thu, 5 Oct 2023 15:08:58 +0100 Subject: [PATCH 2/5] Refactored readme headings --- README.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 6295b1b7..6b734d3b 100644 --- a/README.md +++ b/README.md @@ -134,10 +134,7 @@ We represent these as follows: | | % monthly change | Existing properties | | | % annual change | Existing properties | | | sales volume | Existing properties - -## Developer notes - -### Installing dependencies +## Installing dependencies The app currently depends on Ruby version 2.6.x. The actual version is specified in the `.ruby-version` file, which can be used with @@ -155,7 +152,9 @@ bundle install yarn install ``` -### Accessing the Data API locally +## Running the data API locally + +### Prerequisites The application communicates with the HMLR data API (which uses Sapi-NT) to provide the data to be displayed. The actual API location is specified by the @@ -200,7 +199,7 @@ To use a credential helper for a specific ECR registry[^1], create a The local application can then connect to the triple store via the `data-api` service. -### Running the Data API locally +### Running the API The easiest way to do this is via a local docker container. The `data-api` image can be built from [lr-data-api @@ -265,7 +264,9 @@ If needed, to create the docker network run: docker network create dnet ``` -### Running the app locally +## Running the app + +### Locally Assuming the `Data API` is running on `http://localhost:8888` (the default), to start the app locally: @@ -286,7 +287,7 @@ And then visit . N.B. The default for `RAILS_ENV` is `production` if omitted. -### Running the app locally as a Docker image +### As a Docker image It can be useful to check the compiled Docker image, that will be run by the production installation, locally yourself. Assuming you have the `Data API` @@ -340,7 +341,7 @@ If need be, `config.relative_url_root` may by overridden by means of the `RAILS_RELATIVE_URL_ROOT` environment variable, althought this could also require rebuilding the assets or docker image when running inside docker. -## Running the complete HMLR suite locally +### Running the complete HMLR suite locally Then entire HLMR suite of applications can be run locally as individual rails servers in `development` mode as noted @@ -369,7 +370,7 @@ below[^2]: rake test ``` -## Runtime Configuration environment variables +### Runtime Configuration environment variables A number of environment variables to determine the runtime behaviour of the application: @@ -384,13 +385,11 @@ application: | | This is handled automatically when starting a docker container, or the `server` `make` target | | | `SENTRY_API_KEY` | The Data Source Name (DSN) for sending reports to the Sentry account | None | -## Issues +### Issues Please add issues to the [shared issues list](https://github.com/epimorphics/hmlr-linked-data/issues) -## Additional Information - ### Deployment The detailed deployment mapping is described in `deployment.yml`. At the time From 8e82f60def53732711902827e76180c9ff369df6 Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Thu, 5 Oct 2023 15:09:49 +0100 Subject: [PATCH 3/5] Moved updating geographies into own section --- README.md | 108 +++++++++++++++++++++++++++--------------------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index 6b734d3b..08e81706 100644 --- a/README.md +++ b/README.md @@ -16,60 +16,6 @@ Code in this repository is open-source under the MIT license. The UKHPI data itself is freely available under the terms of the [Open Government License](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) -## Updating geographies - -Over many years, the boundaries of UK Local Authorities (LA) change. Sometimes -multiple adjacent authorities merge to for a new unitary authority (UA), or -sometimes a unitary authority is split off from an existing LA (e.g. City of -Plymouth). - -In this case, we need to update two data sources within the application, and -coordinate this change with HMLR (and, by extension, ONS). - -For example, in 2020 we updated the app due to various changes in LA boundaries, -including the creation of the Bournemouth, Christchurch and Poole UA. The -timeline of these changes was fairly typical, so we've documented it here for -future reference: - -- April 2019: new boundaries go into effect. -- Feb 2020: ONS perform their (single) yearly update of location tables. -- March 2020: HMLR provided Epimorphics with a collection of test data, - including the new regions table, as Turtle (`.ttl`) files. -- March 2020: we loaded the test data into a dev server in order to re-run the - [`rails ukhpi:locations` task](#rails-script-tasks). This task updates the - cached Ruby and Javascript locations tables, to save the application having to - query the API every time a location is referred to. -- In order to update the map UI in UKHPI, we need to regenerate the - `ONS-Geographies` GeoJSON file. [Alex](mailto:alex.coley@epimorphics.com) has - a Feature Migration Engine (FME) script on his system that automates this. - - We need to simplify the outlines, because otherwise the generated GeoJSON file - is enormous. Doing the simplification in FME means that there are no gaps - between adjacent authorities as the outlines are compressed. - - The basic process is: - - 1. Find and download the relevant shapefile from the ONS geographies portal. - 2. Run the FME script to convert the shapefile GeoJSON, simplifying the - outlines along the way. - 3. Once the initial GeoJSON file is produced, there is a simple shellscript - to regularise the property names in the GeoJSON, see - `bin/ons-geojson-cleanup`. - - The core issue is that the application code expects the GSS code for a - location to be `code` and the location name to be `name`. In the ONS - shapefile, the GSS code can be, for example, `rgn18cd` and the name `rgn18nm`. - - Rather than have the code adapt to these, assuming they may change each year, - it was easier to have a data-cleansing task. It could have been a rake task, - but there are some Linux command utilities that do the job very easily. The - code will need to be changed to reference the new GeoJSON file, i.e. - `app/javascript/data/ONS-Geographies-$YEAR.json`. - -- Finally, the application is published on - for the Plymouth team to - test. - ## Outline domain model In the 2017 update, we extended the display to present all of the underlying @@ -357,6 +303,60 @@ the information found in the [simple-web-proxy](https://github.com/epimorphics/simple-web-proxy/edit/main/README.md) repository. +## Updating geographies + +Over many years, the boundaries of UK Local Authorities (LA) change. Sometimes +multiple adjacent authorities merge to for a new unitary authority (UA), or +sometimes a unitary authority is split off from an existing LA (e.g. City of +Plymouth). + +In this case, we need to update two data sources within the application, and +coordinate this change with HMLR (and, by extension, ONS). + +For example, in 2020 we updated the app due to various changes in LA boundaries, +including the creation of the Bournemouth, Christchurch and Poole UA. The +timeline of these changes was fairly typical, so we've documented it here for +future reference: + +- April 2019: new boundaries go into effect. +- Feb 2020: ONS perform their (single) yearly update of location tables. +- March 2020: HMLR provided Epimorphics with a collection of test data, + including the new regions table, as Turtle (`.ttl`) files. +- March 2020: we loaded the test data into a dev server in order to re-run the + [`rails ukhpi:locations` task](#rails-script-tasks). This task updates the + cached Ruby and Javascript locations tables, to save the application having to + query the API every time a location is referred to. +- In order to update the map UI in UKHPI, we need to regenerate the + `ONS-Geographies` GeoJSON file. [Alex](mailto:alex.coley@epimorphics.com) has + a Feature Migration Engine (FME) script on his system that automates this. + + We need to simplify the outlines, because otherwise the generated GeoJSON file + is enormous. Doing the simplification in FME means that there are no gaps + between adjacent authorities as the outlines are compressed. + + The basic process is: + + 1. Find and download the relevant shapefile from the ONS geographies portal. + 2. Run the FME script to convert the shapefile GeoJSON, simplifying the + outlines along the way. + 3. Once the initial GeoJSON file is produced, there is a simple shellscript + to regularise the property names in the GeoJSON, see + `bin/ons-geojson-cleanup`. + + The core issue is that the application code expects the GSS code for a + location to be `code` and the location name to be `name`. In the ONS + shapefile, the GSS code can be, for example, `rgn18cd` and the name `rgn18nm`. + + Rather than have the code adapt to these, assuming they may change each year, + it was easier to have a data-cleansing task. It could have been a rake task, + but there are some Linux command utilities that do the job very easily. The + code will need to be changed to reference the new GeoJSON file, i.e. + `app/javascript/data/ONS-Geographies-$YEAR.json`. + +- Finally, the application is published on + for the Plymouth team to + test. + ### Coding standards `rubocop` should always return a clean status with no warnings. From 3bec24ad8c99b1cd986ffec3de969fdd311ba6ab Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Thu, 5 Oct 2023 15:10:22 +0100 Subject: [PATCH 4/5] Outlining the data model is now a section --- README.md | 129 +++++++++++++++++++++++++++--------------------------- 1 file changed, 65 insertions(+), 64 deletions(-) diff --git a/README.md b/README.md index 08e81706..0ebbed49 100644 --- a/README.md +++ b/README.md @@ -16,70 +16,6 @@ Code in this repository is open-source under the MIT license. The UKHPI data itself is freely available under the terms of the [Open Government License](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/) -## Outline domain model - -In the 2017 update, we extended the display to present all of the underlying -statistics calculated by ONS. In addition to the figures by property type (all, -detached houses, semi-detached, terraced and flat/maisonette), there are also -figures for property status (new build vs. existing), buyer status (first-time -buyer or not), and funding status (paid cash or bought with a mortgage). For -each of these figures, multiple indicators are calculated: the statistical -index, the percentage change (monthly and annual) and the average price. For -some of the statistics, a sales volume indicator is also available. - -We represent these as follows: - -| theme | indicator | statistic | -| --- | --- | --- | -| Type of property | index | All property types | -| | average price | All property types | -| | % monthly change | All property types | -| | % annual change | All property types | -| | sales volume | All property types | -| | index | Detached houses | -| | average price | Detached houses | -| | % monthly change | Detached houses | -| | % annual change | Detached houses | -| | index | Semi-detached houses | -| | average price | Semi-detached houses | -| | % monthly change | Semi-detached houses | -| | % annual change | Semi-detached houses | -| | index | Terraced houses | -| | average price | Terraced houses | -| | % monthly change | Terraced houses | -| | % annual change | Terraced houses | -| | index | Flats and maisonettes | -| | average price | Flats and maisonettes | -| | % monthly change | Flats and maisonettes | -| | % annual change | Flats and maisonettes | -| Buyer status | index | First-time buyers | -| | average price | First-time buyers | -| | % monthly change | First-time buyers | -| | % annual change | First-time buyers | -| | index | Former owner-occupiers | -| | average price | Former owner-occupiers | -| | % monthly change | Former owner-occupiers | -| | % annual change | Former owner-occupiers | -| Funding status | index | Cash purchases | -| | average price | Cash purchases | -| | % monthly change | Cash purchases | -| | % annual change | Cash purchases | -| | sales volume | Cash purchases | -| | index | Mortgage purchases | -| | average price | Mortgage purchases | -| | % monthly change | Mortgage purchases | -| | % annual change | Mortgage purchases | -| | sales volume | Mortgage purchases | -| Property status | index | New build | -| | average price | New build | -| | % monthly change | New build | -| | % annual change | New build | -| | sales volume | New build -| | index | Existing properties | -| | average price | Existing properties | -| | % monthly change | Existing properties | -| | % annual change | Existing properties | -| | sales volume | Existing properties ## Installing dependencies The app currently depends on Ruby version 2.6.x. The actual version is specified @@ -357,6 +293,71 @@ future reference: for the Plymouth team to test. +## Outline domain model + +In the 2017 update, we extended the display to present all of the underlying +statistics calculated by ONS. In addition to the figures by property type (all, +detached houses, semi-detached, terraced and flat/maisonette), there are also +figures for property status (new build vs. existing), buyer status (first-time +buyer or not), and funding status (paid cash or bought with a mortgage). For +each of these figures, multiple indicators are calculated: the statistical +index, the percentage change (monthly and annual) and the average price. For +some of the statistics, a sales volume indicator is also available. + +We represent these as follows: + +| theme | indicator | statistic | +| --- | --- | --- | +| Type of property | index | All property types | +| | average price | All property types | +| | % monthly change | All property types | +| | % annual change | All property types | +| | sales volume | All property types | +| | index | Detached houses | +| | average price | Detached houses | +| | % monthly change | Detached houses | +| | % annual change | Detached houses | +| | index | Semi-detached houses | +| | average price | Semi-detached houses | +| | % monthly change | Semi-detached houses | +| | % annual change | Semi-detached houses | +| | index | Terraced houses | +| | average price | Terraced houses | +| | % monthly change | Terraced houses | +| | % annual change | Terraced houses | +| | index | Flats and maisonettes | +| | average price | Flats and maisonettes | +| | % monthly change | Flats and maisonettes | +| | % annual change | Flats and maisonettes | +| Buyer status | index | First-time buyers | +| | average price | First-time buyers | +| | % monthly change | First-time buyers | +| | % annual change | First-time buyers | +| | index | Former owner-occupiers | +| | average price | Former owner-occupiers | +| | % monthly change | Former owner-occupiers | +| | % annual change | Former owner-occupiers | +| Funding status | index | Cash purchases | +| | average price | Cash purchases | +| | % monthly change | Cash purchases | +| | % annual change | Cash purchases | +| | sales volume | Cash purchases | +| | index | Mortgage purchases | +| | average price | Mortgage purchases | +| | % monthly change | Mortgage purchases | +| | % annual change | Mortgage purchases | +| | sales volume | Mortgage purchases | +| Property status | index | New build | +| | average price | New build | +| | % monthly change | New build | +| | % annual change | New build | +| | sales volume | New build +| | index | Existing properties | +| | average price | Existing properties | +| | % monthly change | Existing properties | +| | % annual change | Existing properties | +| | sales volume | Existing properties + ### Coding standards `rubocop` should always return a clean status with no warnings. From 7059b12c141bda7f0a57cc3ee72b39c9b3e29ba1 Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Thu, 5 Oct 2023 15:10:35 +0100 Subject: [PATCH 5/5] Added additional info heading --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 0ebbed49..0e99ca67 100644 --- a/README.md +++ b/README.md @@ -358,6 +358,8 @@ We represent these as follows: | | % annual change | Existing properties | | | sales volume | Existing properties +## Additional Information + ### Coding standards `rubocop` should always return a clean status with no warnings.