Skip to content

Commit

Permalink
Merge pull request #383 from epimorphics/spike/readme-updates
Browse files Browse the repository at this point in the history 
Spike/readme updates
  • Loading branch information
@bogdanadrianmarc
bogdanadrianmarc authored Oct 6, 2023
2 parents 59aa73a + 7059b12 commit 8eec6c6
Showing 1 changed file with 137 additions and 131 deletions.
268 changes: 137 additions & 131 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ This is the repo for the application that presents UKHPI open data on behalf of
Land Registry (England and Wales), Registers of Scotland, Land and Property
Services (Northern Ireland) and the UK Office for National Statistics (ONS).

Please see the other repositories in the [HM Land Registry Open
Data](https://github.com/epimorphics/hmlr-linked-data/) project for more
details.

Development work was carried out by [Epimorphics
Ltd](http://www.epimorphics.com), funded by [HM Land
Registry](https://www.gov.uk/government/organisations/land-registry).
Expand All @@ -12,128 +16,7 @@ Code in this repository is open-source under the MIT license. The UKHPI data
itself is freely available under the terms of the [Open Government
License](https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/)

## Updating geographies

Over many years, the boundaries of UK Local Authorities (LA) change. Sometimes
multiple adjacent authorities merge to for a new unitary authority (UA), or
sometimes a unitary authority is split off from an existing LA (e.g. City of
Plymouth).

In this case, we need to update two data sources within the application, and
coordinate this change with HMLR (and, by extension, ONS).

For example, in 2020 we updated the app due to various changes in LA boundaries,
including the creation of the Bournemouth, Christchurch and Poole UA. The
timeline of these changes was fairly typical, so we've documented it here for
future reference:

- April 2019: new boundaries go into effect.
- Feb 2020: ONS perform their (single) yearly update of location tables.
- March 2020: HMLR provided Epimorphics with a collection of test data,
including the new regions table, as Turtle (`.ttl`) files.
- March 2020: we loaded the test data into a dev server in order to re-run the
[`rails ukhpi:locations` task](#rails-script-tasks). This task updates the
cached Ruby and Javascript locations tables, to save the application having to
query the API every time a location is referred to.
- In order to update the map UI in UKHPI, we need to regenerate the
`ONS-Geographies` GeoJSON file. [Alex](mailto:[email protected]) has
a Feature Migration Engine (FME) script on his system that automates this.

We need to simplify the outlines, because otherwise the generated GeoJSON file
is enormous. Doing the simplification in FME means that there are no gaps
between adjacent authorities as the outlines are compressed.

The basic process is:

1. Find and download the relevant shapefile from the ONS geographies portal.
2. Run the FME script to convert the shapefile GeoJSON, simplifying the
outlines along the way.
3. Once the initial GeoJSON file is produced, there is a simple shellscript
to regularise the property names in the GeoJSON, see
`bin/ons-geojson-cleanup`.

The core issue is that the application code expects the GSS code for a
location to be `code` and the location name to be `name`. In the ONS
shapefile, the GSS code can be, for example, `rgn18cd` and the name `rgn18nm`.

Rather than have the code adapt to these, assuming they may change each year,
it was easier to have a data-cleansing task. It could have been a rake task,
but there are some Linux command utilities that do the job very easily. The
code will need to be changed to reference the new GeoJSON file, i.e.
`app/javascript/data/ONS-Geographies-$YEAR.json`.

- Finally, the application is published on
<https://hmlr-dev-pres.epimorphics.net/app/ukhpi/> for the Plymouth team to
test.

## Outline domain model

In the 2017 update, we extended the display to present all of the underlying
statistics calculated by ONS. In addition to the figures by property type (all,
detached houses, semi-detached, terraced and flat/maisonette), there are also
figures for property status (new build vs. existing), buyer status (first-time
buyer or not), and funding status (paid cash or bought with a mortgage). For
each of these figures, multiple indicators are calculated: the statistical
index, the percentage change (monthly and annual) and the average price. For
some of the statistics, a sales volume indicator is also available.

We represent these as follows:

| theme | indicator | statistic |
| --- | --- | --- |
| Type of property | index | All property types |
| | average price | All property types |
| | % monthly change | All property types |
| | % annual change | All property types |
| | sales volume | All property types |
| | index | Detached houses |
| | average price | Detached houses |
| | % monthly change | Detached houses |
| | % annual change | Detached houses |
| | index | Semi-detached houses |
| | average price | Semi-detached houses |
| | % monthly change | Semi-detached houses |
| | % annual change | Semi-detached houses |
| | index | Terraced houses |
| | average price | Terraced houses |
| | % monthly change | Terraced houses |
| | % annual change | Terraced houses |
| | index | Flats and maisonettes |
| | average price | Flats and maisonettes |
| | % monthly change | Flats and maisonettes |
| | % annual change | Flats and maisonettes |
| Buyer status | index | First-time buyers |
| | average price | First-time buyers |
| | % monthly change | First-time buyers |
| | % annual change | First-time buyers |
| | index | Former owner-occupiers |
| | average price | Former owner-occupiers |
| | % monthly change | Former owner-occupiers |
| | % annual change | Former owner-occupiers |
| Funding status | index | Cash purchases |
| | average price | Cash purchases |
| | % monthly change | Cash purchases |
| | % annual change | Cash purchases |
| | sales volume | Cash purchases |
| | index | Mortgage purchases |
| | average price | Mortgage purchases |
| | % monthly change | Mortgage purchases |
| | % annual change | Mortgage purchases |
| | sales volume | Mortgage purchases |
| Property status | index | New build |
| | average price | New build |
| | % monthly change | New build |
| | % annual change | New build |
| | sales volume | New build
| | index | Existing properties |
| | average price | Existing properties |
| | % monthly change | Existing properties |
| | % annual change | Existing properties |
| | sales volume | Existing properties

## Developer notes

### Installing dependencies
## Installing dependencies

The app currently depends on Ruby version 2.6.x. The actual version is specified
in the `.ruby-version` file, which can be used with
Expand All @@ -151,7 +34,9 @@ bundle install
yarn install
```

### Accessing the Data API locally
## Running the data API locally

### Prerequisites

The application communicates with the HMLR data API (which uses Sapi-NT) to
provide the data to be displayed. The actual API location is specified by the
Expand Down Expand Up @@ -196,7 +81,7 @@ To use a credential helper for a specific ECR registry[^1], create a
The local application can then connect to the triple store via the `data-api`
service.

### Running the Data API locally
### Running the API

The easiest way to do this is via a local docker container. The `data-api` image
can be built from [lr-data-api
Expand Down Expand Up @@ -261,7 +146,9 @@ If needed, to create the docker network run:
docker network create dnet
```

### Running the app locally
## Running the app

### Locally

Assuming the `Data API` is running on `http://localhost:8888` (the default), to
start the app locally:
Expand All @@ -282,7 +169,7 @@ And then visit <http://localhost:3002/app/ukhpi>.

N.B. The default for `RAILS_ENV` is `production` if omitted.

### Running the app locally as a Docker image
### As a Docker image

It can be useful to check the compiled Docker image, that will be run by the
production installation, locally yourself. Assuming you have the `Data API`
Expand Down Expand Up @@ -336,7 +223,7 @@ If need be, `config.relative_url_root` may by overridden by means of the
`RAILS_RELATIVE_URL_ROOT` environment variable, althought this could also
require rebuilding the assets or docker image when running inside docker.

## Running the complete HMLR suite locally
### Running the complete HMLR suite locally

Then entire HLMR suite of applications can be run locally as individual rails
servers in `development` mode as noted
Expand All @@ -352,6 +239,127 @@ the information found in the
[simple-web-proxy](https://github.com/epimorphics/simple-web-proxy/edit/main/README.md)
repository.

## Updating geographies

Over many years, the boundaries of UK Local Authorities (LA) change. Sometimes
multiple adjacent authorities merge to for a new unitary authority (UA), or
sometimes a unitary authority is split off from an existing LA (e.g. City of
Plymouth).

In this case, we need to update two data sources within the application, and
coordinate this change with HMLR (and, by extension, ONS).

For example, in 2020 we updated the app due to various changes in LA boundaries,
including the creation of the Bournemouth, Christchurch and Poole UA. The
timeline of these changes was fairly typical, so we've documented it here for
future reference:

- April 2019: new boundaries go into effect.
- Feb 2020: ONS perform their (single) yearly update of location tables.
- March 2020: HMLR provided Epimorphics with a collection of test data,
including the new regions table, as Turtle (`.ttl`) files.
- March 2020: we loaded the test data into a dev server in order to re-run the
[`rails ukhpi:locations` task](#rails-script-tasks). This task updates the
cached Ruby and Javascript locations tables, to save the application having to
query the API every time a location is referred to.
- In order to update the map UI in UKHPI, we need to regenerate the
`ONS-Geographies` GeoJSON file. [Alex](mailto:[email protected]) has
a Feature Migration Engine (FME) script on his system that automates this.

We need to simplify the outlines, because otherwise the generated GeoJSON file
is enormous. Doing the simplification in FME means that there are no gaps
between adjacent authorities as the outlines are compressed.

The basic process is:

1. Find and download the relevant shapefile from the ONS geographies portal.
2. Run the FME script to convert the shapefile GeoJSON, simplifying the
outlines along the way.
3. Once the initial GeoJSON file is produced, there is a simple shellscript
to regularise the property names in the GeoJSON, see
`bin/ons-geojson-cleanup`.

The core issue is that the application code expects the GSS code for a
location to be `code` and the location name to be `name`. In the ONS
shapefile, the GSS code can be, for example, `rgn18cd` and the name `rgn18nm`.

Rather than have the code adapt to these, assuming they may change each year,
it was easier to have a data-cleansing task. It could have been a rake task,
but there are some Linux command utilities that do the job very easily. The
code will need to be changed to reference the new GeoJSON file, i.e.
`app/javascript/data/ONS-Geographies-$YEAR.json`.

- Finally, the application is published on
<https://hmlr-dev-pres.epimorphics.net/app/ukhpi/> for the Plymouth team to
test.

## Outline domain model

In the 2017 update, we extended the display to present all of the underlying
statistics calculated by ONS. In addition to the figures by property type (all,
detached houses, semi-detached, terraced and flat/maisonette), there are also
figures for property status (new build vs. existing), buyer status (first-time
buyer or not), and funding status (paid cash or bought with a mortgage). For
each of these figures, multiple indicators are calculated: the statistical
index, the percentage change (monthly and annual) and the average price. For
some of the statistics, a sales volume indicator is also available.

We represent these as follows:

| theme | indicator | statistic |
| --- | --- | --- |
| Type of property | index | All property types |
| | average price | All property types |
| | % monthly change | All property types |
| | % annual change | All property types |
| | sales volume | All property types |
| | index | Detached houses |
| | average price | Detached houses |
| | % monthly change | Detached houses |
| | % annual change | Detached houses |
| | index | Semi-detached houses |
| | average price | Semi-detached houses |
| | % monthly change | Semi-detached houses |
| | % annual change | Semi-detached houses |
| | index | Terraced houses |
| | average price | Terraced houses |
| | % monthly change | Terraced houses |
| | % annual change | Terraced houses |
| | index | Flats and maisonettes |
| | average price | Flats and maisonettes |
| | % monthly change | Flats and maisonettes |
| | % annual change | Flats and maisonettes |
| Buyer status | index | First-time buyers |
| | average price | First-time buyers |
| | % monthly change | First-time buyers |
| | % annual change | First-time buyers |
| | index | Former owner-occupiers |
| | average price | Former owner-occupiers |
| | % monthly change | Former owner-occupiers |
| | % annual change | Former owner-occupiers |
| Funding status | index | Cash purchases |
| | average price | Cash purchases |
| | % monthly change | Cash purchases |
| | % annual change | Cash purchases |
| | sales volume | Cash purchases |
| | index | Mortgage purchases |
| | average price | Mortgage purchases |
| | % monthly change | Mortgage purchases |
| | % annual change | Mortgage purchases |
| | sales volume | Mortgage purchases |
| Property status | index | New build |
| | average price | New build |
| | % monthly change | New build |
| | % annual change | New build |
| | sales volume | New build
| | index | Existing properties |
| | average price | Existing properties |
| | % monthly change | Existing properties |
| | % annual change | Existing properties |
| | sales volume | Existing properties

## Additional Information

### Coding standards

`rubocop` should always return a clean status with no warnings.
Expand All @@ -365,7 +373,7 @@ below[^2]:
rake test
```

## Runtime Configuration environment variables
### Runtime Configuration environment variables

A number of environment variables to determine the runtime behaviour of the
application:
Expand All @@ -380,13 +388,11 @@ application:
| | This is handled automatically when starting a docker container, or the `server` `make` target | |
| `SENTRY_API_KEY` | The Data Source Name (DSN) for sending reports to the Sentry account | None |

## Issues
### Issues

Please add issues to the [shared issues
list](https://github.com/epimorphics/hmlr-linked-data/issues)

## Additional Information

### Deployment

The detailed deployment mapping is described in `deployment.yml`. At the time
Expand Down

0 comments on commit 8eec6c6

Please sign in to comment.