Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

293 rename hame to arho #337

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
16 changes: 8 additions & 8 deletions .env.dev
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
DB_MAINTENANCE_NAME=postgres
DB_MAIN_NAME=hame
DB_MAIN_NAME=arho
DB_INSTANCE_PORT=5434
DB_INSTANCE_ADDRESS=localhost

# Users
SU_USER=postgres
SU_USER_PW=postgres
ADMIN_USER=hame_admin
ADMIN_USER_PW=hame_admin
R_USER=hame_read
R_USER_PW=hame_read
RW_USER=hame_read_write
RW_USER_PW=hame_read_write
ADMIN_USER=arho_admin
ADMIN_USER_PW=arho_admin
R_USER=arho_read
R_USER_PW=arho_read
RW_USER=arho_read_write
RW_USER_PW=arho_read_write
READ_FROM_AWS=0

DB_USERS=hame_admin,hame_read,hame_read_write # Update this to contain all users
DB_USERS=arho_admin,arho_read,arho_read_write # Update this to contain all users

# API keys
SYKE_APIKEY=your_syke_apikey # Add your Syke subscription api key here
Expand Down
65 changes: 38 additions & 27 deletions .tbls.yml
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
include:
- hame.*
- arho.*
- codes.*

viewpoints:
-
name: All tables
desc: All tables that make up maakuntakaava plan data.
tables:
- hame.plan
- hame.land_use_area
- hame.other_area
- hame.line
- hame.land_use_point
- hame.other_point
- hame.plan_regulation_group
- hame.plan_regulation
- hame.plan_proposition
- hame.document
- hame.source_data
- hame.organisation
- hame.lifecycle_date
- arho.plan
- arho.land_use_area
- arho.other_area
- arho.line
- arho.land_use_point
- arho.other_point
- arho.plan_regulation_group
- arho.plan_regulation
- arho.plan_proposition
- arho.document
- arho.source_data
- arho.organisation
- arho.lifecycle_date
- codes.administrative_region
- codes.lifecycle_status
- codes.plan_theme
Expand All @@ -31,32 +31,38 @@ viewpoints:
- codes.type_of_source_data
- codes.type_of_underground
- codes.type_of_verbal_plan_regulation
- codes.category_of_publicity
- codes.name_of_plan_case_decision
- codes.type_of_decision_maker
- codes.type_of_interaction_event
- codes.type_of_processing_event

groups:
-
name: Plan geometry tables
desc: These tables represent all geometries in the plan.
tables:
- hame.plan
- hame.land_use_area
- hame.other_area
- hame.line
- hame.land_use_point
- hame.other_point
- arho.plan
- arho.land_use_area
- arho.other_area
- arho.line
- arho.land_use_point
- arho.other_point
-
name: Plan regulation tables
desc: These tables represent all regulations in the plan.
tables:
- hame.plan_regulation_group
- hame.plan_regulation
- hame.plan_proposition
- arho.plan_regulation_group
- arho.plan_regulation
- arho.plan_proposition
-
name: Plan metadata tables
desc: These tables contain all metadata, documents and other linked information.
tables:
- hame.document
- hame.source_data
- hame.organisation
- hame.lifecycle_date
- arho.document
- arho.source_data
- arho.organisation
- arho.lifecycle_date
-
name: Code tables
desc: These tables contain all national (koodistot.suomi.fi) and local codes used for classifying plan data.
Expand All @@ -72,3 +78,8 @@ viewpoints:
- codes.type_of_source_data
- codes.type_of_underground
- codes.type_of_verbal_plan_regulation
- codes.category_of_publicity
- codes.name_of_plan_case_decision
- codes.type_of_decision_maker
- codes.type_of_interaction_event
- codes.type_of_processing_event
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
test-create-db:
@echo "Creating Hame database..."
@echo "Creating Arho database..."
curl -XPOST "http://localhost:8081/2015-03-31/functions/function/invocations" -d '{"event_type" : 1}'

test-migrate-db:
@echo "Migrating Hame database..."
@echo "Migrating Arho database..."
curl -XPOST "http://localhost:8081/2015-03-31/functions/function/invocations" -d '{"event_type" : 3}'

test-koodistot:
Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# hame-ryhti
# arho-ryhti

[![CI/CD](https://github.com/GispoCoding/hame-ryhti/actions/workflows/ci.yml/badge.svg)](https://github.com/GispoCoding/hame-ryhti/actions/workflows/ci.yml)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)

HAME regional land use planning database and QGIS project compatible with [national Ryhti data model](https://ryhti.syke.fi/alueidenkaytto/tietomallimuotoinen-kaavoitus/) -
[Ryhti-yhteensopiva](https://ryhti.syke.fi/alueidenkaytto/tietomallimuotoinen-kaavoitus/) tietokanta ja QGIS-projekti maakuntakaavoitukseen.
ARHO land use planning database and QGIS project compatible with [national Ryhti data model](https://ryhti.syke.fi/alueidenkaytto/tietomallimuotoinen-kaavoitus/) -
[Ryhti-yhteensopiva](https://ryhti.syke.fi/alueidenkaytto/tietomallimuotoinen-kaavoitus/) tietokanta ja QGIS-projekti yleis- maakunta- ja asemakaavoitukseen.

The database and functions can be run on AWS (Amazon Web Services) cloud platform.

Expand All @@ -22,15 +22,15 @@ The database and functions can be run on AWS (Amazon Web Services) cloud platfor

## Architecture

HAME-Ryhti consists of
ARHO-Ryhti consists of
1. a PostGIS database,
2. various AWS Lambda functions to manage the database and import or export planning data,
3. [X-Road security server sidecar container](https://gofore.com/en/benefits-of-the-x-road-security-server-sidecar/) to connect to Ryhti through Finnish X-Road, and
4. QGIS project to connect to the database and create regional land use plans.

![diagram of AWS resources and their connections to software and APIs](infra/architecture.svg)

To manage Hame-Ryhti AWS resources, check the [infra README](https://github.com/GispoCoding/hame-ryhti/blob/main/infra/README.md#hame-infra) in the infra directory.
To manage Arho-Ryhti AWS resources, check the [infra README](https://github.com/GispoCoding/hame-ryhti/blob/main/infra/README.md#hame-infra) in the infra directory.

## Data model

Expand Down Expand Up @@ -75,12 +75,12 @@ docker network ls --format {{.Name}} |grep pytest | awk '{print $1}' | xargs -I
### Database changes

1. Database is defined using SQLAlchemy, so familiarize yourself with [SQLAlchemy declarative style](https://docs.sqlalchemy.org/en/20/orm/declarative_tables.html).
2. Database is divided into two schemas: `codes` contains all the Ryhti specific [national code lists](https://ryhti.syke.fi/ohjeet-ja-tuki/tietomallit/), while `hame` contains all the data tables (plans, plan objects, plan regulations etc.).
2. Database is divided into two schemas: `codes` contains all the Ryhti specific [national code lists](https://ryhti.syke.fi/ohjeet-ja-tuki/tietomallit/), while `arho` contains all the data tables (plans, plan objects, plan regulations etc.).
3. If you want to change *all* tables in a schema (i.e. edit *all* the code tables, or add a field to *all* the data tables), the abstract base classes are in [base.py](./database/base.py).
4. If you only want to change/add *one* code table or one data table, please edit/add the right table in [codes.py](./database/codes.py) or [models.py](./database/models.py).
5. To get the changes tested and usable in your functions, create a new database revision with `make revision name="describe_your_changes"`, e.g. `make revision name="add_plan_object_table"`. This creates a new random id (`uuid`) for your migration, and a revision file `YYYY-MM-DD-HHMM-uuid-add_plan_object_table` in the [alembic versions dir](./database/migrations/versions). Please check that the autogenerated revision file seems to do approximately sensible things.
- Specifically, when adding geometry fields, please note [GeoAlchemy2 bug with Alembic](https://geoalchemy-2.readthedocs.io/en/latest/alembic.html#interactions-between-alembic-and-geoalchemy-2), which means you will have to *manually remove* `op.create_index` and `op.drop_index` in the revision file. This is because GeoAlchemy2 already automatically creates geometry index whenever adding a geometry column.
6. Run tests with `make pytest` to check that the revision file runs correctly. At minimum, you may have to change the tested table counts (codes_count and hame_count) in [database test setup](./database/test/conftest.py) to reflect the correct number of tables in the database.
6. Run tests with `make pytest` to check that the revision file runs correctly. At minimum, you may have to change the tested table counts (codes_count and arho_count) in [database test setup](./database/test/conftest.py) to reflect the correct number of tables in the database.
7. Run `make rebuild` and `make test-create-db` to start development instance with the new model.
<!-- 8. To update the [database documentation](./backend/databasemodel/dbdoc/README.md) to reflect the changes, install [tbls](https://github.com/k1LoW/tbls) and run `tbls doc --force`. -->
8. Commit your changes and the new revision file in [alembic versions dir](./database/migrations/versions).
Expand Down Expand Up @@ -144,15 +144,15 @@ The data is read from a PostgreSQL service named `postgres` with a QGIS authenti
[postgres]
host=localhost
port=5433
dbname=hame-dev
dbname=arho-dev
```
Save the file to some folder, an example location could be `<your home folder>/hameconfig/`. Name the saved file for example `pg_service_hame_dev.conf` (yes, the suffix '.conf' is part of the file name). Do not save this file as a text file (with a suffix .txt), but instead choose 'All types' from the 'Save as type' dropdown menu.
Save the file to some folder, an example location could be `<your home folder>/arhoconfig/`. Name the saved file for example `pg_service_arho_dev.conf` (yes, the suffix '.conf' is part of the file name). Do not save this file as a text file (with a suffix .txt), but instead choose 'All types' from the 'Save as type' dropdown menu.

![screenshot of pgservice file made with notepad](docs/img/pg_servicefile_notepad.png)

NOTE: the Postgres service file for the dev environment is also included in in this repository under the docs folder, so alternatively you can copy the file from the into a convenient location on your computer.

2. Create a QGIS-profile for each environment. Name the profile for example `ryhti-hame-dev`. A new QGIS window will open to this profile, use that in the following.
2. Create a QGIS-profile for each environment. Name the profile for example `ryhti-arho-dev`. A new QGIS window will open to this profile, use that in the following.

![screenshot of new profile menu](docs/img/qgis-new-profile.png)

Expand Down
6 changes: 3 additions & 3 deletions database/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ class VersionedBase(Base):
"""

__abstract__ = True
__table_args__ = {"schema": "hame"}
__table_args__ = {"schema": "arho"}

# Go figure. We have to *explicitly state* id is a mapped column, because id will
# have to be defined inside all the subclasses for relationship remote_side
Expand Down Expand Up @@ -162,11 +162,11 @@ class PlanObjectBase(PlanBase):
index=True,
)
plan_id: Mapped[Optional[uuid.UUID]] = mapped_column(
ForeignKey("hame.plan.id", name="plan_id_fkey"), index=True
ForeignKey("arho.plan.id", name="plan_id_fkey"), index=True
)
plan_regulation_group_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey(
"hame.plan_regulation_group.id", name="plan_regulation_group_id_fkey"
"arho.plan_regulation_group.id", name="plan_regulation_group_id_fkey"
),
index=True,
)
Expand Down
2 changes: 1 addition & 1 deletion database/codes.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class TypeOfAdditionalInformation(CodeBase):
"""
Kaavamääräyksen lisätiedon laji

HAME-Ryhti-tietomallissa tämän koodiston arvot jaetaan useaan monivalintakenttään,
ARHO-Ryhti-tietomallissa tämän koodiston arvot jaetaan useaan monivalintakenttään,
joista jokaisesta käyttäjä voi valita yhden koodin.

Toteutetaan kentät niin, että jokaisessa on filtteri, joka tarkistaa sopivan
Expand Down
28 changes: 14 additions & 14 deletions database/db_manager/db_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from psycopg2.sql import SQL, Identifier

"""
Hame-ryhti database manager, adapted from Tarmo db_manager.
Arho-ryhti database manager, adapted from Tarmo db_manager.
"""

LOGGER = logging.getLogger()
Expand Down Expand Up @@ -51,10 +51,10 @@ def configure_schemas_and_users(
conn: psycopg2.extensions.connection, users: dict[User, dict]
) -> str:
"""
Configures given database with hame schemas and users.
Configures given database with arho schemas and users.
"""
with conn.cursor() as cur:
cur.execute(SQL("CREATE SCHEMA codes; CREATE SCHEMA hame;"))
cur.execute(SQL("CREATE SCHEMA codes; CREATE SCHEMA arho;"))
cur.execute(SQL("CREATE EXTENSION postgis WITH SCHEMA public;"))
for key, user in users.items():
if key == User.SU:
Expand All @@ -78,7 +78,7 @@ def configure_schemas_and_users(
).format(username=Identifier(user["username"])),
vars={"password": user["password"]},
)
msg = "Added hame schemas and users."
msg = "Added arho schemas and users."
return msg


Expand All @@ -97,7 +97,7 @@ def configure_permissions(
pass
if key == User.ADMIN:
# admin user should be able to edit all tables
# (hame and code tables etc.)
# (arho and code tables etc.)
cur.execute(
SQL(
"ALTER DEFAULT PRIVILEGES FOR USER {SU_user} GRANT ALL PRIVILEGES ON TABLES TO {username};" # noqa
Expand All @@ -107,11 +107,11 @@ def configure_permissions(
)
)
elif key == User.READ_WRITE:
# read and write user should be able to edit hame tables and
# read and write user should be able to edit arho tables and
# read code tables
cur.execute(
SQL(
"ALTER DEFAULT PRIVILEGES FOR USER {SU_user} IN SCHEMA hame GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO {username};" # noqa
"ALTER DEFAULT PRIVILEGES FOR USER {SU_user} IN SCHEMA arho GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO {username};" # noqa
).format(
SU_user=Identifier(users[User.SU]["username"]),
username=Identifier(user["username"]),
Expand All @@ -126,18 +126,18 @@ def configure_permissions(
)
)
else:
# default user should be able to read hame tables and code tables
# default user should be able to read arho tables and code tables
cur.execute(
SQL(
"ALTER DEFAULT PRIVILEGES FOR USER {SU_user} IN SCHEMA hame, codes GRANT SELECT ON TABLES TO {username};" # noqa
"ALTER DEFAULT PRIVILEGES FOR USER {SU_user} IN SCHEMA arho, codes GRANT SELECT ON TABLES TO {username};" # noqa
).format(
SU_user=Identifier(users[User.SU]["username"]),
username=Identifier(user["username"]),
)
)
# Finally, all users must have schema usage permissions
cur.execute(
SQL("GRANT USAGE ON SCHEMA hame to {username}").format(
SQL("GRANT USAGE ON SCHEMA arho to {username}").format(
username=Identifier(user["username"])
)
)
Expand All @@ -157,7 +157,7 @@ def database_exists(conn: psycopg2.extensions.connection, db_name: str) -> bool:
return cur.fetchone()[0] == 1


def migrate_hame_db(db_helper: DatabaseHelper, version: str = "head") -> str:
def migrate_arho_db(db_helper: DatabaseHelper, version: str = "head") -> str:
"""Migrates an existing db to the latest scheme, or provided version. Also
configures database permissions.

Expand Down Expand Up @@ -260,14 +260,14 @@ def handler(event: Event, _) -> Response:

event_type = event.get("event_type", EventType.CREATE_DB.value)
if event_type == EventType.CREATE_DB.value:
msg = migrate_hame_db(db_helper)
msg = migrate_arho_db(db_helper)
elif event_type == EventType.CHANGE_PWS.value:
msg = change_passwords(db_helper)
elif event_type == EventType.MIGRATE_DB.value:
version = str(event.get("version", ""))
if version:
msg = migrate_hame_db(db_helper, version)
msg = migrate_arho_db(db_helper, version)
else:
msg = migrate_hame_db(db_helper)
msg = migrate_arho_db(db_helper)
response["body"] = json.dumps(msg)
return response
4 changes: 2 additions & 2 deletions database/migrations/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ def include_object(object, name, type_, reflected, compare_to):
# Check our schemas
def include_name(name, type_, parent_names):
if type_ in "schema":
return name in ["hame", "codes"]
return name in ["arho", "codes"]
else:
return True

Expand All @@ -118,7 +118,7 @@ def get_url(connection_params: dict):
"host", os.environ.get("DB_INSTANCE_ADDRESS", "localhost")
)
port = connection_params.get("port", os.environ.get("DB_INSTANCE_PORT", "5434"))
dbname = connection_params.get("dbname", os.environ.get("DB_MAIN_NAME", "hame"))
dbname = connection_params.get("dbname", os.environ.get("DB_MAIN_NAME", "arho"))
return f"postgresql://{user}:{password}@{host}:{port}/{dbname}"


Expand Down
Loading
Loading