5 initial backend database setup

[kusalananda]

This PR closes #5

What this PR adds:

.
├── backend
│   ├── .gitignore
│   ├── Dockerfile
│   ├── backend.env.example
│   ├── entrypoint.d
│   │   ├── apply-migrations
│   │   ├── main
│   │   └── restore-database
│   ├── initdb.d
│   │   └── README.md
│   ├── migration.d
│   │   ├── 0000-initial-schema.sql
│   │   ├── 0001-create-table-test.sql
│   │   ├── 0002-drop-table-test.sql
│   │   └── README.md
│   └── scripts
│       └── dumpdb
└── docker-compose.yml

At the top level, it adds the project docker-compose.yml file. Eventually, this file will be expanded to include services relating to the frontend, but it currently only provides a short definition of the backend service. Note that this PR is only concerned about the database setup and that there is no actual API code yet, so the docker-compose.yml file does not enable listening on any particular ports. However, it adds a volume, backend-volume, which the backend uses to store a persistent database.

In the backend directory, we find the Dockerfile used to build the backend container image and the entrypoint.d directory containing scripts for performing database restore and migration when the container starts up.

There is an initdb.d directory into which one may optionally put an SQL file called init.sql. If this file exists and the container's volume does not contain a database, the file will be used to initialize the database.

The scripts directory contains the single script dumpdb, which creates an SQL file containing the statements necessary for restoring the database to its current state (e.g., by putting a dump in initdb.d as previously mentioned).

The migration.d directory contains a set of database migrations. These will be applied to the database as the container starts up. The entypoint.d/apply-migrations scripts will automatically skip already applied migrations.

The entrypoint.d and migration.d directories are copied into the container when the container is built. The initdb.d directory is bind-mounted (read-only) into the container, allowing an operator to start the service with a previously backed-up database dump.

Testing this PR

To test this PR, you would want to copy the backend.env.example file to backend.env, run docker compose build followed by docker compose up -d to run the service in the background. You would expect to see the following:

$ cd backend
$ cp backend.env.example backend.env
$ docker compose build
[...]
$ docker compose up -d
[+] Running 3/3
 ✔ Network meerkats-git-workshop_default          Created                            0.1s
 ✔ Volume "meerkats-git-workshop_backend-volume"  Created                            0.0s
 ✔ Container meerkats-git-workshop-backend-1      Started                            0.5s
$ docker compose logs
backend-1  | No database dump found at "./initdb.d/init.sql", skipping restore
backend-1  | Applying migration "0000-initial-schema.sql"
backend-1  | Applying migration "0001-create-table-test.sql"
backend-1  | Applying migration "0002-drop-table-test.sql"
backend-1  | + sleep infinity

Note that the "server" is simulated by a simple shell script that just runs sleep infinity, which never terminates.

Dumping the database at this point should look like this:

$ ./scripts/dumpdb
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE migration (
        id      INTEGER PRIMARY KEY AUTOINCREMENT,
        ts      TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
        hash    TEXT NOT NULL,
        name    TEXT NOT NULL,

        UNIQUE(name)
);
INSERT INTO migration VALUES(1,'2024-08-31 15:06:07','b3ff83e21ef618f42623a0531dc6e60e','0000-initial-schema.sql');
INSERT INTO migration VALUES(2,'2024-08-31 15:06:07','9d234e3fa84282ea30ed6358c9280843','0001-create-table-test.sql');
INSERT INTO migration VALUES(3,'2024-08-31 15:06:07','c711d0aaebf49ec7e934e6bee7d70478','0002-drop-table-test.sql');
DELETE FROM sqlite_sequence;
INSERT INTO sqlite_sequence VALUES('migration',3);
COMMIT;

Stopping the service and starting it again should inform you that migrations have already been applied:

$ docker compose down
[+] Running 2/2
 ✔ Container meerkats-git-workshop-backend-1  Removed                               10.3s
 ✔ Network meerkats-git-workshop_default      Removed                                0.4s
$ docker compose up -d
[+] Running 2/2
 ✔ Network meerkats-git-workshop_default      Created                                0.1s
 ✔ Container meerkats-git-workshop-backend-1  Started                                0.4s
$ docker compose logs
backend-1  | Database found at "/vol/backend-data.db", skipping restore
backend-1  | Skipping application of migration "0000-initial-schema.sql"
backend-1  | Skipping application of migration "0001-create-table-test.sql"
backend-1  | Skipping application of migration "0002-drop-table-test.sql"
backend-1  | + sleep infinity

You may also try stopping the services with docker compose down -v, which will remove the persistent volume and the database therein. Another thing to test is to start the service (which applies the migrations in migration.d if needed), stop everything yet again and then modify one of the migration files (just add a space or whatever). Rebuilding and restarting the services should now look like this:

$ docker compose down
[+] Running 2/2
 ✔ Container meerkats-git-workshop-backend-1  Removed                               10.3s
 ✔ Network meerkats-git-workshop_default      Removed                                0.4s
$ vi migration.d/0001-create-table-test.sql
$ docker compose build
[...]
$ docker compose up -d
[+] Running 2/2
 ✔ Network meerkats-git-workshop_default      Created                                0.1s
 ✔ Container meerkats-git-workshop-backend-1  Started                                0.4s
$ docker compose logs
backend-1  | Database found at "/vol/backend-data.db", skipping restore
backend-1  | Skipping application of migration "0000-initial-schema.sql"
backend-1  | Migration "0001-create-table-test.sql" was changed after it was applied
backend-1  | Expected MD5 "2c3e59ec04b6a3fcc2ef693aa3babb4e", got "9d234e3fa84282ea30ed6358c9280843"
backend-1  | Migrations failed
$ docker compose ps
NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS

The last command above shows that the service isn't running as it encountered an issue when starting up. Since migrations are applied to a copy of the database, no data will have been changed in the database.