From b14562b13c954b0978bae13a0ead836612b6000f Mon Sep 17 00:00:00 2001 From: Mark Boyd Date: Wed, 31 Jul 2024 10:29:53 -0400 Subject: [PATCH] Update database access docs (#2526) * rewrite docs on accessing cloud.gov databases to improve clarity * reorganize docs on restoring databases * update and clarify docs on opening datbase shells * update headers * update link text * remove redundant note * update link * update header * reorganize docs --- _docs/services/relational-database.md | 249 +++++++++++------- ...22-log4j_vulnerability_bod_22-02_update.md | 2 +- 2 files changed, 154 insertions(+), 97 deletions(-) diff --git a/_docs/services/relational-database.md b/_docs/services/relational-database.md index 0b55852f7..c317b23fd 100644 --- a/_docs/services/relational-database.md +++ b/_docs/services/relational-database.md @@ -379,114 +379,152 @@ When the restore process is completely finished, we'll notify you and ask you to When we remove the old database instance, we will not retain snapshots or backups of it unless we're explicitly asked to do so. We'll remind you of this when coordinating a specific date and time to remove the old instance. -You can also create manual backups using the [export process](#exporting-a-database) described below. In general, you are responsible for making sure that your backup procedures are adequate for your needs; see CP-9 in the cloud.gov SSP. +You can also create manual backups using the [export process](#exporting-a-database-manually) described below. In general, you are responsible for making sure that your backup procedures are adequate for your needs; see CP-9 in the cloud.gov SSP. -## Access the data in the database +## Tunneling to your database -To access a service database, use the [cf-service-connect plugin](https://github.com/18F/cf-service-connect#readme) or the new [cg-manage-rds](https://github.com/cloud-gov/cg-manage-rds) application and the corresponding CLI (command line interface) tools for the database service you are using, +Databases on cloud.gov can only be connected to from other resources within our internal network. You cannot make a connection directly from your local machine to a database hosted on cloud.gov. -The examples below show working with a PostgreSQL database, but should be similar for MySQL or others. +To connect to databases hosted on cloud.gov, you need to use an SSH tunnel. Two options for creating an SSH tunnel are outlined below. -### Using cg-manage-rds +### Using `cf connect-to-service` to open a tunnel -#### Exporting from a service instance +First, open a terminal and connect to an instance using the [cf-service-connect plugin](https://github.com/cloud-gov/cf-service-connect#usage) to create a SSH tunnel: -The `cg-manage-rds` application is meant to simplify and streamline import, export and cloning operations on service instances. Full usage docs can be found on the github [readme](https://github.com/cloud-gov/cg-manage-rds#usage) +```sh +$ cf connect-to-service -no-client ${APP_NAME} ${SERVICE_NAME} +... +Host: localhost +Port: ... +Username: ... +Password: ... +Name: ... +``` -To perform a basic export of a postgres instance using the compressed format: +If this fails to open a SSH tunnel, try deleting any existing `connect-to-service` service keys first: ```sh -cg-manage-rds export -o "-F c" -f ./backup.pg ${SERVICE_NAME} +cf delete-service-key ${SERVICE_NAME} SERVICE_CONNECT ``` -This will create an export using `pg_dump` named `backup.pg`. Other options for the pg_dump command can be pased as a string with the `-o` option. +Then try the previous step again. -#### Importing to a service instance +Once the SSH tunnel is created, keep it running in that terminal window to use the tunnel to make database connections. -This is a simple example of importing a previous export to database service instance. -By default `cg-manage-rds` adds options to remove ownership and create new objects to make porting easy. +The `Port` specified in the terminal output is the port on your local machine where the tunnel to your database is open. You should use this port when making connections to the database on `localhost` for utilities like `pg_dump`, `mysqldump`, or `mysqlsh`. -```sh -cg-manage-rds import -o "-F c" -f ./backup.pg ${SERVICE_NAME} -``` +### Using `cf ssh` -#### Cloning a service instance +> Note: you can find all the information for accessing your database (username, password, host, database name) by running `cf env app_name` for your app and looking at the `credentials` for your RDS database -This is a simple example of replicating database service instance to another instance. The destination database must be created beforehand. The export is downloaded locally as in the `export` command. +Open an SSH tunnel to your database: -```sh -cg-manage-rds clone ${SERVICE_NAME_SOURCE} ${SERVICE_NAME_DEST} +```shell +cf ssh -N -L :: ``` -### Using cf-service-connect plugin +with these values: -#### Exporting a database +- `` - port to open on your local machine for accessing your database. This value can be any port that is available for use on your local machine. +- `` - port database is running on in RDS +- `` - AWS host for accessing your database +- `` - your application name -First, open a terminal and connect to an instance using the [cf-service-connect plugin](https://github.com/cloud-gov/cf-service-connect#usage) to create a SSH tunnel: +If the `cf ssh` command succeeded, you should see an empty shell prompt that looks something like, which indicates that the SSH tunnel is open: -```sh -$ cf connect-to-service -no-client ${APP_NAME} ${SERVICE_NAME} -... -Host: localhost -Port: ... -Username: ... -Password: ... -Name: ... +```shell +vcap@abc123-de45-fg67-hi89-jk10:~$ ``` -If this fails to open a SSH tunnel, try deleting any existing `connect-to-service` service keys first: +Once the SSH tunnel is open, your database should be available for connections on `localhost:`. -```sh -cf delete-service-key ${SERVICE_NAME} SERVICE_CONNECT -``` +## Exporting your database -Then try the previous step again. +### Exporting using `cg-manage-rds` -Once the SSH tunnel is created, keep it running in that terminal window and open a separate terminal session in another window/tab, then create the backup file using the parameters provided by the plugin in the new terminal session, e.g. (be sure to tailor the backup/export command to your specific needs): +To perform a basic export of a PostgreSQL instance using the compressed format using [`cg-manage-rds`][cg-manage-rds]: ```sh -$ pg_dump -F c \ - --no-acl \ - --no-owner \ - -f backup.pg \ - postgresql://${USERNAME}:${PASSWORD}@${HOST}:${PORT}/${NAME} +cg-manage-rds export -o "-F c" -f ./backup.pg ${SERVICE_NAME} ``` -This will create the `backup.pg` file on your local machine in whatever your current working directory is. +This will create an export using `pg_dump` named `backup.pg`. Other options for the pg_dump command can be pased as a string with the `-o` option. -When you are finished, you can terminate the SSH tunnel. You should also clean up the `SERVICE_KEY` created by the cf-service-connect plugin so that you are able to reconnect in the future: +### Exporting a database manually -```sh -cf delete-service-key ${SERVICE_NAME} SERVICE_CONNECT -``` +1. [Create an SSH tunnel to your database](#tunneling-to-your-database) and keep it running in that terminal window +1. Open a separate terminal session in another window/tab +1. View the credentials for accessing your database by running `cf env app_name` for the app connected to your database and looking at the `credentials` for your RDS database +1. In the separate terminal window/tab, create the backup file (be sure to tailor the backup/export command to your specific needs). + + For example, to create a dump of a PostgreSQL database: + + ```sh + $ pg_dump -F c \ + --no-acl \ + --no-owner \ + -f backup.pg \ + postgresql://:${PASSWORD}@localhost:/ + ``` -### Restoring to a local database + with the values: -Continuing with the PostgreSQL example and the `backup.pg` file, load the dump into your local database using the [pg_restore](https://www.postgresql.org/docs/current/static/app-pgrestore.html) tool. If objects exist in a local copy of the database already, you might run into inconsistencies when doing a `pg_restore`. This pg_restore invocation does not drop all of the objects in the database when loading the dump: + - `` - username for accessing your database + - `` - port opened for SSH tunnel to your database + - `` - database name + + This command will create the `backup.pg` file on your local machine in the current working directory. + +1. When you are finished, you can terminate the SSH tunnel. + +## Restoring to a database + +### Restoring to a database using `cg-manage-rds` + +This is a simple example of importing a previous export to database service instance. +By default [`cg-manage-rds`][cg-manage-rds] adds options to remove ownership and create new objects to make porting easy. ```sh -pg_restore --clean --no-owner --no-acl --dbname={database name} backup.pg +cg-manage-rds import -o "-F c" -f ./backup.pg ${SERVICE_NAME} ``` -### Importing to a service instance - Windows +### Restoring to a database manually -> Note: you can find all the information for accessing your database (username, password, host, database name) by running `cf env app_name` for your app and looking at the `credentials` for your RDS database +Once you have a database backup file, you can import the dump into another database. -Open an SSH tunnel to your database: +1. If you want to restore to a cloud.gov hosted database, [you will need to first open a tunnel to that database which will expose a port on your local machine for connecting to the database](#tunneling-to-your-database). + - If you are restoring to a local database, opening a tunnel is not necessary. +1. View the credentials necessary for accessing your database by running `cf env ` on the app connected to your database. +1. Use the commands for PostgreSQL/MySQL below to restore your database. -```shell -cf ssh -N -L port:host:port application_name + - For importing to a cloud.gov database, the `` value should be the port opened by the SSH tunnel on your local machine. + - For importing to a local database, the `` value should be whatever is configured for that database service + +#### PostgreSQL + +This pg_restore invocation does not drop all of the objects in the database when loading the dump, so if objects exist in a local copy of the database already, you might run into inconsistencies when doing a `pg_restore`: + +```sh +pg_restore --clean --no-owner --no-acl \ + -h localhost \ + -p \ + -U + --dbname= backup.pg ``` -with these values: +If you want to drop and recreate the database before doing a restore, thus avoiding any data collisions, use the `--create` flag: -- `port` - port for accessing your database -- `host` - AWS host for accessing your database -- `application_name` - your application name +```sh +pg_restore --create --clean --no-owner --no-acl \ + -h localhost \ + -p \ + -U + --dbname= backup.pg +``` -Once the SSH tunnel is open, your database should be available for connections on `localhost:`. +#### MySQL -Now you can run a command in a separate prompt to import a data backup into the database. For example, using the [`mysqlsh` tool](https://dev.mysql.com/doc/mysql-shell/8.0/en/): +Run this command to import a database backup into a MySQL database using the [`mysqlsh` tool](https://dev.mysql.com/doc/mysql-shell/8.0/en/): ```shell mysqlsh -u username -p -h host -P port -D db_name -f path-to-file.sql @@ -500,9 +538,9 @@ with these values: - `db_name` - database name for accessing your database - `path-to-file.sql` - Full path to the database backup file on your machine -### Database import errors +## Database import errors -#### MySQL +### MySQL If you get an error like `ERROR: 1227` when trying to import to your database: @@ -532,6 +570,55 @@ If you get an error like `ERROR: 1227` when trying to import to your database: If those steps do not help, additional remediation steps can be found in the [AWS knowledge center article on how to resolve this error](https://repost.aws/knowledge-center/definer-error-mysqldump). +## Cloning a service instance using `cg-manage-rds` + +This is a simple example of replicating database service instance to another instance using [`cg-manage-rds`][cg-manage-rds]. The destination database must be created beforehand. The export is downloaded locally as in the `export` command. + +```sh +cg-manage-rds clone ${SERVICE_NAME_SOURCE} ${SERVICE_NAME_DEST} +``` + +## Opening a shell to your cloud.gov database + +### Using `cf connect-to-service` to start a database shell + +Run this command: + +```shell +cf connect-to-service +``` + +with the values: + +- `` - the name of the app connected to your database +- `` - the name of your database services + +By default, `cf connect-to-service` should open a database shell for the relevant database type if you have the utility installed on your machine (e.g. `psql` for PostgreSQL databases). + +### Using `cf ssh` to start a database shell + +1. [Create a tunnel to your database](#tunneling-to-your-database) +2. In another terminal window, connect to the database: + + ```shell + myapp_guid=$(cf app --guid ) + + creds=$(cf curl /v2/apps/$myapp_guid/env \ + | jq -r '[.system_env_json.VCAP_SERVICES."aws-rds"[0].credentials \ + | .username, .password] \ + | join(":")') + + dbname=$(cf curl /v2/apps/$myapp_guid/env \ + | jq -r '.system_env_json.VCAP_SERVICES."aws-rds"[0].credentials \ + | .name') + + psql postgres://$creds@localhost:5432/$dbname + ``` + + with values: + + - `` - The name of the app connected to your database + ## Encryption Every RDS instance configured through cloud.gov is [encrypted at rest](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html). We use the industry standard AES-256 encryption algorithm to encrypt your data on the server that hosts your RDS instance. The RDS then handles authenticating access and decrypting your data, with minimal performance impact and without requiring you to modify your applications. @@ -621,38 +708,6 @@ Now connect using `sqlplus random-username/secretpassword@host:port/ORCL`, where Then you can use SQLPLUS commands like `SELECT table_name FROM user_tables;` -## Connect to databases without use of `connect-to-service` - -Example for app name `hello-doe` - -```shell -myapp_guid=$(cf app --guid hello-doe) - -tunnel=$(cf curl /v2/apps/$myapp_guid/env \ - | jq -r '[.system_env_json.VCAP_SERVICES."aws-rds"[0].credentials \ - | .host, .port] \ - | join(":")') - -cf ssh -N -L 5432:$tunnel hello-doe -``` - -Another window: - -```shell -myapp_guid=$(cf app --guid hello-doe) - -creds=$(cf curl /v2/apps/$myapp_guid/env \ - | jq -r '[.system_env_json.VCAP_SERVICES."aws-rds"[0].credentials \ - | .username, .password] \ - | join(":")') - -dbname=$(cf curl /v2/apps/$myapp_guid/env \ - | jq -r '.system_env_json.VCAP_SERVICES."aws-rds"[0].credentials \ - | .name') - -psql postgres://$creds@localhost:5432/$dbname -``` - ## Version information The software versions listed in the table above are for new instances of those plans. @@ -665,4 +720,6 @@ For Oracle plans, minor upgrades are not automatic. To upgrade an existing Oracl ## The broker in GitHub -You can find the broker here: [https://github.com/18F/aws-broker](https://github.com/18F/aws-broker). +You can find the broker here: [https://github.com/cloud-gov/aws-broker](https://github.com/cloud-gov/aws-broker). + +[cg-manage-rds]: https://github.com/cloud-gov/cg-manage-rds#usage diff --git a/_posts/2021-12-22-log4j_vulnerability_bod_22-02_update.md b/_posts/2021-12-22-log4j_vulnerability_bod_22-02_update.md index 4e12ae98e..4ab975967 100644 --- a/_posts/2021-12-22-log4j_vulnerability_bod_22-02_update.md +++ b/_posts/2021-12-22-log4j_vulnerability_bod_22-02_update.md @@ -9,7 +9,7 @@ Update 2021-12-27: In compliance with directives from the FedRAMP JAB, cloud.gov cloud.gov Customer Responsibilities * Customers are responsible for ensuring that their Java applications are fully patched or otherwise have this vulnerability mitigated. For guidance, refer to [CISA’s Log4J guidance](https://www.cisa.gov/news-events/cybersecurity-advisories/aa21-356a). -* If you have reporting requirements, e.g. [ED 22-02](https://www.cisa.gov/emergency-directive-22-02), regarding cloud.gov components that you leverage: +* If you have reporting requirements, e.g. [ED 22-02](https://www.cisa.gov/news-events/directives/ed-22-02-mitigate-apache-log4j-vulnerability-closed), regarding cloud.gov components that you leverage: * cloud.gov brokered AWS Elasticsearch components were patched at the latest by Dec 21, 2021. * All other cloud.gov brokered components, AWS RDS, AWS S3, AWS Elasticache Redis, External Domains, Service Accounts and Identity Provider, were patched by Dec 13, 2021.