Skip to content

Commit

Permalink
Update documentation for Getting Started with data migration (#9030)
Browse files Browse the repository at this point in the history 
* Update documentation for Getting Started with data migration

Signed-off-by: Jugal Chauhan <[email protected]>

* Update _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Move list of deployed stacks to beginning of Step 5.

Signed-off-by: Naarcha-AWS <[email protected]>

* Update phrasing of Step 4

Signed-off-by: Jugal Chauhan <[email protected]>

* Update getting-started-data-migration.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Jugal Chauhan <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Co-authored-by: Naarcha-AWS <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
  • Loading branch information
@jugal-chauhan @Naarcha-AWS @natebower
3 people authored Jan 17, 2025
1 parent 2957a1f commit a4ae08c
Showing 1 changed file with 52 additions and 45 deletions.
97 changes: 52 additions & 45 deletions ...ation-assistant/deploying-migration-assistant/getting-started-data-migration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Before using this quickstart, make sure you fulfill the following prerequisites:
* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/#migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported.
* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin.
* The target cluster must be deployed.
* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html).

The steps in this guide assume the following:

Expand Down Expand Up @@ -70,6 +71,7 @@ Use the following steps to set up Bootstrap instance access:
{% include copy.html %}

3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**.
4. Attach the newly created policy to your EC2 instance's IAM role.

---

Expand Down Expand Up @@ -107,14 +109,23 @@ To use these steps, make sure you fulfill the following prerequisites:

## Step 4: Configure and deploy RFS (~20 minutes)

Use the following steps to configure and deploy RFS:
To deploy Migration Assistant with RFS, the following stacks must be deployed:

1. Add the target cluster password to AWS Secrets Manager as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment.
2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory:
These commands deploy the following stacks:

* `Migration Assistant network` stack
* `RFS` stack
* `Migration console` stack

Use the following steps to configure and deploy RFS, deploy Migration Assistant, and verify installation of the required stacks:

1. Add the source and target cluster password as separate **Secrets** in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment.
2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory and configure the following settings:

```json
{
"migration-assistant": {
"stage": "dev",
"vpcId": "<TARGET CLUSTER VPC ID>",
"targetCluster": {
"endpoint": "<TARGET CLUSTER ENDPOINT>",
Expand All @@ -126,14 +137,14 @@ Use the following steps to configure and deploy RFS:
},
"sourceCluster": {
"endpoint": "<SOURCE CLUSTER ENDPOINT>",
"version": "<SOURCE ENGINE VERSION>",
"auth": {
"type": "basic",
"username": "<TARGET CLUSTER USERNAME>",
"passwordFromSecretArn": "<TARGET CLUSTER PASSWORD SECRET>"
}
},
"reindexFromSnapshotExtraArgs": "<RFS PARAMETERS (see below)>",
"stage": "dev",
"otelCollectorEnabled": true,
"migrationConsoleServiceEnabled": true,
"reindexFromSnapshotServiceEnabled": true,
Expand All @@ -145,61 +156,55 @@ Use the following steps to configure and deploy RFS:

The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`.

3. Bootstrap the account with the following command:
3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command:

```bash
cdk bootstrap --c contextId=migration-assistant --require-approval never
```
{% include copy.html %}

4. Deploy the stacks:
4. Deploy Migration Assistant using the following command:

```bash
cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 5
```
{% include copy.html %}

5. Verify that all CloudFormation stacks were installed successfully.

### RFS parameters

If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values:

5. From the same Bootstrap instance shell, verify that all CloudFormation stacks were installed successfully:

```bash
--s3-repo-uri s3://<bucket-name>/<repo> --s3-region <region> --snapshot-name <name>
aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table
```
{% include copy.html %}

You should receive a similar output for your Region:

You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket.

---

## Step 5: Deploy Migration Assistant

To deploy Migration Assistant, use the following steps:
```bash
------------------------------------------------------------------------
| ListStacks |
+--------------------------------------------------+-------------------+
| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE |
| MigrationBootstrap | CREATE_COMPLETE |
| CDKToolkit | CREATE_COMPLETE |
+--------------------------------------------------+-------------------+
```

1. Bootstrap the account:

```bash
cdk bootstrap --c contextId=migration-assistant --require-approval never --concurrency 5
```
{% include copy.html %}
### RFS parameters

2. Deploy the stacks when `cdk.context.json` is fully configured:

```bash
cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 3
```
{% include copy.html %}
If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values:

These commands deploy the following stacks:
```bash
"reindexFromSnapshotExtraArgs": "--s3-repo-uri s3://<bucket-name>/<repo> --s3-region <region> --snapshot-name <name>"
```

* Migration Assistant network stack
* `Reindex-from-snapshot` stack
* Migration console stack
You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket.

---

## Step 6: Access the migration console
## Step 5: Access the migration console

Run the following command to access the migration console:

Expand All @@ -214,7 +219,7 @@ Run the following command to access the migration console:

---

## Step 7: Verify the connection to the source and target clusters
## Step 6: Verify the connection to the source and target clusters

To verify the connection to the clusters, run the following command:

Expand All @@ -226,15 +231,17 @@ console clusters connection-check
You should receive the following output:

```bash
* **Source Cluster:** Successfully connected!
* **Target Cluster:** Successfully connected!
SOURCE CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='')
TARGET CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='')
```

To learn more about migration console commands, see [Migration commands].

---

## Step 8: Create a snapshot
## Step 7: Create a snapshot

Run the following command to initiate snapshot creation from the source cluster:

Expand Down Expand Up @@ -263,7 +270,7 @@ To learn more about snapshot creation, see [Snapshot Creation].

---

## Step 9: Migrate metadata
## Step 8: Migrate metadata

Run the following command to migrate metadata:

Expand All @@ -276,7 +283,7 @@ For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migr

---

## Step 10: Migrate documents with RFS
## Step 9: Migrate documents with RFS

You can now use RFS to migrate documents from your original cluster:

Expand Down Expand Up @@ -312,7 +319,7 @@ For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assi

---

## Step 11: Backfill monitoring
## Step 10: Backfill monitoring

Use the following command for detailed monitoring of the backfill process:

Expand All @@ -339,7 +346,7 @@ Logs and metrics are available in Amazon CloudWatch in the `OpenSearchMigrations

---

## Step 12: Verify that all documents were migrated
## Step 11: Verify that all documents were migrated

Use the following query in CloudWatch Logs Insights to identify failed documents:

Expand Down

0 comments on commit a4ae08c

Please sign in to comment.