diff --git a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md index f260a28701..2608a6eb44 100644 --- a/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md +++ b/_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md @@ -20,6 +20,7 @@ Before using this quickstart, make sure you fulfill the following prerequisites: * Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/#migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported. * The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin. * The target cluster must be deployed. +* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about how to bootstrap your AWS account in the required AWS Region, see [the CDKToolkit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html). The steps in this guide assume the following: @@ -70,6 +71,7 @@ Use the following steps to set up Bootstrap instance access: {% include copy.html %} 3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**. +4. Attach the newly created policy to your EC2 instance's IAM role. --- @@ -107,14 +109,23 @@ To use these steps, make sure you fulfill the following prerequisites: ## Step 4: Configure and deploy RFS (~20 minutes) -Use the following steps to configure and deploy RFS: +To deploy Migration Assistant with RFS, the following stacks must be deployed: -1. Add the target cluster password to AWS Secrets Manager as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment. -2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory: +These commands deploy the following stacks: + +* `Migration Assistant network` stack +* `RFS` stack +* `Migration console` stack + +Use the following steps to configure and deploy RFS, deploy Migration Assistant, and verify installation of the required stacks: + +1. Add the source and target cluster password as separate **Secrets** in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment. +2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory and configure the following settings: ```json { "migration-assistant": { + "stage": "dev", "vpcId": "", "targetCluster": { "endpoint": "", @@ -126,6 +137,7 @@ Use the following steps to configure and deploy RFS: }, "sourceCluster": { "endpoint": "", + "version": "", "auth": { "type": "basic", "username": "", @@ -133,7 +145,6 @@ Use the following steps to configure and deploy RFS: } }, "reindexFromSnapshotExtraArgs": "", - "stage": "dev", "otelCollectorEnabled": true, "migrationConsoleServiceEnabled": true, "reindexFromSnapshotServiceEnabled": true, @@ -145,61 +156,55 @@ Use the following steps to configure and deploy RFS: The source and target cluster authorization can be configured to have no authorization, `basic` with a username and password, or `sigv4`. -3. Bootstrap the account with the following command: +3. After the `cdk.context.json` file is fully configured, bootstrap the account and deploy the required stacks using the following command: ```bash cdk bootstrap --c contextId=migration-assistant --require-approval never ``` {% include copy.html %} -4. Deploy the stacks: +4. Deploy Migration Assistant using the following command: ```bash cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 5 ``` {% include copy.html %} - -5. Verify that all CloudFormation stacks were installed successfully. - -### RFS parameters - -If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values: + +5. From the same Bootstrap instance shell, verify that all CloudFormation stacks were installed successfully: ```bash - --s3-repo-uri s3:/// --s3-region --snapshot-name + aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table ``` + {% include copy.html %} + +You should receive a similar output for your Region: -You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket. - ---- - -## Step 5: Deploy Migration Assistant - -To deploy Migration Assistant, use the following steps: +```bash +------------------------------------------------------------------------ +| ListStacks | ++--------------------------------------------------+-------------------+ +| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE | +| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE | +| MigrationBootstrap | CREATE_COMPLETE | +| CDKToolkit | CREATE_COMPLETE | ++--------------------------------------------------+-------------------+ +``` -1. Bootstrap the account: - - ```bash - cdk bootstrap --c contextId=migration-assistant --require-approval never --concurrency 5 - ``` - {% include copy.html %} +### RFS parameters -2. Deploy the stacks when `cdk.context.json` is fully configured: - - ```bash - cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 3 - ``` - {% include copy.html %} +If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values: -These commands deploy the following stacks: +```bash + "reindexFromSnapshotExtraArgs": "--s3-repo-uri s3:/// --s3-region --snapshot-name " +``` -* Migration Assistant network stack -* `Reindex-from-snapshot` stack -* Migration console stack +You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket. --- -## Step 6: Access the migration console +## Step 5: Access the migration console Run the following command to access the migration console: @@ -214,7 +219,7 @@ Run the following command to access the migration console: --- -## Step 7: Verify the connection to the source and target clusters +## Step 6: Verify the connection to the source and target clusters To verify the connection to the clusters, run the following command: @@ -226,15 +231,17 @@ console clusters connection-check You should receive the following output: ```bash -* **Source Cluster:** Successfully connected! -* **Target Cluster:** Successfully connected! +SOURCE CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') +TARGET CLUSTER +ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='') ``` To learn more about migration console commands, see [Migration commands]. --- -## Step 8: Create a snapshot +## Step 7: Create a snapshot Run the following command to initiate snapshot creation from the source cluster: @@ -263,7 +270,7 @@ To learn more about snapshot creation, see [Snapshot Creation]. --- -## Step 9: Migrate metadata +## Step 8: Migrate metadata Run the following command to migrate metadata: @@ -276,7 +283,7 @@ For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migr --- -## Step 10: Migrate documents with RFS +## Step 9: Migrate documents with RFS You can now use RFS to migrate documents from your original cluster: @@ -312,7 +319,7 @@ For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migration-assi --- -## Step 11: Backfill monitoring +## Step 10: Backfill monitoring Use the following command for detailed monitoring of the backfill process: @@ -339,7 +346,7 @@ Logs and metrics are available in Amazon CloudWatch in the `OpenSearchMigrations --- -## Step 12: Verify that all documents were migrated +## Step 11: Verify that all documents were migrated Use the following query in CloudWatch Logs Insights to identify failed documents: