[BUG] Replication tests are flaky (index [books-names] blocked by: [FORBIDDEN/1000/index read-only(cross-cluster-replication))

[dblock]

What is the bug?

OpenSearch 2.18.0


ERROR   autofollow.yaml (tests/plugins/replication/autofollow.yaml)
WARNING Multiple paths detected, please group similar tests together and move paths not being tested to prologues or epilogues.
  /_plugins/_replication/_autofollow
  /_plugins/_replication/autofollow_stats

    ERROR   EPILOGUES
        PASSED  POST /_plugins/_replication/{index}/_stop
        ERROR   DELETE /books-names (index [books-names] blocked by: [FORBIDDEN/1000/index read-only(cross-cluster-replication)];)
        PASSED  DELETE /books


ERROR   index.yaml (tests/plugins/replication/index.yaml)
WARNING Multiple paths detected, please group similar tests together and move paths not being tested to prologues or epilogues.
  /_plugins/_replication/{index}/_start
  /_plugins/_replication/{index}/_update

    ERROR   CHAPTERS
        ERROR   Start replication.
            PASSED  PARAMETERS
                PASSED  index
            PASSED  REQUEST BODY
            ERROR   RESPONSE STATUS (Expected status 200, but received 400: application/json. Cant use same index again for replication. Delete the index:books-names)
            SKIPPED RESPONSE PAYLOAD BODY
            SKIPPED RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES
        SKIPPED Update settings.
    ERROR   EPILOGUES
        ERROR   POST /_plugins/_replication/{index}/_stop (No replication in progress for index:books-names)
        PASSED  DELETE /books-names
        PASSED  DELETE /books

PASSED  stats.yaml (tests/plugins/replication/stats.yaml)
WARNING Multiple paths detected, please group similar tests together and move paths not being tested to prologues or epilogues.
  /_plugins/_replication/{index}/_status
  /_plugins/_replication/leader_stats
  /_plugins/_replication/follower_stats
  /_plugins/_replication/autofollow_stats


Tested 7/5[28](https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12014616796/job/33491014455?pr=694#step:7:29) paths.
Writing coverage/test-spec-coverage-2.18.0-218a289a83d99f103c6e063e594[29](https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12014616796/job/33491014455?pr=694#step:7:30)0a9879b80c1b2c08[30](https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12014616796/job/33491014455?pr=694#step:7:31)15fa9752b7a36c9db.json ...
Error: Process completed with exit code 1.

How can one reproduce the bug?

https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12014616796/job/33491014455?pr=694

What is the expected behavior?

Tests to always pass.

What is your host/environment?

GHA.

Do you have any screenshots?

Do you have any additional context?

Maybe a simple retry can work around this or use books in one set of tests and movies in another?

[dblock]

@Tokesh maybe you want to take a look when you get a chance

[Tokesh]

@dblock
Oh! I'm sorry about that. I've already prepared a PR for your review. You can check it now.

[dblock]

It failed again in https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12243422490/job/34152955207 :(

[Tokesh]

Most probably, we need to realize 'timeout' parameter. If necessary, I can do this after 100% coverage of test cases. Perhaps some APIs like replication require about 5-20 seconds of waiting and they do not always have time to complete.

[dblock]

failure logs https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12670482466/artifacts/2401168664