[BUG] Not able to change translog related replication settings.

[dewsingh]

What is the bug?
Not able to change translog related replication settings plugins.replication.translog.retention_lease.pruning.enabled and plugins.replication.translog.retention_size

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Install OpenSearch leader and follower clusters with cross cluster replication plugin enabled and configured.
2. Verify the replication is working and the follower is able to replicate from leader cluster.
3. Try changing the plugins.replication.translog.retention_size replication setting.
4. Below error is observed.

curl -u admin:xxx -XPUT -k "https://opensearch.xxx:9200/_cluster/settings" -H 'Content-Type: application/json' -d '
{
"persistent": {
"plugins.replication.translog.retention_size": "256mb"
}
}'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"persistent setting [plugins.replication.translog.retention_size], not recognized"}],"type":"illegal_argument_exception","reason":"persistent setting [plugins.replication.translog.retention_size], not recognized"},"status":400}

What is the expected behavior?
Retention size of the translog should be changed.

What is your host/environment?

• Version - 2.12.0.0

[bharath-techie]

@dewsingh can you verify that the plugin is installed properly using cat plugins - https://opensearch.org/docs/latest/api-reference/cat/cat-plugins/ . Maybe paste the output here.

[ Triage attendees - 1 2 3 4]

[dewsingh]

@bharath-techie , here is the output
From the leader cluster

curl -u admin:xxxx -XGET -k "https://opensearch.xxxx:9200/_cat/plugins?v"
name component version
ifd-leader-client-6bcd6dc768-vd6nl ingest-attachment 2.12.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-cross-cluster-replication 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-custom-codecs 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-index-management 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-job-scheduler 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-reports-scheduler 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl opensearch-security 2.12.0.0
ifd-leader-client-6bcd6dc768-vd6nl prometheus-exporter 2.12.0.0
ifd-leader-manager-0 ingest-attachment 2.12.0
ifd-leader-manager-0 opensearch-cross-cluster-replication 2.12.0.0
ifd-leader-manager-0 opensearch-custom-codecs 2.12.0.0
ifd-leader-manager-0 opensearch-index-management 2.12.0.0
ifd-leader-manager-0 opensearch-job-scheduler 2.12.0.0
ifd-leader-manager-0 opensearch-reports-scheduler 2.12.0.0
ifd-leader-manager-0 opensearch-security 2.12.0.0
ifd-leader-manager-0 prometheus-exporter 2.12.0.0
ifd-leader-data-0 ingest-attachment 2.12.0
ifd-leader-data-0 opensearch-cross-cluster-replication 2.12.0.0
ifd-leader-data-0 opensearch-custom-codecs 2.12.0.0
ifd-leader-data-0 opensearch-index-management 2.12.0.0
ifd-leader-data-0 opensearch-job-scheduler 2.12.0.0
ifd-leader-data-0 opensearch-reports-scheduler 2.12.0.0
ifd-leader-data-0 opensearch-security 2.12.0.0
ifd-leader-data-0 prometheus-exporter 2.12.0.0

From follower cluster

curl -u admin:xxxx -XGET -k "https://opensearch.xxxx:9200/_cat/plugins?v"
name component version
ifd-follower-manager-0 ingest-attachment 2.12.0
ifd-follower-manager-0 opensearch-cross-cluster-replication 2.12.0.0
ifd-follower-manager-0 opensearch-custom-codecs 2.12.0.0
ifd-follower-manager-0 opensearch-index-management 2.12.0.0
ifd-follower-manager-0 opensearch-job-scheduler 2.12.0.0
ifd-follower-manager-0 opensearch-reports-scheduler 2.12.0.0
ifd-follower-manager-0 opensearch-security 2.12.0.0
ifd-follower-manager-0 prometheus-exporter 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz ingest-attachment 2.12.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-cross-cluster-replication 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-custom-codecs 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-index-management 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-job-scheduler 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-reports-scheduler 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz opensearch-security 2.12.0.0
ifd-follower-client-7c468c8cfc-cxfhz prometheus-exporter 2.12.0.0
ifd-follower-data-0 ingest-attachment 2.12.0
ifd-follower-data-0 opensearch-cross-cluster-replication 2.12.0.0
ifd-follower-data-0 opensearch-custom-codecs 2.12.0.0
ifd-follower-data-0 opensearch-index-management 2.12.0.0
ifd-follower-data-0 opensearch-job-scheduler 2.12.0.0
ifd-follower-data-0 opensearch-reports-scheduler 2.12.0.0
ifd-follower-data-0 opensearch-security 2.12.0.0
ifd-follower-data-0 prometheus-exporter 2.12.0.0

[skumarp7]

Hi @bharath-techie ,

Any updates?

I also could not change any translog related settings:

plugins.replication.translog.retention_size
plugins.replication.translog.retention_lease.pruning.enabled

But im able to set these setting at the index level :

curl -k -u <user>:<pass> -XPUT
https://opensearch:9200/log-sanjay
-d '{"settings": {"index.plugins.replication.translog.retention_lease.pruning.enabled": "true"}}' -H 'Content-Type: application/json'
{"acknowledged":true,"shards_acknowledged":true,"index":"log-sanjay"}

[bharath-techie]

@nisgoel-amazon @ankitkala can you folks help here ?

[ankitkala]

Hi, This is expected. These settings are available at index level(and not cluster level)

[skumarp7]

Hi @ankitkala ,

These settings are still documented in the latest documentation page of Cross-Cluster Replication.

Ref: https://opensearch.org/docs/latest/tuning-your-cluster/replication-plugin/settings/

Can i raise a PR to remove this from the documentation?

[ankitkala]

Thanks @skumarp7, please go ahead with the update. We can either split the settings into cluster/index. Or add a column specifying the type of setting.