IndexReplicationTask will never terminate when the index is no more

[kranthikirang]

What is the bug?
Opensearch data nodes keep trying to the index which is already deleted from leader and follower nodes but the IndexReplicationTask will never terminate.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create the auto-follow replication against leader indexes.
2. Leader indexes will have a policy which will delete the indexes after some time (let's say 1 hour).
3. Then in follower cluster delete the same index manually as its not there in leader cluster.
4. Look into the data node logs

[2025-01-16T20:33:18,371][ERROR][o.o.r.t.i.IndexReplicationTask] [opensearch-cluster-data-hot-data-3] [cluster_westus2_vm-uat-000011] Error in getting the required metadata [cluster_westus2_vm-uat-000011] IndexNotFoundException[no such index [cluster_westus2_vm-uat-000011]] at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1070) at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1007) at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:963) at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:255) at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:235) at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:115) at org.opensearch.action.admin.indices.settings.get.TransportGetSettingsAction.checkBlock(TransportGetSettingsAction.java:101) at org.opensearch.action.admin.indices.settings.get.TransportGetSettingsAction.checkBlock(TransportGetSettingsAction.java:64) at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.checkForBlock(TransportClusterManagerNodeAction.java:409) at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.doStart(TransportClusterManagerNodeAction.java:270) at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction.tryAction(TransportClusterManagerNodeAction.java:239) at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:200) at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction.doExecute(TransportClusterManagerNodeAction.java:88) at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:325) at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:166) at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:133) at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:129) at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:207) at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:317) at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:155) at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:866) at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44) at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:63) at org.opensearch.ratelimitting.admissioncontrol.transport.AdmissionControlTransportHandler.messageReceived(AdmissionControlTransportHandler.java:68) at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) at org.opensearch.transport.NativeMessageHandler.handleRequest(NativeMessageHandler.java:278) at org.opensearch.transport.NativeMessageHandler.handleMessage(NativeMessageHandler.java:146) at org.opensearch.transport.NativeMessageHandler.messageReceived(NativeMessageHandler.java:126) at org.opensearch.transport.InboundHandler.messageReceivedFromPipeline(InboundHandler.java:121) at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:113) at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:800) at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.forwardFragments(NativeInboundBytesHandler.java:157) at org.opensearch.transport.nativeprotocol.NativeInboundBytesHandler.doHandleBytes(NativeInboundBytesHandler.java:94) at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:143) at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:119) at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1473) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(Thread.java:1583)

What is the expected behavior?
If the index is no more then the task should canceled or terminated appropriately.

What is your host/environment?
opensearch-cluster-master-0 opensearch-cross-cluster-replication 2.16.0.0
opensearch 2.16.0