Partial Batch Retry

[stefan-fast]

Hello everyone,

the docs in the README state the following:

If an exception is thrown while processing the batch, all messages in the batch will be returned to the queue, to be retried with the standard retry system. There is no guarantee that the messages will be retried again in the same batch.

I would like to use batch processing, however I would also like to be able to only return messages inside the batch to queue that failed. Let's say I have a batch size of 5. If message 1 and 2 were successful but message 3 failed, 1 and 2 will also be send back to queue. Also 4 and 5 will be send back even though processing of them never started in the first place.

So my question is: Is there an option to only do partial batch retries where only failed messages are send back to queue?

Thanks in advance!

[rkolesnev]

Hi @stefan-fast,
I think the main thinking behind handling of batch processing results is the expectation that processing is done on the whole batch - i.e. SQL statement or web service call etc that either succeeds or fails for the batch as a whole.
Hence the results being handled on the whole batch as well.

Could you please give more information on how in batch processing there are only partial failures or messages not processed at all?

Partial batch retries could be implemented with new custom exception where user function would populate the exception with details of which messages failed and which succeeded, and then Parallel Consumer could mark messages according to those details - some as failed and some as succeeded.
It would still need to mark the messages that weren't even tried to process as failed as well to allow them to be retried - but generally there shouldn't be messages like that i would guess.

But it sounds very niche - hence i would like to understand why this behaviour is needed at all?

[stefan-fast]

Hi @rkolesnev,

Thank you for your quick and detailed response and explanation.

I could probably modify my code to process the entire batch within a single transaction. Currently, I handle each message in a batch individually. I haven’t encountered any issues so far, but I was experimenting to see what potential problems might arise. One scenario I tested involved a new message on the topic that couldn’t be processed due to deserialization issues, such as a JSON string that couldn’t be deserialized into a known object. However, this issue can be managed in various ways without causing the entire batch to fail.

Thanks again for providing the technical details. I now have a clearer understanding of the intended use of the batch feature. In my case, I should probably either stick to a batch size of 1 to handle messages individually or, as you suggested, handle the entire batch in one transaction, which aligns with the intended purpose of the batch feature.