Custom Python Source Connector: Handling Transformations Beyond Stream Slices in Airbyte

[Kressaa]

I’m working on creating a custom Python source connector in Airbyte to extract data from an HTTP-based API. I’ve implemented specific stream-slicing logic, and the API responses are received and processed slice by slice. I’ve also added custom transformations for the responses, which are applied per slice.

However, I’d like to apply transformations to the entire response instead of applying transformation to each response slice individually. Is there a way to achieve this in Airbyte?

Additionally, I’m curious about how Airbyte internally handles loading source data to the destination when stream slicing is implemented. Can anyone share insights or best practices for this?