Streaming S3 files directly to Axum clients

[Gio2241]

have an Axum server where one of the routes accesses an S3 storage and returns a JSON file based on the request. The current implementation downloads the file to the server via the rust-s3 crate and returns Result<Json, StatusCode> from the Axum route.
This works, but I want to make it more efficient by streaming the file directly from S3 to the client.

I have the following code to get a stream from S3:

/// Downloads a file from an S3 bucket as a stream.
/// This function returns a stream of the requested file from the specified S3 bucket.
pub async fn download_stream(bucket_name: &str, remote_path: &str) -> Result<ResponseDataStream> {
    // Create S3 region and credentials from environment variables.
    ...

    // Connect to the S3 bucket.
    let bucket = Bucket::new(bucket_name, region, credentials)?;

    // Fetch the file from the S3 bucket as a stream.
    let object_stream = bucket.get_object_stream(remote_path).await?;

    Ok(object_stream)
}

The return value is ResponseDataStream 1.

Now, I want to return the stream from axum route:

pub async fn do_analysis(
    Path((sha, id)): Path<(String, String)>,
) -> Result<Json<Value>, StatusCode> {
...
match s3::download_stream(&s3_bucket, &dynamic).await {
    Ok(stream) => Ok(Json(stream)), // invalid code
    Err(_) => Err(StatusCode::NOT_FOUND),
}
}

I'm not sure if the return type is correct or what to return from the do_analysis route. What's the best way to implement this logic?

[davidpdrsn]

axum does not provide a way to stream a single JSON object. The closest thing we have is JsonLines but that's a stream of JSON with one value per line. Which would be faster than buffering an array and sending that all at once.

Why do you wanna return JSON in the first place? Can't you just stream the data as is?

[Gio2241]

Yes, returning as data would work

How couls I do that?

[davidpdrsn]

Give https://docs.rs/axum/0.6.18/axum/body/struct.StreamBody.html a shot

[Gio2241]

tried.

let body = StreamBody::new(stream.bytes());

error:

   --> src/routes.rs:243:32
    |
243 |                     let body = StreamBody::new(data_stream.bytes());
    |                                ^^^^^^^^^^^^^^^ expected `Bytes`, found `Result<_, _>`
    |
    = note: expected struct `axum::body::Bytes`
                 found enum `Result<_, _>`
    = note: required for `&mut Pin<Box<(dyn Stream<Item = axum::body::Bytes> + 'static)>>` to implement `TryStream`
    ...
  error[E0271]: type mismatch resolving `<&mut Pin<Box<dyn Stream<Item = Bytes>>> as Stream>::Item == Result<_, _>`
   --> src/routes.rs:243:48
    |
243 |                     let body = StreamBody::new(data_stream.bytes());
    |                                --------------- ^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `Bytes`
    |                                |
    |                                required by a bound introduced by this call
    |
    = note: expected enum `Result<_, _>`
             found struct `axum::body::Bytes`
    = note: required for `&mut Pin<Box<(dyn Stream<Item = axum::body::Bytes> + 'static)>>` to implement `TryStream`

[Gio2241]

Version 0.33 of rust-s3 does not implement SEND, beta4 and master branch works!
https://docs.rs/rust-s3/0.34.0-beta3/s3/request/request_trait/struct.ResponseDataStream.html

[davidpdrsn]

The issue you got before was that your stream must yield Result where the Ok value implements Into<Bytes> (typically you'd just yield Bytes values).