APIGatewayRestResolver: How to receive a gzipped file?

[AuxiliumCDNG]

Hi!

I am currently having issues with receiving a binary file with APIGatewayRestResolver. I have to use gzip to avoid the 6MB Lambda payload limit but I cannot decompress it in my Lambda function.

I saved the exact contents of the body I get via router.current_event.body or router.current_event.decoded_body or router.current_event.raw_event.get("Body") to S3 and downloaded them. While they are identical, they are different to the gzip file I sent via Postman.

Am I missing a way to just access the "original" binary data?
Maybe also something I have to change in AWS?

I currently have it set up as just a Lambda Proxy Event. The rest of the function works, Content-Type header is set to application/gzip, I don't change anything about the event data...

Help would be greatly appreciated.

[rubenfonseca]

Hi @AuxiliumCDNG do you have any sample code that we can see to understand the situation a little bit better?

[AuxiliumCDNG]

I sadly can't show to much, but I will try to piece together something.

Main Lambda handler:

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver()

app.include_router(router_items, prefix="/items")

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    try:
        authContext = event["requestContext"]["authorizer"]
    except KeyError:
        authContext = event["auth-context"]

    app.append_context(shop=authContext["shopId"])
    return app.resolve(event, context)

router_items file:

router_items = Router()

@router_items.post("/<typ>/<language_code>")
def add(typ: str, language_code: str):
    ...
    if router_items.current_event.headers.get("Content-Type") == "application/json":
        products: list = router_items.current_event.json_body["Products"]
    elif router_items.current_event.headers.get("Content-Type") == "application/gzip":
        client = boto3.client("s3")
        client.put_object(
            Body=router_items.current_event.raw_event.get("Body"),
            Bucket="opensearchapidev",
            ContentType="application/gzip",
            Key=token_urlsafe(16)
        )
        products: list = json.loads(
            gzip.decompress(router_items.current_event.raw_event["body"].encode())  # <-  What do I put here?
        )

    # Further processing...

Hope that helps! Feel free to ask if you need anything else.

[rubenfonseca]

Awesome this already helps me to debug it. Last question: does the code work? Or do you get an error when you try to gzip.decompress it?

[AuxiliumCDNG]

No, decompression does not work. I also cannot open the file I upload to S3 with 7zip. Both say it's not a gzip file.

[rubenfonseca]

Got it, will get back to you later

[rubenfonseca]

All right found the answer: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-make-request-with-compressed-payload.html

To make an API request with a compressed payload, the client must set the Content-Encoding header with one of the supported content codings.
When API Gateway receives the request, it verifies if the specified content coding is supported. Then, it attempts to decompress the payload with the specified content coding. If the decompression is successful, it dispatches the request to the integration endpoint.

So when you're passing the correct HTTP headers, API Gateway will actually do the inflate for you automatically before calling your function.

I've tested it with this code:

import json

from aws_lambda_powertools import Tracer, Logger
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.event_handler import (
    APIGatewayRestResolver,
)

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver()


@app.post("/items")
def add():
    products: list = app.current_event.json_body
    logger.info("List of products reeived", products=products)
    return products


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event, context):
    print(json.dumps(event))
    return app.resolve(event, context)

A POST request with just Content-Type: application/json and a JSON list on the body, correctly returns the list of products.
A POST request with both Content-Type: application/json and Content-Encoding: gzip and a gzipped JSON body, also correctly returns the list of products.

Can you give it a try?

[AuxiliumCDNG]

Thanks so much for your help so far!
The problem is that I actually need to decompress the JSON inside the lambda function because I have payloads larger than 6mb which is the limit for lambda invocation. Thats the main reason I use compression for, to get more data into lambda.

[rubenfonseca]

I see. Right now I don't have a good solution since that use case seems unsupported. But I'll try to see if I can figure out something.