v1.21.0

[heitorlessa]

Summary

After some vacation period, we're back with a new minor release with major features:

• Bring your own boto3 sessions for cross-account operations & snapshot testing
• New features on Feature Flags
• Idempotency unit testing made easier
• JSON Schema Validation utility contains new data elements to more easily construct a validation error
• New utility: we're now exposing our internal custom JMESPath Functions so you can easily decode and deserialize JSON objects found in various formats within Lambda Event Sources.

New Contributors

I'd like to personally thank our new contributors to the project :)

• @Tankanow made their first contribution in fix(idempotency): use ExpressionAttributeNames in DynamoDBPersistenceLayer _put_record #697
• @DanyC97 made their first contribution in docs: Terraform reference for SAR Lambda Layer #716
• @gwlester made their first contribution in feat(feature-flags): improve "IN/NOT_IN"; new rule actions #710

Detailed changes

Boto3 sessions

You can now pass in your own boto3 session when using Parameters, Batch, and Idempotency.

This is helpful in two typical scenarios: 1/ You want to run an operation in another account like fetching secrets/parameters somewhere else, 2/ Use snapshot testing tools like Placebo that will replay session data that happened earlier when doing unit testing.

from aws_lambda_powertools.utilities import parameters
import boto3

boto3_session = boto3.session.Session()
ssm_provider = parameters.SSMProvider(boto3_session=boto3_session)

def handler(event, context):
    # Retrieve a single parameter
    value = ssm_provider.get("/my/parameter")
    ...

Feature flags

There's been three main improvements in Feature flags utility as part of this release: New rule conditions, Bring your own Logger for debugging, and Getting a copy of fetched configuration from the store

New rule conditions

You can now use the following new rule conditions to evaluate your feature flags for inequality, comparison, and more explicit contains logic, where a is the key and b is the value passed as a context input for evaluation:

Action	Equivalent expression
KEY_GREATER_THAN_VALUE	lambda a, b: a > b
KEY_GREATER_THAN_OR_EQUAL_VALUE	lambda a, b: a >= b
KEY_LESS_THAN_VALUE	lambda a, b: a < b
KEY_LESS_THAN_OR_EQUAL_VALUE	lambda a, b: a <= b
KEY_IN_VALUE	lambda a, b: a in b
KEY_NOT_IN_VALUE	lambda a, b: a not in b
VALUE_IN_KEY	lambda a, b: b in a
VALUE_NOT_IN_KEY	lambda a, b: b not in a

Example

Feature flag schema

{
    "premium_features": {
        "default": false,
        "rules": {
            "customer tier equals premium": {
                "when_match": true,
                "conditions": [
                    {
                        "action": "VALUE_IN_KEY",
                        "key": "groups",
                        "value": "PAID_PREMIUM",
                    }
                ]
            }
        }
    },
    "ten_percent_off_campaign": {
        "default": false
    }
}

App

from aws_lambda_powertools.utilities.feature_flags import FeatureFlags, AppConfigStore

app_config = AppConfigStore(
    environment="dev",
    application="product-catalogue",
    name="features"
)

feature_flags = FeatureFlags(store=app_config)

def lambda_handler(event, context):
    # groups: ["FREE_TIER", "PAID_BASIC", "PAID_PREMIUM"]
    ctx={"tenant_id": "6", "username": "a", "groups": event.get("groups", [])}

    # Evaluate whether customer's tier has access to premium features
    # based on `has_premium_features` rules
    has_premium_features: bool = feature_flags.evaluate(name="premium_features",
                                                        context=ctx, default=False)
    if has_premium_features:
        # enable premium features
        ...

Accessing raw configuration fetched

Previously, if you were using a single application configuration and a feature schema in a single AppConfig key, we would only use the feature flags schema and discard the rest.

You can now access the raw configuration with a new property get_raw_configuration within AppConfig Store:

from aws_lambda_powertools.utilities.feature_flags import FeatureFlags, AppConfigStore

app_config = AppConfigStore(
    environment="dev",
    application="product-catalogue",
    name="configuration",
    envelope = "feature_flags"
)

feature_flags = FeatureFlags(store=app_config)

config = app_config.get_raw_configuration

Unit testing idempotency

We have improved how you can unit test your code when using @idempotent and @idempotent_function decorators.

You can now disable all interactions with the idempotence store using POWERTOOLS_IDEMPOTENCY_DISABLED environment variable, and monkeypatch the DynamoDB resource client Idempotency utility uses if you wish to either use DynamoDB Local or mock all I/O operations.

import boto3

import app

def test_idempotent_lambda():
    # Create our own Table resource using the endpoint for our DynamoDB Local instance
    resource = boto3.resource("dynamodb", endpoint_url='http://localhost:8000')
    table = resource.Table(app.persistence_layer.table_name)
    app.persistence_layer.table = table

    result = app.handler({'testkey': 'testvalue'}, {})
    assert result['payment_id'] == 12345

New data elements for JSON Schema validation errors

When validating input/output with the Validator, you can now access new properties in SchemaValidationError to more easily construct your custom errors based on what went wrong.

Property | Type | Description
------------------------------------------------- | ---------------------------------------------------------------------------------
message | str | Powertools formatted error message
validation_message | str, optional | Containing human-readable information what is wrong, e.g. data.property[index] must be smaller than or equal to 42
name | str, optional | name of a path in the data structure, e.g. data.property[index]
path | List, optional | path as an array in the data structure, e.g. ['data', 'property', 'index']
value | Any, optional | The invalid value, e.g. {"message": "hello world"}
definition | Any, optional | JSON Schema definition
rule | str, optional | rule which the data is breaking (e.g. maximum, required)
rule_definition | Any, optional | The specific rule definition (e.g. 42, ['message', 'username'])

Sample

from aws_lambda_powertools.utilities.validation import validate
from aws_lambda_powertools.utilities.validation.exceptions import SchemaValidationError

import schemas

def handler(event, context):
    try:
        validate(event=event, schema=schemas.INPUT)
    except SchemaValidationError as e:
        message = "data must contain ['message', 'username'] properties"

        assert str(e.value) == e.value.message
        assert e.value.validation_message == message
        assert e.value.name == "data"
        assert e.value.path is not None
        assert e.value.value == data
        assert e.value.definition == schema
        assert e.value.rule == "required"
        assert e.value.rule_definition == schema.get("required")

        raise

    return event

New JMESPath Powertools functions

Last but not least, as part of a documentation revamp in Idempotency by @walmsles, we're now exposing an internal feature used by many Lambda Powertools utilities which is the ability to extract and decode JSON objects.

You can now use JMESPath (JSON Query language) Lambda Powertools functions to easily decode and deserialize JSON often found as compressed (Kinesis, CloudWatch Logs, etc), as strings (SNS, SQS, EventBridge, API Gateway, etc), or as base64 (Kinesis).

We're exposing three custom JMESPath functions you can use such as powertools_json, powertools_base64, powertools_base64_gzip, and a new standalone function that will use JMESPath to search and extract the data you want called extract_data_from_envelope.

Sample

from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope
from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope="powertools_json(body)")
    customer = payload.get("customerId")  # now deserialized
    ...

SNS sample using built-in envelope

from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope, envelopes
from aws_lambda_powertools.utilities.typing import LambdaContext


def handler(event: dict, context: LambdaContext):
    payload = extract_data_from_envelope(data=event, envelope=envelopes.SNS)
    customer = payload.get("customerId")  # now deserialized
    ...

Sample event

{
    "Records": [
        {
            "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
            "receiptHandle": "MessageReceiptHandle",
            "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\",\"booking\":{\"id\":\"5b2c4803-330b-42b7-811a-c68689425de1\",\"reference\":\"ySz7oA\",\"outboundFlightId\":\"20c0d2f2-56a3-4068-bf20-ff7703db552d\"},\"payment\":{\"receipt\":\"https:\/\/pay.stripe.com\/receipts\/acct_1Dvn7pF4aIiftV70\/ch_3JTC14F4aIiftV700iFq2CHB\/rcpt_K7QsrFln9FgFnzUuBIiNdkkRYGxUL0X\",\"amount\":100}}",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1523232000000",
                "SenderId": "123456789012",
                "ApproximateFirstReceiveTimestamp": "1523232000001"
            },
            "messageAttributes": {},
            "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
            "awsRegion": "us-east-1"
        }
    ]
}

Changes

🌟New features and non-breaking changes

• feat: expose jmespath powertools functions (feat: expose jmespath powertools functions #736) by @heitorlessa
• feat(feature-flags): Added inequality conditions (feat(feature_flags): Added inequality conditions #721) by @gwlester
• feat(feature-flags): Bring your own logger for debug (feat(feature-flags): Bring your own logger for debug #709) by @gwlester
• feat(feature-flags): improve "IN/NOT_IN"; new rule actions (feat(feature-flags): improve "IN/NOT_IN"; new rule actions #710) by @gwlester
• feat(idempotency): makes customers unit testing easier (feat(idempotency): makes customers unit testing easier #719) by @cakepietoast
• feat(feature-flags): get_raw_configuration property in Store (feat(feature-flags): get_raw_configuration property in Store #720) by @heitorlessa
• feat: boto3 sessions in batch, parameters & idempotency (feat: boto3 sessions in batch, parameters & idempotency #717) by @cakepietoast
• feat(validator): include missing data elements from a validation error (feat(validator): include missing data elements from a validation error #686) by @michaelbrewer

🌟 Minor Changes

• fix(idempotency): use ExpressionAttributeNames in DynamoDBPersistenceLayer _put_record (fix(idempotency): use ExpressionAttributeNames in DynamoDBPersistenceLayer _put_record #697) by @Tankanow

📜 Documentation updates

• chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 (chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 #730) by @dependabot
• chore(deps-dev): bump coverage from 5.5 to 6.0 (chore(deps-dev): bump coverage from 5.5 to 6.0 #732) by @dependabot
• docs(parser): fix incorrect import in root_validator example (docs(parser): fix incorrect import in root_validator example #735) by @heitorlessa
• docs: Terraform reference for SAR Lambda Layer (docs: Terraform reference for SAR Lambda Layer #716) by @DanyC97
• docs(idempotency): fix misleading idempotent examples (docs(idempotency): fix misleading idempotent examples #661) by @walmsles
• docs(event-handler): document catch-all routes (docs(event-handler): document catch-all routes  #705) by @heitorlessa
• refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse (refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse #643) by @michaelbrewer
• fix(data-classes): use correct asdict funciton (fix(data-classes): use correct asdict funciton #666) by @michaelbrewer

🐛 Bug and hot fixes

• fix(feature-flags): rules should evaluate with an AND op (fix(feature-flags): rules should evaluate with an AND op #724) by @risenberg-cyberark
• fix(logger): push extra keys to the end (fix(logger): push extra keys to the end #722) by @heitorlessa
• fix(mypy): a few return types, type signatures, and untyped areas (fix(mypy): a few return types, type signatures, and untyped areas #718) by @heitorlessa

🔧 Maintenance

• chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 (chore(deps-dev): bump pytest-cov from 2.12.1 to 3.0.0 #730) by @dependabot
• chore(deps-dev): bump coverage from 5.5 to 6.0 (chore(deps-dev): bump coverage from 5.5 to 6.0 #732) by @dependabot
• chore(deps): bump boto3 from 1.18.51 to 1.18.54 (chore(deps): bump boto3 from 1.18.51 to 1.18.54 #733) by @dependabot
• chore(deps-dev): bump flake8-bugbear from 21.9.1 to 21.9.2 (chore(deps-dev): bump flake8-bugbear from 21.9.1 to 21.9.2 #712) by @dependabot
• chore(deps): bump boto3 from 1.18.49 to 1.18.51 (chore(deps): bump boto3 from 1.18.49 to 1.18.51 #713) by @dependabot
• chore(deps): bump boto3 from 1.18.41 to 1.18.49 (chore(deps): bump boto3 from 1.18.41 to 1.18.49 #703) by @dependabot
• chore(deps): bump codecov/codecov-action from 2.0.2 to 2.1.0 (chore(deps): bump codecov/codecov-action from 2.0.2 to 2.1.0 #675) by @dependabot
• chore(deps-dev): bump mkdocs-material from 7.2.8 to 7.3.0 (chore(deps-dev): bump mkdocs-material from 7.2.8 to 7.3.0 #695) by @dependabot
• chore(deps-dev): bump mkdocs-material from 7.2.6 to 7.2.8 (chore(deps-dev): bump mkdocs-material from 7.2.6 to 7.2.8 #682) by @dependabot
• chore(deps-dev): bump flake8-bugbear from 21.4.3 to 21.9.1 (chore(deps-dev): bump flake8-bugbear from 21.4.3 to 21.9.1 #676) by @dependabot
• chore(deps): bump boto3 from 1.18.38 to 1.18.41 (chore(deps): bump boto3 from 1.18.38 to 1.18.41 #677) by @dependabot
• chore(deps-dev): bump radon from 4.5.2 to 5.1.0 (chore(deps-dev): bump radon from 4.5.2 to 5.1.0 #673) by @dependabot
• chore(deps): bump boto3 from 1.18.32 to 1.18.38 (chore(deps): bump boto3 from 1.18.32 to 1.18.38 #671) by @dependabot
• refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse (refactor(data-classes): clean up internal logic for APIGatewayAuthorizerResponse #643) by @michaelbrewer
• chore(deps-dev): bump xenon from 0.7.3 to 0.8.0 (chore(deps-dev): bump xenon from 0.7.3 to 0.8.0 #669) by @dependabot

This release was made possible by the following contributors:

@DanyC97, @Tankanow, @cakepietoast, @dependabot, @dependabot[bot], @gwlester, @heitorlessa, @michaelbrewer, @risenberg-cyberark and @walmsles

Full Changelog: v1.20.2...v1.21.0

---

This discussion was created from the release v1.21.0.