Amazon Kinesis to Fluent log transfer Terraform module

Terraform module and Lambda for sending JSON log records from Kinesis Data Streams to a Fluent endpoint.

Prerequisites

1. Records in Kinesis stream must be valid JSON data. Non-JSON data will be saved with unknown prefix.
   1. gzipped JSON, CloudWatch Logs subscription filters log format are supported.
   2. Logs without either of necessary keys listed below will be saved as unknown as well.
2. JSON data must have the following keys (key names are modifiable via variables):
   1. log_type: Log type identifier. Failed logs will be saved by this key: %log_type%/YYYY-MM/DD/.
   2. log_id: Any unique identifier. Used to avoid file overwrites on S3. Also is useful to search for a specific log record.
   3. time: Any timestamp supported by dateutil.parser.parse. ISO8601 with milli/microseconds recommended. This timestamp is used when emitting logs with fluentd protocol; message timestamp will be set to this timestamp.

Usage

data "aws_vpc" "vpc" {
  filter {
    name   = "tag:Name"
    values = ["main"]
  }
}

data "aws_subnet_ids" "private" {
  vpc_id = "${data.aws_vpc.vpc.id}"
  
  filter {
    name   = "tag:Role"
    values = ["private"]
  }
}

resource "aws_security_group" "all_outbound" {
  name   = "kinesis_to_fluent-sg"
  vpc_id = "${data.aws_vpc.vpc.id}"
  
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_kinesis_stream" "stream" {
  name             = "stream"
  shard_count      = 1
  retention_period = 24
}

module "fluent" {
  source  = "baikonur-oss/lambda-kinesis-to-fluent/aws"
  version = "1.0.0"

  lambda_package_url = "https://github.com/baikonur-oss/terraform-aws-lambda-kinesis-fluent/releases/download/v1.0.0/lambda_package.zip"
  name               = "kinesis_to_fluent"

  batch_size         = 100
  kinesis_stream_arn = "${aws_kinesis_stream.stream.arn}"
  fluent_tag         = "dev"
  fluent_target_url  = "10.20.30.40"
  # can be internal DNS/private IP because this module creates an VPC Lambda

  security_group_ids = ["${aws_security_group.all_outbound.id}"]
  subnet_ids         = ["${data.aws_subnet_ids.private.ids}"]

  failed_log_s3_bucket = "failed-logs"
  failed_log_s3_prefix = "fluent"
}

Version pinning

Terraform Module Registry

Use version parameter to pin to a specific version, or to specify a version constraint when pulling from Terraform Module Registry (source = baikonur-oss/lambda-kinesis-to-fluent/aws). For more information, refer to Module Versions section of Terraform Modules documentation.

GitHub URI

Make sure to use ?ref= version pinning in module source URI when pulling from GitHub. Pulling from GitHub is especially useful for development, as you can pin to a specific branch, tag or commit hash. Example: source = github.com/baikonur-oss/terraform-aws-lambda-kinesis-to-fluent?ref=v1.0.0

For more information on module version pinning, see Selecting a Revision section of Terraform Modules documentation.

Inputs

Name	Description	Type	Default	Required
batch_size	Maximum number of records passed for a single Lambda invocation	string	n/a	yes
failed_log_s3_bucket	S3 bucket name for saving failed logs (ES API errors etc.)	string	n/a	yes
failed_log_s3_prefix	Path prefix for failed logs	string	n/a	yes
fluent_sink_endpoint	Target Fluent sink endpoint (IP or hostname)	string	n/a	yes
fluent_tag	Second-level fluent tag name (first level is log type)	string	n/a	yes
handler	Lambda Function handler (entrypoint)	string	"main.handler"	no
kinesis_stream_arn	Source Kinesis Data Streams stream name	string	n/a	yes
lambda_package_url	Lambda package URL (see Usage in README)	string	n/a	yes
log_id_field	Key name for unique log ID	string	"log_id"	no
log_retention_in_days	Lambda Function log retention in days	string	"30"	no
log_timestamp_field	Key name for log timestamp	string	"time"	no
log_type_field	Key name for log type	string	"log_type"	no
log_type_field_whitelist	Log type whitelist (if empty, all types will be processed)	list	<list>	no
log_type_unknown_prefix	Log type prefix for logs without log type field	string	"unknown"	no
memory	Lambda Function memory in megabytes	string	"256"	no
name	Resource name	string	n/a	yes
runtime	Lambda Function runtime	string	"python3.7"	no
security_group_ids	VPC Security group IDs	list	n/a	yes
starting_position	Kinesis ShardIterator type (see: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html )	string	"TRIM_HORIZON"	no
subnet_ids	VPC Subnet IDs	list	n/a	yes
tags	Tags for Lambda Function	map	<map>	no
timeout	Lambda Function timeout in seconds	string	"60"	no
timezone	tz database timezone name (e.g. Asia/Tokyo)	string	"UTC"	no
tracing_mode	X-Ray tracing mode (see: https://docs.aws.amazon.com/lambda/latest/dg/API_TracingConfig.html )	string	"PassThrough"	no

Contributing

Make sure to have following tools installed:

• Terraform
• terraform-docs
• pre-commit

macOS

brew install pre-commit terraform terraform-docs

# set up pre-commit hooks by running below command in repository root
pre-commit install