Goal

You have your Lambda on AWS. Now you want to view the logs in Cloudwatch and you probably know this is not the most pleasant thing to do. But wait, there are things like Logz.io - an ELK service that makes viewing logs a breeze. Let’s see how to connect logz.io log shipper that will push your Lambda logs from cloudwatch to their ELK stack.

Logz.io shipper lambda

Let’s start with a Logz.io Shipper Lambda CDK code:

package com.piotrnowicki.cdk;

import software.amazon.awscdk.Duration;
import software.amazon.awscdk.services.lambda.Runtime;
import software.amazon.awscdk.services.lambda.*;
import software.constructs.Construct;

import java.util.Map;

public class LogzioForwarderLambda extends Construct {

    private static final Map<String, String> CONFIG = Map.of(
            "TOKEN", System.getenv("LOGZIO_TOKEN"),
            "REGION", "EU",
            "TYPE", "logzio_cloudwatch_lambda",
            "FORMAT", "json");

    private static final int MEMORY = 512;
    private static final Duration TIMEOUT = Duration.seconds(60);

    private final Function function;

    public LogzioForwarderLambda(Construct scope, String functionName) {
        super(scope, "LogzIoShipperLambda");
        this.function = createFunction(functionName);
    }

    Function createFunction(String functionName) {
        return Function.Builder.create(this, functionName)
                .runtime(Runtime.PYTHON_3_7)
                .code(Code.fromAsset("logzio-cloudwatch.zip"))
                .handler("lambda_function.lambda_handler")
                .memorySize(LogzioForwarderLambda.MEMORY)
                .functionName(functionName)
                .environment(LogzioForwarderLambda.CONFIG)
                .timeout(TIMEOUT)
                .build();
    }

    public Function getFunction() {
        return this.function;
    }
}

LOGZIO_TOKEN is the env-var that we need to pass to CDK (e.g. from CI/CD pipeline).

The mentioned logzio-cloudwatch.zip is a ZIP file produced by invoking following code (taken from the official docs):

git clone https://github.com/logzio/logzio_aws_serverless.git \
&& cd logzio_aws_serverless/python3/cloudwatch/ \
&& mkdir -p dist/python3/shipper; cp -r ../shipper/shipper.py dist/python3/shipper \
&& cp src/lambda_function.py dist \
&& cd dist/ \
&& zip logzio-cloudwatch lambda_function.py python3/shipper/*

CDK Stack using logz.io shipper

Here is a sample CDK stack that is using the freshly defined Logz.io shipper:

package pl.simplestep.assistant.cloud.infrastructure;

import software.amazon.awscdk.Stack;
import software.amazon.awscdk.services.logs.SubscriptionFilter;
import software.amazon.awscdk.services.logs.destinations.LambdaDestination;
import software.constructs.Construct;

import static pl.simplestep.assistant.cloud.infrastructure.CdkApp.APP_NAME;
import static software.amazon.awscdk.services.logs.FilterPattern.allEvents;

public class QuarkusStack extends Stack {

    public QuarkusStack(Construct construct, String id) {
        super(construct, id + "-quarkus-stack");

        QuarkusLambda quarkusLambda = new QuarkusLambda(this, APP_NAME + "-function");

        LogzioForwarderLambda logzioShipper = new LogzioForwarderLambda(this, APP_NAME + "-cloudwatch-forwarder");

        SubscriptionFilter.Builder.create(this, APP_NAME + "-cloudwatch-trigger")
                .logGroup(quarkusLambda.getFunction().getLogGroup())
                .filterPattern(allEvents())
                .destination(LambdaDestination.Builder
                        .create(logzioShipper.getFunction())
                        .build())
                .build();

    }
}

The QuarkusLambda is a separate construct that represents business-code Lambda we want to ship logs to logz.io.

The whole magic happens in the SubscriptionFilter that is binding the logGroup created implicitly when we create a Lambda (software.amazon.awscdk.services.lambda.Function) with the log shipper.

Summary

This quick sample shows how to create logz.io lambda that ships the logs from given Cloudwatch log group and forwards it to the logz.io for easier processing.