Context

For at least a few weeks now I was trying to figure out how serverless developers are:

  • testing their applications,
  • developing them locally.

My interest lies in AWS, Python and Serverless Framework so I’ll focus on them.

What is Already on The Market

There are a couple of players on this market like:

  • localstack that mocks up all AWS APIs and start their local instances (can be used for local environment and started automatically for tests),
  • AWS SAM (AWS Serverless Application Model that mimics AWS Lambda and API Gateway locally,
  • serverless-offline that basically does the same as AWS SAM but for Serverless Framework based applications (it’s a plugin in serverless.yml),
  • serverless-dynamodb-local (+ serverless-dynamodb-client) that lets you use offline DynamoDB (and used with serverless-offline and serverless-dynamodb-client also knows when you’re running application locally or in AWS and points to correct DynamoDB instance),
  • AWS DynamoDB Local Docker image that is coming directly from AWS and allows you to run it in your local environment,
  • moto a Python library that mocks AWS services for using them in tests.

I’m pretty sure there are more of them. The issue I had is that I couldn’t find a solution that will be mature enough, often updated and will work with most current Serverless Framework and AWS APIs (e.g. S3 not available yet, Lambda not being found in localstack, Issue with DynamoDB Client). I ended up googling one issue after another spending time not how I believe I should be.

What I Needed was a Mindset Shift

And then I just realized that I’m trying to find a perfect solution similar to the environments I was working until now and try to run everything locally. Serverless might be a different story. It composes of:

  • many small, simple functions that should be well unit-tested and,
  • a lot of integration between your code and gazillion of AWS services.

In this fast pace world it is hard for any mock-platform provider to catch up with the cloud provider (take a look at AWS This Week to see how fast those services are evolving. Not as fast as new JavaScript frameworks are created but quite fast :-).

Understanding this made me to think about using direct AWS account / environment per-developer.

With proper separation (e.g. separate accounts within one organization) you can properly ring-fence yourself from the production environment. With proper configuration of resources and architecture the costs of such per-developer environments can be also reduced – e.g. do not use EC2 instances or configure DynamoDB instances always to work in ‘On-Demand’ payment mode.

At the same time you’re shortening the distance between local and production environment and gaining the possibility to find bugs with the integration layer sooner than before.

The developers are working all the time on real AWS environment we’ll use in production.

Disadvantages

Of course, it’s not so perfect if you take a closer look at the downsides of such solution.

Local development is slower.

For resources you need to go to AWS and depending on your internet connection, the information loop (write, run, test, verify, fix) will be longer. Imagine that when you change a piece of code, you need to deploy it to AWS and then execute it and repeat this routine many times during every-day development.

I tend to think that not being able to develop and test everything locally (as in your local-machine) is a huge downside. However, I have reviewed my approach – nowadays we got fast internet connection almost from every place on the world, At the same time, if you properly unit-test your lambda business code, developers are able to test them locally.

If you’re working on Lambdas triggered mainly via HTTP then a hybrid model is also tempting.
You could use some solutions that are already on the market and are working fine in this limited set of capabilities. This will give you a development boost.

What I mean is you could use serverless-offline and work on your Lambdas offline, test it using API Gateway on localhost then modify the code, refresh the web page and you’ll see the results.

At the same time other resources (DynamoDB, SNS, SQS, Cognito, S3 are taken from real AWS endpoint.

My Current Local Development and Testing Process

So this is how my current development looks like:

  • Design your Lambdas thinking about Hexagonal architecture,
  • Write most unit tests just as you’d do for non-serverless application,
  • Write some unit tests that works with mocked AWS services (e.g. moto in Python),
  • Write integration tests that will be deployed using AWS and tested against real AWS resources on dedicated environment (either per-developer or per-test environment),
  • Run local application by using serverless-offline (runs Lambda + API Gateway) while rest of AWS Service are taken from AWS dedicated environment (per-developer).

Summary

I’m definitely not saying the above assumptions are the best but that’s what right now I think makes your local development as close as possible to the production environment. At the same time it gives you at least a bit of a boost in development in form of those basic offline AWS services (Lambda + API GW).
Hence, that’s what I decided to go and verify how it works in a longer run.

Wonder if you guys have any experience with your own approach to serverless applications local development and testing. I would love to read it in comments!