If you build applications with the Serverless Framework and deploy them to AWS, it's likely that at some point, as your apps grow in size and complexity, you'll bump up against the fact that CloudFormation stacks can contain a maximum of 200 resources. Given 200 resources is quite a lot, by the time you reach this point the chances are your application is relatively mature, running in production and serving real users. It's not a convenient time to discover that you can't deploy that new killer feature because you've hit the stack resource limit!

Do you learn better with a more hands-on approach?

If a face-to-face approach to learning works better for you or your team, orangejellyfish run a popular Serverless workshop which can take place remotely or on-site with you, giving you an opportunity to go deeper into some more advanced Serverless concepts.

What is CloudFormation?

CloudFormation is an AWS technology that "provides a common language for you to model and provision AWS and third party application resources in your cloud environment". It's used behind the scenes by the Serverless Framework. For example, each function in your service is defined in CloudFormation as a resource of type AWS::Lambda::Function. If your function responds to HTTP events from API Gateway you'll have a AWS::ApiGateway::Resource and a AWS::ApiGateway::Method resource as well. Throw in resources for permissions and it's easy to see how the total number of resources in a stack can rapidly grow.

Unfortunately, AWS enforce a limit of 200 resources per CloudFormation stack. Unlike many AWS limits, this is not one that can be increased upon request.

Working around the limit

Fortunately, AWS are aware of the fact that many real-world applications will have CloudFormation stacks that need to contain more than 200 resources and provide a solution:

To specify more resources, separate your template into multiple templates by using, for example, nested stacks.

And since the Serverless Framework has a rich ecosystem of tools and plugins it's no surprise that an open-source plugin exists to do exactly that. Enter serverless-plugin-split-stacks by Doug Moscrop.

The plugin provides a number of strategies for splitting a stack, including "per Lambda" to split based on resources associated with a given function and "per type" to split based on resource type, such as those mentioned in the previous section. In our experience, splitting by Lambda function has been the most effective approach. You can configure the plugin in serverless.yml:

custom:
  splitStacks:
    perFunction: true
    perType: false

plugins:
  - serverless-plugin-split-stacks

With this configuration, running sls package to build your service and produce the CloudFormation templates results in something like this:

The output from the Serverless Framework CLI on packaging

Examining the generated CloudFormation templates in the .serverless directory shows us that the new nested stack for a function contains all resources related to that function and the root stack contains common resources that are likely used across functions, such as API Gateway deployments and the S3 bucket used by the Serverless Framework itself. Each of these nested stacks can now grow up to 200 resources which is a much harder limit to reach.

The orangejellyfish Serverless starter kit includes the aforementioned plugin by default and contains an example Lambda function along with a range of other best-practice configuration defaults and helpers.

If you're starting from scratch with a new project you'll be able to deploy this in its current state. Unfortunately, existing CloudFormation stacks cannot be retroactively split by the per-function strategy so if you're doing this because you've hit the limit in an existing service you'll have to work around this restriction.

Splitting an existing stack

Attempting to deploy a stack which is now split on top of an existing, previously unsplit, stack, will result in a problem. Notice in the following screenshot how there is no output related to split stacks as there is in the previous screenshot. This is the same command as above, with the same split stack configuration, but running against a service that has already been deployed without split stacks:

The output from the Serverless Framework CLI on packaging an existing stack

It is possible to work around this error but the easiest approach would be to deploy a whole new stage of your service. Unfortunately, in some cases, this won't be possible. For example, if your stack contains resources with data that can't easily be ported over to a new stack (such as a Cognito User Pool, which stores user data including passwords that cannot easily be transferred) then you will have to find another approach.

Start by removing all of your existing functions and replacing them with a single new function:

functions:
  # hello: ${file(src/functions/hello/index.yml):hello}
  workaround: ${file(src/functions/workaround/index.yml):workaround}

Because the new function has not previously been part of a CloudFormation stack it can be split into a nested stack successfully:

The output from the Serverless Framework CLI on packaging the workaround

After deploying this version of the service you can safely remove the workaround function, reinstate your real functions and deploy again to split the real functions into new nested stacks.

Avoiding circular dependencies

One final potential issue you might run into with the per-function stack split strategy is the problem of circular dependencies on CloudFormation resources. This can happen when you have custom non-function resources that depend on functions and is common with AWS Cognito if you use any Cognito triggers to customise the authentication flow.

Cognito triggers are Lambda functions and now have first-class support as event sources in the Serverless Framework. These function resources need to be included in the same stack as the Cognito User Pool resource itself to avoid a circular dependency where the stack defining the User Pool depends on the stack defining the trigger functions, and the stack defining the trigger functions depends on the stack defining the User Pool.

Luckily for us, the split-stacks plugin provides a mechanism by which we can define custom splits. We can write a JavaScript file called stacks-map.js in the root of our Serverless project and use it to exclude Cognito trigger functions from the default per-function split behaviour:

// Custom migration for the serverless-plugin-split-stacks module. We use this
// to produce nested CloudFormation stacks to work around the hard limit of 200
// resources per stack. The default "per function" behaviour of the plugin is
// fine in most cases but results in circular dependencies in some cases, so we
// need to leave Cognito-specific functions in the root stack alongside the
// Cognito resources.
//
// Returning false from this function means the resource in question remains in
// the root stack (created by the Serverless Framework). Returning an object
// means the resource is moved into a nested stack, using the "destination" key
// as part of the stack name. Returning falsy (but not the value false itself)
// results in the default behaviour as defined by the plugin.
//
// See https://github.com/dougmoscrop/serverless-plugin-split-stacks.
// See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cognito-userpool-lambdaconfig.html.
const COGNITO_TRIGGERS = [
  'CreateAuthChallenge',
  'CustomMessage',
  'DefineAuthChallenge',
  'PostAuthentication',
  'PostConfirmation',
  'PreAuthentication',
  'PreSignUp',
  'PreTokenGeneration',
  'UserMigration',
  'VerifyAuthChallengeResponse',
];

module.exports = (resource, logicalId) => {
  if (COGNITO_TRIGGERS.some((trigger) => logicalId.startsWith(trigger))) {
    return false;
  }

  return null;
};

Going further

This approach of splitting resources into nested CloudFormation stacks will buy you time but if your service continues to grow in size, especially with custom non-function resources, you may find yourself hitting the limit again in the root stack before too long. If you are in this situation the best option is most likely to consider splitting your service itself into multiple smaller services based on features or functionality of your app. The Serverless Framework is not geared towards building monolithic apps in a single service.