Aravind V
Dev Post

Dev Post

๐Ÿ’ฅ AWS CDK 101 - ๐Ÿฃ Event source mapping with Cfn property override

๐Ÿ’ฅ AWS CDK 101 - ๐Ÿฃ Event source mapping with Cfn property override

Aravind V's photo
Aravind V
ยทMay 24, 2022ยท

6 min read

Subscribe to my newsletter and never miss my upcoming articles

Play this article

Table of contents

๐Ÿ”ฐ Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

๐Ÿ” Original previous post at ๐Ÿ”— Dev Post

๐Ÿ” Reposted the previous post at ๐Ÿ”— dev to @aravindvcyber

In this article, we will be making a simple change in our CDK stack which eventually helps us with more optimization and control in processing streams. The high-level configuration which we have made to connect dynamodb streams event source with lambda in the last two articles will be optimized using event source mapping L2 construct with simple override with its cfn L1 construct as well.

Why we need this refactoring ๐Ÿž

In our last article, you can find that we are using dynamodb streams from stgMessages to delete objects from S3. The current configuration does not filter or distinguish the event based on the event name at the source, instead, we invoke the lambda unnecessary even for events other than REMOVE like INSERT and we end up, making unwanted batch invocations, which could have otherwise been avoided.

Benefits we achieve ๐ŸŒ‹

  • In a previous couple of articles we directly interfaced with our dynamodb streams without any filter and our lambda is processing every single one of them and optionally choosing business logic based on the CRUD operation which leads to more number of stream records processed by the lambda eventually.
  • The above scenario can be avoided by filtering using the eventName in the dynamodb stream record and selectively using only the most appropriate record needed to be processed in our specific lambda handler.
  • Here we not only get a chance to filter with CRUD operation, but also you can filter with the other fields which are part of the record object, this will help us filter these messages to the fullest.
  • One more thing we achieve here would be that we can offload the filtering from inside the lambda to the invocation source mapping itself and hence time and cost are saved.

Necessary imports in the stack ๐Ÿ“™

Here we will be making use of the below imports in our stack file.


import {
  CfnEventSourceMapping,
  EventSourceMapping,
  StartingPosition,
} from "aws-cdk-lib/aws-lambda";

Previous event source mappings ๐ŸŒท

Earlier we have used something like the below to configure the source to the lambda. This does not have the filtering enabled and all the streams are used to invoke the lambda, which leads to unwanted executions.

The below L2 construct does not yet provide a provision to add the filter and hopefully, we can expect this soon.


const stgTableSourceProps: DynamoEventSourceProps = {
    startingPosition: lambda.StartingPosition.LATEST,
    batchSize:100,
    maxBatchingWindow: Duration.seconds(60),
}
stgMessageStreamFunc.addEventSource(new DynamoEventSource(stgMessages, stgTableSourceProps));

Generic event source mapping construct ๐ŸŒฐ

Due to the shortcomings discussed above, we are going to try another generic L2 construct.

We will replace this with a more generic event source mapping construct as shown below for both the lambda handlers used in the previous 2 articles. Here we have added a few changes besides filtering.

  • startingPosition is changed to StartingPosition.TRIM_HORIZON to get the oldest item first.
  • bisectBatchOnError will split the record chunk into half and retry each half so that we can identify and isolate the problematic record from the other records while processing them inside the handler.
const msgDeleteEventSourceMap = new EventSourceMapping(this, "msgDeleteEventSourceMap", {
      target: messageStreamFunc,
      eventSourceArn: messages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

msgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

const stgMsgDeleteEventSourceMap = new EventSourceMapping(this, "stgMsgDeleteEventSourceMap", {
      target: stgMessageStreamFunc,
      eventSourceArn: stgMessages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

stgMsgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

Granting stream read to handler functions ๐Ÿ„

In the last two articles, we have not performed this explicitly, since the dynamodb event source addition to the lambda construct, automatically applied for the necessary permissions. Whereas in this case, we may have to grant the privileges as shown below.


messages.grantStreamRead(messageStreamFunc);

stgMessages.grantStreamRead(stgMessageStreamFunc);

Cloudformation property override ๐ŸŒผ

Currently, we could not add the filter criteria as this EventSourceMapping L2 construct directly, so here we have to access its L1 construct and add a property override as shown below for both the lambda handlers.

Once do note that there is much scope in the filtering pattern used here where we could include more fields to achieve the desired effect and it is not only limited to eventName only.

const cfnMsgDeleteEventSourceMap = msgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["INSERT"],
          }),
        },
      ],
});

insert map


const cfnStgMsgDeleteEventSourceMap = stgMsgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnStgMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["REMOVE"],
          }),
        },
      ],
});

remove map

Issue while deploying the stack ๐Ÿญ

You may occasionally get an error failure to deploy the stack, by then it is necessary for us to manually delete the event source mapping in the lambda either with the AWS console or using the AWS CLI as shown below.

The stack named CommonEventStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:dynamodb:ap-south-1:*****:table/stgMessagesTable/stream/2022-05-21T03:39:54.488 ") and function (" CommonEventStack-stgMessageStreamFuncF95CB7BF-877BwMFlctoc ") provided mapping already exists. Please update or delete the existing mapping with UUID 43eccb79-7a6f-4fd0-82f1-c176cff317da

AWS console to delete the mapping ๐ŸŽ

delete aws console

AWS cli to delete mapping ๐Ÿฃ

Verify the event source mapping if it is existing as shown below

aws lambda delete-event-source-mapping --uuid ********

get 2

Then we can delete it by executing the below command

aws lambda delete-event-source-mapping --uuid ********

del 2

Conclusion ๐Ÿค

Here we have demonstrated two things as shown below.

  • Ability to filter streams inside the generic event source mapping which is not only limited to dynamodb stream, this has a lot of other applications to be used similarly for kinesis streams, SQS source, and a lot more sources.
  • Additionally we have learned how we can use the L1 construct for an L2 resource and override certain properties, this will have a lot of use cases beyond this example.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

โญ We have our next article in serverless, do check out

๐ŸŽ‰ Thanks for supporting! ๐Ÿ™

Would be great if you like to โ˜• Buy Me a Coffee, to help boost my efforts ๐Ÿ˜.

Buy Me a Coffee at ko-fi.com

๐Ÿ” Original post at ๐Ÿ”— Dev Post

๐Ÿ” Reposted at ๐Ÿ”— dev to @aravindvcyber

๐Ÿ„ AWS CDK 101 - ๐Ÿฒ GraphQL using AppSync with dynamodb
{ by @Aravind_V7 }

Checkout more such in my pagehttps://t.co/CuYxnKr0Ighttps://t.co/phzOKFlXXO#aws #typescript #graphql #dynamodb #awscdk https://t.co/yeqeLZQt8A

โ€” Aravind V (@Aravind_V7) May 30, 2022

Did you find this article valuable?

Support Aravind V by becoming a sponsor. Any amount is appreciated!

See recent sponsors |ย Learn more about Hashnode Sponsors
ย 
Share this

Impressum

Also you can find my dev.to profile at โœจ @aravindvcyber for similar posts and contributions to community.

Are you wondering about reading more posts like this or writing your own post with hashnode.

Hashnode is free to ๐Ÿ”— sign-up, using my link helps to enable more features in my blog.

Would you like to โ˜• Buy Me a Coffee to help boost my efforts.

Also you can follow me on twitter, to find my latest articles and shares ๐Ÿ“ฃ @Aravind_V7