Imagine a scenario where a Lambda function polls messages from an SQS queue and uses the payload from those messages to make a POST request to an AWS OpenSearch domain via the REST API endpoint.
Issue
During batch processing, multiple Lambda instances are triggered simultaneously as soon as messages arrive in the queue. This led to some requests being processed too quickly, causing a 429 (Too Many Requests) error due to rate limiting.
Fix
We implemented two solutions:
Reserved Concurrency: We limited the Lambda function’s concurrency to 1 to ensure that only one instance processes messages at a time. This was done using CloudFormation:
Lambda:
Type: AWS::Serverless::Function
Properties:
ReservedConcurrentExecutions: 1
Queue Delay: We set a value for DelaySeconds in the SQS queue to introduce a delay before the next message is processed, allowing OpenSearch more time to handle requests. This was also configured using CloudFormation.
Queue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 40
DelaySeconds: 30
