Introduction to Serverless Framework

Serverless Framework is a tool designed to streamline the development and deployment of serverless applications, including functions and infrastructure, by abstracting away the need to manage servers. 

We define desired infrastructure in serverless yaml files and then deploy it by executing:

sls deploy

This command parses serverless yaml file into larger AWS CloudFormation template which automatically gets filled with values from the yaml. 

The sls deploy command in the Serverless Framework is effectively idempotent at the infrastructure level, but with important nuances:

How it works: 

sls deploy packages our service and deploys it via AWS CloudFormation. CloudFormation itself is designed to be idempotent: if we deploy the same stack with the same configuration and code, AWS will detect no changes and will not modify our resources. If there are changes, only those changes are applied.

What this means:

Repeated runs of sls deploy with no changes will not create duplicate resources or apply unnecessary updates.

If we make changes (to code, configuration, or infrastructure), only the differences are deployed.

Side effects in Lambda code: While infrastructure deployment is idempotent, our Lambda functions themselves must be written to handle repeated invocations safely if we want end-to-end idempotency. The deployment command itself does not guarantee idempotency at the application logic level.

Limitations:

If we use sls deploy function (to update a single function without CloudFormation), this command simply swaps out the function code and is also idempotent in the sense that re-uploading the same code does not cause issues.

If we use plugins or custom resources, their behavior may not always be idempotent unless explicitly designed that way.

To conclude:

• sls deploy is idempotent for infrastructure: Re-running it with no changes is safe and does not cause duplicate resources or unintended side effects at the CloudFormation level.
• Application-level idempotency is our responsibility: Ensure our Lambda functions and integrations handle repeated events if that is a requirement for our use case

Serverless Yaml Configuration File

serverless yaml file defines a serverless service. It is a good idea to break up the serverless project into multiple services, each of which is defined by its own serverless yaml file. We don't want to have everything in one big infrastructure stack. 

Example:

• database e.g. DynamoDB
• Rest API e.g. which handles the submitted web form and stores data in DynamoDB
• front-end website which e.g. stores React app website in s3 bucket

Services can be deployed in multiple regions. (Multi-region architecture is supported)

serverless.yml example:

service: my-service

frameworkVersion: "3"

useDotenv: true

plugins: 

  - serverless-plugin-log-subscription

  - serverless-dotenv-plugin

provider:

  name: aws

  runtime: nodejs14.x

  region: eu-east-1

  memorySize: 512

  timeout: 900

  deploymentBucket:

    name: my-serverless-deployments

  vpc: 

    securityGroupIds: 

      - "sg-0123cf34f6c6354cb"

    subnetIds: 

      - "subnet-01a23493f9e755207"

      - "subnet-02b234dbd7d66d33c"

      - "subnet-03c234712e99ae1fb"

  iam: 

    role:

      statements:

        - Effect: Allow

          Action:

            - lambda:InvokeFunction

          Resource: arn:aws:lambda:eu-east-1:123456789099:function:my-database

package:

  patterns:

    - "out/**"

    - "utils.js"

    - "aws-sdk"

functions:

  my-function:

    handler: lambda.handler

    events:

      - schedule:

          name: "my-service-${opt:stage, self:provider.stage}"

          description: "Periodically run my-service lambdas"

          rate: rate(4 hours)

          inputTransformer:

            inputTemplate: '{"Records":[{"EventSource":"aws:rate","EventVersion":"1.0","EventSubscriptionArn":"arn:aws:sns:eu-east-1:{{accountId}}:ExampleTopic","Sns":{"Type":"Notification","MessageId":"95df01b4-1234-5678-9903-4c221d41eb5e","TopicArn":"arn:aws:sns:eu-east-1:123456789012:ExampleTopic","Subject":"example subject","Message":"example message","Timestamp":"1970-01-01T00:00:00.000Z","SignatureVersion":"1","Signature":"EXAMPLE","SigningCertUrl":"EXAMPLE","UnsubscribeUrl":"EXAMPLE","MessageAttributes":{"type":{"Type":"String","Value":"populate_unsyncronised"},"count":{"Type":"Number","Value":"400"}}}}]}'

      - sns:

          arn: arn:aws:sns:us-east-2:123456789099:trigger-my-service

      - http: 

custom:

  dotenv:

    dotenvParser: env.loader.js

  logSubscription:

      enabled: true

      destinationArn: ${env:KINESIS_SUBSCRIPTION_STREAM}

      roleArn: ${env:KINESIS_SUBSCRIPTION_ROLE}

• service: - name of the service
• useDotenv: boolean (true|false)
• configValidationMode: error
• frameworkVersion: e.g. "3"
• provider - 
• name - provider name e.g. aws
• runtime - e.g. nodejs18.x
• region e.g. us-east-1
• memorySize - how much memory will have the machine on which Lambda will be running e.g. 1024 (MB). It is good to check the actual memory usage and adjust the required memory size - downsizing can lower the costs!
• timeout: (number) e.g. 60 [seconds] - the maximum amount of time, in seconds, that a serverless function (such as an AWS Lambda function) is allowed to run before it is forcibly terminated by the AWS platform. This setting ensures that our function does not run indefinitely. If the function execution exceeds 60 seconds, the serverless platform will automatically stop it and return a timeout error. The timeout property is commonly used to control resource usage and prevent runaway executions. It is especially important for functions that interact with external services or perform long-running tasks. If not specified, most serverless platforms (like AWS Lambda) use a default timeout (for AWS Lambda, the default is 3 seconds, and the maximum is 900 seconds or 15 minutes).
• httpApi:
• id:
• apiGateway:
• minimumCompressionSize: 1024
• shouldStartNameWithService: true
• restApiId: ""
• restApiRootResourceId: ""
• stage: - name of the environment e.g. production; 
• iamManagedPolicies: a list of ARNs of policies that will be associated to the Lambda's computing instance e.g. policy which allows access to S3 buckets etc...
• lambdaHashingVersion
• environment: dictionary of environment variable names and values
• vpc
• securityGroupIds: list 
• subnetIds - typically a list of private subnets with NAT gateway. 
• functions: a dictionary which defines the AWS Lambda functions that are deployed as part of this Serverless service. This is where we define the AWS Lambda functions that our Serverless service will deploy. 
• <function_name>: string, a logical name of the function (e.g., my-function). This name is used to reference the function within the Serverless Framework and in deployment outputs. A name of the provisioned Lambda function is in format: <service_name>-<stage>-<function_name>. Each function entry under functions specifies:
• handler - tells Serverless which file and exported function to execute as the Lambda entry point (e.g., src/fn/lambda.handler which points to handler export in the src/fn/lambda module). Specifies the entry point for the Lambda function. When the function is invoked, AWS Lambda will execute this handler.
• events - (optional, array) a list of events that trigger this function
• Some triggers:
• schedule, scheduled events: for periodic invocation (cron-like jobs)
• sns: for invocation via an AWS SNS topic
• HTTP endpoints,
• S3 events
• messages from a Kafka topic in an MSK cluster (msk)
• If the array is empty, that means that the function currently has no event sources configured and will not be triggered automatically by any AWS event.
• plugins: a list of serverless plugins e.g. 
• serverless-webpack
• serverless-esbuild
• serverless-offline [https://www.serverless.com/plugins/serverless-offline, https://github.com/dherault/serverless-offline]
• emulates AWS Lambda and API Gateway. It starts an HTTP server that handles the request's lifecycle like APIG does and invokes the handlers.
• sls offline --help
• serverless-plugin-log-subscription
• custom: - section for serverless plugins settings e.g. for esbuild, logSubscription, webpack etc...
• example: serverless-plugin-log-subscription plugin has the settings:

logSubscription: {
    enabled: true,
    destinationArn: process.env.SUBSCRIPTION_STREAM,
    roleArn: process.env.SUBSCRIPTION_ROLE,
}


• example: serverless-domain-manager - used to define stage-specific domains.

domains: {

  production: {

   url: "app.api.example.com",

   certificateArn: "arn:aws:acm:us-east-2:123456789012:certificate/a8f8f8e2-95fe-4934-abf2-19dc08138f1f",

},

  staging: {

    url: "app.staging.example.com",

    certificateArn: "arn:aws:acm:us-east-2:123456789012:certificate/a32e9708-7aeb-495b-87b1-8532a2592eeb",

},

  dev: { 

    url: "", 

    certificateArn: "" 

  },

}, 

• 
  

Introduction to Amazon Simple Queue Service (SQS)

Amazon Simple Queue Service (SQS) is a fully managed message queuing service provided by Amazon Web Services (AWS). It enables decoupling and scaling of microservices, distributed systems, and serverless applications. 

Here's an overview of how Amazon SQS works:

Key Concepts

• Queue:
• A queue is a temporary storage location for messages waiting to be processed (polled by consumers). There are two types of queues in SQS:
• Standard Queue: Offers maximum throughput, best-effort ordering, and at-least-once delivery.
• FIFO Queue: Ensures exactly-once processing and preserves the exact order of messages.
• Message:
• A message is the data that is sent between different components. It can be up to 256 KB in size and contains the information needed for processing.
• Producer:
• The producer (or sender) sends messages to the queue. Producers can be applications, microservices and other AWS services.
• Consumer:
• The consumer (or receiver) retrieves and processes messages from the queue. Consumers can be Lambda functions, EC2 instances and other AWS services
• Visibility Timeout:
• A period during which a message is invisible to other consumers after a consumer retrieves it from the queue. This prevents other consumers from processing the same message concurrently.
• Dead-Letter Queue (DLQ):
• A queue for messages that could not be processed successfully after a specified number of attempts. This helps in isolating and analyzing problematic messages.

Workflow

• Sending Messages:
• A producer sends messages to an SQS queue using the SendMessage action. Each message is assigned a unique ID and placed in the queue.
• Receiving Messages:
• A consumer retrieves messages from the queue using the ReceiveMessage action. This operation can specify:
• number of messages to retrieve (up to 10) 
• duration to wait if no messages are available
• Processing Messages:
• After receiving a message, the consumer processes it. The message remains invisible to other consumers for a specified visibility timeout.
• Deleting Messages:
• Once processed, the consumer deletes the message from the queue using the DeleteMessage action. If not deleted within the visibility timeout, the message becomes visible again for other consumers to process.
• Handling Failures:
• If a message cannot be processed successfully within a specified number of attempts, it is moved to the Dead-Letter Queue for further investigation.

Additional Features

• Long Polling:
• Reduces the number of empty responses by allowing the ReceiveMessage action to wait for a specified amount of time until a message arrives in the queue.
• Message Attributes:
• Metadata about the message that can be used for filtering and routing.
• Batch Operations:
• SQS supports batch sending, receiving, and deleting of messages, which can improve efficiency and reduce costs.

Security and Access Control

• IAM Policies:
• Use AWS Identity and Access Management (IAM) policies to control access to SQS queues.
• Encryption:
• Messages can be encrypted in transit using SSL/TLS and at rest using AWS Key Management Service (KMS).

Use Cases

• Decoupling Microservices:
• SQS allows microservices to communicate asynchronously, improving scalability and fault tolerance.
• Work Queues:
• Distributing tasks to multiple workers for parallel processing.
• Event Sourcing:
• Storing a series of events to track changes in state over time.

Example Scenario

Order Processing System:

• An e-commerce application has separate microservices for handling orders, inventory, and shipping.
• The order service sends an order message to an SQS queue.
• The inventory service retrieves the message, processes it (e.g., reserves stock), and then sends an updated message to another queue.
• The shipping service retrieves the updated message and processes it (e.g., ships the item).

By using Amazon SQS, these microservices can operate independently and scale as needed, ensuring reliable and efficient order processing.

What is best-effort ordering?

Best-effort ordering is the default delivery logic for Amazon SQS Standard queues. Under this model, SQS attempts to deliver messages in the same order they were sent, but it does not guarantee it. 

How It Works

• General Alignment: SQS uses a highly distributed architecture to achieve nearly unlimited throughput. While it tries to maintain a "loose FIFO" (First-In, First-Out) flow, messages may occasionally be delivered out of sequence.
• Cause of Reordering: Out-of-order delivery typically occurs due to the way messages are stored across multiple servers and availability zones for redundancy. Factors like high throughput, network delays, or failure recovery can cause a message sent later to be available for retrieval before an earlier one. 

Comparison with FIFO Queues

If your application requires strict ordering, you must use SQS FIFO queues instead of Standard queues. 

Feature                     Best-Effort Ordering (Standard)       Strict Ordering (FIFO)

----------                       ------------------------------------------          -----------------------------

Ordering Guarantee    No (messages may arrive out of order)    Yes (exact order preserved)

Throughput             Nearly unlimited                                       Limited (unless High Throughput mode is used)

Delivery Model     At-least-once (duplicates possible)       Exactly-once (no duplicates)

Cost                             Lower                                                       Slightly higher

Best Practices for Best-Effort Ordering

• Idempotency: Ensure your application can handle the same message multiple times without unintended side effects.
• Tolerance for Shuffle: Use Standard queues for workloads where order isn't critical, such as processing log data, real-time analytics, or distributing independent background tasks.
• Application-Level Logic: If you need some ordering but want the high throughput of Standard queues, you can include sequence numbers in your message attributes and handle the reordering logic within your consumer application. 

How can some AWS Service send messages to a queue?

AWS services send messages to an Amazon SQS queue through three primary methods: direct API calls, event-driven notifications, or as a downstream target for messaging services. 

1. Direct API Integration (Producer Model)

Many compute services act as "producers" by calling the SendMessage or SendMessageBatch API actions directly using an AWS SDK (like Boto3 for Python or the SDK for Node.js). 

• AWS Lambda: A function can use an SDK to programmatically push results or tasks into a queue for further processing.
• Amazon EC2 & ECS: Applications running on virtual machines or in containers can send messages to SQS to decouple from backend systems.
• AWS Step Functions: You can use a "Task" state to publish a message directly to SQS as part of a workflow. 

2. Event-Driven Notifications

Certain services can be configured to automatically "push" notifications into a queue when specific events occur. 

• Amazon S3: You can set up S3 Event Notifications to send a message to SQS whenever an object is created, deleted, or restored in a bucket.
• Amazon EventBridge: You can create rules that match specific system events and route them to an SQS queue as a target. 

3. Messaging Service Fan-out

SQS often acts as a subscriber or target for other messaging and integration services. 

• Amazon SNS: Using the "fan-out" pattern, a message published to an SNS topic can be automatically delivered to multiple SQS queues simultaneously.
• Amazon API Gateway: You can integrate an API endpoint directly with SQS. This allows external clients to send messages to your queue via a REST API without needing a Lambda function in between. 

Crucial Requirement: Permissions 

For any service to send messages, it must have the sqs:SendMessage permission granted via an IAM Policy. Additionally, the SQS Queue Access Policy must explicitly allow the sending service or account to perform that action.

Dead Letter Queues (DLQ)

In the context of AWS SQS, DLQ stands for Dead Letter Queue. 

It is not a special type of queue; it is simply a standard SQS queue that is designated as a "holding pen" for messages that could not be processed successfully by your consumer application.

How it Works

When a message is picked up from your main queue, the consumer tries to process it. If the consumer fails (crashes, times out, or throws an error), the message returns to the main queue to be tried again. Without a DLQ, a "poison pill" message (one that causes a crash every single time) could cycle through your system forever, wasting resources. A DLQ solves this by setting a Redrive Policy.

Main Queue: Receives the message.

Maximum Receives: You define a limit (e.g., 3 or 5). If a message fails this many times, SQS gives up.

The Move: SQS automatically moves that specific message from the main queue into the Dead Letter Queue.

Why use a DLQ?

Isolate Problematic Data: It separates "bad" messages from the "good" ones so your main pipeline can keep flowing.

Debugging: You can inspect the DLQ to see exactly what caused the failure (e.g., malformed JSON or an unexpected null value).

Manual Recovery: Once you fix the bug in your code, you can "drive" the messages back from the DLQ into the main queue to be processed correctly.

Common SQS Attributes for DLQs

Term                        Meaning

-------                        ------------

Source Queue          The original queue where messages arrive first.

Redrive Policy         The configuration that links the source queue to the DLQ.

maxReceiveCount   The number of times a message can fail before being sent to the DLQ.

If queues ending in -dlq have 0.0 activity (number of messages written in past X days), this is actually a sign of a healthy system! It means:

• Your consumers are processing messages successfully.
• No "poison pill" messages have failed enough times to be kicked over to the dead letter storage.

Always set an alarm on your DLQs for ApproximateNumberOfMessagesVisible > 0. You want to know immediately when a message lands there, as it usually means a bug in your consumer code or a problem with your data.

Is DLQ created automatically for each SQS queue?

No, a Dead Letter Queue (DLQ) is not created automatically. It is a configuration you must set up yourself.

In AWS, a DLQ is just a regular SQS queue that you have "pointed" another queue toward. If you create a queue named orders-queue and don't explicitly tell it to use a DLQ, it simply won't have one.

The Two Steps to Creating a DLQ

To set up a DLQ, you have to perform two distinct actions:

1. Create the DLQ itself: You create a second, standalone SQS queue (usually named with a -dlq suffix).
2. Configure the Redrive Policy: You go back to your source queue and update its settings to point to the ARN (Amazon Resource Name) of the DLQ you just created.

Key Requirements

For a DLQ relationship to work, a few rules must be followed:

• Same Region: The source queue and the DLQ must be in the same AWS Region and the same AWS Account
• Same Type: A FIFO queue must use a FIFO queue as its DLQ. A Standard queue must use a Standard queue as its DLQ.
• Permissions: If you are using a custom KMS key for encryption, the source queue needs permission to use that key to move messages to the DLQ.

What happens if you don't have a DLQ?

If a message fails to process and you haven't configured a DLQ:

1. The message returns to the source queue after the Visibility Timeout expires.
2. The consumer picks it up again.
3. If the message is "poison" (causes a crash), this loop repeats indefinitely until the message's Message Retention Period (default 4 days) expires.
4. Once the retention period is up, the message is simply deleted by AWS and lost forever.

In professional environments it is "Best Practice" to create a DLQ for every production queue. We should have 1:1 ratio of queue and -dlq names in our list of SQS queues. Engineers should create and link them to ensure no data is lost during a processing failure.

Any queues in our list that don't have a matching -dlq might be candidates for a quick configuration update!

Message Queuing Service - Amazon Simple Queue Service - AWS