Friday, 13 June 2025

Introduction to Serverless Framework



Serverless Framework is a tool designed to streamline the development and deployment of serverless applications, including functions and infrastructure, by abstracting away the need to manage servers. 

We define desired infrastructure in serverless yaml files and then deploy it by executing:

sls deploy

This command parses serverless yaml file into larger AWS CloudFormation template which automatically gets filled with values from the yaml. 


Serverless Yaml Configuration File


serverless yaml file defines a serverless service. It is a good idea to break up the serverless project into multiple services, each of which is defined by its own serverless yaml file. We don't want to have everything in one big infrastructure stack. 

Example:
  • database e.g. DynamoDB
  • Rest API e.g. which handles the submitted web form and stores data in DynamoDB
  • front-end website which e.g. stores React app website in s3 bucket

Services can be deployed in multiple regions. (Multi-region architecture is supported)


serverless.yml example:


service: my-service
frameworkVersion: "3"
useDotenv: true
plugins: 
  - serverless-plugin-log-subscription
  - serverless-dotenv-plugin
provider:
  name: aws
  runtime: nodejs14.x
  region: eu-east-1
  memorySize: 512
  timeout: 900
  deploymentBucket:
    name: my-serverless-deployments
  vpc: 
    securityGroupIds: 
      - "sg-0123cf34f6c6354cb"
    subnetIds: 
      - "subnet-01a23493f9e755207"
      - "subnet-02b234dbd7d66d33c"
      - "subnet-03c234712e99ae1fb"
  iam: 
    role:
      statements:
        - Effect: Allow
          Action:
            - lambda:InvokeFunction
          Resource: arn:aws:lambda:eu-east-1:123456789099:function:my-database
package:
  patterns:
    - "out/**"
    - "utils.js"
    - "aws-sdk"
functions:
  my-function:
    handler: lambda.handler
    events:
      - schedule:
          name: "my-service-${opt:stage, self:provider.stage}"
          description: "Periodically run my-service lambdas"
          rate: rate(4 hours)
          inputTransformer:
            inputTemplate: '{"Records":[{"EventSource":"aws:rate","EventVersion":"1.0","EventSubscriptionArn":"arn:aws:sns:eu-east-1:{{accountId}}:ExampleTopic","Sns":{"Type":"Notification","MessageId":"95df01b4-1234-5678-9903-4c221d41eb5e","TopicArn":"arn:aws:sns:eu-east-1:123456789012:ExampleTopic","Subject":"example subject","Message":"example message","Timestamp":"1970-01-01T00:00:00.000Z","SignatureVersion":"1","Signature":"EXAMPLE","SigningCertUrl":"EXAMPLE","UnsubscribeUrl":"EXAMPLE","MessageAttributes":{"type":{"Type":"String","Value":"populate_unsyncronised"},"count":{"Type":"Number","Value":"400"}}}}]}'
      - sns:
          arn: arn:aws:sns:us-east-2:123456789099:trigger-my-service
      - http: 
custom:
  dotenv:
    dotenvParser: env.loader.js
  logSubscription:
      enabled: true
      destinationArn: ${env:KINESIS_SUBSCRIPTION_STREAM}
      roleArn: ${env:KINESIS_SUBSCRIPTION_ROLE}



  • provider
    • vpc
      • securityGroupIds 
      • subnetIds - typically a list of private subnets with NAT gateway. 
  • functions - defines the AWS Lambda functions that are deployed as part of this Serverless service. This is where we define the AWS Lambda functions that our Serverless service will deploy. 
    • <function_name> (e.g., my-function) Each function entry under functions specifies:
      • handler - tells Serverless which file and exported function to execute as the Lambda entry point (e.g., lambda.handler)
      • events - a list of events that trigger this function, such as:
        • schedule: for periodic invocation (cron-like jobs)
        • sns: for invocation via an AWS SNS topic
        • http

Thursday, 12 June 2025

Useful Kibana DevTools Queries





To perform a search operation on a specific index:

GET /my_index/_search 

By itself (without a request body), it returns the first 10 documents by default. This request is the same as the above one:

GET /my_index/_search
{
  "query": {
    "match_all": {}
  }
}


To get the number of documents in an Elasticsearch index, you can use the _count API or the _stats API.

GET /my_index/_count

This will return a response like:

{
  "count": 12345,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  }
}


To get a certain number of documents, use size argument:

GET my_index/_search?size=900

We can also use _cat API:

GET /_cat/count/my_index?v

This will return output like:

epoch      timestamp count
1718012345 10:32:25  12345


GET /my_index/_stats

"indices": {
  "my_index": {
    "primaries": {
      "docs": {
        "count": 12345,
        "deleted": 12
      }
    }
  }
}


To get the union of all values of some field e.g. channel_type field across all documents in the my_index index, we can use an Elasticsearch terms aggregation:


GET my_index/_search
{
  "size": 0, 
  "aggs": {
    "unique_channel_types": {
      "terms": {
        "field": "channel_type.keyword",
        "size": 10000  // increase if you expect many unique values
      }
    }
  }
}


Explanation:
  • "size": 0: No documents returned, just aggregation results.
  • "terms": Collects unique values.
  • "channel_type.keyword": Use .keyword to aggregate on the raw value (not analyzed text).
  • "size": 10000: Max number of buckets (unique values) to return. Adjust as needed.

Response example:

{
  "aggregations": {
    "unique_channel_types": {
      "buckets": [
        { "key": "email", "doc_count": 456 },
        { "key": "push", "doc_count": 321 },
        { "key": "sms", "doc_count": 123 }
      ]
    }
  }
}

The "key" values in the buckets array are your union of channel_type values.


Let's assume that my_index has the timestamp field (as the root field...but it can be at any path in which case we'd need to adjust the query) is correctly mapped as a date type.


To find the oldest document:

GET my_index/_search
{
  "size": 1,
  "sort": [
    { "timestamp": "asc" }
  ]
}


To find the newest document:

GET my_index/_search
{
  "size": 1,
  "sort": [
    { "timestamp": "desc" }
  ]
}

----