Managing AWS S3 using Terraform

 

Configuration for provisioning an S3 bucket:

main.tf:

resource "aws_s3_bucket" "images" {

    bucket = "images-123456"

    tags = {

        Description = "Images for training ML model"

    }

}

Both bucket and tags are optional. If bucket is not provided, TF will generate random and unique bucket name. 

 

Bucket name must be DNS-compliant: it cannot contain underscores (but it can contain dashes - minus signs). If name contains underscores (e.g. bucket = "images_123456"), terraform apply will issue an error:

 

aws_s3_bucket.images: Creating...

Error: error creating S3 Bucket (images_123456): InvalidBucketName: The specified bucket is not valid.
        status code: 400, request id: 2D91CDB6D21FDEF1, host id: MzRISOwyjmnup2D91CDB6D21FDEF17/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp

  on main.tf line 1, in resource "aws_s3_bucket" "images":
   1: resource "aws_s3_bucket" "images" {

 

To specify objects we want to be uploaded to the bucket, we need to use aws_s3_bucket_object resource type:

resource "aws_s3_bucket_object" {

    content = "/root/images/cat001.jpg"

    key = "cat001.jpg"

    bucket = aws_s3_bucket.images.id

}

content is the local path to the file. key is the name of the object being uploaded. bucket is the bucket id and we use the reference expression to point to the previously created bucket. terraform apply uploads the file to S3 bucket.

Bucket access policy defines who can access this bucket.

If there is IAM group named "ML Engineers" created manually, terraform.tfstate is not aware of it. To get this resource under TF control we need to create its data source so TF can read its attributes:

data "aws_iam_group" "data-ml-engineers" {

    group_name = "ml-engineers"

}

Finally, let's define S3 bucket policy which allows ML Engineers group an access to all files in the images bucket:

resource "aws_s3_bucket_policy" "policy-images" {

    bucket = aws_s3_bucket. images.id

    policy = <<EOF

    {

        "Version": "2022-05-16",

        "Statement": [

            {

                "Action": "*",

                "Effect": "Allow",

                "Resource": "arn:aws:s3:::${aws_s3_bucket.images.id}/*",

                "Principal": {

                    "AWS": [

                        "${data.aws_iam_group.data-ml-engineers.arn}"

                    ]

                }

            }

        ] 

    }

    EOF

}

---