Feel free to check more updates from re:Invent and pre:Invent 2025 here: https://aws-news.com/

CloudTrail events in CloudWatch: fewer moving parts, more visibility

AWS added a more straightforward way to enable CloudTrail events in CloudWatch using service-linked channels (SLCs).

Why I care

Because half the time you want CloudTrail events in CloudWatch, you’re not trying to build an archival strategy. You’re trying to answer:

• “Who changed this security group?”

• “Why did this API call spike?”

• “Can I alert on this before it becomes a Slack incident?”

SLCs also include features such as safety checks and termination protection.

The “AWS fine print”

You still pay:

• CloudTrail event delivery charges and

• CloudWatch Logs ingestion (custom logs pricing).

Yes, it’s simpler, but don’t turn everything on everywhere and then act surprised.

Database Savings Plans

AWS introduced Database Savings Plans (up to 35% savings).
This is essentially AWS acknowledging that databases are expensive and that “please right-size” is not a strategy.

What to do with it

If you have steady-state usage (Aurora/RDS/others in the eligible set), you can treat it like:

• commit for the baseline,

• keep spikes on-demand.

AWS also integrated this into the billing console recommendations flow (Savings Plans recommendations + purchase analyzer).

Lambda Managed Instances: serverless DX, EC2-shaped compute

This one is spicy: Lambda Managed Instances lets you run Lambda functions on your Amazon EC2 instances via a capacity provider model. This update is very controversial, since I believe in Lambda being serverless, but you can still use Lambda (as it was initially intended) and ignore this release 😂

You define:

• VPC config

• optional instance requirements

• scaling policies
  …and then attach Lambda functions to that capacity provider via console, API, or IaC.

Why this matters

This is a new middle layer for teams that:

• love Lambda event sources + tooling (CloudWatch, X-Ray, Config…)

• but want more control over compute shape or cost for steady workloads.

Also, supported runtimes include the latest Java, Node.js, Python, and .NET.

The “this will come up in architecture review” part

Third-party reporting states that pricing is standard EC2 + a compute management fee + request pricing, and that this eliminates the usual Lambda duration charge (since you’re paying for EC2).

API Gateway adds MCP proxy support (AI-adjacent, but it’s really “API-as-a-tool”)

API Gateway now supports MCP proxy capability, which provides protocol translation so a REST API can communicate with MCP clients/agents without requiring app changes or additional infrastructure.

AWS frames it alongside Bedrock AgentCore Gateway services, but the “boring” value is:

• governance,

• auth,

• throttling,

• making APIs discoverable/usable as tools

If you’re not building “agents”, you can still read this as: API Gateway now supports another integration pattern that used to require custom glue (not the service).

AWS Interconnect (multicloud) preview: private connectivity to other clouds

AWS announced a preview of AWS Interconnect – multicloud: “simple, resilient, high-speed private connections” to other cloud providers.

It starts with Google Cloud as the first partner, and AWS says Azure comes later in 2026.

Why this matters

Multicloud connectivity is usually:

• slow to procure,

• annoying to operate,

• and “fun” during incident response.

The promise here is: private links between clouds without weeks of paperwork and waiting.

We will see how this evolves over time.

Amazon S3 Vectors is GA: S3 continues to absorb the universe

S3 Vectors is now generally available, and AWS says it’s available in 14 Regions (up from 5 in preview).

Even if you don’t want to say “embeddings” out loud, vector storage shows up in:

• similarity search,

• dedupe,

• recommendations,

• anomaly detection,

• and hybrid search patterns.

AWS also highlights integration patterns in which OpenSearch can manage vector storage in S3 to optimize hybrid search costs.

This is an update where I want to make a showcase blog post in the upcoming weeks.

Lambda Durable Functions: long-running workflows without paying for “waiting”

AWS added durable functions for Lambda to build multi-step workflows that can run for seconds to one year, without incurring idle compute costs while waiting for humans/external systems.

I am still a fan of Step Functions, and TBH, by using Step Functions’ native integration, there is no need for Lambda functions for everything (i.e., put DynamoDB item). I see the value here, but for me, Step Functions still win in that case.

Why this matters

Because today, teams tend to choose between:

• Step Functions (great, but can become “JSON orchestration art”)

• DIY state in DynamoDB + retries + idempotency + sadness

Durable functions are AWS's way of saying: “What if the workflow was code, but also reliable?”

The AWS News Blog post explicitly positions it for long-running multi-step coordination and not paying while waiting.

Conclusion

It’s easy to get swept up in the big re:Invent moments, but these quieter updates are the ones that tend to stick. They refine the everyday experience of building on AWS, and that’s often where the real gains show up.

Feel free to reach out if you have suggestions for my next blog post.

Till the next time, stay safe and have fun! ❤️

What we’ll see:

• How to provision S3 Tables with Terraform, including table buckets, namespaces, and Iceberg tables.

• Set up a workflow with Step Functions and Glue Jobs to process CSV files and ingest the data into our Iceberg tables.

• Managing the access with Lake Formation (basic)

• Ability to query our Iceberg tables from Athena, which will make the provisioning of Lake Formation mandatory🥲

We will not see the S3 Tables table maintenance in this post. You can read more about table maintenance here.

This is a Level 300 post; following along with the post and deploying the infrastructure to your AWS account will cost approximately ~3.5$, given that you run the Glue Job around 10 times and have around 1GB of files in S3 buckets. You can follow along with the code in my repo.

What are S3 Tables

Amazon S3 Tables is a purpose-built storage solution within Amazon S3, explicitly optimized for analytics workloads that use tabular data. Unlike general-purpose S3 buckets, S3 Tables introduces a new bucket type—table buckets—designed to efficiently store and manage data in a table-like structure (rows and columns), such as transaction logs, sensor data, or event streams.

S3 Tables natively supports the Apache Iceberg format, enabling advanced features like schema evolution, partition evolution, and time travel queries. This means you can query your data using standard SQL with analytics engines that support Iceberg, such as Amazon Athena, Amazon Redshift, and Apache Spark.

Key features of S3 Tables include:

• Optimized performance for high-throughput analytics queries.

• Automated table maintenance (compaction, snapshot management, and cleanup of unused files).

• Fine-grained access control using IAM and Lake Formation

  

• Seamless integration with AWS analytics services, allowing easy discovery and querying of your tabular data.

What You’ll Learn

In this post, you’ll build an automated way of processing CSV files into Iceberg using Step Functions, Lambdas, Glue Jobs, and most importantly, S3 Tables.

Key takeaways:

• Lake Formation is mandatory when you want to expose your data to Athena and other analytical services within AWS

• Unfortunately, there will be some manual work from the Console regarding Lake Formation

• No need to activate lake formation if the Iceberg tables are only for your Spark jobs

Setting the Stage - Terraform

As you can see in the diagram above, we will set up a landing bucket for our data, and when we upload a CSV file, a lambda will be triggered. This lambda will trigger a step function (this is in case you want to expand the functionality) that will run the Glue Job to process the CSV file. The data will land in the S3 Table, and we will be able (with proper access) to query the data in Athena as well.

Terraform

In the Terraform code below, I will remove the IAM statements and things we have covered multiple times in our previous posts. I will focus on the essential things regarding our pipeline. You can find the full code here.

As always, we will start from the S3 buckets listed in s3.tf. We create two buckets, one for landing our CSV files named raw_data_bucket and one for the artifacts of the Glue Job.

module "artifacts_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 4.8"

  bucket = "${local.project_name}-glue-artifacts-${local.environment}"

  force_destroy = true
  acl           = "private"

  # Add ownership controls
  control_object_ownership = true
  object_ownership         = "ObjectWriter"


  tags = local.tags
}

module "raw_data_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 4.10"

  bucket = local.raw_data_bucket_name

  force_destroy = true
  acl           = "private"

  # Add ownership controls
  control_object_ownership = true
  object_ownership         = "ObjectWriter"

  tags = local.tags
}

Now let’s create the Glue Job, which will take the file from the raw bucket and convert it to Iceberg format. File: glue.tf

# Upload the S3 Tables Iceberg connector JAR
resource "aws_s3_object" "s3_tables_connector" {
  bucket = module.artifacts_bucket.s3_bucket_id
  key    = "jars/s3-tables-catalog-for-iceberg-runtime-0.1.5.jar"
  source = "${path.module}/jars/s3-tables-catalog-for-iceberg-runtime-0.1.5.jar"
  etag   = filemd5("${path.module}/jars/s3-tables-catalog-for-iceberg-runtime-0.1.5.jar")
}

# Upload the Glue job script to S3
resource "aws_s3_object" "glue_job_script" {
  depends_on = [aws_s3_object.s3_tables_connector]
  bucket     = module.artifacts_bucket.s3_bucket_id
  key        = "scripts/csv_to_iceberg.py"
  source     = "${path.module}/scripts/csv_to_iceberg.py"
  etag       = filemd5("${path.module}/scripts/csv_to_iceberg.py")
}

# Glue job definition
resource "aws_glue_job" "csv_to_iceberg" {
  depends_on = [aws_s3_object.glue_job_script]
  name       = "${local.project_name}-csv-to-iceberg"
  role_arn   = aws_iam_role.glue_job_role.arn

  command {
    name            = "glueetl"
    script_location = "s3://${module.artifacts_bucket.s3_bucket_id}/${aws_s3_object.glue_job_script.key}"
    python_version  = "3"
  }

  default_arguments = {
    "--job-language"                     = "python"
    "--job-bookmark-option"              = "job-bookmark-enable"
    "--enable-metrics"                   = "true"
    "--enable-continuous-cloudwatch-log" = "true"
    "--TempDir"                          = "s3://${module.raw_data_bucket.s3_bucket_id}/temp/"
    "--extra-jars"                       = "s3://${module.artifacts_bucket.s3_bucket_id}/jars/s3-tables-catalog-for-iceberg-runtime-0.1.5.jar"
  }

  execution_property {
    max_concurrent_runs = 2
  }

  glue_version      = "5.0"
  worker_type       = "G.1X"
  number_of_workers = 2
  timeout           = 15
}

resource "aws_iam_role" "glue_job_role" {
  name = "${local.project_name}-glue-job-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "glue.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "glue_service" {
  role       = aws_iam_role.glue_job_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

This is pretty much the same as what we used in our previous post (see below), the only difference is that we upload the JAR file for managing the Iceberg tables.

Regarding the Step Function, it is as simple as it gets; it is composed of a single step where it runs the Glue Job. Feel free to extend this and add more capabilities and let me know in the comments below what you did. File: state_machine.tf

# Step Functions state machine definition
module "etl_state_machine" {
  source  = "terraform-aws-modules/step-functions/aws"
  version = "~> 4.2.1"

  name = "${local.project_name}-etl-workflow"

  attach_policy_json = true
  policy_json        = data.aws_iam_policy_document.step_functions_glue_policy.json

  definition = jsonencode({
    Comment = "ETL workflow to process CSV to Iceberg and crawl the data",
    StartAt = "StartGlueJob",
    States = {
      "StartGlueJob" = {
        Type     = "Task",
        Resource = "arn:aws:states:::glue:startJobRun.sync",
        Parameters = {
          JobName = aws_glue_job.csv_to_iceberg.name,
          Arguments = {
            "--source_s3_path.$"   = "$.source_s3_path",
            "--table_namespace.$"  = "$.table_namespace",
            "--table_name.$"       = "$.table_name",
            "--table_bucket_arn.$" = "$.table_bucket_arn"
          }
        },
        ResultPath = "$.glueJobResult",
        End        = true
      }
    }
  })
}

Now let’s put everything together. Let’s create a trigger on the S3 bucket every time we upload a CSV and trigger the step function accordingly using a Lambda. File: lambdas.tf

# ECR Docker image for Lambda
module "docker_image" {
  source = "terraform-aws-modules/lambda/aws//modules/docker-build"

  ecr_repo    = module.ecr.repository_name
  source_path = "${path.module}/lambdas"

  use_image_tag = true
}

module "ecr" {
  source = "terraform-aws-modules/ecr/aws"

  repository_name         = "${local.project_name}-ecr"
  repository_force_delete = true

  create_lifecycle_policy = false

  repository_lambda_read_access_arns = [module.trigger_step_function.lambda_function_arn]
}


module "trigger_step_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.20"

  function_name = "${local.project_name}-trigger-step-function"
  description   = "Lambda function to trigger Step Function when a file is uploaded to S3"

  create_package = false
  image_uri      = module.docker_image.image_uri
  package_type   = "Image"

  timeout     = 300
  memory_size = 512

  environment_variables = {
    GLUE_JOB_NAME     = aws_glue_job.csv_to_iceberg.name
    STATE_MACHINE_ARN = module.etl_state_machine.state_machine_arn
    TABLE_NAMESPACE   = aws_s3tables_namespace.iceberg_namespace.namespace
    TABLE_NAME        = local.table_name
    TABLE_BUCKET_ARN  = module.s3_tables_bucket.s3_table_bucket_arn
  }

  image_config_command = ["trigger_step_function.handler"]

  attach_policies = true
  policies = [
    "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
    aws_iam_policy.lambda_glue_access.arn,
    aws_iam_policy.lambda_step_functions_policy.arn
  ]
  number_of_policies = 3

  tags = local.tags
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = module.raw_data_bucket.s3_bucket_id

  lambda_function {
    lambda_function_arn = module.trigger_step_function.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "input/"
    filter_suffix       = ".csv"
  }

  depends_on = [aws_lambda_permission.allow_bucket]
}

resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = module.trigger_step_function.lambda_function_arn
  principal     = "s3.amazonaws.com"
  source_arn    = "arn:aws:s3:::${module.raw_data_bucket.s3_bucket_id}"
}

Again, make sure you check the files, because I am vomiting the IAM resources.

Now let’s see the protagonist of our post, the S3 Table. File: main.tf

module "s3_tables_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws//modules/table-bucket"
  version = "~> 4.10"

  table_bucket_name = local.s3_tables_bucket_name
  encryption_configuration = {
    kms_key_arn   = module.kms.key_arn
    sse_algorithm = "aws:kms"
  }

  maintenance_configuration = {
    iceberg_unreferenced_file_removal = {
      status = "enabled"

      settings = {
        non_current_days  = 7
        unreferenced_days = 3
      }
    }
  }

  create_table_bucket_policy = true
  table_bucket_policy        = data.aws_iam_policy_document.s3_tables_bucket_policy.json
}

# S3 Tables Namespace - requires a table bucket
resource "aws_s3tables_namespace" "iceberg_namespace" {
  namespace        = local.namespace_name
  table_bucket_arn = module.s3_tables_bucket.s3_table_bucket_arn
}

S3 Table is a different kind of S3 bucket under the Table Buckets section in the AWS console. Once you have created the table, you will find it there. You can see that I have added some simple maintenance (data housekeeping), but more functionality can be added if you want to.

Since we want to query our Iceberg table with Athena, we must enable the integration with the AWS analytics services. This will use Lake Formation for access management and Cataloging.

To convert the CSV file from the S3 bucket to the Iceberg, we will use the following code (part of csv_to_iceberg.py)

# Initialize spark session with integration to analytics services
spark = SparkSession.builder.appName("SparkIcebergSQL") \
       .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
       .config("spark.sql.defaultCatalog", "s3tables") \
       .config("spark.sql.catalog.s3tables", "org.apache.iceberg.spark.SparkCatalog") \
       .config("spark.sql.catalog.s3tables.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \
       .config("spark.sql.catalog.s3tables.glue.id", f"{ACCOUNT_ID}:s3tablescatalog/{TABLE_BUCKET_NAME}") \
       .config("spark.sql.catalog.s3tables.warehouse", f"s3://{TABLE_BUCKET_NAME}/warehouse/") \
       .getOrCreate()  

First, we initiate the Spark Session with the integration to analytics services.

dynamic_frame = glueContext.create_dynamic_frame.from_options(
        format_options={
            "quoteChar": '"',
            "withHeader": True,
            "separator": ",",
            "optimizePerformance": False,
        },
        connection_type="s3",
        format="csv",
        connection_options={
            "paths": [SOURCE_S3_PATH],
            "recurse": True
        },
        transformation_ctx="read_csv"
    )

Then we read the CSV file from the S3 bucket and place it into a dynamic frame.

columns = df.dtypes
columns_sql = ", ".join([f"{slugify(name)} {dtype.upper()}" for name, dtype in columns])

table_identifier = f"{TABLE_NAMESPACE}.{TABLE_NAME}"
    
create_table_sql = f"""
    CREATE TABLE IF NOT EXISTS {table_identifier} (
        {columns_sql}
    )
"""

spark.sql(create_table_sql)

Then we infer the schema of the CSV and create the table in the Table Bucket.

df.createOrReplaceTempView("temp_data_to_insert")
insert_sql = f"""
    INSERT INTO {table_identifier}
    SELECT * FROM temp_data_to_insert
"""
spark.sql(insert_sql)

Last but not least, we insert the data into the table.

Set up the environment

Now, to put everything in place, we will use Terraform. As I said in the beginning, we will have some manual Steps.

First of all, run:

terraform apply

This will create the resources, but you will not be able to run the script since we will need to enable Lake Formation and provide the necessary access to our Glue Role.

First of all, you will need to enable the Integration:

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.

2. In the left navigation pane, choose Table buckets.

3. Click on Enable integration

Once you do that, your Bucket will be registered as a catalog in Lake formation with the name s3tablescatalog. Navigate to the Lake Formation page and do the following:

Go to Permissions and then click on Data permissions and then click Grant

• Select Principals and then Iam user and Roles. Select your Glue Role

• Then select Named Data Catalog resources and

  • Select your catalog with the bucket name at the end

  • Select your database

• Click at the bottom Super access for Database permissions and Grantable permissions (You can narrow down the scope of the access if you want to by adding only Describe, Create table, and Alter)

We do the same thing, but this time we select all Tables as well

I’ve tried to do this with Terraform, but it seems there is a bug for referencing an S3 Tables catalog.

Last but not least, you will need to uncomment the resource here.

resource "aws_lakeformation_permissions" "data_location" {
  principal   = aws_iam_role.glue_job_role.arn
  permissions = ["DATA_LOCATION_ACCESS"]

  data_location {
    arn = module.s3_tables_bucket.s3_table_bucket_arn
  }
}

And run again

terraform apply

And you are set. Upload a CSV file in your raw bucket under the prefix input and see it run 😄

Conclusion

Exploring S3 Tables has been a game-changer in my thinking about storing and analyzing tabular data in the cloud. The performance, built-in Iceberg support, and “seamless integration” (besides Lake Formation) with AWS analytics tools made setting up a modern, scalable data lake easy.

If you’re curious about the next generation of data lake storage, I definitely recommend giving S3 Tables a try. It’s been a fun and eye-opening experience, and I’m excited to see how this new approach will shape future analytics projects!

To destroy what we have created today, simply run

terraform destroy  

Feel free to reach out if you encounter any problems or have suggestions.

Till the next time, stay safe and have fun! ❤️

We will use the following services with Terraform, where the majority of those are from Terraform AWS Modules:

• Amazon S3: for storing our raw and processed data

• AWS Glue:

  • Glue Job: For processing files

  • Glue Crawler for cataloging the data

• Step-function: For executing the whole workflow

• Lambda: For triggering the step function upon file upload in the S3 bucket

This is a Level 200 post; following along with the post and deploying the infrastructure to your AWS account will cost approximately ~$1.5 per month, given you run ~10 times a month, the Glue job and the Crawler. You can follow along with the code in my repo.

What is AWS Glue

AWS Glue is a fully managed, serverless data integration service that eliminates the complexity of building and managing data infrastructure. It automatically provisions, scales, and manages the compute resources needed for your data workloads, allowing you to focus on extracting value from your data rather than managing servers.

🕷️ Crawlers - Automatically discover and catalog your data across various sources (S3, databases, data lakes), inferring schemas and populating metadata without manual intervention.

📚 Data Catalog - A centralized metadata repository that is a persistent store for table definitions, schema information, and data location details, making your data discoverable and queryable.

⚙️ Jobs - Serverless ETL (Extract, Transform, Load) and ELT workflows that process your data using familiar programming languages like Python and Scala, with built-in monitoring and error handling.

What You'll Learn

In this post, you'll build an automated AWS Glue ETL pipeline that transforms CSV files to Parquet format and catalogs them for analytics.

Key takeaways:

• Glue Jobs - Create serverless ETL scripts and use Glue Jobs to process your data

• Glue Crawlers - Set up automatic schema discovery and data cataloging for your processed datasets

• Event-Driven Processing - Trigger Glue jobs automatically when new files arrive in S3, creating a seamless data ingestion pipeline

• Serverless Spark - Leverage AWS Glue's managed Spark environment for scalable data transformations without cluster management

Setting the Stage - Terraform

As we can see in the Diagram above, we are going to use Glue Jobs to convert a file from CSV to Parquet. You can replace the Glue code with whatever you like; I am just using this example to showcase the Job.

We will start by creating the buckets for our use case. File: s3.tf. We have created multiple buckets using Anton’s module, so I will only show you the definition of the RAW bucket. Most importantly, I will later show you how to create the trigger.

module "raw_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 4.8"

  bucket = "${local.project_name}-raw-data-${local.environment}"

  force_destroy = true
  acl           = "private"

  # Add ownership controls
  control_object_ownership = true
  object_ownership         = "ObjectWriter"


  tags = local.tags
}

Now, let’s create our Glue Job, which will take the files from the raw bucket and place them as Parquet in a processed bucket. File: glue.tf

# Upload the Glue job script to S3
resource "aws_s3_object" "glue_job_script" {
  bucket = module.artifacts_bucket.s3_bucket_id
  key    = "scripts/csv_to_parquet.py"
  source = "${path.module}/scripts/csv_to_parquet.py"
  etag   = filemd5("${path.module}/scripts/csv_to_parquet.py")
}

# Glue job definition
resource "aws_glue_job" "csv_to_parquet" {
  depends_on = [aws_s3_object.glue_job_script]
  name       = "${local.project_name}-csv-to-parquet"
  role_arn   = aws_iam_role.glue_job_role.arn

  command {
    name            = "glueetl"
    script_location = "s3://${module.artifacts_bucket.s3_bucket_id}/${aws_s3_object.glue_job_script.key}"
    python_version  = "3"
  }

  default_arguments = {
    "--job-language"                     = "python"
    "--job-bookmark-option"              = "job-bookmark-enable"
    "--enable-metrics"                   = "true"
    "--enable-continuous-cloudwatch-log" = "true"
    "--TempDir"                          = "s3://${module.parquet_bucket.s3_bucket_id}/temp/"
    "--input_path"  = "s3://${module.raw_bucket.s3_bucket_id}/input/"
    "--output_path" = "s3://${module.parquet_bucket.s3_bucket_id}/data/"
  }

  execution_property {
    max_concurrent_runs = 1
  }

  glue_version      = "5.0"
  worker_type       = "G.1X"
  number_of_workers = 2
  timeout           = 10 # minutes
}

A lot to digest here 😅, let’s go line by line and see the configuration of our crawler.

resource "aws_s3_object" "glue_job_script" {
  bucket = module.artifacts_bucket.s3_bucket_id
  key    = "scripts/csv_to_parquet.py"
  source = "${path.module}/scripts/csv_to_parquet.py"
  etag   = filemd5("${path.module}/scripts/csv_to_parquet.py")
}

This is to upload the script that Glue will run, in this script I have used both Spark and GlueSpark context.

command {
    name            = "glueetl"
    script_location = "s3://${module.artifacts_bucket.s3_bucket_id}/${aws_s3_object.glue_job_script.key}"
    python_version  = "3"
}

Let’s define the execution environment and script location. We specify where Glue can find the script and what the Python version is

default_arguments = {
    "--job-language"                     = "python"
    "--job-bookmark-option"              = "job-bookmark-enable"
    "--enable-metrics"                   = "true"
    "--enable-continuous-cloudwatch-log" = "true"
    "--TempDir"                          = "s3://${module.parquet_bucket.s3_bucket_id}/temp/"
    "--input_path"  = "s3://${module.raw_bucket.s3_bucket_id}/input/"
    "--output_path" = "s3://${module.parquet_bucket.s3_bucket_id}/data/"
}

Now for the job behavior and data flow paths. We set up the observability features for logging and enable bookmarks. Last but not least, we define the locations we need to interact with our files.

execution_property {
  max_concurrent_runs = 1
}

This prevents the same job from running multiple times. You can set this to whatever value you need; I’ve used 1 to ensure the cost estimate above is accurate.

glue_version      = "5.0"
worker_type       = "G.1X"
number_of_workers = 2
timeout           = 10 # minutes

We define the compute environment and resource allocation where we set the Glue runtime version (determines Spark version, features available), the allocated compute resources (worker type and count), and some safety limits (timeout) to prevent runaway costs.

After the Glue Job, we create a Glue Crawler to catalog our data from the destination bucket. You can find the definition here; no need to describe this again, since we have seen it multiple times in our previous posts.

Now we need to define our Step Function. It will operate as shown in the definition below.

Feel free to copy the definition from here. File: state_machine.tf

# Step Functions state machine definition
module "etl_state_machine" {
  source  = "terraform-aws-modules/step-functions/aws"
  version = "~> 4.2.1"

  name = "${local.project_name}-etl-workflow"

  attach_policy_json = true
  policy_json        = data.aws_iam_policy_document.step_functions_glue_policy.json

  definition = jsonencode({
    Comment = "ETL workflow to process CSV to Parquet and crawl the data",
    StartAt = "StartGlueJob",
    States = {
      "StartGlueJob" = {
        Type     = "Task",
        Resource = "arn:aws:states:::glue:startJobRun.sync",
        Parameters = {
          JobName = aws_glue_job.csv_to_parquet.name,
          Arguments = {
            "--input_path.$"  = "$.input_path",
            "--output_path.$" = "$.output_path"
          }
        },
        ResultPath = "$.glueJobResult",
        Next       = "StartGlueCrawler"
      },
      ...
      "Success" = {
        Type = "Pass",
        End  = true
      }
    }
  })
}

Let’s see now how we create the trigger along with the Lambda. The Lambda will be Dockerized. File lambda.tf This is the Lambda that will trigger the step function above.

We first define the Docker image and Registry.

# ECR Docker image for Lambda
module "docker_image" {
  source = "terraform-aws-modules/lambda/aws//modules/docker-build"

  ecr_repo = module.ecr.repository_name
  # image_tag       = "latest"
  source_path = "${path.module}/lambdas"

  # cache_from = ["${module.ecr.repository_url}:latest"]
  # Use the pre-built image from ECR
  use_image_tag = true
}

module "ecr" {
  source = "terraform-aws-modules/ecr/aws"

  repository_name         = "${local.project_name}-ecr"
  repository_force_delete = true

  create_lifecycle_policy = false

  repository_lambda_read_access_arns = [module.trigger_step_function.lambda_function_arn]
}

Then we create the Lambda.

module "trigger_step_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.20"

  function_name = "${local.project_name}-trigger-step-function"
  description   = "Lambda function to trigger Step Function when a file is uploaded to S3"

  # Docker image config
  create_package = false
  image_uri      = module.docker_image.image_uri
  package_type   = "Image"

  # Lambda settings
  timeout     = 300
  memory_size = 512

  # Environment variables
  environment_variables = {
    GLUE_JOB_NAME     = aws_glue_job.csv_to_parquet.name
    OUTPUT_BUCKET     = module.parquet_bucket.s3_bucket_id
    STATE_MACHINE_ARN = module.etl_state_machine.state_machine_arn
  }

  image_config_command = ["trigger_step_function.handler"]

  # IAM policy statements
  attach_policies = true
  policies = [
    "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
    aws_iam_policy.lambda_glue_access.arn,
    aws_iam_policy.lambda_step_functions_policy.arn
  ]
  number_of_policies = 3

  tags = local.tags
}

Last but not least, we create the notification from the bucket we have created above.

# S3 event notification to trigger Lambda
resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = module.raw_bucket.s3_bucket_id

  lambda_function {
    lambda_function_arn = module.trigger_step_function.lambda_function_arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "input/"
    filter_suffix       = ".csv"
  }

  depends_on = [aws_lambda_permission.allow_bucket]
}

# Permission for S3 to invoke Lambda
resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = module.trigger_step_function.lambda_function_arn
  principal     = "s3.amazonaws.com"
  source_arn    = "arn:aws:s3:::${module.raw_bucket.s3_bucket_id}"
}

You can edit your deployment by configuring the variables in locals.tf. Additional files, not included in the post, are:

• iam.tf: to define the IAM access of our resources

• trigger_step_function.py: the Lambda code

• Dockerfile: the Lambda Image

And that’s it. You are now ready to deploy the infrastructure.

terraform plan

and once we validate the plan, we can run

terraform apply

Using Glue Jobs

How Glue Jobs Execute Spark Code Serverlessly

AWS Glue Jobs provide a serverless Apache Spark environment that automatically provisions the exact compute resources you need, executes your PySpark transformations across distributed workers, and cleans up when complete. You simply upload your script to S3 and define job parameters - AWS handles cluster management, scaling, and infrastructure entirely behind the scenes, charging you only for actual compute time used.

Job Configuration Essentials

Data Processing Units (DPUs) are the core compute building blocks, where each DPU provides 4 vCPUs and 16 GB of RAM. You can choose from different worker types (Standard, G.1X, G.2X) and scale from 2 to 100 DPUs based on your data volume. Job bookmarks intelligently track what data has been processed to avoid reprocessing on subsequent runs - essential for incremental data pipelines. Retry mechanisms and timeout controls provide operational safety, while custom arguments let you parameterize your jobs for different environments and datasets.

Conclusion

AWS Glue transforms what once required complex cluster management, infrastructure planning, and operational overhead into a simple, declarative experience.

It is a very powerful tool to know, but it still has its limits. You need to evaluate and validate your configuration carefully.

To destroy what we have created today, simply run

terraform destroy  

Feel free to reach out if you encounter any problems or have suggestions.

Till the next time, stay safe and have fun! ❤️

We are going to enable CUR 2.0 and set up an Infrastructure via Terraform to do the following:

• Set up Amazon Athena

• Create a Glue Crawler to catalog our data

• Create a Lambda to send us monthly reports based on some SQL queries

This is a Level 200 post, following along with the post and deploying the infrastructure to your AWS account will cost approximately ~$1 per month (not including the CUR reports in S3)

If you haven’t seen my previous post about Amazon Athena, I would recommend doing so, since I will not dive deep into that section.

What is CUR, and what is CUR 2.0

CUR is AWS's Cost and Usage Reports, and this name is now deprecated. The official name of the service is Data Exports. With Data Exports, you can extract AWS accounts’/orgs’ data to S3 in a readable and queryable format. There are many flavors of exports as mentioned below:

• Cost and Usage Report 2.0 (CUR 2.0)
  Enhanced, detailed cost and usage data. Recommended over legacy CUR.

• Cost Optimization Recommendations
  Data from the Cost Optimization Hub to identify savings opportunities.

• FOCUS 1.0 with AWS Columns
  Structured cost data export aligned with the FOCUS standard.

• Carbon Emissions
  Track AWS carbon footprint data for sustainability reporting.

• Cost and Usage Dashboard (QuickSight)
  Pre-built dashboard export for cost visualization in Amazon QuickSight.

• Legacy Cost and Usage Report (Legacy CUR)
  Older CUR format, supported with different API actions.

We will focus today on the CUR 2.0 reports, but after this post, you can enable whichever data export, follow the same process, use the same infrastructure, and query your data as you did with CUR 2.0.

Cost and Usage Reports 2.0 provides the following improvements over Cost and Usage Reports (Legacy):

• Consistent schema: CUR 2.0 contains a fixed set of columns, whereas the columns included for CUR can vary monthly depending on your usage of AWS services, cost categories, and resource tags.

• Nested data: CUR 2.0 reduces data sparsity by collapsing certain columns from CUR into individual columns with key-value pairs of the collapsed columns. The nested keys can optionally be queried in Data Exports as separate columns to match the original CUR schema and data.

Enabling CUR 2.0 ~ Data Exports

Now let’s enable our CUR 2.0 in our AWS account. You can follow this guide from AWS, which shows how to do that via the console, but you can always use the CLI.

First, we need to create a bucket, make sure you put a proper bucket name here:

aws s3 mb s3://my-cur-bucket-example --region eu-central-1

Then we need to attach a bucket policy so that Data Exports can write to the bucket. You can save the following to a .json file, let’s call it bucket-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnableAWSDataExportsToWriteToS3AndCheckPolicy",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "billingreports.amazonaws.com",
                    "bcm-data-exports.amazonaws.com"
                ]
            },
            "Action": [
                "s3:PutObject",
                "s3:GetBucketPolicy"
            ],
            "Resource": [
                "arn:aws:s3:::${bucket_name}/*",
                "arn:aws:s3:::${bucket_name}"
            ],
            "Condition": {
                "StringLike": {
                    "aws:SourceAccount": "${accountId}",
                    "aws:SourceArn": [
                        "arn:aws:cur:us-east-1:${accountId}:definition/*",
                        "arn:aws:bcm-data-exports:us-east-1:${accountId}:export/*"
                    ]
                }
            }
        }
    ]
}

Replace ${bucket_name} with the name you used above and ${accountId} with your AWS account ID.

Then we want to attach this policy to our newly created bucket. Make sure you put the correct bucket name in the command.

aws s3api put-bucket-policy --bucket my-cur-bucket-example --policy file://bucket-policy.json

After we have our bucket, we can set up the data export. Follow this guide. At this point, you should have a CUR 2.0 data export, and within 24 hours, it will start generating data.

Disclaimer: There is no backfill option, meaning you will start seeing data from the day you enabled the data export.

Setting the stage - Terraform

As we can see, we will create the following resources. The Terraform files and sample queries can be found in the GitHub repo.

• 2 S3 buckets: One for Athena to save the query results and one for Athena data in case we want to CTAS. The third bucket is the one we created in the previous section.

• A Glue Crawler to catalog the CUR 2.0 data to Glue

• Amazon Athena workgroup and database for the CUR table

• And a Lambda that will be triggered monthly and send us an email report with the untagged resources. The Lambda is dockerized.

You can configure the infrastructure by editing the locals.tf file.

• Update aws_region to your preferred region

• Set name to your project's name

• Set cur_bucket_name to your CUR bucket name

• Set cur_prefix to your CUR data prefix

• Make sure table_name cur_{data}, data is the folder Glue is looking

• Set sender_email and recipient_emails for the Lambda to send the cur report

• Set the tag_key_to_analyze to the tag you want to make sure you are searching for

I will skip the creation of the S3 Buckets, AWS Glue Crawler, and Amazon Athena, since it is the same as my previous post. Feel free to follow the guide there.

Let’s first add an identity to SES to send our email reports.

resource "aws_ses_email_identity" "sender" {
  email = local.sender_email
}

We will need to verify our identity in order to send emails.

Then we will create the Lambda function and the Docker image. We use Anton’s Terraform modules to make our lives easier.

# ECR Docker image for Lambda
module "docker_image" {
  source = "terraform-aws-modules/lambda/aws//modules/docker-build"

  ecr_repo        = module.ecr.repository_name
  source_path     = "${path.module}/lambdas"

  use_image_tag = true
}

module "ecr" {
  source = "terraform-aws-modules/ecr/aws"

  repository_name         = local.ecr_repository_name
  repository_force_delete = true

  create_lifecycle_policy = false

  repository_lambda_read_access_arns = [module.lambda_function.lambda_function_arn]
}


# Lambda function using terraform-aws-modules/lambda/aws module
module "lambda_function" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.20"

  function_name = local.lambda_name
  description   = local.lambda_description

  # Docker image config
  create_package = false
  image_uri      = module.docker_image.image_uri
  package_type   = "Image"

  # Lambda settings
  timeout     = 300
  memory_size = 512

  # Environment variables
  environment_variables = {
    DATABASE_NAME      = aws_athena_database.athena_database.name
    TABLE_NAME         = local.table_name
    WORKGROUP          = aws_athena_workgroup.athena_workgroup.name
    OUTPUT_BUCKET      = module.athena_results_bucket.s3_bucket_id
    SENDER_EMAIL       = local.sender_email
    RECIPIENT_EMAILS   = local.recipient_emails
    TAG_KEY_TO_ANALYZE = local.tag_key_to_analyze
  }

  # IAM policy statements
  attach_policy_statements = true
  policy_statements = {
    cloudwatch_logs = {
      effect = "Allow",
      actions = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      resources = ["arn:aws:logs:*:*:*"]
    },
    athena = {
      effect = "Allow",
      actions = [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults"
      ],
      resources = ["*"]
    },
    glue = {
      effect = "Allow",
      actions = [
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetTable",
        "glue:GetTables",
        "glue:GetPartition",
        "glue:GetPartitions",
        "glue:BatchGetPartition"
      ],
      resources = ["*"]
    },
    s3 = {
      effect = "Allow",
      actions = [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:PutObject"
      ],
      resources = [
        "arn:aws:s3:::${module.athena_results_bucket.s3_bucket_id}",
        "arn:aws:s3:::${module.athena_results_bucket.s3_bucket_id}/*"
      ]
    },
     s3_cur = {
      effect = "Allow",
      actions = [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      resources = [
        "arn:aws:s3:::${local.cur_bucket_name}",
        "arn:aws:s3:::${local.cur_bucket_name}/*"
      ]
    },
    ses = {
      effect = "Allow",
      actions = [
        "ses:SendEmail",
        "ses:SendRawEmail"
      ],
      resources = ["*"]
    }
  }

  tags = local.tags
}

In the code above, we first set up the Docker build and push it to ECR. Then we create the Lambda function alongside all the permissions it needs to execute its job.

Last but not least, we introduce the scheduling part of our architecture by utilizing a CloudWatch event rule.

# CloudWatch Event Rule to trigger Lambda on a schedule (monthly)
resource "aws_cloudwatch_event_rule" "monthly_trigger" {
  name                = "lambda-monthly-trigger"
  description         = "Triggers the untagged resources reporter Lambda function on the 3rd day of each month"
  schedule_expression = "cron(0 8 3 * ? *)" # 8:00 AM UTC on the 3rd day of each month

  tags = local.tags
}

# CloudWatch Event Target
resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.monthly_trigger.name
  target_id = "TriggerLambda"
  arn       = module.lambda_function.lambda_function_arn
}

# Lambda permission to allow CloudWatch Events to invoke the function
resource "aws_lambda_permission" "allow_cloudwatch" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = module.lambda_function.lambda_function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.monthly_trigger.arn
}

In the code above, we create a CloudWatch event rule triggered at 8:00 AM UTC every 3rd day of the month. 3rd day because we want to make sure all the data points from CUR are available from the previous month, usually it takes 24 hours. Then, we create the target and the permissions to CloudWatch to trigger the Lambda function.

And that’s it. You are now ready to deploy the infrastructure.

terraform plan

and once we validate the plan, we can run

terraform apply

If we are impatient and want to see the work we’ve done, we can trigger the Lambda function with a test and see the results.

As you can see, I vibe-coded the sh*t out of it for the report visuals 😅😅😅Who likes to write HTML in python for emails? Let me know in the comments below!

You can find the Lambda code inside the lambdas folder.

You can modify the Lambda function to do whatever you like, you now have all of the CUR data at your fingertips 🤤

Useful queries

In this section, I will showcase some Athena queries on the CUR 2.0 data that I found helpful. The complete collection of the CUR queries can be found here.

You will notice these lines in some queries I show you. Pretty much this excludes the credits to show some data 😅 If you want the credits to be applied in your queries, remove those lines.

line_item_line_item_type != 'Credit' AND
line_item_line_item_type != 'Refund'

S3 Costs by bucket and usage type

SELECT
    line_item_resource_id AS bucket_name,
    line_item_usage_type AS usage_type,
    SUM(line_item_unblended_cost) AS cost
FROM cur_data
WHERE
    line_item_product_code = 'AmazonS3' AND
    line_item_resource_id <> '' AND
    bill_billing_period_start_date = DATE '2025-05-01' AND
    line_item_line_item_type != 'Credit' AND
    line_item_line_item_type != 'Refund'
GROUP BY 1, 2
ORDER BY 3 DESC
LIMIT 20;

Saving from Reserved Instances

SELECT
    bill_billing_period_start_date AS billing_period,
    reservation_reservation_a_r_n AS reservation_arn,
    SUM(reservation_effective_cost) AS effective_cost,
    SUM(reservation_unused_amortized_upfront_fee_for_billing_period) AS unused_upfront_fee,
    SUM(reservation_unused_recurring_fee) AS unused_recurring_fee
FROM cur_data
WHERE
    reservation_reservation_a_r_n <> '' AND
    bill_billing_period_start_date = DATE '2025-05-01'
GROUP BY 1, 2
ORDER BY 3 DESC;

Identify untagged resources

SELECT
    line_item_product_code AS service,
    line_item_resource_id AS resource_id,
    product_region_code AS region,
    SUM(line_item_unblended_cost) AS cost
FROM cur_data
WHERE
    CARDINALITY(MAP_KEYS(resource_tags)) = 0 AND
    line_item_resource_id <> '' AND
    line_item_line_item_type = 'Usage' AND
    bill_billing_period_start_date = DATE '2025-05-01'
GROUP BY 1, 2, 3
HAVING SUM(line_item_unblended_cost) > 0
ORDER BY 4 DESC
LIMIT 50;

Cost trend current vs previous month

SELECT
    '2025-05' AS current_month,
    '2025-04' AS previous_month,
    SUM(CASE WHEN bill_billing_period_start_date = DATE '2025-05-01' AND line_item_line_item_type NOT IN ('Credit', 'Refund') THEN line_item_unblended_cost ELSE 0 END) AS may_cost,
    SUM(CASE WHEN bill_billing_period_start_date = DATE '2025-04-01' AND line_item_line_item_type NOT IN ('Credit', 'Refund') THEN line_item_unblended_cost ELSE 0 END) AS april_cost,
    SUM(CASE WHEN bill_billing_period_start_date = DATE '2025-05-01' AND line_item_line_item_type NOT IN ('Credit', 'Refund') THEN line_item_unblended_cost ELSE 0 END) - 
    SUM(CASE WHEN bill_billing_period_start_date = DATE '2025-04-01' AND line_item_line_item_type NOT IN ('Credit', 'Refund') THEN line_item_unblended_cost ELSE 0 END) AS cost_difference
FROM cur_data
WHERE
    bill_billing_period_start_date IN (DATE '2025-05-01', DATE '2025-04-01');

Identify resources with the highest cost increase

Weekly analysis for the last 2 weeks

WITH current_week_costs AS (
    SELECT
        line_item_resource_id AS resource_id,
        SUM(line_item_unblended_cost) AS cost
    FROM cur_data
    WHERE
        line_item_usage_start_date BETWEEN DATE('2025-05-12') AND DATE('2025-05-18') AND
        line_item_resource_id <> ''
    GROUP BY 1
),
prev_week_costs AS (
    SELECT
        line_item_resource_id AS resource_id,
        SUM(line_item_unblended_cost) AS cost
    FROM cur_data
    WHERE
        line_item_usage_start_date BETWEEN DATE('2025-05-05') AND DATE('2025-05-11') AND
        line_item_resource_id <> ''
    GROUP BY 1
),
resource_details AS (
    SELECT DISTINCT
        line_item_resource_id AS resource_id,
        line_item_product_code AS service,
        product_region_code AS region
    FROM cur_data
    WHERE
        line_item_resource_id <> '' AND
        line_item_usage_start_date BETWEEN DATE('2025-05-05') AND DATE('2025-05-18')
)
SELECT
    r.resource_id,
    r.service,
    r.region,
    -- Week costs
    COALESCE(p.cost, 0) AS prev_week_cost,
    COALESCE(c.cost, 0) AS current_week_cost,
    COALESCE(c.cost, 0) - COALESCE(p.cost, 0) AS week_over_week_change,
    
    -- Percentage change
    CASE 
        WHEN COALESCE(p.cost, 0) > 0 
        THEN ROUND(((COALESCE(c.cost, 0) - COALESCE(p.cost, 0)) / COALESCE(p.cost, 0)) * 100, 2)
        WHEN COALESCE(p.cost, 0) = 0 AND COALESCE(c.cost, 0) > 0
        THEN NULL
        ELSE 0
    END AS percentage_change,
    
    -- Growth classification
    CASE
        WHEN COALESCE(p.cost, 0) = 0 AND COALESCE(c.cost, 0) > 0 
        THEN 'New Resource'
        WHEN COALESCE(c.cost, 0) - COALESCE(p.cost, 0) > 0 
        THEN 'Increasing'
        WHEN COALESCE(c.cost, 0) - COALESCE(p.cost, 0) < 0 
        THEN 'Decreasing'
        ELSE 'Stable'
    END AS cost_trend
FROM resource_details r
LEFT JOIN current_week_costs c ON r.resource_id = c.resource_id
LEFT JOIN prev_week_costs p ON r.resource_id = p.resource_id
WHERE 
    -- Show resources with costs in either week
    COALESCE(c.cost, 0) > 0 OR COALESCE(p.cost, 0) > 0
ORDER BY week_over_week_change DESC
LIMIT 20;

Athena usage and cost by workgroup and operation

SELECT
    CAST(line_item_usage_start_date AS DATE) AS usage_date,
    line_item_operation AS operation,
    line_item_resource_id AS workgroup,
    SUM(line_item_usage_amount) AS data_scanned_bytes,
    SUM(line_item_unblended_cost) AS cost
FROM cur_data
WHERE
    line_item_product_code = 'AmazonAthena' AND
    bill_billing_period_start_date = DATE '2025-05-01'
GROUP BY 1, 2, 3
ORDER BY 1 DESC, 5 DESC;

List all tags with their associated cost

WITH flattened_tags AS (
  SELECT
    line_item_resource_id,
    line_item_unblended_cost,
    k AS tag_key,
    resource_tags[k] AS tag_value
  FROM cur_data
  CROSS JOIN UNNEST(MAP_KEYS(resource_tags)) AS t(k)
  WHERE
    bill_billing_period_start_date = DATE '2025-05-01'
    line_item_resource_id <> '' AND
    resource_tags[k] <> ''
)
SELECT
  tag_key,
  tag_value,
  COUNT(DISTINCT line_item_resource_id) AS resource_count,
  SUM(line_item_unblended_cost) AS total_cost
FROM flattened_tags
GROUP BY 1, 2
ORDER BY 1, 4 DESC;

Tagged vs Untagged resources and their cost

This is one of the queries we used in the Lambda, edit the tag_key_to_search and then you can see how many resources are not tagged with that particular tag and how many they are tagged properly.

WITH tag_key_to_search AS (
    SELECT 'user_creator' AS key -- Change this to your desired tag key
),
resource_tagging AS (
    SELECT
        line_item_product_code AS service,
        line_item_resource_id,
        line_item_unblended_cost,
        CASE 
            WHEN CARDINALITY(MAP_KEYS(resource_tags)) > 0 THEN 'Tagged'
            ELSE 'Untagged'
        END AS general_tagging_status,
        CASE
            WHEN resource_tags[(SELECT key FROM tag_key_to_search)] IS NOT NULL 
                 AND resource_tags[(SELECT key FROM tag_key_to_search)] <> '' THEN 'Has Tag Key'
            ELSE 'Missing Tag Key'
        END AS specific_tag_status
    FROM cur_data
    WHERE
        line_item_resource_id <> '' AND
        bill_billing_period_start_date = DATE '2025-05-01' AND
        line_item_line_item_type != 'Credit' AND
        line_item_line_item_type != 'Refund' AND
        line_item_line_item_type = 'Usage'
)
SELECT
    service,
    general_tagging_status,
    specific_tag_status,
    COUNT(DISTINCT line_item_resource_id) AS resource_count,
    SUM(line_item_unblended_cost) AS total_cost
FROM resource_tagging
GROUP BY 1, 2, 3
HAVING COUNT(DISTINCT line_item_resource_id) > 0
ORDER BY 1, 2, 3;

The other query used for the Lambda, listing the untagged services, is query 15 in the sample_queries.sql file.

To generate some of the queries, I’ve used an LLM, and it seems that, providing the right context (and some editing after the proposed query), you can pretty much effortlessly create queries on top of the CUR 2.0. It is much nicer than searching 300+ (legacy cur) columns and their definition to start using the data.

Disclaimer: While vibe-coding is trendy and can bootstrap your code, please do not forget to take a closer look at your query and understand that the fields being queried are correct.

In my queries, I’ve used the unblinded cost, which is the raw, undiscounted cost.

CUR 2.0 dictionary: https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2.html

Here is a very nice cheat sheet to help you better understand the AWS Cost Dataset.

Conclusion

Well, that’s all, folks. I hope you found this post helpful and that it leaves you with a bit more knowledge and confidence when navigating your AWS cost reports.

It will be a shame not to mention some already existing automation/dashboard/etc, a considerable effort from the AWS team to streamline and trivialize FinOps (to an extent of course.)

Here you can find the Cloud Intelligence dashboards.

Here you can find a cost optimization workshop.

Feel free to reach out if you encounter any problems or have suggestions.

Till the next time, stay safe and have fun! ❤️

Whether you're a data engineer looking to implement a new analytics solution or a developer seeking to optimize existing Athena workloads, in this post, I will provide insights and examples to help you leverage Athena's full potential.

I hope that after you read this, you have a better understanding on how Athena works. So without further ado, let’s dive in!

If you enjoy this content, take a moment to subscribe; it means a lot to me.

Subscribe now

We will not discuss Iceberg in this blog post because I want to cover this topic in detail in a future post about S3 Tables.

What is Amazon Athena

Amazon Athena is an interactive query service that simplifies data analysis in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to set up or manage, and you only pay for the resources your query needs to run.

The above definition is from AWS; believe me, no additional words describe Athena. You put data, catalog data, and query data, and that is it, it is completely serverless (There is a provisioned flavor, but we are not going to dive into this today) without the hassle of spawning up resources, maintaining the cluster, etc.

Athena shines in the following use cases:

• Log analysis and monitoring

• Cost and Usage Analysis

• Data Lake analytics

  • Ad-hoc analysis on a massive amount of data

  • Data processing and cleaning

• Security and compliance

  • Security logs

  • Audit trails

• ML Data Preparation

Athena’s pricing is very straightforward, 5$ per TB of scan, meaning you pay not for compute but based on the amount of data you scanned to generate the required result. You can always use provisioned capacity, but it usually makes sense if you have a large data lake that constantly scans TBs of data. Minimum commitment is 24 DPUs, and a 24/7 usage will cost around 5,000$ per month, so in that case, if you scan more than 1.2PBs per month, provisioned capacity makes sense.

What you’ll learn

This post will show a simple use case on loading, cataloging, and querying data.

We are going to take a look at the following:

• Basics of setting up Amazon Athena

• Configure Athena properly

• Create a Glue crawler for cataloging our Data

• Deploying our IaC in Terraform

This is a Level 200 difficulty, meaning you will need to know some basic concepts of AWS and Terraform

You will need the following:

• An AWS Account and access to credentials of your AWS (either a service account or configure SSO)

• Terraform installed

The whole project will cost approximately: 2-3$

Terraform into Play

For the infrastructure setup, we will create the following resources:

• 2 S3 buckets, one for Athena to save the results and one for sample data

• An Athena workgroup and a database where we will run our queries

• A Glue crawler to catalog our sample data

• IAM policies to provide access

You can find the whole code for this post here, where you can edit the locals.tf file to configure it your way and follow along. So let’s begin, I will showcase some Terraform and explain its usage to make it easier to follow.

First, we need to take a look at the locals.tf file and adjust line 6 and line 17:

• name: add a name for your project, which will be propagated in the resource names

• sample_data_prefix: is the “folder” where we will put the data in S3

Setting up the buckets (main.tf).

module "athena_results_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 4.8"

  bucket = "${local.name}-query-results"
  force_destroy = true
  acl = "private"

  # Add ownership controls
  control_object_ownership = true
  object_ownership         = "ObjectWriter"

  tags = local.tags
}

# Create a bucket for sample data
module "athena_data_bucket" {
  source  = "terraform-aws-modules/s3-bucket/aws"
  version = "~> 4.8"

  bucket = "${local.name}-data"
  force_destroy = true
  acl = "private"

  # Add ownership controls
  control_object_ownership = true
  object_ownership         = "ObjectWriter"

  tags = local.tags
}

In the code above, we are creating two buckets:

• query-results: Athena needs a bucket to save the generated results in a CSV format and serve them to the user. In this bucket, usually we have a lifecycle policy where we discard the contents after an X number of days.

• data: Athena also needs a bucket where the data is saved and ready for querying. Additionally, Athena can use this bucket to generate data with CTAS statements.

Next, we will create an Athena workgroup and a database to save the data (athena.tf).

resource "aws_athena_workgroup" "athena_workgroup" {
  name        = local.name
  description = "Athena workgroup for ${local.name}"

  configuration {
    enforce_workgroup_configuration    = local.athena_workgroup.enforce_workgroup_configuration
    publish_cloudwatch_metrics_enabled = local.athena_workgroup.publish_cloudwatch_metrics_enabled
    bytes_scanned_cutoff_per_query     = local.athena_workgroup.bytes_scanned_cutoff_per_query

    engine_version {
      selected_engine_version = local.athena_workgroup.engine_version
    }

    result_configuration {
      output_location = "s3://${module.athena_results_bucket.s3_bucket_id}/"

      encryption_configuration {
        encryption_option = local.athena_workgroup.encryption_option
      }
    }
  }

  tags = local.tags
}

resource "aws_athena_database" "athena_database" {
  name   = replace(local.name, "-", "_")
  bucket = module.athena_data_bucket.s3_bucket_id
  force_destroy = true

  properties = {
    location = "s3://${module.athena_data_bucket.s3_bucket_id}/tables/"
  }
} 

You can use Athena workgroups to separate workloads, control team access, enforce configuration, track query metrics, and control costs.

As for the database, we have selected the bucket as our data bucket, but this is primarily for the database metadata. We also defined a location where the data (CTAS) will be saved. We are using force_destroy in case we want to delete the database, avoid this in production workloads.

It is time to create the Glue Crawler to catalog our new data (crawler.tf).

# Create Glue crawler for CUR data
resource "aws_glue_crawler" "demo_crawler" {
  name          = "${local.name}-demo-crawler"
  database_name = aws_athena_database.athena_database.name
  role          = aws_iam_role.glue_crawler.arn
  table_prefix  = "demo_"  # This will prefix all tables created by this crawler

  s3_target {
    path = "s3://${module.athena_data_bucket.s3_bucket_id}/${local.sample_data_prefix}"
  }

  schema_change_policy {
    delete_behavior = "LOG"
    update_behavior = "UPDATE_IN_DATABASE"
  }

  configuration = jsonencode({
    Version = 1.0
    CrawlerOutput = {
      Partitions = { AddOrUpdateBehavior = "InheritFromTable" }
      Tables     = { AddOrUpdateBehavior = "MergeNewColumns" }
    }
  })

  tags = local.tags
} 

The above Terraform code is pretty standard for a Glue Crawler. The most important parts here are:

• s3_target: where we are pointing to the data we have uploaded (not done that yet)

• schema_change_policy: where we provide the behaviour of the catalog when the schema changes, here we say to update the database, but also keep the existing schema on the catalog

Last but not least, we have the IAM policies and roles (iam.tf). There is not much to say here.

Once we are ready, we can proceed with

terraform plan

and once we validate the plan, we can run

terraform apply

And voila! You have a working environment to start putting data and experimenting!

Using Athena

Now that everything is ready, we can populate our S3 bucket with some data. Our choice will be parquet format, and we will upload the iconic Titanic Dataset, which you can find in the repo.

So, navigate to your S3 bucket and upload the dataset under the prefix (data) you have selected above while creating the Glue Crawler. I will use a CLI command, but you are more than welcome to use the console.

aws s3 cp titanic.parquet s3://<bucket_name>/data/titanic.parquet

Then we go to our newly generated crawler and run it to catalog our file. Again, I will use a CLI command.

aws glue start-crawler --name <crawler-name>

This will create a table within our database in Amazon Athena.

We can now run a simple query on our Athena Interface. Use your database name in your case.

SELECT * FROM "thelastdev_athena_demo"."demo_data" limit 10;

And, believe it or not, this is pretty much it…. This is how Athena works. You can expand your knowledge with the AWS Documentation regarding Athena or by asking Q Developer for more complex implementations.

Closing this post, I will share my notes regarding optimizing Athena.

Athena Optimizations

When it comes to Optimizations, we are going to talk about:

• Cost Optimizations

• Performance Optimizations

So, let’s begin and see the low-hanging fruits:

Partitioning strategies

By properly partitioning your data, you will significantly benefit both in cost and performance optimization. If you utilize a partition in your query (where statement on that field), Athena will only scan the data existing in this partition.

For example, (let’s exaggerate) let’s say you have a bucket with log data going back 3 years, totaling 970 TBs, and the bucket is split into partitions by week. If you search for a specific week or a collection of weeks, Athena will only scan the data and not the 970 TBs to find your results.

To utilize partitioning, you can split your data into prefixes (“folders”) within your bucket

# with one partition
data/week=1/test.parquet
data/week=2/test2.parquet

or

# with multiple partitions
data/year=2025/week=1/day=1/test.partquet

File format selection (Parquet, ORC)

Utilizing a proper format for your data is very very crucial. I personally prefer parquet due to its columnar representation, which allows me to be optimal in select statements (and not *). The difference between CSV files and Parquet regarding performance is significant! Additionally, especially in the CSV format, Athena will open the entire file, while in Parquet, it will be more cost optimized due to its columnar format.

Compression techniques

Because Athena charges you by the amount of TBs you scan, compressing your data will result in lower costs, but not necessarily better performance. My favourite is snappy, fast, and lightweight.

Result caching

You can reuse query results in Athena. When you enable result reuse for a query, Athena looks for a previous query execution within the same workgroup. If Athena finds corresponding stored query results, it does not rerun the query, but points to the previous result location or fetches data from it.

Now, let’s move to more complex optimizations and focus on the 20% of the 80/20 rule. Trying to reach almost optimal performance and cost, but with more effort than the low-hanging fruits we mentioned before.

Query planning - Explain

You can analyze your query execution and discover how Athena scans your data for further optimization.

Workgroup configurations

Cost control settings: You can limit the amount of data that can be scanned in a single query

Performance settings: Use engine version “Athena version 3”, which is currently the latest, but make sure you check the benefits when a new engine comes out.

Publish metrics in CloudWatch to properly monitor Athena usage.

You can see Athena in action with read data in my AWS CUR processing post!

Conclusion

And this is it, this is Amazon Athena. A very powerful and cheap database where you can query your structured and semi-structured data. As a next step, I will go through the S3 Tables to see Athena in action with Iceberg, and move on to the remaining data stack of AWS, like AWS Glue Jobs, Lake Formation, Redshift, and many more.

Feel free to reach out if you encounter any problems or have suggestions.

Till the next time, stay safe and have fun!

If you enjoy this content, take a moment to subscribe; it means a lot to me.

Subscribe now

News

There have been two exciting announcements about Amazon Q Developer in the past week. It seems that AWS properly focuses on LLMs and is starting to catch up with the competition. First of all, Amazon Q Developer CLI now supports MCP, allowing you to have a customized response regarding code generation. Additionally, Amazon Q Developer offers an agentic experience in IDEs. The new coding experience provides intelligent task execution, enabling Q Developer to perform actions beyond code suggestions, such as modifying files, generating code diffs, and running commands based on your natural language instructions. Available in VS Code and JetBrains.

Last but not least, a massive quality of life upgrade in my opinion, is that EC2 Image Builder now integrates with SSM Parameter Store. This streamlines the way you build your custom images and their maintenance. This new feature comes at no additional cost.

Events

20th of May: The Mondelez International Cloud Engineering team will speak at the AWS User Group in Athens. FinOps in Practice: Managing AWS Costs at Scale

7-9 May: Panathenea event. You can get tickets from here. The agenda can be found here.

AWS Community Day Adria: Registration has opened. This event will be on the 5th of September, and it will have excellent speakers! Jeff Barr will be doing the keynote. There is also a call for speakers.

Interesting Posts

The Cloud Engineers

Cloud Migration Strategies: Choosing the Right Path for Your Organization

Moving to the cloud is no longer a question of "if" but "when" and "how." As organizations continue to embrace digital transformation, understanding different cloud migration strategies is crucial for a successful transition. In this article, we'll explore the "6 R's" of cloud migration and help you determine which strategy might work best for your orga…

Read more

a year ago · 1 like · 2 comments · Lefteris Karageorgiou

is exploring the 6 R’s and showcasing the different approaches to migrating your applications to the cloud. This is something that anyone who aspires to migrate workloads to the cloud (any cloud) should know.

DataConscious – A mindful approach to analytics

Delta Lake vs. Apache Iceberg: A Mindful Choice for Your Data Lakehouse

Thanks for reading DataConscious – A mindful approach to analytics! Subscribe for free to receive new posts and support my work…

Read more

a year ago · 1 like · Antonios Angelakis

is comparing Delta Lake and Apache Iceberg. Understanding how these two data formats work to build a Data Lake is good. My favourite is Apache Iceberg, which makes it very easy to manage with Amazon S3 Tables.

Last but not last, this is my first official blog post after a looooong time. I hope you like it!

A sneak peek of my next posts:

• Managing data at scale with a fraction of the cost

• Optimizing your data format

• Reduce operational burden for managing Apache Iceberg

Till the next time, stay safe and have fun!

The purpose of this blog post is to showcase some capabilities of AWS ECS and Terraform modules, and by doing so, why not end up with a functional Vanilla Minecraft server way overpriced, where we can also play 😁

Disclaimer: While coming from an FPS background as a kid, I chose Minecraft for two reasons: first, it is one of the greatest games, and second, I play with my daughter.

What You’ll Learn

We are going to take a look at the following:

• Basics of ECS (Fargate, Task Definition, Services)

• Configuring our container in ECS

• Utilizing Terraform modules1 (not resources) by Anton Babenko

• Deploying our IaC in Terraform

This is a Level 200 difficulty, meaning you will need to know some basic concepts of AWS, Docker, and VPCs

You will need the following:

• An AWS Account and access to credentials of your AWS (either a service account or configure SSO)

• Terraform installed

Setting the Stage

This is the architecture we are going to implement

As shown above, we have created a VPC with both public and private subnets. We have placed our Network Load Balancer in the public subnets, and in the private subnet, we have our Minecraft server and the EFS storage (a type of persistent storage).

This will cost approximately 65$ per month! 😨😨Obviously, we can reduce the cost to 25$ per month, but we are not trying to make a cheap Minecraft server, we are trying to learn ECS and how it works!

The full code for this project is in this GitHub Repo. I have parameterized the infrastructure and added some locals in the file locals.tf.

Terraform into play

DISCLAIMER: I will explain the different modules I have used; however, for a complete example, please visit the GitHub repository. Files are missing, like

• locals.tf

• data.tf

• etc

We first start by creating a VPC for our server. We are using this module.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.19.0"

  name = "minecraft-vpc"
  cidr = local.cidr

  azs             = local.azs
  private_subnets = local.private_subnets
  public_subnets  = local.public_subnets

  # Enable NAT Gateway for private subnets
  enable_nat_gateway = true
  single_nat_gateway = true # Use a single NAT Gateway to save costs

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

With the code above, we create a VPC with two private and two public subnets spanning two availability zones. Additionally, we enable the NAT gateway because we are going to place our container in the private subnet, and it will need access to the internet in order to download the Minecraft Docker image.

Once we have the VPC in place, we will need to add the EFS (Elastic File Storage) for persistent storage. We will use this module.

module "efs" {
  source  = "terraform-aws-modules/efs/aws"
  version = "~> 1.8"

  # File system
  name           = "minecraft-volume"
  creation_token = "minecraft-volume"
  encrypted      = true
  kms_key_arn    = module.kms.key_arn

  # File system policy
  attach_policy                      = true
  bypass_policy_lockout_safety_check = false

  # policy statements will be added after the ECS Service
  # <THIS IS WHERE WE WILL PUT LATER THE STATEMENT>
  
  # Mount targets / security group
  mount_targets              = { for k, v in zipmap(module.vpc.azs, module.vpc.private_subnets) : k => { subnet_id = v } }
  security_group_description = "EFS security group for minecraft server"
  security_group_vpc_id      = module.vpc.vpc_id

  security_group_rules = {
    vpc = {
      # relying on the defaults provided for EFS/NFS (2049/TCP + ingress)
      description = "NFS ingress from VPC private subnets"
      cidr_blocks = module.vpc.private_subnets_cidr_blocks
    }
  }

  access_points = {
    vanilla_minecraft = {
      posix_user = {
        gid = 1000
        uid = 1000
      }
      root_directory = {
        path = "/vanilla"
        creation_info = {
          owner_gid   = 1000
          owner_uid   = 1000
          permissions = "755"
        }
      }
    }
  }

  # Backup policy
  enable_backup_policy = false
  # Replication configuration
  create_replication_configuration = false

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}


module "kms" {
  source  = "terraform-aws-modules/kms/aws"
  version = "~> 1.0"

  aliases               = ["efs/minecraft-volume"]
  description           = "EFS customer managed key"
  enable_default_policy = true
}

In the code above, we are creating our EFS. First of all, we are encrypting our storage, not that we need to for a personal Minecraft server, but again, we are here to learn. We have omitted the policy statements for now, as the ECS task has not been created yet. We create the mount targets in our private subnets to allow us to attach EFS to resources existing in the private subnets, then we create the security group rules. Lastly, we create the access points for the folder where Minecraft will store its data. You do not necessarily need the access point, but by doing so, you can create multiple servers within the same cluster, all of which use the same EFS filesystem (spoilers).

Now that we have our EFS, we will need to create the Network Load Balancer to forward the traffic to our ECS. We are using this module.

 module "nlb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 9.16"

  name = "minecraft-nlb"

  load_balancer_type = "network"

  vpc_id                     = module.vpc.vpc_id
  subnets                    = module.vpc.public_subnets
  enable_deletion_protection = false
  create_security_group      = false
  security_groups            = [aws_security_group.nlb.id]

  listeners = {
    minecraft = {
      port     = local.container_port
      protocol = "TCP"
      forward = {
        target_group_key = "minecraft-vanilla"
      }
    }
  }

  target_groups = {
    minecraft-vanilla = {
      name              = "minecraft-vanilla"
      protocol          = "TCP"
      port              = local.container_port
      target_type       = "ip"
      create_attachment = false
      health_check = {
        enabled             = true
        interval            = 30
        healthy_threshold   = 3
        unhealthy_threshold = 3
        protocol            = "TCP"
        timeout             = 10
      }
    }
  }


  tags = {
    Environment = "Development"
    Project     = "Example"
  }
}

resource "aws_security_group" "nlb" {
  name        = "minecraft-nlb-sg"
  description = "Security group for Minecraft NLB"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port   = local.container_port
    to_port     = local.container_port
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Minecraft server port"
  }

  egress {
    from_port   = local.container_port
    to_port     = local.container_port
    protocol    = "tcp"
    description = "Minecraft port to VPC"
    cidr_blocks = [module.vpc.vpc_cidr_block]
  }

  tags = {
    Name        = "minecraft-nlb-sg"
    Environment = "Development"
    Project     = "Example"
  }
} 

This is a very straightforward approach. We create a network load balancer in the public subnets and attach a security group that allows all connections from anywhere to the Minecraft server port. We then forward these connections to resources only within our VPC. Then, we create the listener and a target group. This target group will be connected to the ECS module.

Now, the final part, let’s create the ECS Cluster with our Minecraft server! 💪We are going to use this module.

module "ecs" {
  source  = "terraform-aws-modules/ecs/aws"
  version = "~> 5.12"

  cluster_name = "minecraft-servers"

  cluster_configuration = {
    execute_command_configuration = {
      logging = "OVERRIDE"
      log_configuration = {
        cloud_watch_log_group_name = "/aws/ecs/minecraft-servers"
      }
    }
  }

  fargate_capacity_providers = {
    FARGATE_SPOT = {
      default_capacity_provider_strategy = {
        weight = 100
      }
    }
  }

  services = {
    minecraft-vanilla = {
      cpu    = 4096
      memory = 8192

      volume = [
        {
          name = "minecraft-storage"

          efs_volume_configuration = {
            file_system_id     = module.efs.id
            transit_encryption = "ENABLED"
            root_directory     = "/"
            authorization_config = {
              iam             = "ENABLED"
              access_point_id = module.efs.access_points["vanilla_minecraft"].id
            }
          }
        }
      ]

      # Move to private subnets and remove public IP
      assign_public_ip = false
      subnet_ids       = module.vpc.private_subnets

      # Configure load balancer
      load_balancer = {
        service = {
          target_group_arn = module.nlb.target_groups["minecraft-vanilla"].arn
          container_name   = "minecraft-vanilla-task"
          container_port   = local.container_port
        }
      }

      # Use the dedicated security group
      security_group_ids = [aws_security_group.ecs_service.id]

      # add access to EFS and KMS in the task role
      tasks_iam_role_policies = {
        efs_access = aws_iam_policy.efs_access_policy.arn
        kms_access = aws_iam_policy.efs_kms_access_policy.arn
        ssm_access = aws_iam_policy.ssm_session_manager_policy.arn
      }

      # Container definition(s)
      container_definitions = {

        minecraft-vanilla-task = {
          cpu    = 4096
          memory = 8192
          image  = "itzg/minecraft-server"

          port_mappings = [
            {
              name          = "minecraft-vanilla-container"
              containerPort = local.container_port
              hostPort      = local.container_port
              protocol      = "tcp"
            }
          ]

          environment = [
            {
              name  = "EULA"
              value = "TRUE"
            },
            {
              name  = "WHITELIST"
              value = local.whitelist_list
            },
            {
              name  = "DIFFICULTY"
              value = local.difficulty
            }
          ]

          mount_points = [
            {
              sourceVolume  = "minecraft-storage"
              containerPath = "/data"
              readOnly      = false
            }
          ]

          # Example image used requires access to write to root filesystem
          readonly_root_filesystem = false
          memory_reservation       = 100

          # Enable SSM Session Manager
          enable_execute_command = true
        }
      }

    }
  }


  # Create task execution role and attach policies for EFS  create_task_exec_iam_role = true
  create_task_exec_iam_role = true
  task_exec_iam_role_name   = "minecraft-exec-role"
  task_exec_iam_role_policies = {
    efs_access = aws_iam_policy.efs_access_policy.arn
    kms_access = aws_iam_policy.efs_kms_access_policy.arn
  }

  tags = {
    Environment = "Development"
    Project     = "Example"
  }
}

# Create a dedicated security group for the ECS service
resource "aws_security_group" "ecs_service" {
  name        = "minecraft-ecs-service-sg"
  description = "Security group for Minecraft ECS service"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port       = local.container_port
    to_port         = local.container_port
    protocol        = "tcp"
    description     = "Minecraft port from NLB"
    security_groups = [aws_security_group.nlb.id]
  }

  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    description     = "NFS Port"
    security_groups = [module.efs.security_group_id]
  }

  egress {
    from_port       = local.container_port
    to_port         = local.container_port
    protocol        = "tcp"
    description     = "Minecraft port to NLB"
    security_groups = [aws_security_group.nlb.id]
  }

  egress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    description     = "NFS Port"
    security_groups = [module.efs.security_group_id]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    description = "SSM Session Manager"
    cidr_blocks = ["0.0.0.0/0"] # SSM endpoints are AWS managed
  }

  tags = {
    Name        = "minecraft-ecs-service-sg"
    Environment = "Development"
    Project     = "Example"
  }
}

The above code may seem intimidating, but once we go through it, it will become clearer. Let’s dive in.

First, we define the logs and the capacity providers for our cluster. We select FARGATE_SPOT to save a few bucks and because we can afford some outage. If you run business-critical applications that are not self-healing and resumable, use FARGATE.

Then, we define our service where our Minecraft docker instance will run. We set the CPU and RAM available for this service, and we define the attachment to our EFS. Now, once the service is deployed, it will fail. We will need to add the policy to our EFS (remember the comment on the EFS module). "

volume = [
        {
          name = "minecraft-storage"

          efs_volume_configuration = {
            file_system_id     = module.efs.id
            transit_encryption = "ENABLED"
            root_directory     = "/"
            authorization_config = {
              iam             = "ENABLED"
              access_point_id = module.efs.access_points["vanilla_minecraft"].id
            }
          }
        }
      ]

We disable public access to the service, which is why we have the NLB, and we also place everything that will be spawned within the service in private subnets.

assign_public_ip = false
subnet_ids       = module.vpc.private_subnets

We attach the load balancer to our service and ensure that we reference the target group and the container to which we will forward the traffic.

load_balancer = {
  service = {
    target_group_arn = module.nlb.target_groups["minecraft-vanilla"].arn
    container_name   = "minecraft-vanilla-task"
    container_port   = local.container_port
  }
}

Lastly, for the service, we will define the policies and security groups. You can find the IAM roles here in the iam.tf file. The security group is mentioned at the end.

# Use the dedicated security group
security_group_ids = [aws_security_group.ecs_service.id]

# add access to EFS and KMS in the task role
tasks_iam_role_policies = {
  efs_access = aws_iam_policy.efs_access_policy.arn
  kms_access = aws_iam_policy.efs_kms_access_policy.arn
  ssm_access = aws_iam_policy.ssm_session_manager_policy.arn
}

Now let’s move to the container definitions. We create our container with the name minecraft-vanilla-task, set up the port mappings, image, and CPU and RAM

minecraft-vanilla-task = {
  cpu    = 4096
  memory = 8192
  image  = "itzg/minecraft-server"

  port_mappings = [
    {
      name          = "minecraft-vanilla-container"
      containerPort = local.container_port
      hostPort      = local.container_port
      protocol      = "tcp"
    }
  ]
...

Then, the environment variables, mount points to our EFS, and we also enable the execute command, in case we get stuck in a cave mining nearly dead 😉

...
environment = [
  {
    name  = "EULA"
    value = "TRUE"
  },
  {
    name  = "WHITELIST"
    value = local.whitelist_list
  },
  {
    name  = "DIFFICULTY"
    value = local.difficulty
  }
]

mount_points = [
  {
    sourceVolume  = "minecraft-storage"
    containerPath = "/data"
    readOnly      = false
  }
]

# Example image used requires access to write to root filesystem
readonly_root_filesystem = false
memory_reservation       = 100

# Enable SSM Session Manager
enable_execute_command = true

Now that we are done with the container definitions, we set up the task execution role, so our container will have access to the EFS and the encryption key.

 create_task_exec_iam_role = true
  task_exec_iam_role_name   = "minecraft-exec-role"
  task_exec_iam_role_policies = {
    efs_access = aws_iam_policy.efs_access_policy.arn
    kms_access = aws_iam_policy.efs_kms_access_policy.arn
  }

And that is all! 🎉🎉🎉

Let’s deploy this thing now.

Deploy and Test

I assume you already have Terraform installed and connected to your AWS account.

First, we run init to load our modules

terraform init

Then we plan and see what it will be created

terraform plan

Then we deploy

terraform apply

Once everything is deployed, we will need to add the policy to our EFS module.

policy_statements = [
    {
      sid = "Example"
      actions = [
        "elasticfilesystem:ClientMount",
        "elasticfilesystem:ClientWrite",
        "elasticfilesystem:ClientRootAccess",
        "elasticfilesystem:DescribeFileSystems"
      ]
      principals = [
        {
          type        = "AWS"
          identifiers = [module.ecs.task_exec_iam_role_arn]
        }
      ]
    }
  ]

And then deploy again.

By that point, you should have everything provisioned, with your service running smoothly and healthily.

Log in to your AWS account and go to the ECS service to check your cluster.

If everything is healthy, you can run this command from your terminal (AWS CLI command that requires logging in to AWS via the CLI) to get the public DNS of your NLB.

aws elbv2 describe-load-balancers --names minecraft-nlb --query 'LoadBalancers[0].DNSName' --output text

Or, you can go to the EC2 service, search for the load balancer, and get it from there.

Now the only thing left is to spin up your Minecraft and connect to the server by providing the NLB DNS.

🎉Congratulations!🎉

You have a working Minecraft server, which is way overcomplicated and way more expensive, but you learned ECS along the way!

Clean up

DO NOT FORGET TO CLEAN UP YOUR ENVIRONMENT!!! It is expensive!

Run the command:

terraform destroy

What comes next?

Since this is an expensive solution, I would like to create a mechanism that allows me to open and close the Minecraft server on demand. In the future, I will place the Minecraft server on public subnets to avoid the NAT gateway cost and create a hosted zone for my vanilla Minecraft server.

Feel free to reach out if you encounter any problems or have suggestions.

Till the next time, stay safe and have fun!

Appendix

• News: Any interesting news I find and their impact in the landscape.

• Events: Any upcoming events

• Posts: Any interesting posts from my fellow peers

News

🎆This is huge! Amazon Redshift introduces history mode support for eight third-party SaaS applications. Having a history mode in the data will allow you to run multiple analyses on your data, taking into account historicity (i.e., Churn). Pretty much this is the fundamental property of analytical data 😁

AWS DMS is one of my favourites, while my worst service (don’t tell me this makes no sense, for me it does). It’s always nice to see features that will allow you to have one less thing to worry about when you use DMS. AWS DMS Serverless now supports automatic storage scaling. 🎉

And for those who are trying to save a “few” bucks 💸 (few and Redshift do not always go together), Amazon Redshift Serverless has announced a new discounted pricing. AWS offers up to a 24% discount for the all-upfront option and a 20% discount on the no-upfront option. All the options are for a one-year term.

Events

29th of April: Ctrl+Alt+Migrate: Rebooting to the Cloud from AWS User Group Athens. Do not miss this if you live in Athens!

AWS Community Day Adria: Registration has opened. This event will be on the 5th of September, and it is going to have excellent speakers! Jeff Barr will be doing the keynote!

Feel free to reach out if you want to add your upcoming event here

Posts

Are We Really a Team? by . A really nice post about what it really means to be a team. I agree that the team is so much more than being together

Being a team isn’t about how often we’re together.

It’s about how deeply we’ve got each other’s backs—even when we’re apart.

Scaling Global: 8 Use Cases Where CloudFront Performs Best by . An informative post about Amazon’s CDN solution, CloudFront. Lefteris presents eight use cases where CloudFront shines.

Remember that effective CDN implementation is not just about deployment—it's about continuous optimization.

Till the next time, stay safe and have fun!

News

• Amazon ECS adds the ability to set a default log driver blocking mode

• Amazon S3 Tables now support server-side encryption using AWS KMS with customer-managed keys

• Amazon Q Business launches support for hallucination mitigation in chat responses

• AWS simplifies Amazon VPC Peering billing

• Load Balancer Capacity Unit Reservation for Gateway Load Balancers

• Amazon S3 Express One Zone reduces storage and request prices

• AWS Compute Optimizer now supports 57 new Amazon EC2 instance types

Blog Posts

• Build a FinOps agent using Amazon Bedrock with multi-agent capability and Amazon Nova as the foundation model

• Accelerate your analytics with Amazon S3 Tables and Amazon SageMaker Lakehouse

• 2025 AWS Summit London: Top 5 recommended sessions for business leaders

• Migrating files to Amazon FSx for Windows File Server using Robocopy

• Validate recovery readiness with AWS Backup restore testing

Events

• 29th of April: Ctrl+Alt+Migrate: Rebooting to the Cloud

• AWS Community Day Adria: Registration has opened

Till the next time, stay safe and have fun!

• Amazon EKS Adds Support for Bottlerocket FIPS AMIs in Managed Node Groups

• Amazon Security Lake now supports Internet Protocol Version 6 (IPv6)

• Amazon SNS now supports Internet Protocol Version 6 (IPv6)

• Amazon QuickSight launches dashboard versioning and publish any analysis to any dashboard

• Amazon QuickSight now supports Highlighting

• Amazon QuickSight launches Amazon Q in embedded QuickSight

Blog Posts

• Validate Your Lambda Runtime with CloudFormation Lambda Hooks

• Simplify AWS Cost Data Analysis with Amazon Q in QuickSight

• Build low-latency, resilient applications with Amazon MemoryDB Multi-Region

• Streamlining content compliance: Automating media analysis with Amazon Nova

• Beyond the server room: Transform your IT career with AWS

• Simplifying private API integrations with Amazon EventBridge and AWS Step Functions

Events in Greece

• 10-12 April: DEVOXX GREECE

Till the next time, stay safe and have fun!

• Amazon EventBridge Scheduler now supports AWS PrivateLink

• AWS CloudFormation now supports targeted resource scans in the IaC generator

• Deploy Storage Browser for Amazon S3 quickly with AWS sample application

• Amazon SageMaker introduces metadata rules to enforce standards and improve data governance

• Amazon DataZone now supports metadata rules for publishing

Blog Posts

• Simplifying private API integrations with Amazon EventBridge and AWS Step Functions

Events in Greece

• DDD Greece: #37: Beyond the Hype: Practical Insights on Data Mesh Adoption - April 2nd at Orfium

• Ask the CTO - April 3rd at Kaizen Campus

News

• Amazon Bedrock now supports RAG Evaluation (generally available)

Blog Posts

• Using Amazon S3 Tables with Amazon Redshift to query Apache Iceberg tables

• Optimize your Amazon QuickSight implementation: a guide to usage analytics and cost management

• Optimizing network footprint in serverless applications

DEVOXX Greece 2025

Register: https://devoxx.gr/

DDD Meetup 2nd of April: #37: Beyond the Hype: Practical Insights on Data Mesh Adoption

Now before we begin, please let me know what you want to see (obviously new posts) in this substack by answering the following poll.

I have been using SageMaker for some time now, both for personal and professional use. SageMaker provides you a series of tools that every Data Scientist needs, but no obligations to use them all to produce a complete result. For example, SageMaker has a tool to tune your hyperparameters of your model called Automatic Model Tuning. You can use the Model Tunner if you want, but if you skip this tool, you still have the complete result. In other words, you can go and use whatever tool you may need, even in your local machine (using the SageMaker SDK) and create, train, and deploy your model seamlessly.

While I usually create and train a model in SageMaker, I want to have a complete overview of the progress to spot problems in the training process. For example, I want to know if my model is in a plateau or when I have an exploding tensor, etc When you are in your PC and your train the model in your own GPU, you can monitor everything, but when you are on the cloud and you train your model in a cluster of machines it is a bit difficult to monitor and debug everything. If something went wrong you need to know why and how to correct it. SageMaker now provides a tool called Amazon SageMaker Debugger that helps with the progress of the model’s training in a very detailed manner. I will use the Debugger on a previous post that I have made for “classifying the Fashion-MNIST dataset” (I will update this post and repost it here).

Difficulty and costs

• Level: 300

• Total Cost: Let’s say 1$ (with no Endpoint deployed)

  • Amazon SageMaker Notebook: ml.system for 24 hours -> 0.00364 x 24 -> 0.087$ + ml.t3.medium for 12 hours -> 0.7$ = 0.8$

  • Amazon SageMaker Training instance: 122 seconds in ml.c5.2xlarge -> 0.0019 x 122 = 0.14

  • Debugger Instance: 244 seconds in ml.t3.medium -> 0.003$

Amazon SageMaker Debugger

The Debugger is the fourth component in the equation, and it monitors the model, saves the metrics to S3, and can evaluate the metrics using something called Rules. Let’s take one piece at a time. You can add a Debugger Hook to your model on the circled number 1 in Figure 1. Once the job starts, that Hook will save metrics from your model as a tensor. Then (2) the hook listens for the requested metrics, creates the tensors, and saves them to S3. The debugger (3) can evaluate the model in real-time based on provided (built-in or custom) rules and decide if the training process was successful.

Add a Debugging Hook

Amazon SageMaker provides us a series of built-in Collections in order to monitor and save metrics (in a tensor format) of our model. Here is the list that we can use. For this example, we are saving the biases, the weights, and the metrics of our model.

from sagemaker.debugger import CollectionConfig, DebuggerHookConfig

bucket = sess.default_bucket()

collection_config_biases = CollectionConfig(name='biases')
collection_config_weights = CollectionConfig(name='weights')
collection_config_metrics = CollectionConfig(name='metrics')

debugger_hook_config = DebuggerHookConfig(
    s3_output_path=f"s3://{bucket}/fashion-mnist/hook",
    hook_parameters={
        'save_interval': '100'
    },
    collection_configs=[
        collection_config_biases,
        collection_config_weights,
        collection_config_metrics
    ]
)

To integrate the Debugger Hook to your model, add it to the estimator.

hyperparameters = {'epochs': 10, 'batch_size': 256, 'learning_rate': 0.001}

estimator = TensorFlow(
    entry_point='model.py',
    train_instance_type='ml.p3.2xlarge',
    train_instance_count=1,
    model_dir='/opt/ml/model',
    hyperparameters=hyperparameters,
    role=sagemaker.get_execution_role(),
    base_job_name='tf-fashion-mnist',
    framework_version='1.15',
    py_version='py3',
    script_mode=True,
    debugger_hook_config=debugger_hook_config,
)

In addition, you can create custom Debugger Hooks as shown in the following snippet. This snippet is from an AWS example on Debugger called Visualizing Debugging Tensors.

debugger_hook_config = DebuggerHookConfig(
    s3_output_path=f"s3://{bucket}/fashion-mnist/hook",
    collection_configs=[
        CollectionConfig(
            name="all_tensors",
            parameters={
                "include_regex": ".*",
                "save_steps": "1, 2, 3"
            }
        )
    ]
)

In this example, we are creating a Hook that will save all the tensors of our model. The AWS example has an amazing result, and I urge you to go and read it! More on custom Hooks here.

Analyze the Tensors

To analyze the tensors you need a SageMaker package that is called sm_debug and the documentation can be found here. With smdebug we fetch the tensors and explore them.

from smdebug.trials import create_trial
from smdebug import modes
import numpy as np
import matplotlib.pyplot as plt


# Get the tensors from S3
s3_output_path = estimator.latest_job_debugger_artifacts_path()

# Create a Trial https://github.com/awslabs/sagemaker-debugger/blob/master/docs/analysis.md#Trial
trial = create_trial(s3_output_path)

# Get all the tensor names
trial.tensor_names()

# Get the values of the tensor `val_acc`for mode GLOBAL (validation accuracy)
values = trial.tensor("val_acc").values(modes.GLOBAL)

# Convert it to numpy array
values_eval = np.array(list(values.items()))

fig = plt.figure()
plt.plot(values_eval[:, 1])
fig.suptitle('Validation Accuracy', fontsize=20)
plt.xlabel('Intervals of sampling', fontsize=18)
plt.ylabel('Acuracy', fontsize=16)
fig.savefig('temp.jpg')

First things first, let’s talk about what we just did. Initially, at line 8, we are fetching the location of the tensors in S3, then we create a trial to be able to query the tensors (line 11). Once we have the trial we are able to fetch all the tensor names (line 14) and with a specific name, in our case acc we are fetching the values for the validation accuracy (line 17), we are also using the modes to select the validation tensors. Finally, we convert the values to a NumPy array and we plot it.

There are so many possibilities for using tensors. Once you define a trial, you can use the SageMaker trials page to create all of your plots. This feature is available only on SageMaker Studio.

Add a Debugging Rule

Now that we have a Hook to create tensors from our model, we need to evaluate the model to see if the training was successful. We will set a SageMaker Rule that will read the tensors from our Debugger Hook and it will evaluate them. Be careful, you need to set up a hook for a Rule to extract and evaluate the tensors. Some amazing built-in rules will really help you evaluate your model. You can find the built-in rules here. I have only used built-in rules so far, so we will not dive into the custom ones. I will leave this for a future post, for now, you can take a look on awslabs examples here.

from sagemaker.debugger import Rule, rule_configs

estimator = TensorFlow(
    entry_point='model.py',
    train_instance_type=train_instance_type,
    train_instance_count=1,
    model_dir=model_dir,
    hyperparameters=hyperparameters,
    role=sagemaker.get_execution_role(),
    base_job_name='tf-fashion-mnist',
    framework_version='1.15',
    py_version='py3',
    script_mode=True,
    debugger_hook_config=debugger_hook_config,
    rules=[
        Rule.sagemaker(rule_configs.overfit()),
        Rule.sagemaker(rule_configs.loss_not_decreasing())
    ],
)

We have created two separate rules (more examples here).

1. The overfit rule where SageMaker will know when the model is overfitted and it will mark the training job as failed

2. The loss_not_decreasing rule where Sagemaker will monitor the loss and if it is stale then it will mark the training job as failed

Furthermore, we can go to the tensors, explore them, and find out at what step this problem occurred, why, and how we can prevent it.

Deliberately, I have increased the epochs of my example to 100, this may cause the model to overfit. Let’s see how SageMaker will react to that.

We can see that once the Training Job has started, we can go to the Experiments, search the trials nad find our Job. In the last tab, there is Debugger, that holds all the rules, both built-in and custom. We will see the verdict of the debugger once the job is done. SageMaker correctly captured the overfit. If we go to the tensors we can see that the training accuracy is very close to 1 and the validation accuracy is remaining the same.

You can also, create alerts when a rule condition is met. you can receive messages, stop the training job, etc. All of these will be covered in a future post. For now, you can follow awslabs examples here.

Conclusion

This post taught us how to debug our machine learning model in SageMaker. As you can see, once you get a grip on it, finding what you are looking for will be easy. Personally, I am very excited about this addition to SageMaker (lol, blast from the past, it’s been 5 years!). Debugger exists in the ecosystem of AWS and it can communicate with several other AWS services, such as Lambda. I hope I have shed some light on SageMaker Debugger in this post. If you have any questions, suggestions or requests, please let me know in the comments section below or message me at my BlueSky account @thelastdev.com or LinkedIn.

Till the next time, stay safe and have fun!

News

The most exciting thing is the S3 Tables, which I want to cover in a later blog post, explaining how they work and building a simple use-case.

• Amazon S3 Tables add create and query table support in the S3 console

• Amazon S3 Tables integration with SageMaker Lakehouse is now generally available

• Accelerate serverless development with ready-to-use Serverless Land Patterns in Visual Studio Code

• Announcing support of AWS Glue Data Catalog views with AWS Glue 5.0

• Amazon S3 Access Grants simplify authentication when using both IAM and Identity Provider permissions

• Amazon Data Firehose now delivers real-time streaming data into Amazon S3 Tables

Blog Posts

• Connect Snowflake to S3 Tables using the SageMaker Lakehouse Iceberg REST endpoint

• Building multi-arch containers with GitHub Actions in AWS

• Amazon S3 Tables integration with Amazon SageMaker Lakehouse is now generally available

AWS Developer Day 2025

Recordings: https://aws.amazon.com/blogs/devops/watch-the-recordings-from-aws-developer-day-2025/

Oh Gosh…. it’s been some time….some time….but now I am back!

I’m launching a brand-new Substack where I’ll be diving into DevOps, Cloud, Data Mesh, and more! Whether you're a tech enthusiast, builder, or curious mind, this space will be packed with insights, strategies, and real-world lessons you won’t want to miss. I will start re-uploading my old (OLD) posts from thelastdev.com to familiarize myself with posting here but also, I do not want to lose them :)

Subscribe now

📅 Stay tuned for the first post! In the meantime, hit Subscribe so you don’t miss a thing.

Drop a 🔥 in the comments if you're excited! Let me know what topics you'd love to see covered.

Till the next time, stay safe and have fun!