Skip to main content
This guide explains how to debug issues using CloudWatch logs and other AWS tools.

CloudWatch logs

Log groups

EnvironmentLog groupDescription
Staging/ecs/staging/address-apiAll staging logs
Production/ecs/prod/address-apiAll production logs

Log streams

Each ECS task creates two log streams:
Stream prefixDescription
ecs-init/Database migration logs
ecs/Application logs

Viewing logs in AWS Console

  1. Go to CloudWatchLog groups
  2. Select /ecs/{env}/address-api
  3. Click on a log stream
  4. Use the search bar to filter logs

Useful filters

Filter by log level:
{ $.level = "error" }
Filter by endpoint:
{ $.message = "*geoencode*" }
Filter by status code:
{ $.status = 500 }

Viewing logs with AWS CLI

Tail logs in real-time

aws logs tail /ecs/staging/address-api \\
  --follow \\
  --region ap-south-1 \\
  --profile staging

Search logs

aws logs filter-log-events \\
  --log-group-name /ecs/staging/address-api \\
  --filter-pattern "error" \\
  --start-time $(date -u -d '1 hour ago' +%s)000 \\
  --region ap-south-1

Get logs for specific time range

aws logs filter-log-events \\
  --log-group-name /ecs/staging/address-api \\
  --start-time 1734518400000 \\
  --end-time 1734522000000 \\
  --region ap-south-1

Log format

The Address API uses structured JSON logging with zerolog:
{
  "level": "info",
  "time": 1734518400,
  "message": "request completed",
  "method": "POST",
  "path": "/api/v1/geoencode",
  "status": 200,
  "duration_ms": 145,
  "ip": "203.0.113.1"
}

Log levels

LevelDescriptionWhen to use
debugDetailed informationDevelopment only
infoGeneral informationNormal operations
warnWarning messagesPotential issues
errorError messagesFailures that need attention
fatalFatal errorsServer crashes

Common log patterns

Successful geocoding request

{
  "level": "info",
  "time": 1734518400,
  "message": "geocoding request",
  "address": "94108, CA, US",
  "cached": false,
  "duration_ms": 234
}

Failed geocoding request

{
  "level": "error",
  "time": 1734518400,
  "message": "geocoding failed",
  "address": "invalid address",
  "error": "ZERO_RESULTS",
  "provider": "google"
}

Database connection error

{
  "level": "error",
  "time": 1734518400,
  "message": "database connection failed",
  "error": "connection refused",
  "host": "staging-address-api-postgres.xxxxx.rds.amazonaws.com"
}

Migration logs

{
  "level": "info",
  "time": 1734518400,
  "message": "applying migration",
  "version": "20251217091243",
  "description": "initializing_iso_3166"
}

CloudWatch Insights queries

Top error messages

fields @timestamp, message, error
| filter level = "error"
| stats count() by message
| sort count desc
| limit 10

Request latency percentiles

fields @timestamp, duration_ms
| filter ispresent(duration_ms)
| stats avg(duration_ms), pct(duration_ms, 50), pct(duration_ms, 95), pct(duration_ms, 99)

Requests by endpoint

fields @timestamp, path, status
| filter ispresent(path)
| stats count() by path, status
| sort count desc

Error rate over time

fields @timestamp, level
| stats count() as total, sum(level = "error") as errors by bin(5m)
| fields @timestamp, (errors / total * 100) as error_rate

Better Stack monitoring

Better Stack provides external monitoring and alerting.

Metrics tracked

MetricDescriptionAlert threshold
API callsTotal requests per minute-
Status codesDistribution of HTTP status codes5xx > 5%
Response timeAverage response time> 1000ms
UptimeService availability< 99.9%

Accessing Better Stack

  1. Go to Better Stack dashboard
  2. Select the Address API project
  3. View metrics and alerts

Alert channels

Alerts are sent to:

ECS Exec for live debugging

ECS Exec allows you to run commands inside running containers.

Enable ECS Exec

ECS Exec is already enabled for the Address API service.

Connect to a running task

  1. List running tasks:
aws ecs list-tasks \\
  --cluster staging-ecs-cluster \\
  --service-name address-api \\
  --region ap-south-1
  1. Connect to a task:
aws ecs execute-command \\
  --cluster staging-ecs-cluster \\
  --task <task-id> \\
  --container address-api \\
  --interactive \\
  --command "/bin/sh"
  1. Run commands:
# Check environment variables
env | grep RDS

# Test database connection
psql $DATABASE_URL -c "SELECT 1"

# Check disk usage
df -h

# View running processes
ps aux

Performance monitoring

RDS Performance Insights

  1. Go to RDSDatabases
  2. Select {env}-address-api-postgres
  3. Click Performance Insights tab
Metrics available:
  • Top SQL queries
  • Database load
  • Wait events
  • Connection count

ECS Service metrics

  1. Go to ECSClusters{env}-ecs-cluster
  2. Select address-api service
  3. Click Metrics tab
Metrics available:
  • CPU utilization
  • Memory utilization
  • Network in/out
  • Task count

Debugging checklist

When investigating an issue:
  • Check CloudWatch logs for errors
  • Review CloudWatch alarms
  • Check ECS service health
  • Verify RDS is running
  • Check Better Stack metrics
  • Review recent deployments
  • Test endpoints manually
  • Check security group rules
  • Verify secrets are correct
  • Review Performance Insights

Next steps