Deployment

​

Deployment overview

GitHub Release Created
        │
        ▼
  GitHub Actions Triggered
        │
   ┌────┴────────────────┐
   │                     │
   ▼                     ▼
Build Docker        Run Tests
   │                     │
   └────┬────────────────┘
        │
        ▼
  Push to ECR
        │
        ▼
  Terraform Apply
  (Update Task Definition)
        │
        ▼
  ECS Rolling Deployment
  (Circuit Breaker Enabled)
        │
        ▼
  Smoke Test (/healthz)
        │
        ▼
  Deployment Complete

​

Prerequisites

1. PR is merged: Your changes are in the main branch
2. Tests pass: All CI checks are green
3. AWS credentials: GitHub Actions has access to AWS (configured in repository secrets)
4. Secrets configured: Required secrets are in AWS Secrets Manager

​

Deployment to staging

​

Step 1: Create a pre-release

1. Go to the Releases page
2. Click Draft a new release
3. Fill in the release form:
   • Tag: v0.1.0-rc.1 (increment the RC number for each staging release)
   • Release title: v0.1.0-rc.1 - Staging Release
   • Description: Describe what changed
   • ✅ Check: “Set as a pre-release”
4. Click Publish release

​

Step 2: Monitor the deployment

1. Checkout code - Clone the repository
2. Set up Go - Install Go 1.23.1
3. Run tests - Execute integration tests
4. Build Docker image - Create container image
5. Push to ECR - Upload image to staging ECR
6. Terraform apply - Update infrastructure
7. Wait for deployment - ECS rolling deployment
8. Smoke test - Verify /healthz endpoint

​

Step 3: Verify the deployment

# Check health endpoint
curl https://address.in.staging.commenda.io/healthz

# Expected response:
{
  "api": true,
  "database_pool": {
    "total_connections": 12,
    "acquired_connections": 0,
    "idle_connections": 12,
    "max_connections": 48,
    "acquire_duration_ms": 2
  }
}

​

Step 4: Test your changes

# Test geocoding endpoint
curl -X POST https://address.in.staging.commenda.io/api/v1/geoencode \\
  -H "x-commenda-key: your-staging-api-key" \\
  -H "Content-Type: application/json" \\
  -d '{"address": "94108, CA, US"}'

​

Deployment to production

​

Step 1: Verify staging

1. Test thoroughly on staging
2. Check CloudWatch logs for errors
3. Monitor CloudWatch alarms for anomalies
4. Get approval from team lead (if required)

​

Step 2: Create a full release

1. Go to the Releases page
2. Click Draft a new release
3. Fill in the release form:
   • Tag: v0.1.0 (same version as staging, without -rc.X)
   • Release title: v0.1.0 - Production Release
   • Description: Describe what changed
   • ❌ Do NOT check: “Set as a pre-release”
4. Click Publish release

​

Step 3: Monitor the deployment

​

Step 4: Verify the deployment

# Check health endpoint
curl https://address.in.commenda.io/healthz

# Test geocoding endpoint
curl -X POST https://address.in.commenda.io/api/v1/geoencode \\
  -H "x-commenda-key: your-production-api-key" \\
  -H "Content-Type: application/json" \\
  -d '{"address": "94108, CA, US"}'

​

Step 5: Monitor production

• CloudWatch logs: /ecs/prod/address-api
• CloudWatch alarms: Check for any triggered alarms
• Better Stack: Monitor API call metrics and error rates
• ECS service: Verify all tasks are healthy

​

Deployment workflow details

​

GitHub Actions workflow

name: Deploy Application

on:
  release:
    types: [published]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ contains(github.ref, 'rc') && 'staging' || 'prod' }}
    
    steps:
      - name: Checkout code
      - name: Set up Go
      - name: Run integration tests
      - name: Configure AWS credentials
      - name: Build and push Docker image
      - name: Deploy with Terraform
      - name: Wait for ECS deployment
      - name: Smoke test

​

Environment detection

​

Docker image tagging

​

Terraform apply

terraform workspace select $ENVIRONMENT
terraform apply -auto-approve \\
  -var="app_image_tag=$TAG" \\
  -var="app_secrets=$SECRETS_JSON"

• ECS task definition (with new image tag)
• Secrets Manager (with latest secrets)
• Any infrastructure changes from code

​

ECS rolling deployment

1. Start new tasks with new image
2. Wait for new tasks to pass health checks
3. Drain connections from old tasks
4. Stop old tasks
5. Repeat until all tasks are updated

​

Smoke test

curl -f https://address.in.$ENVIRONMENT.commenda.io/healthz

​

Rollback

​

Automatic rollback

• New tasks fail health checks
• New tasks crash repeatedly
• Deployment doesn’t stabilize within 10 minutes

​

Manual rollback

1. Find the previous release tag (e.g., v0.0.9)
2. Redeploy the previous version:
   • Go to Releases
   • Find the previous release
   • Click “Edit release”
   • Click “Update release” (this triggers a new deployment)

# Switch to the environment
terraform workspace select staging

# Apply with previous image tag
terraform apply -var="app_image_tag=v0.0.9"

​

Deployment checks

​

Deployment frequency

​

Deployment notifications

• GitHub Actions: Workflow status and logs
• Slack (if configured): Deployment start/success/failure notifications
• Better Stack: Deployment events and metrics

​

Troubleshooting deployments

​

Deployment stuck at “Waiting for ECS deployment”

1. Check CloudWatch logs for errors
2. Verify the health check endpoint is responding
3. Check security group rules
4. Verify secrets are correct

​

Error: “Image not found in ECR”

1. Check the “Build and push Docker image” step in GitHub Actions
2. Verify ECR repository exists
3. Check AWS credentials

​

Error: “Terraform apply failed”

1. Review the Terraform plan output
2. Check for resource conflicts
3. Verify AWS permissions

​

Circuit breaker triggered rollback

1. Check CloudWatch logs for crash reasons
2. Verify environment variables and secrets
3. Test the Docker image locally
4. Check database connectivity

​

Best practices

​

Before deploying

• ✅ Run tests locally
• ✅ Test on staging first
• ✅ Review Terraform plan
• ✅ Check for breaking changes
• ✅ Update documentation

​

During deployment

• ✅ Monitor GitHub Actions workflow
• ✅ Watch CloudWatch logs
• ✅ Check ECS service events
• ✅ Monitor CloudWatch alarms

​

After deployment

• ✅ Verify health endpoint
• ✅ Test critical endpoints
• ✅ Monitor error rates
• ✅ Check Better Stack metrics
• ✅ Update release notes

​

Next steps

• Database connection
• Debugging guide
• Common failures