tercul-backend/.github/workflows/README.md

# GitHub Actions CI/CD Documentation

## Overview

This document describes the GitHub Actions CI/CD pipeline for the Tercul backend
project, updated to 2025 best practices. The pipeline ensures code quality,
security, and reliable deployments through automated testing, linting, security
scanning, and containerized deployments.

### Quick Reference

| Workflow | File | Purpose | Triggers |
|----------|------|---------|----------|
| **Lint** | `lint.yml` | Code quality & style | Push/PR to main, develop |
| **Test** | `test.yml` | Unit tests & compatibility | Push/PR to main, develop |
| **Build** | `build.yml` | Binary compilation | Push/PR to main, develop |
| **Security** | `security.yml` | CodeQL scanning | Push/PR to main + Weekly |
| **Docker Build** | `docker-build.yml` | Container images | Push to main, Tags, PRs |
| **Deploy** | `deploy.yml` | Production deployment | Tags (v*), Manual |

## Architecture

The CI/CD pipeline follows the **Single Responsibility Principle** with focused workflows:

1. **Lint** (`lint.yml`) - Code quality and style enforcement
2. **Test** (`test.yml`) - Unit tests and compatibility matrix
3. **Build** (`build.yml`) - Binary compilation and verification
4. **Security** (`security.yml`) - CodeQL security scanning
5. **Docker Build** (`docker-build.yml`) - Container image building and publishing
6. **Deploy** (`deploy.yml`) - Production deployment orchestration

## Workflows

### Lint Workflow (`lint.yml`)

**Purpose**: Ensures code quality and consistent style across the codebase.

**Triggers**:

- Push to `main` and `develop` branches
- Pull requests targeting `main` and `develop` branches

**Jobs**:

- `golangci-lint`: Go linting with golangci-lint
  - Checkout code
  - Setup Go 1.25 with caching
  - Install dependencies
  - Tidy modules (ensures go.mod/go.sum are clean)
  - Run linter with 5-minute timeout

**Configuration**:

- **Timeout**: 5 minutes
- **Target**: All Go files (`./...`)
- **Cache**: Enabled for faster runs

### Test Workflow (`test.yml`)

**Purpose**: Validates code functionality through comprehensive testing.

**Triggers**:

- Push to `main` and `develop` branches
- Pull requests targeting `main` and `develop` branches

**Jobs**:

#### Unit Tests

- **Environment**: Ubuntu with PostgreSQL 15 and Redis 7
- **Features**:

  - Race detection enabled
  - Code coverage reporting (atomic mode)
  - HTML coverage report generation
  - Test result summaries in GitHub UI
  - 30-day artifact retention

**Services**:

- PostgreSQL 15 with health checks
- Redis 7-alpine with health checks

#### Compatibility Matrix

- **Trigger**: Push to `main` branch only
- **Strategy**: Tests across Go versions 1.22, 1.23, 1.24, 1.25
- **Purpose**: Ensures compatibility with multiple Go versions
- **Fail-fast**: Disabled (all versions tested even if one fails)

### Build Workflow (`build.yml`)

**Purpose**: Compiles the application binary and validates the build.

**Triggers**:

- Push to `main` and `develop` branches
- Pull requests targeting `main` and `develop` branches

**Jobs**:

- `build-binary`: Binary compilation and verification
  - Dependency verification with `go mod verify`
  - Build to `bin/tercul-backend`
  - Binary validation test
  - Artifact upload (30-day retention)

**Permissions**:

- `contents: read` - Read repository code
- `attestations: write` - Future SLSA attestation support
- `id-token: write` - OIDC token for attestations

### Security Workflow (`security.yml`)

**Purpose**: Automated security vulnerability detection with CodeQL.

**Triggers**:

- Push to `main` branch
- Pull requests targeting `main` branch
- Scheduled: Every Monday at 14:20 UTC

**Jobs**:

- `codeql-analysis`: CodeQL security scanning for Go
  - Initialize CodeQL with Go language support
  - Build code for analysis
  - Perform security scan
  - Category: "backend-security" for tracking

**CodeQL Configuration**:

The workflow can be customized with additional query suites:

```yaml
- name: Initialize CodeQL
  uses: github/codeql-action/init@v3
  with:
    languages: go
    # Run security-extended suite for more comprehensive scanning
    queries: security-extended
    # Or use security-and-quality for maintainability checks
    # queries: security-and-quality
```

**Available Query Suites**:

- `security-extended`: Default queries plus lower severity/precision queries
- `security-and-quality`: Security queries plus maintainability and reliability

**Custom Query Packs**:

Add custom CodeQL query packs for specialized analysis:

```yaml
- name: Initialize CodeQL
  uses: github/codeql-action/init@v3
  with:
    languages: go
    packs: my-org/go-security-queries@1.0.0
```

### Docker Build Workflow (`docker-build.yml`)

**Purpose**: Builds and publishes multi-architecture Docker images.

**Triggers**:

- Push to `main` branch
- Tag pushes with `v*` pattern
- Pull requests targeting `main` branch

**Jobs**:

- `build-image`: Multi-platform Docker image building

  - Docker Buildx setup for multi-arch builds
  - Login to GitHub Container Registry
  - Metadata extraction for tags and labels
  - Build for AMD64 and ARM64 architectures
  - Push to registry (except for PRs)
  - Generate build provenance attestation

**Image Tagging Strategy**:

- `main` branch pushes → `main` + `sha-<hash>` tags
- Tag pushes (`v1.2.3`) → `v1.2.3`, `1.2.3`, `1.2`, `sha-<hash>` tags
- Pull requests → `pr-<number>` tag (build only, not pushed)

**Push Behavior**:

- **Pushes to main/tags**: Build and push to registry
- **Pull requests**: Build only (validation, no push)

**Platforms**: linux/amd64, linux/arm64

### Deploy Workflow (`deploy.yml`)

**Purpose**: Production deployment orchestration to Docker Swarm.

**Triggers**:

- Tag pushes with `v*` pattern
- Manual dispatch with version input

**Jobs**:

- `deploy-production`: Deployment to production environment

  - Version extraction from tag or manual input
  - Docker Swarm service update (SSH-based deployment template)
  - Deployment summary with timestamp
  - Environment protection and tracking

**Environment**:

- Name: `production`
- URL: Configurable production URL
- Protection: Supports required reviewers and wait timers

**Manual Deployment**:

Deployments can be triggered manually from the Actions tab with a specific version.

**Required Secrets**:

- `SWARM_HOST`: Docker Swarm manager hostname/IP
- `SWARM_SSH_KEY`: SSH private key for Swarm access

## Workflow Execution Order

**On Pull Request**:

1. Lint → Validates code style
2. Test → Runs unit tests with coverage
3. Build → Compiles binary
4. Security → CodeQL analysis (main branch PRs only)
5. Docker Build → Builds image (no push)

**On Push to main**:

1. Lint → Code quality check
2. Test → Unit tests + compatibility matrix
3. Build → Binary compilation
4. Security → CodeQL scan
5. Docker Build → Build and push image

**On Tag Push (v\*)**:

1. Docker Build → Build and push versioned image
2. Deploy → Deploy to production

## Security Features

### Permissions Management

- **Principle of Least Privilege**: Each workflow has minimal required permissions
- **GITHUB_TOKEN Restrictions**: Read-only by default, elevated only when necessary
- **Workflow Separation**: Each workflow operates independently
- **Attestation Permissions**: For build provenance and SLSA compliance

### Code Security

- **CodeQL Integration**: Automated security scanning for Go code
- **Dependency Verification**: `go mod verify` ensures integrity
- **Module Tidying**: `go mod tidy` prevents dependency drift

### Container Security

- **Multi-platform Builds**: Ensures compatibility and security across architectures
- **Provenance Attestation**: Cryptographic proof of build integrity
- **Registry Security**: GitHub Container Registry with token-based authentication

### Secrets Management

- **No Hardcoded Secrets**: All sensitive data uses GitHub secrets
- **Environment Variables**: Proper isolation of configuration
- **GITHUB_TOKEN**: Automatic authentication for package registries
- **Granular Permissions**: Package-level access control

### Package Registry Security

- **GITHUB_TOKEN Authentication**: No personal access tokens required
- **Automatic Permissions**: Packages inherit repository visibility
- **Repository Scoped**: Packages linked to source repository
- **Granular Access**: Fine-grained permissions per package
- **Artifact Attestation**: Cryptographic proof of build provenance
- **OIDC Support**: Token-based authentication without long-lived credentials

## Best Practices Implemented

### 2025 Updates

- **Semantic Versioning**: Actions pinned to major versions (e.g., `@v5`) instead of SHA
- **Caching Optimization**: Go module and Docker layer caching
- **Matrix Testing**: Cross-version compatibility validation
- **Service Health Checks**: Database and Redis readiness verification
- **Artifact Management**: Proper retention policies and naming

### Performance Optimizations

- **Dependency Caching**: Reduces setup time significantly
- **Parallel Jobs**: Independent jobs run concurrently
- **Conditional Execution**: Security scans only on main branch
- **Artifact Upload**: Efficient storage and retrieval

### Reliability Features

- **Timeout Configuration**: Prevents hanging jobs
- **Error Handling**: Proper exit codes and logging
- **Health Checks**: Service readiness validation
- **Retention Policies**: Balanced storage management

## Configuration Details

### Go Version

- Primary: Go 1.25
- Matrix: Go 1.22, 1.23, 1.24, 1.25

### Services

- **PostgreSQL**: Version 15 with health checks
- **Redis**: Version 7-alpine with ping health checks

### Tools

- **Linter**: golangci-lint latest with 5-minute timeout
- **Testing**: `go test` with race detection and coverage
- **Building**: `go build` with verbose output
- **Security**: CodeQL for Go analysis

### Caching

- **Go Modules**: Automatic caching via setup-go action
- **Docker Layers**: GitHub Actions cache with GHA type
- **CodeQL Databases**: Stored in `${{ github.runner_temp }}/codeql_databases`

## Maintenance

### Dependabot Configuration

- **GitHub Actions**: Weekly updates with "ci" prefix
- **Go Modules**: Weekly updates with "deps" prefix
- **Automated PRs**: Keeps dependencies current and secure

### Monitoring

- **Workflow Runs**: GitHub Actions tab for execution monitoring
- **Security Alerts**: Code scanning results and dependency alerts
- **Coverage Reports**: Artifact downloads for test coverage analysis

### Troubleshooting

#### Common Issues

1. **Cache Misses**: Clear caches if corruption suspected
2. **Service Failures**: Check health check configurations
3. **Permission Errors**: Verify GITHUB_TOKEN scopes
4. **Timeout Issues**: Adjust timeout values in workflow configurations
5. **Docker Push Failures**: Check package write permissions
6. **Registry Authentication**: Ensure `packages: write` permission is set

#### Package Registry Issues

**Problem**: Cannot push to GitHub Container Registry

```yaml
# Solution: Ensure proper permissions
permissions:
  contents: read
  packages: write
  id-token: write
  attestations: write
```

**Problem**: Package not visible after push

- Check package visibility settings (public/private/internal)
- Verify repository is linked to package
- Ensure workflow completed successfully

**Problem**: Cannot pull package in workflow

```yaml
# Solution: Login to registry first
- name: Login to GitHub Container Registry
  uses: docker/login-action@v3
  with:
    registry: ghcr.io
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}
```

#### Debugging

- **Manual Dispatch**: Use `workflow_dispatch` for testing
- **Log Analysis**: Review step outputs for error details
- **Artifact Inspection**: Download build artifacts for verification
- **Package Logs**: Check package activity in GitHub UI

## Future Enhancements

### Planned Features

- **SLSA Integration**: Enhanced build attestation
- **Dependency Review**: Automated dependency vulnerability checking
- **Performance Testing**: Load testing integration
- **Multi-environment Deployment**: Staging and production separation

### Scalability Considerations

- **Self-hosted Runners**: For resource-intensive jobs
- **Job Parallelization**: Further optimization of concurrent execution
- **Cache Optimization**: Advanced caching strategies

## Docker Image Usage

### Pulling Images

Pull the latest image:

```bash
docker pull ghcr.io/<owner>/<repository>:latest
```

Pull a specific version:

```bash
docker pull ghcr.io/<owner>/<repository>:1.2.3
```

### Running Locally

Run the container:

```bash
docker run -d \
  --name tercul-backend \
  -p 8080:8080 \
  -e DATABASE_URL="postgres://..." \
  ghcr.io/<owner>/<repository>:latest
```

### Docker Swarm Deployment

Deploy as a stack:

```bash
docker stack deploy -c docker-compose.yml tercul
```

Update running service:

```bash
docker service update \
  --image ghcr.io/<owner>/<repository>:1.2.3 \
  tercul_backend
```

### Verifying Attestations

Verify build provenance:

```bash
gh attestation verify \
  oci://ghcr.io/<owner>/<repository>:latest \
  --owner <owner>
```

## Contributing

When modifying workflows:

1. Test changes using `workflow_dispatch`
2. Ensure backward compatibility
3. Update this documentation
4. Follow security best practices
5. Use semantic versioning for action references
6. Test Docker builds locally before pushing
7. Verify package permissions after changes
8. Review CodeQL alerts before merging PRs
9. Update query packs regularly for latest security rules
10. Test configuration changes with `workflow_dispatch`

## CodeQL Advanced Configuration

### Custom Configuration File

For complex CodeQL setups, use a configuration file (`.github/codeql/codeql-config.yml`):

```yaml
name: "CodeQL Config"

# Disable default queries to run only custom queries
disable-default-queries: false

# Specify query packs
packs:
  - scope/go-security-pack
  - scope/go-compliance-pack@1.2.3

# Add custom queries
queries:
  - uses: security-and-quality
  - uses: ./custom-queries

# Filter queries by severity
query-filters:
  - exclude:
      problem.severity:
        - warning
        - recommendation
  - exclude:
      id: go/redundant-assignment

# Scan specific directories
paths:
  - internal
  - cmd
  - pkg

paths-ignore:
  - "**/*_test.go"
  - vendor
  - "**/testdata/**"

# Extend threat model (preview)
threat-models: local
```

Reference the config in your workflow:

```yaml
- uses: github/codeql-action/init@v3
  with:
    config-file: ./.github/codeql/codeql-config.yml
```

### Inline Configuration

Alternatively, specify configuration inline:

```yaml
- uses: github/codeql-action/init@v3
  with:
    languages: go
    config: |
      disable-default-queries: false
      queries:
        - uses: security-extended
      query-filters:
        - exclude:
            problem.severity:
              - recommendation      
```

### Scheduling CodeQL Scans

Run CodeQL on a schedule for regular security audits:

```yaml
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    # Run at 14:20 UTC every Monday
    - cron: '20 14 * * 1'
```

### Avoiding Unnecessary Scans

Skip CodeQL for specific file changes:

```yaml
on:
  pull_request:
    branches: [main]
    paths-ignore:
      - '**/*.md'
      - '**/*.txt'
      - 'docs/**'
```

### Analysis Categories

Categorize multiple analyses in monorepos:

```yaml
- name: Perform CodeQL Analysis
  uses: github/codeql-action/analyze@v3
  with:
    category: "backend-api"
```

### External Query Packs

Use query packs from GitHub Enterprise Server:

```yaml
- uses: github/codeql-action/init@v3
  with:
    registries: |
      - url: https://containers.GHEHOSTNAME/v2/
        packages:
          - my-company/*
        token: ${{ secrets.GHES_TOKEN }}      
    packs: my-company/go-queries
```

### Query Suite Examples

**Security-focused**:

```yaml
queries:
  - uses: security-extended
```

**Quality and security**:

```yaml
queries:
  - uses: security-and-quality
```

**Custom suite**:

```yaml
queries:
  - uses: ./custom-queries/critical-security.qls
```

## Advanced Workflow Techniques

### Workflow Commands

GitHub Actions supports workflow commands for advanced functionality:

#### Debug Logging

Enable detailed debugging with the `ACTIONS_STEP_DEBUG` secret:

```yaml
- name: Debug step
  run: echo "::debug::Detailed debugging information"
```

#### Annotations

Create annotations for notices, warnings, and errors:

```yaml
# Notice annotation
- run: echo "::notice file=app.go,line=10::Consider refactoring"

# Warning annotation
- run: echo "::warning file=main.go,line=5,col=10::Deprecated function"

# Error annotation
- run: echo "::error file=handler.go,line=20,title=Build Error::Missing import"
```

#### Grouping Log Lines

Organize logs with collapsible groups:

```yaml
- name: Build application
  run: |
    echo "::group::Compiling Go code"
    go build -v ./...
    echo "::endgroup::"    
```

#### Masking Secrets

Prevent sensitive values from appearing in logs:

```yaml
- name: Generate token
  run: |
    TOKEN=$(generate_token)
    echo "::add-mask::$TOKEN"
    echo "TOKEN=$TOKEN" >> $GITHUB_ENV    
```

#### Job Summaries

Add Markdown summaries to workflow runs:

```yaml
- name: Test summary
  run: |
    echo "### Test Results :white_check_mark:" >> $GITHUB_STEP_SUMMARY
    echo "" >> $GITHUB_STEP_SUMMARY
    echo "- Total: 150 tests" >> $GITHUB_STEP_SUMMARY
    echo "- Passed: 148" >> $GITHUB_STEP_SUMMARY
    echo "- Failed: 2" >> $GITHUB_STEP_SUMMARY    
```

### Environment Files

Use environment files for dynamic configuration:

```yaml
# Set environment variable for subsequent steps
- name: Set build info
  run: |
    echo "BUILD_TIME=$(date +'%Y-%m-%d %H:%M:%S')" >> $GITHUB_ENV
    echo "COMMIT_SHA=${GITHUB_SHA:0:7}" >> $GITHUB_ENV    

# Use in later steps
- name: Deploy
  run: echo "Deploying $COMMIT_SHA built at $BUILD_TIME"
```

### Output Parameters

Share data between steps:

```yaml
- name: Calculate version
  id: version
  run: echo "VERSION=1.2.3" >> $GITHUB_OUTPUT

- name: Use version
  run: echo "Building version ${{ steps.version.outputs.VERSION }}"
```

### Multiline Values

Handle multiline strings safely:

```yaml
- name: Store API response
  run: |
    {
      echo 'API_RESPONSE<<EOF'
      curl https://api.example.com/data
      echo EOF
    } >> $GITHUB_ENV    
```

## References

- [GitHub Actions Documentation](https://docs.github.com/en/actions)
- [Go CI/CD Best Practices](https://github.com/golang/go/wiki/Go-Release-Cycle)
- [Security Hardening Guide](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions)
- [Dependency Caching](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows)
- [Workflow Commands Reference](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions)
- [Publishing Packages with Actions](https://docs.github.com/en/packages/managing-github-packages-using-github-actions-workflows/publishing-and-installing-a-package-with-github-actions)
- [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)
- [Artifact Attestations](https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds)
- [CodeQL Configuration Reference](https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning)
- [CodeQL Query Suites](https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/codeql-query-suites)
- [CodeQL CLI Reference](https://docs.github.com/en/code-security/codeql-cli)