tercul-backend/PRODUCTION-TASKS.md

Tercul Backend - Production Readiness Tasks

Generated: November 27, 2025 Current Status: Most core features implemented, needs production hardening

⚠️ MIGRATED TO GITHUB ISSUES

All production readiness tasks have been migrated to GitHub Issues for better tracking. See issues #30-38 in the repository: https://github.com/SamyRai/backend/issues

This document is kept for reference only and should not be used for task tracking.

---

📊 Current Reality Check

✅ What's Actually Working

• ✅ Full GraphQL API with 90%+ resolvers implemented
• ✅ Complete CQRS pattern (Commands & Queries)
• ✅ Auth system (Register, Login, JWT, Password Reset, Email Verification)
• ✅ Work CRUD with authorization
• ✅ Translation management with analytics
• ✅ User management and profiles
• ✅ Collections, Comments, Likes, Bookmarks
• ✅ Contributions with review workflow
• ✅ Analytics service (views, likes, trending)
• ✅ Clean Architecture with DDD patterns
• ✅ Comprehensive test coverage (passing tests)
• ✅ CI/CD pipelines (build, test, lint, security, docker)
• ✅ Docker setup and containerization
• ✅ Database migrations and schema

⚠️ What Needs Work

• ⚠️ Search functionality (stub implementation) → Issue #30
• ⚠️ Observability (metrics, tracing) → Issues #31, #32, #33
• ⚠️ Production deployment automation → Issue #36
• ⚠️ Performance optimization → Issues #34, #35
• ⚠️ Security hardening → Issue #37
• ⚠️ Infrastructure as Code → Issue #38

---

🎯 EPIC 1: Search & Discovery (HIGH PRIORITY)

Story 1.1: Full-Text Search Implementation

Priority: P0 (Critical) Estimate: 8 story points (2-3 days) Labels: enhancement, search, backend

User Story:

As a user exploring literary works,
I want to search across works, translations, and authors by keywords,
So that I can quickly find relevant content in my preferred language.

Acceptance Criteria:

• Implement Weaviate-based full-text search for works
• Index work titles, content, and metadata
• Support multi-language search (Russian, English, Tatar)
• Search returns relevance-ranked results
• Support filtering by language, category, tags, authors
• Support date range filtering
• Search response time < 200ms for 95th percentile
• Handle special characters and diacritics correctly

Technical Tasks:

1. Complete internal/app/search/service.go implementation
2. Implement Weaviate schema for Works, Translations, Authors
3. Create background indexing job for existing content
4. Add incremental indexing on create/update operations
5. Implement search query parsing and normalization
6. Add search result pagination and sorting
7. Create integration tests for search functionality
8. Add search metrics and monitoring

Dependencies:

• Weaviate instance running (already in docker-compose)
• internal/platform/search client (exists)
• internal/domain/search interfaces (exists)

Definition of Done:

• All acceptance criteria met
• Unit tests passing (>80% coverage)
• Integration tests with real Weaviate instance
• Performance benchmarks documented
• Search analytics tracked

---

Story 1.2: Advanced Search Filters

Priority: P1 (High) Estimate: 5 story points (1-2 days) Labels: enhancement, search, backend

User Story:

As a researcher or literary enthusiast,
I want to filter search results by multiple criteria simultaneously,
So that I can narrow down to exactly the works I'm interested in.

Acceptance Criteria:

• Filter by literature type (poetry, prose, drama)
• Filter by time period (creation date ranges)
• Filter by multiple authors simultaneously
• Filter by genre/categories
• Filter by language availability
• Combine filters with AND/OR logic
• Save search filters as presets (future)

Technical Tasks:

1. Extend SearchFilters domain model
2. Implement filter translation to Weaviate queries
3. Add faceted search capabilities
4. Implement filter validation
5. Add filter combination logic
6. Create filter preset storage (optional)
7. Add tests for all filter combinations

---

🎯 EPIC 2: API Documentation (HIGH PRIORITY)

Story 2.1: Comprehensive GraphQL API Documentation

Priority: P1 (High) Estimate: 5 story points (1-2 days) Labels: documentation, api, devex

User Story:

As a frontend developer or API consumer,
I want complete documentation for all GraphQL queries and mutations,
So that I can integrate with the API without constantly asking questions.

Acceptance Criteria:

• Document all 80+ GraphQL resolvers
• Include example queries for each operation
• Document input types and validation rules
• Provide error response examples
• Document authentication requirements
• Include rate limiting information
• Add GraphQL Playground with example queries
• Auto-generate docs from schema annotations

Technical Tasks:

1. Add descriptions to all GraphQL types in schema
2. Document each query/mutation with examples
3. Create api/README.md with comprehensive guide
4. Add inline schema documentation
5. Set up GraphQL Voyager for schema visualization
6. Create API changelog
7. Add versioning documentation
8. Generate OpenAPI spec for REST endpoints (if any)

Deliverables:

• api/README.md - Complete API guide
• api/EXAMPLES.md - Query examples
• api/CHANGELOG.md - API version history
• Enhanced GraphQL schema with descriptions
• Interactive API explorer

---

Story 2.2: Developer Onboarding Documentation

Priority: P1 (High) Estimate: 3 story points (1 day) Labels: documentation, devex

User Story:

As a new developer joining the project,
I want clear setup instructions and architecture documentation,
So that I can become productive quickly without extensive hand-holding.

Acceptance Criteria:

• Updated README.md with quick start guide
• Architecture diagrams and explanations
• Development workflow documentation
• Testing strategy documentation
• Contribution guidelines
• Code style guide
• Troubleshooting common issues

Technical Tasks:

1. Update root README.md with modern structure
2. Create docs/ARCHITECTURE.md with diagrams
3. Document CQRS and DDD patterns used
4. Create docs/DEVELOPMENT.md workflow guide
5. Document testing strategy in docs/TESTING.md
6. Create CONTRIBUTING.md guide
7. Add package-level README.md for complex packages

Deliverables:

• Refreshed README.md
• docs/ARCHITECTURE.md
• docs/DEVELOPMENT.md
• docs/TESTING.md
• CONTRIBUTING.md

---

🎯 EPIC 3: Observability & Monitoring (CRITICAL FOR PRODUCTION)

Story 3.1: Distributed Tracing with OpenTelemetry

Priority: P0 (Critical) Estimate: 8 story points (2-3 days) Labels: observability, monitoring, infrastructure

User Story:

As a DevOps engineer monitoring production,
I want distributed tracing across all services and database calls,
So that I can quickly identify performance bottlenecks and errors.

Acceptance Criteria:

• OpenTelemetry SDK integrated
• Automatic trace context propagation
• All HTTP handlers instrumented
• All database queries traced
• All GraphQL resolvers traced
• Custom spans for business logic
• Traces exported to OTLP collector
• Integration with Jaeger/Tempo

Technical Tasks:

1. Add OpenTelemetry Go SDK dependencies
2. Create internal/observability/tracing package
3. Instrument HTTP middleware with auto-tracing
4. Add database query tracing via GORM callbacks
5. Instrument GraphQL execution
6. Add custom spans for slow operations
7. Set up trace sampling strategy
8. Configure OTLP exporter
9. Add Jaeger to docker-compose for local dev
10. Document tracing best practices

Configuration:

// Example trace configuration
type TracingConfig struct {
    Enabled       bool
    ServiceName   string
    SamplingRate  float64
    OTLPEndpoint  string
}

---

Story 3.2: Prometheus Metrics & Alerting

Priority: P0 (Critical) Estimate: 5 story points (1-2 days) Labels: observability, monitoring, metrics

User Story:

As a site reliability engineer,
I want detailed metrics on API performance and system health,
So that I can detect issues before they impact users.

Acceptance Criteria:

• HTTP request metrics (latency, status codes, throughput)
• Database query metrics (query time, connection pool)
• Business metrics (works created, searches performed)
• System metrics (memory, CPU, goroutines)
• GraphQL-specific metrics (resolver performance)
• Metrics exposed on /metrics endpoint
• Prometheus scraping configured
• Grafana dashboards created

Technical Tasks:

1. Enhance existing Prometheus middleware
2. Add HTTP handler metrics (already partially done)
3. Add database query duration histograms
4. Create business metric counters
5. Add GraphQL resolver metrics
6. Create custom metrics for critical paths
7. Set up metric labels strategy
8. Create Grafana dashboard JSON
9. Define SLOs and SLIs
10. Create alerting rules YAML

Key Metrics:

# HTTP Metrics
http_requests_total{method, path, status}
http_request_duration_seconds{method, path}

# Database Metrics
db_query_duration_seconds{query}
db_connections_current
db_connections_max

# Business Metrics
works_created_total{language}
searches_performed_total{type}
user_registrations_total

# GraphQL Metrics
graphql_resolver_duration_seconds{operation, resolver}
graphql_errors_total{operation, error_type}

---

Story 3.3: Structured Logging Enhancements

Priority: P1 (High) Estimate: 3 story points (1 day) Labels: observability, logging

User Story:

As a developer debugging production issues,
I want rich, structured logs with request context,
So that I can quickly trace requests and identify root causes.

Acceptance Criteria:

• Request ID in all logs
• User ID in authenticated request logs
• Trace ID/Span ID in all logs
• Consistent log levels across codebase
• Sensitive data excluded from logs
• Structured fields for easy parsing
• Log sampling for high-volume endpoints

Technical Tasks:

1. Enhance HTTP middleware to inject request ID
2. Add user ID to context from JWT
3. Add trace/span IDs to logger context
4. Audit all logging statements for consistency
5. Add field name constants for structured logging
6. Implement log redaction for passwords/tokens
7. Add log sampling configuration
8. Create log aggregation guide (ELK/Loki)

Log Format Example:

{
  "level": "info",
  "ts": "2025-11-27T10:30:45.123Z",
  "msg": "Work created successfully",
  "request_id": "req_abc123",
  "user_id": "user_456",
  "trace_id": "trace_xyz789",
  "span_id": "span_def321",
  "work_id": 789,
  "language": "en",
  "duration_ms": 45
}

---

🎯 EPIC 4: Performance Optimization (MEDIUM PRIORITY)

Story 4.1: Read Models (DTOs) for Efficient Queries

Priority: P1 (High) Estimate: 8 story points (2-3 days) Labels: performance, architecture, refactoring

User Story:

As an API consumer,
I want fast query responses with only the data I need,
So that my application loads quickly and uses less bandwidth.

Acceptance Criteria:

• Create DTOs for all list queries
• DTOs include only fields needed by API
• Avoid N+1 queries with proper joins
• Reduce payload size by 30-50%
• Query response time improved by 20%
• No breaking changes to GraphQL schema

Technical Tasks:

1. Create internal/app/work/dto package
2. Define WorkListDTO, WorkDetailDTO
3. Create TranslationListDTO, TranslationDetailDTO
4. Define AuthorListDTO, AuthorDetailDTO
5. Implement optimized SQL queries for DTOs
6. Update query services to return DTOs
7. Update GraphQL resolvers to map DTOs
8. Add benchmarks comparing old vs new
9. Update tests to use DTOs
10. Document DTO usage patterns

Example DTO:

// WorkListDTO - Optimized for list views
type WorkListDTO struct {
    ID              uint
    Title           string
    AuthorName      string
    AuthorID        uint
    Language        string
    CreatedAt       time.Time
    ViewCount       int
    LikeCount       int
    TranslationCount int
}

// WorkDetailDTO - Full information for single work
type WorkDetailDTO struct {
    *WorkListDTO
    Content         string
    Description     string
    Tags            []string
    Categories      []string
    Translations    []TranslationSummaryDTO
    Author          AuthorSummaryDTO
    Analytics       WorkAnalyticsDTO
}

---

Story 4.2: Redis Caching Strategy

Priority: P1 (High) Estimate: 5 story points (1-2 days) Labels: performance, caching, infrastructure

User Story:

As a user browsing popular works,
I want instant page loads for frequently accessed content,
So that I have a smooth, responsive experience.

Acceptance Criteria:

• Cache hot works (top 100 viewed)
• Cache author profiles
• Cache search results (5 min TTL)
• Cache translations by work ID
• Automatic cache invalidation on updates
• Cache hit rate > 70% for reads
• Cache warming for popular content
• Redis failover doesn't break app

Technical Tasks:

1. Refactor internal/data/cache with decorator pattern
2. Create CachedWorkRepository decorator
3. Implement cache-aside pattern
4. Add cache key versioning strategy
5. Implement selective cache invalidation
6. Add cache metrics (hit/miss rates)
7. Create cache warming job
8. Handle cache failures gracefully
9. Document caching strategy
10. Add cache configuration

Cache Key Strategy:

work:{version}:{id}
author:{version}:{id}
translation:{version}:{work_id}:{lang}
search:{version}:{query_hash}
trending:{period}

---

Story 4.3: Database Query Optimization

Priority: P2 (Medium) Estimate: 5 story points (1-2 days) Labels: performance, database

User Story:

As a user with slow internet,
I want database operations to complete quickly,
So that I don't experience frustrating delays.

Acceptance Criteria:

• All queries use proper indexes
• No N+1 query problems
• Eager loading for related entities
• Query time < 50ms for 95th percentile
• Connection pool properly sized
• Slow query logging enabled
• Query explain plans documented

Technical Tasks:

1. Audit all repository queries
2. Add missing database indexes
3. Implement eager loading with GORM Preload
4. Fix N+1 queries in GraphQL resolvers
5. Optimize joins and subqueries
6. Add query timeouts
7. Configure connection pool settings
8. Enable PostgreSQL slow query log
9. Create query performance dashboard
10. Document query optimization patterns

---

🎯 EPIC 5: Deployment & DevOps (CRITICAL FOR PRODUCTION)

Story 5.1: Production Deployment Automation

Priority: P0 (Critical) Estimate: 8 story points (2-3 days) Labels: devops, deployment, infrastructure

User Story:

As a DevOps engineer,
I want automated, zero-downtime deployments to production,
So that we can ship features safely and frequently.

Acceptance Criteria:

• Automated deployment on tag push
• Blue-green or rolling deployment strategy
• Health checks before traffic routing
• Automatic rollback on failures
• Database migrations run automatically
• Smoke tests after deployment
• Deployment notifications (Slack/Discord)
• Deployment dashboard

Technical Tasks:

1. Complete .github/workflows/deploy.yml implementation
2. Set up staging environment
3. Implement blue-green deployment strategy
4. Add health check endpoints (/health, /ready)
5. Create database migration runner
6. Add pre-deployment smoke tests
7. Configure load balancer for zero-downtime
8. Set up deployment notifications
9. Create rollback procedures
10. Document deployment process

Health Check Endpoints:

GET /health       -> {"status": "ok", "version": "1.2.3"}
GET /ready        -> {"ready": true, "db": "ok", "redis": "ok"}
GET /metrics      -> Prometheus metrics

---

Story 5.2: Infrastructure as Code (Kubernetes)

Priority: P1 (High) Estimate: 8 story points (2-3 days) Labels: devops, infrastructure, k8s

User Story:

As a platform engineer,
I want all infrastructure defined as code,
So that environments are reproducible and version-controlled.

Acceptance Criteria:

• Kubernetes manifests for all services
• Helm charts for easy deployment
• ConfigMaps for configuration
• Secrets management with sealed secrets
• Horizontal Pod Autoscaling configured
• Ingress with TLS termination
• Persistent volumes for PostgreSQL/Redis
• Network policies for security

Technical Tasks:

1. Enhance deploy/k8s manifests
2. Create Deployment YAML for backend
3. Create Service and Ingress YAMLs
4. Create ConfigMap for app configuration
5. Set up Sealed Secrets for sensitive data
6. Create HorizontalPodAutoscaler
7. Add resource limits and requests
8. Create StatefulSets for databases
9. Set up persistent volume claims
10. Create Helm chart structure
11. Document Kubernetes deployment

File Structure:

deploy/k8s/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   └── hpa.yaml
├── overlays/
│   ├── staging/
│   └── production/
└── helm/
    └── tercul-backend/
        ├── Chart.yaml
        ├── values.yaml
        └── templates/

---

Story 5.3: Disaster Recovery & Backups

Priority: P1 (High) Estimate: 5 story points (1-2 days) Labels: devops, backup, disaster-recovery

User Story:

As a business owner,
I want automated backups and disaster recovery procedures,
So that we never lose user data or have extended outages.

Acceptance Criteria:

• Daily PostgreSQL backups
• Point-in-time recovery capability
• Backup retention policy (30 days)
• Backup restoration tested monthly
• Backup encryption at rest
• Off-site backup storage
• Disaster recovery runbook
• RTO < 1 hour, RPO < 15 minutes

Technical Tasks:

1. Set up automated database backups
2. Configure WAL archiving for PostgreSQL
3. Implement backup retention policy
4. Store backups in S3/GCS with encryption
5. Create backup restoration script
6. Test restoration procedure
7. Create disaster recovery runbook
8. Set up backup monitoring and alerts
9. Document backup procedures
10. Schedule regular DR drills

---

🎯 EPIC 6: Security Hardening (HIGH PRIORITY)

Story 6.1: Security Audit & Vulnerability Scanning

Priority: P0 (Critical) Estimate: 5 story points (1-2 days) Labels: security, compliance

User Story:

As a security officer,
I want continuous vulnerability scanning and security best practices,
So that user data and the platform remain secure.

Acceptance Criteria:

• Dependency scanning with Dependabot (already active)
• SAST scanning with CodeQL
• Container scanning with Trivy
• No high/critical vulnerabilities
• Security headers configured
• Rate limiting on all endpoints
• Input validation on all mutations
• SQL injection prevention verified

Technical Tasks:

1. Review existing security workflows (already good!)
2. Add rate limiting middleware
3. Implement input validation with go-playground/validator
4. Add security headers middleware
5. Audit SQL queries for injection risks
6. Review JWT implementation for best practices
7. Add CSRF protection for mutations
8. Implement request signing for sensitive operations
9. Create security incident response plan
10. Document security practices

Security Headers:

X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: default-src 'self'

---

Story 6.2: API Rate Limiting & Throttling

Priority: P1 (High) Estimate: 3 story points (1 day) Labels: security, performance, api

User Story:

As a platform operator,
I want rate limiting to prevent abuse and ensure fair usage,
So that all users have a good experience and our infrastructure isn't overwhelmed.

Acceptance Criteria:

• Rate limiting per user (authenticated)
• Rate limiting per IP (anonymous)
• Different limits for different operations
• 429 status code with retry-after header
• Rate limit info in response headers
• Configurable rate limits
• Redis-based distributed rate limiting
• Rate limit metrics and monitoring

Technical Tasks:

1. Implement rate limiting middleware
2. Use redis for distributed rate limiting
3. Configure different limits for read/write
4. Add rate limit headers to responses
5. Create rate limit exceeded error handling
6. Add rate limit bypass for admins
7. Monitor rate limit usage
8. Document rate limits in API docs
9. Add tests for rate limiting
10. Create rate limit dashboard

Rate Limits:

Authenticated Users:
- 1000 requests/hour (general)
- 100 writes/hour (mutations)
- 10 searches/minute

Anonymous Users:
- 100 requests/hour
- 10 writes/hour
- 5 searches/minute

---

🎯 EPIC 7: Developer Experience (MEDIUM PRIORITY)

Story 7.1: Local Development Environment Improvements

Priority: P2 (Medium) Estimate: 3 story points (1 day) Labels: devex, tooling

User Story:

As a developer,
I want a fast, reliable local development environment,
So that I can iterate quickly without friction.

Acceptance Criteria:

• One-command setup (make setup)
• Hot reload for Go code changes
• Database seeding with realistic data
• GraphQL Playground pre-configured
• All services start reliably
• Clear error messages when setup fails
• Development docs up-to-date

Technical Tasks:

1. Create comprehensive make setup target
2. Add air for hot reload in docker-compose
3. Create database seeding script
4. Add sample data fixtures
5. Pre-configure GraphQL Playground
6. Add health check script
7. Improve error messages in Makefile
8. Document common setup issues
9. Create troubleshooting guide
10. Add setup validation script

---

Story 7.2: Testing Infrastructure Improvements

Priority: P2 (Medium) Estimate: 5 story points (1-2 days) Labels: testing, devex

User Story:

As a developer writing tests,
I want fast, reliable test execution without external dependencies,
So that I can practice TDD effectively.

Acceptance Criteria:

• Unit tests run in <5 seconds
• Integration tests isolated with test containers
• Parallel test execution
• Test coverage reports
• Fixtures for common test scenarios
• Clear test failure messages
• Easy to run single test or package

Technical Tasks:

1. Refactor internal/testutil for better isolation
2. Implement test containers for integration tests
3. Add parallel test execution
4. Create reusable test fixtures
5. Set up coverage reporting
6. Add golden file testing utilities
7. Create test data builders
8. Improve test naming conventions
9. Document testing best practices
10. Add make test-fast and make test-all

---

📋 Task Summary & Prioritization

Sprint 1 (Week 1): Critical Production Readiness

1. Search Implementation (Story 1.1) - 8 pts
2. Distributed Tracing (Story 3.1) - 8 pts
3. Prometheus Metrics (Story 3.2) - 5 pts
4. Total: 21 points

Sprint 2 (Week 2): Performance & Documentation

1. API Documentation (Story 2.1) - 5 pts
2. Read Models/DTOs (Story 4.1) - 8 pts
3. Redis Caching (Story 4.2) - 5 pts
4. Structured Logging (Story 3.3) - 3 pts
5. Total: 21 points

Sprint 3 (Week 3): Deployment & Security

1. Production Deployment (Story 5.1) - 8 pts
2. Security Audit (Story 6.1) - 5 pts
3. Rate Limiting (Story 6.2) - 3 pts
4. Developer Docs (Story 2.2) - 3 pts
5. Total: 19 points

Sprint 4 (Week 4): Infrastructure & Polish

1. Kubernetes IaC (Story 5.2) - 8 pts
2. Disaster Recovery (Story 5.3) - 5 pts
3. Advanced Search Filters (Story 1.2) - 5 pts
4. Total: 18 points

Sprint 5 (Week 5): Optimization & DevEx

1. Database Optimization (Story 4.3) - 5 pts
2. Local Dev Environment (Story 7.1) - 3 pts
3. Testing Infrastructure (Story 7.2) - 5 pts
4. Total: 13 points

🎯 Success Metrics

Performance SLOs

• API response time p95 < 200ms
• Search response time p95 < 300ms
• Database query time p95 < 50ms
• Cache hit rate > 70%

Reliability SLOs

• Uptime > 99.9% (< 8.7 hours downtime/year)
• Error rate < 0.1%
• Mean Time To Recovery < 1 hour
• Zero data loss

Developer Experience

• Setup time < 15 minutes
• Test suite runs < 2 minutes
• Build time < 1 minute
• Documentation completeness > 90%

---

Next Steps:

1. Review and prioritize these tasks with the team
2. Create GitHub issues for Sprint 1 tasks
3. Add tasks to project board
4. Begin implementation starting with search and observability

This is a realistic, achievable roadmap based on the ACTUAL current state of the codebase! 🚀