turash/docs/implementation/ARCHITECTURAL_REFACTORING_PLAN.md

Architectural Refactoring Plan: From Math Model to Backend Foundation

Date: November 1, 2025 Status: Analysis Complete - Implementation Plan Ready Goal: Transform mathematical calculation engine into scalable backend foundation

Executive Summary

The current models package has evolved from a simple mathematical calculator into the core computational engine of the Turash platform. However, its architecture reflects its origins as a CLI math tool rather than a backend service foundation. This document outlines a comprehensive refactoring plan to introduce proper layered architecture, dependency injection, interfaces, and scalability patterns required for a production backend system.

Current Architecture Analysis

🏗️ Current Structure

models/
├── calc.go              # Core orchestration (monolithic)
├── params/              # Configuration structs (no DI)
├── customer/            # Business logic (direct calls)
├── revenue/             # Business logic (direct calls)
├── cost/                # Business logic (direct calls)
├── impact/              # Business logic (direct calls)
├── unit/                # Business logic (direct calls)
├── profitability/       # Business logic (direct calls)
├── transport/           # Business logic (direct calls)
├── match/               # Business logic (direct calls)
├── validator/           # Cross-cutting (mixed concerns)
├── cli/                 # Presentation layer (tightly coupled)
└── scenarios/           # Batch processing (direct calls)

🔍 Current Issues

1. Tight Coupling & Direct Dependencies

// Current: Direct function calls everywhere
custMetrics := customer.CalculateCustomerMetrics(year, p)
tierDist := customer.CalculateTierDistribution(year, custMetrics.PayingOrgs, p)
revBreakdown := revenue.CalculateRevenue(year, custMetrics, tierDist, p)

• Problem: Impossible to test in isolation, swap implementations, or mock dependencies

2. Mixed Concerns & Responsibilities

• Business logic, data access, validation, and presentation all mixed
• No separation between domain models and infrastructure
• Error handling inconsistent across packages

3. No Abstraction Layers

• No interfaces for different implementations
• Hard-coded algorithms and data sources
• Impossible to extend or modify without breaking changes

4. Configuration Management

• Parameters passed as structs everywhere
• No dependency injection container
• Configuration scattered across multiple places

5. Synchronous Request-Response Only

• No event-driven architecture
• No background processing capabilities
• No scalability patterns for high-throughput scenarios

---

🏛️ Proposed Architecture: Clean Architecture + DDD

Layered Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                    PRESENTATION LAYER                   │
│  ┌─────────────────────────────────────────────────┐   │
│  │ HTTP/GraphQL API │ CLI │ Event Handlers │ Jobs │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                   APPLICATION LAYER                     │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Use Cases │ Commands │ Queries │ Event Handlers │  │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                    DOMAIN LAYER                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │ Entities │ Value Objects │ Domain Services │    │   │
│  │ Events │ Repositories │ Specifications │       │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                 INFRASTRUCTURE LAYER                    │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Repositories │ External APIs │ Message Queues │   │
│  │  Cache │ Files │ Databases │ Event Store │     │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Package Structure Refactor

internal/
├── domain/
│   ├── model/              # Domain entities and value objects
│   │   ├── calculation.go  # Core domain model
│   │   ├── customer.go     # Customer aggregate
│   │   ├── resource.go     # Resource flow aggregate
│   │   ├── match.go        # Match aggregate
│   │   └── events.go       # Domain events
│   ├── service/            # Domain services
│   │   ├── calculation_service.go
│   │   ├── matching_service.go
│   │   └── validation_service.go
│   ├── repository/         # Repository interfaces
│   │   ├── calculation_repository.go
│   │   ├── customer_repository.go
│   │   └── match_repository.go
│   └── specification/      # Domain specifications
│       ├── customer_specs.go
│       └── match_specs.go
├── application/
│   ├── command/            # CQRS Commands
│   │   ├── calculate_model_command.go
│   │   ├── create_match_command.go
│   │   └── update_calculation_command.go
│   ├── query/              # CQRS Queries
│   │   ├── get_calculation_query.go
│   │   ├── list_matches_query.go
│   │   └── get_model_summary_query.go
│   ├── event/              # Event handlers
│   │   ├── calculation_completed_handler.go
│   │   └── match_created_handler.go
│   └── dto/                # Data transfer objects
│       ├── calculation_dto.go
│       └── match_dto.go
├── infrastructure/
│   ├── repository/         # Repository implementations
│   │   ├── postgres/
│   │   │   ├── calculation_repository.go
│   │   │   └── match_repository.go
│   │   ├── neo4j/
│   │   │   ├── graph_repository.go
│   │   │   └── matching_repository.go
│   │   └── in_memory/      # For testing
│   ├── messaging/          # Message queue implementations
│   │   ├── nats/
│   │   │   ├── publisher.go
│   │   │   └── consumer.go
│   │   └── kafka/          # For scale phase
│   ├── cache/              # Cache implementations
│   │   ├── redis/
│   │   └── in_memory/
│   ├── config/             # Configuration management
│   │   ├── config.go
│   │   └── environment.go
│   └── logging/            # Logging infrastructure
├── interfaces/
│   ├── http/               # HTTP API layer
│   │   ├── handlers/
│   │   ├── middleware/
│   │   └── router.go
│   ├── cli/                # CLI interface (refactored)
│   ├── graphql/            # GraphQL API (future)
│   └── events/             # Event handling
└── shared/
    ├── kernel/             # Dependency injection container
    ├── errors/             # Error handling
    ├── validation/         # Validation framework
    └── types/              # Shared types and utilities

---

🔧 Key Architectural Improvements

1. Dependency Injection & IoC Container

Current Problem:

// Tight coupling everywhere
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    custMetrics := customer.CalculateCustomerMetrics(year, p)
    // Direct dependency - can't mock, test, or swap
}

Proposed Solution:

// interfaces/calculation_service.go
type CalculationService interface {
    CalculateYear(ctx context.Context, year int) (*domain.CalculationResult, error)
    CalculateModel(ctx context.Context, req *application.CalculateModelRequest) (*domain.ModelResult, error)
}

// infrastructure/kernel/container.go
type Container struct {
    CalculationService application.CalculationService
    CustomerRepository  domain.CustomerRepository
    MatchRepository     domain.MatchRepository
    EventPublisher      messaging.EventPublisher
    Cache               cache.Cache
    Logger              logging.Logger
}

// Constructor with proper DI
func NewCalculationService(
    customerRepo domain.CustomerRepository,
    revenueSvc domain.RevenueService,
    costSvc domain.CostService,
    validator domain.ValidationService,
    publisher messaging.EventPublisher,
    cache cache.Cache,
    logger logging.Logger,
) CalculationService {
    return &calculationService{
        customerRepo: customerRepo,
        revenueSvc:   revenueSvc,
        costSvc:      costSvc,
        validator:    validator,
        publisher:   publisher,
        cache:       cache,
        logger:      logger,
    }
}

2. Domain-Driven Design (DDD) Entities

Current Problem:

// Flat structs with no behavior
type YearResult struct {
    Year       int                       `json:"year"`
    Customer   customer.CustomerMetrics  `json:"customer"`
    Revenue    revenue.RevenueBreakdown  `json:"revenue"`
    // No business logic, just data containers
}

Proposed Solution:

// domain/model/calculation.go
type Calculation struct {
    id          uuid.UUID
    year        int
    customer    *Customer
    revenue     *Revenue
    costs       *Costs
    impact      *Impact
    status      CalculationStatus
    createdAt   time.Time
    completedAt *time.Time
    events      []domain.Event
}

func (c *Calculation) CalculateProfit() (float64, error) {
    if c.revenue == nil || c.costs == nil {
        return 0, errors.New("revenue and costs must be calculated first")
    }
    return c.revenue.Total() - c.costs.Total(), nil
}

func (c *Calculation) MarkCompleted() error {
    if c.status != CalculationStatusInProgress {
        return errors.New("calculation not in progress")
    }
    c.status = CalculationStatusCompleted
    c.completedAt = &time.Time{}
    c.events = append(c.events, CalculationCompletedEvent{
        CalculationID: c.id,
        CompletedAt:   *c.completedAt,
    })
    return nil
}

3. Repository Pattern for Data Access

Current Problem:

// Direct data manipulation - no abstraction
func CalculateCustomerMetrics(year int, p *params.Params) CustomerMetrics {
    totalOrgs := p.Adoption.TotalOrgs.GetYear(year)
    // Direct parameter access - tightly coupled to data structure
}

Proposed Solution:

// domain/repository/customer_repository.go
type CustomerRepository interface {
    GetByYear(ctx context.Context, year int) (*Customer, error)
    GetAdoptionMetrics(ctx context.Context, year int) (*AdoptionMetrics, error)
    Save(ctx context.Context, customer *Customer) error
}

// infrastructure/repository/postgres/customer_repository.go
type postgresCustomerRepository struct {
	db     *gorm.DB
	logger logging.Logger
}

func (r *postgresCustomerRepository) GetByYear(ctx context.Context, year int) (*domain.Customer, error) {
	var customer domain.Customer
	
	err := r.db.WithContext(ctx).
		Where("year = ?", year).
		First(&customer).Error
	
	if err != nil {
		if errors.Is(err, gorm.ErrRecordNotFound) {
			return nil, domain.ErrCustomerNotFound
		}
		r.logger.Error("Failed to get customer by year", "year", year, "error", err)
		return nil, fmt.Errorf("failed to get customer: %w", err)
	}

	return &customer, nil
}

4. CQRS Pattern for Complex Operations

Current Problem:

// Single method does everything - hard to test, extend, monitor
func Calculate(p *params.Params) (*ModelResult, error) {
    // 50+ lines of mixed logic
    // Hard to debug, test, or extend
}

Proposed Solution:

// application/command/calculate_model_command.go
type CalculateModelCommand struct {
    Years          []int  `json:"years"`
    Scenario       string `json:"scenario,omitempty"`
    IncludeMatches bool   `json:"include_matches"`
}

type CalculateModelHandler struct {
    calculationSvc domain.CalculationService
    matchSvc       domain.MatchService
    eventPublisher messaging.EventPublisher
    cache          cache.Cache
}

func (h *CalculateModelHandler) Handle(ctx context.Context, cmd *CalculateModelCommand) (*application.CalculationResponse, error) {
    // Command validation
    if len(cmd.Years) == 0 {
        return nil, errors.New("years cannot be empty")
    }

    // Check cache first
    cacheKey := fmt.Sprintf("calculation:%v", cmd.Years)
    if cached, err := h.cache.Get(cacheKey); err == nil {
        return cached.(*application.CalculationResponse), nil
    }

    // Execute business logic
    result, err := h.calculationSvc.CalculateModel(ctx, cmd.Years)
    if err != nil {
        return nil, fmt.Errorf("calculation failed: %w", err)
    }

    // Include matches if requested
    if cmd.IncludeMatches {
        matches, err := h.matchSvc.FindMatches(ctx, cmd.Years)
        if err != nil {
            h.eventPublisher.Publish(ctx, MatchCalculationFailedEvent{
                Years:  cmd.Years,
                Reason: err.Error(),
            })
            return nil, fmt.Errorf("match calculation failed: %w", err)
        }
        result.Matches = matches
    }

    // Publish success event
    h.eventPublisher.Publish(ctx, ModelCalculationCompletedEvent{
        Years:       cmd.Years,
        Result:      result,
        CompletedAt: time.Now(),
    })

    // Cache result
    h.cache.Set(cacheKey, result, 1*time.Hour)

    return result, nil
}

5. Event-Driven Architecture

Current Problem:

// Everything synchronous - no background processing
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    // Synchronous calculation only
}

Proposed Solution:

// domain/events.go
type CalculationCompletedEvent struct {
    CalculationID uuid.UUID `json:"calculation_id"`
    Year         int        `json:"year"`
    Result       *CalculationResult `json:"result"`
    CompletedAt  time.Time  `json:"completed_at"`
}

// interfaces/events/calculation_handler.go
type CalculationCompletedHandler struct {
    notificationSvc notification.Service
    analyticsSvc    analytics.Service
}

func (h *CalculationCompletedHandler) Handle(ctx context.Context, event *domain.CalculationCompletedEvent) error {
    // Send notifications asynchronously
    go h.notificationSvc.NotifyCalculationComplete(ctx, event.CalculationID)

    // Update analytics asynchronously
    go h.analyticsSvc.RecordCalculationMetrics(ctx, event.Result)

    // Trigger dependent calculations
    if event.Result.RequiresMatchAnalysis {
        h.eventPublisher.Publish(ctx, TriggerMatchAnalysisEvent{
            CalculationID: event.CalculationID,
            Year:         event.Year,
        })
    }

    return nil
}

6. Configuration Management

Current Problem:

// Parameters scattered and passed everywhere
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    // p is passed through every function
}

Proposed Solution:

// infrastructure/config/config.go
type Config struct {
    Database    DatabaseConfig    `yaml:"database"`
    Cache       CacheConfig       `yaml:"cache"`
    Messaging   MessagingConfig   `yaml:"messaging"`
    Calculation CalculationConfig `yaml:"calculation"`
    Matching    MatchingConfig    `yaml:"matching"`
}

type CalculationConfig struct {
    DefaultDiscountRate     float64 `yaml:"default_discount_rate"`
    MaxYears               int     `yaml:"max_years"`
    CacheEnabled           bool    `yaml:"cache_enabled"`
    AsyncProcessingEnabled bool    `yaml:"async_processing_enabled"`
}

// infrastructure/kernel/container.go
func NewContainer(cfg *config.Config) (*Container, error) {
    // Initialize all dependencies based on configuration
    db, err := setupDatabase(cfg.Database)
    if err != nil {
        return nil, fmt.Errorf("database setup failed: %w", err)
    }

    cache, err := setupCache(cfg.Cache)
    if err != nil {
        return nil, fmt.Errorf("cache setup failed: %w", err)
    }

    // Wire all services with proper dependencies
    customerRepo := postgres.NewCustomerRepository(db, logger)
    calculationSvc := application.NewCalculationService(
        customerRepo,
        cache,
        cfg.Calculation,
        logger,
    )

    return &Container{
        CalculationService: calculationSvc,
        CustomerRepository: customerRepo,
        // ... other services
    }, nil
}

---

📦 Dependency Stack & Technology Choices

Final Dependency List (2025 Recommendations)

Based on documentation requirements and 2025 Go best practices, here is the comprehensive dependency stack:

Key Technology Decisions:

• ✅ Echo v4 - HTTP framework (clean API, excellent middleware)
• ✅ GORM - ORM for PostgreSQL (associations, hooks, query builder)
• ✅ golang-migrate/v4 - Database migrations (SQL-based, version control, rollback support)
• ⚠️ DO NOT use GORM AutoMigrate - Use golang-migrate for production migrations

1. Core Framework & HTTP

// HTTP Framework
github.com/labstack/echo/v4 v4.14.0     // ✅ FINAL CHOICE: Echo - Clean API, excellent middleware, good performance
// Decision: Echo chosen for clean API design and middleware support
// Alternative considered: Gin (more mature ecosystem), Fiber (lower latency)
// Rationale: Echo provides better middleware chaining and cleaner handler signatures

// Dependency Injection (2025 Recommended)
go.uber.org/fx v1.24.0                   // ✅ FINAL CHOICE: Uber's fx - Clean, functional DI
// Alternatives considered:
//   - github.com/google/wire (code generation) - Too verbose for MVP
//   - github.com/samber/do (minimal) - Too simple for complex needs
// Rationale: fx provides lifecycle hooks, graceful shutdown, and clean functional composition

// Configuration Management
github.com/spf13/viper v1.21.0          // ✅ FINAL CHOICE: YAML/JSON/ENV support
github.com/spf13/cobra v1.10.1          // Already in use - CLI framework

2. Database & Data Access

// Graph Database (Primary)
github.com/neo4j/neo4j-go-driver/v5 v5.23.0  // ✅ Neo4j driver with connection pooling

// ORM (Relational Database)
gorm.io/gorm v1.25.12                 // ✅ FINAL CHOICE: GORM - Full-featured ORM with migrations, associations, hooks
gorm.io/driver/postgres v1.5.9        // ✅ GORM PostgreSQL driver (uses pgx/v5 under the hood)
// Decision: GORM chosen for ActiveRecord-style ORM, automatic migrations, associations
// Rationale: GORM provides excellent developer experience with hooks, associations, and migrations
// Alternative considered: pgx/v5 direct (more control, less convenience) - GORM chosen for productivity

// PostgreSQL Driver (Used by GORM)
github.com/jackc/pgx/v5 v5.7.2        // ✅ Used by GORM driver - Best performance PostgreSQL driver
// Note: GORM uses pgx/v5 as the underlying driver for optimal performance

// Spatial/Geographic Support
github.com/twpayne/go-geom v1.6.0      // PostGIS support for geographic queries

// Database Migrations
github.com/golang-migrate/migrate/v4 v4.18.1 // ✅ FINAL CHOICE: golang-migrate - Industry standard, SQL-based migrations
// Decision: Use golang-migrate instead of GORM AutoMigrate for production-grade migrations
// Rationale: SQL-based migrations provide version control, rollback support, and full control over schema changes
// Features: Supports PostgreSQL, Neo4j (via Cypher), version control, up/down migrations, CLI tool
// Alternative considered: pressly/goose (good DX but golang-migrate is more widely adopted)
// Note: GORM AutoMigrate should NOT be used in production - only for rapid prototyping

3. Caching & Message Queue

// Cache (MVP)
github.com/redis/go-redis/v9 v9.7.0     // ✅ FINAL CHOICE: Official Redis client, fast and reliable

// Message Queue (MVP)
github.com/nats-io/nats.go v1.35.0     // ✅ FINAL CHOICE: Go-native, 60% simpler than Kafka
// Migration path: github.com/segmentio/kafka-go v0.4.48 (at 1000+ business scale)

// Distributed Task Queue
github.com/hibiken/asynq v0.24.1        // ✅ Redis-based task queue for background jobs

4. Validation & Error Handling

// Validation
github.com/go-playground/validator/v10 v10.22.1  // ✅ FINAL CHOICE: Struct tags, widely used
// Go 1.25 generics can supplement for type-safe validation

// Error Handling
github.com/pkg/errors v0.9.1            // ✅ Enhanced error wrapping and stack traces
// Standard errors package for Go 1.25 error handling features

5. Logging & Observability

// Logging (2025 Recommended)
github.com/rs/zerolog v1.33.0           // ✅ FINAL CHOICE: Fast structured logging, zero allocations
// Alternative considered: logrus - zerolog is 10x faster

// Observability
go.opentelemetry.io/otel v1.33.0        // ✅ FINAL CHOICE: Industry standard observability
go.opentelemetry.io/otel/trace v1.33.0
go.opentelemetry.io/otel/metric v1.33.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0

// Metrics
github.com/prometheus/client_golang v1.20.5    // ✅ Prometheus metrics integration

6. Testing Framework

// Testing
github.com/stretchr/testify v1.9.0       // ✅ Already in use - assertions and mocks
github.com/golang/mock v1.6.0            // ✅ Mock generation from interfaces
// Alternative: github.com/vektra/mockery/v2 - testify/mock is simpler

// Test Containers (Integration Testing)
github.com/testcontainers/testcontainers-go v0.34.0  // ✅ Docker-based integration tests
github.com/testcontainers/testcontainers-go/modules/postgres v0.34.0
github.com/testcontainers/testcontainers-go/modules/neo4j v0.34.0

7. API & GraphQL

// GraphQL (Future)
github.com/99designs/gqlgen v0.17.47     // ✅ FINAL CHOICE: Schema-first, code generation
// Alternative considered: graphql-go (runtime-first) - gqlgen is more type-safe

// WebSocket
nhooyr.io/websocket v1.8.11              // ✅ FINAL CHOICE: Fast, minimal dependencies
// Alternative considered: gorilla/websocket - nhooyr is more modern and faster

// HTTP Client
github.com/go-resty/resty/v2 v2.14.0    // ✅ Convenient HTTP client (optional, for external APIs)

8. Event-Driven & Background Processing

// Event Sourcing (Future)
github.com/EventStore/EventStore-Client-Go v1.4.1  // Optional: For event sourcing

// Background Jobs
github.com/hibiken/asynq v0.24.1        // Already listed above - Redis-based task queue

// Rate Limiting
golang.org/x/time v0.10.0               // Rate limiting for API endpoints

9. Security & Authentication

// JWT Authentication
github.com/golang-jwt/jwt/v5 v5.2.1     // ✅ FINAL CHOICE: Official JWT library (v5 is latest)
// Alternative: github.com/form3tech-oss/jwt-go - JWT v5 is official and maintained

// Password Hashing
golang.org/x/crypto v0.35.0              // ✅ bcrypt and other crypto functions

// OAuth2
golang.org/x/oauth2 v0.28.0             // ✅ OAuth2 client library

10. Configuration & Serialization

// YAML (Already in use)
gopkg.in/yaml.v3 v3.0.1                 // ✅ Already in use

// JSON (Go 1.25 Experimental)
// encoding/json/v2                       // ✅ Use if Go 1.25 experimental features stable
// Fallback: standard encoding/json

// UUID Generation
github.com/google/uuid v1.6.0            // ✅ UUID generation for entities

11. Utilities & Helpers

// Time Utilities
github.com/jonboulle/clockwork v0.4.0    // ✅ Time mocking for tests

// String Utilities
github.com/google/go-cmp v0.6.0         // ✅ Better diffing for tests

// Context & Timeouts
golang.org/x/sync v0.11.0               // ✅ errgroup, semaphore for concurrency

12. Development Tools

// Code Generation
github.com/google/wire v0.6.0           // Optional: For advanced DI code generation

// Linting
// golangci-lint (external tool)          // ✅ Recommended: Use in CI/CD

// Documentation
github.com/swaggo/swag v1.16.3          // ✅ Swagger documentation generation
github.com/swaggo/echo-swagger v1.4.1    // ✅ Echo Swagger integration

📋 Complete go.mod (MVP Phase)

module github.com/damirmukimov/city_resource_graph

go 1.25.3

require (
    // Core Framework
    github.com/labstack/echo/v4 v4.14.0
    go.uber.org/fx v1.24.0
    
    // Database
    github.com/neo4j/neo4j-go-driver/v5 v5.23.0
    gorm.io/gorm v1.25.12
    gorm.io/driver/postgres v1.5.9
    github.com/jackc/pgx/v5 v5.7.2        // Used by GORM driver
    github.com/twpayne/go-geom v1.6.0
    
    // Database Migrations (REQUIRED - Do NOT use GORM AutoMigrate)
    github.com/golang-migrate/migrate/v4 v4.18.1  // ✅ SQL-based migrations with version control
    
    // Cache & Messaging
    github.com/redis/go-redis/v9 v9.7.0
    github.com/nats-io/nats.go v1.35.0
    github.com/hibiken/asynq v0.24.1
    
    // Validation & Errors
    github.com/go-playground/validator/v10 v10.22.1
    github.com/pkg/errors v0.9.1
    
    // Logging & Observability
    github.com/rs/zerolog v1.33.0
    go.opentelemetry.io/otel v1.33.0
    go.opentelemetry.io/otel/trace v1.33.0
    go.opentelemetry.io/otel/metric v1.33.0
    github.com/prometheus/client_golang v1.20.5
    
    // Testing
    github.com/stretchr/testify v1.9.0
    github.com/golang/mock v1.6.0
    github.com/testcontainers/testcontainers-go v0.34.0
    
    // API & WebSocket
    nhooyr.io/websocket v1.8.11
    github.com/go-resty/resty/v2 v2.14.0
    
    // Security
    github.com/golang-jwt/jwt/v5 v5.2.1
    golang.org/x/crypto v0.35.0
    golang.org/x/oauth2 v0.28.0
    
    // Configuration
    github.com/spf13/viper v1.21.0
    github.com/spf13/cobra v1.10.1
    gopkg.in/yaml.v3 v3.0.1
    
    // Utilities
    github.com/google/uuid v1.6.0
    github.com/jonboulle/clockwork v0.4.0
    golang.org/x/sync v0.11.0
    golang.org/x/time v0.10.0
)

// Development-only dependencies
require (
    github.com/swaggo/swag v1.16.3
    github.com/swaggo/echo-swagger v1.4.1
)

🎯 Technology Decision Matrix

Category	Choice	Rationale	Alternatives Considered
HTTP Framework	Echo v4	✅ FINAL: Clean API, excellent middleware chaining, good performance	Gin (mature ecosystem), Fiber (lower latency) - Echo chosen for cleaner API
DI Container	fx (Uber)	Functional composition, lifecycle hooks, graceful shutdown	wire (too verbose), dig (deprecated)
Graph DB Driver	neo4j-go-driver/v5	Official, connection pooling, transaction support	Official driver only viable option
ORM	GORM	✅ FINAL: Full-featured ORM with associations, hooks, query builder - Excellent developer experience	pgx/v5 direct (more control, less convenience) - GORM chosen for productivity
Migration Tool	golang-migrate/v4	✅ FINAL: Industry standard, SQL-based migrations, version control, rollback support	GORM AutoMigrate (NOT for production), pressly/goose (good DX but less adopted)
PostgreSQL Driver	pgx/v5	Used by GORM driver - 30% faster than database/sql, PostGIS support	database/sql + pq (standard but slower)
Cache	go-redis/v9	Official client, fastest, reliable	redigo (older), rueidis (newer but less stable)
Message Queue (MVP)	NATS	Go-native, 60% simpler than Kafka, perfect for MVP	Kafka (too complex for MVP), Redis Streams (limited features)
Logging	zerolog	Zero allocations, 10x faster than logrus	logrus (slower), zap (more complex)
Validation	validator/v10	Struct tags, widely adopted, comprehensive	custom generics (future enhancement)
Testing	testify + golang/mock	Simple, widely used, good mocking	mockery (more features but more complex)
WebSocket	nhooyr.io/websocket	Fast, minimal, modern	gorilla/websocket (older, more dependencies)
Observability	OpenTelemetry	Industry standard, vendor-agnostic	Prometheus-only (less flexible)

🔧 Echo & GORM Integration Notes

Echo Framework Benefits

• Clean Handler Signatures: func Handler(c echo.Context) error - consistent error handling
• Middleware Chaining: Built-in support for CORS, JWT, rate limiting, logging
• Context Propagation: Echo's context extends Go's context.Context for request-scoped values
• Performance: Excellent performance with minimal overhead
• Validation Integration: Works seamlessly with validator/v10 for struct validation

GORM Benefits

• Associations: Built-in support for has-one, has-many, many-to-many relationships
• Hooks: BeforeSave, AfterCreate, etc. for business logic triggers
• Query Builder: Fluent API for complex queries: db.Where().Joins().Preload()
• Performance: Uses pgx/v5 under the hood, so no performance penalty
• Transaction Support: Built-in transaction management with rollback support
• ⚠️ AutoMigrate Note: AutoMigrate should NOT be used in production - use golang-migrate for schema management

golang-migrate Benefits (Migration Management)

• SQL-Based Migrations: Write explicit SQL migrations for full control
• Version Control: Track migration versions and status
• Rollback Support: Automatic down migrations for safe rollbacks
• Multiple Databases: Supports PostgreSQL, Neo4j (Cypher), MySQL, SQLite, etc.
• CLI Tool: migrate command-line tool for easy migration management
• Library API: Programmatic migration support for automated deployments
• Developer Experience:
  • Simple file naming: 000001_create_customers.up.sql and 000001_create_customers.down.sql
  • Migration verification before execution
  • Force version support for fixing migration issues
  • Dry-run mode for testing migrations

Migration Workflow with golang-migrate

// Example: Migration setup and execution
import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

// Migration directory structure:
// migrations/
//   ├── 000001_create_customers.up.sql
//   ├── 000001_create_customers.down.sql
//   ├── 000002_add_customer_indexes.up.sql
//   ├── 000002_add_customer_indexes.down.sql
//   └── ...

func RunMigrations(databaseURL string, migrationsPath string) error {
    m, err := migrate.New(
        "file://"+migrationsPath,
        databaseURL,
    )
    if err != nil {
        return fmt.Errorf("failed to initialize migrations: %w", err)
    }
    defer m.Close()

    // Run all pending migrations
    if err := m.Up(); err != nil {
        if err == migrate.ErrNoChange {
            // No pending migrations - this is OK
            return nil
        }
        return fmt.Errorf("failed to run migrations: %w", err)
    }

    return nil
}

// Example SQL migration files:
// 000001_create_customers.up.sql:
// CREATE TABLE customers (
//     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
//     year INTEGER NOT NULL,
//     total_orgs INTEGER NOT NULL,
//     paying_orgs INTEGER NOT NULL,
//     created_at TIMESTAMP DEFAULT NOW(),
//     updated_at TIMESTAMP DEFAULT NOW()
// );
// CREATE INDEX idx_customers_year ON customers(year);

// 000001_create_customers.down.sql:
// DROP INDEX IF EXISTS idx_customers_year;
// DROP TABLE IF EXISTS customers;

golang-migrate CLI Usage

# Install golang-migrate CLI
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

# Create a new migration
migrate create -ext sql -dir migrations -seq create_customers

# Run all pending migrations
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" up

# Rollback one migration
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" down 1

# Check migration status
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" version

# Force version (for fixing migration issues)
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" force 1

Migration Best Practices

• ✅ Always create up AND down migrations - Enables safe rollbacks
• ✅ Version migrations sequentially - Use sequential numbering: 000001, 000002, etc.
• ✅ Keep migrations idempotent - Use IF NOT EXISTS and IF EXISTS clauses
• ✅ Test migrations - Test both up and down migrations before deploying
• ✅ Review SQL before committing - Never auto-generate migrations blindly
• ✅ Keep migrations small - One logical change per migration
• ✅ Use transactions - Wrap migrations in transactions when possible (PostgreSQL)
• ⚠️ Never modify existing migrations - Create new migrations to fix issues

Architecture Integration

// Example: Repository pattern with GORM
import (
    "context"
    "errors"
    "fmt"
    
    "gorm.io/gorm"
)

type customerRepository struct {
    db *gorm.DB
    logger logging.Logger
}

func (r *customerRepository) GetByYear(ctx context.Context, year int) (*domain.Customer, error) {
    var customer domain.Customer
    err := r.db.WithContext(ctx).
        Where("year = ?", year).
        First(&customer).Error
    
    if err != nil {
        if errors.Is(err, gorm.ErrRecordNotFound) {
            return nil, domain.ErrCustomerNotFound
        }
        r.logger.Error("Failed to get customer", "year", year, "error", err)
        return nil, fmt.Errorf("failed to get customer: %w", err)
    }
    
    return &customer, nil
}

// Echo handler integration
import (
    "net/http"
    "strconv"
    
    "github.com/labstack/echo/v4"
)

func (h *customerHandler) GetCustomer(c echo.Context) error {
    year, err := strconv.Atoi(c.Param("year"))
    if err != nil {
        return echo.NewHTTPError(http.StatusBadRequest, "invalid year")
    }
    
    customer, err := h.customerRepo.GetByYear(c.Request().Context(), year)
    if err != nil {
        if errors.Is(err, domain.ErrCustomerNotFound) {
            return echo.NewHTTPError(http.StatusNotFound, "customer not found")
        }
        return echo.NewHTTPError(http.StatusInternalServerError, "failed to get customer")
    }
    
    return c.JSON(http.StatusOK, customer)
}

// Echo router setup with middleware
func setupEchoRouter(h *customerHandler) *echo.Echo {
    e := echo.New()
    
    // Middleware
    e.Use(middleware.Logger())
    e.Use(middleware.Recover())
    e.Use(middleware.CORS())
    
    // Routes
    api := e.Group("/api/v1")
    api.GET("/customers/:year", h.GetCustomer)
    
    return e
}

📈 Migration Path for Scale Phase

When reaching 1000+ businesses:

• Kafka: Replace NATS with github.com/segmentio/kafka-go v0.4.48
• TimescaleDB: Add github.com/influxdata/influxdb-client-go/v2 for time-series
• Service Mesh: Consider istio or linkerd for inter-service communication
• GORM: Continue using GORM - it scales well with connection pooling and query optimization

---

📋 Implementation Roadmap

Phase 1: Foundation (2 weeks)

1. Create Domain Layer

   • Define domain entities (Customer, Calculation, Match)
   • Create value objects and domain events
   • Define repository interfaces

2. Setup Infrastructure

   • Create DI container structure with fx
   • Implement configuration management with viper
   • Setup logging with zerolog and error handling
   • Setup Database Migrations with golang-migrate
     • Create migrations directory structure
     • Setup initial PostgreSQL migrations (customers, calculations, matches)
     • Setup Neo4j migrations (graph schema, constraints, indexes)
     • Create migration runner service with DI integration

3. Create Repository Pattern

   • Implement in-memory repositories for current data
   • Define repository interfaces
   • Migrate parameter loading to repositories
   • ⚠️ DO NOT use GORM AutoMigrate - Use golang-migrate for all schema changes

Phase 2: Application Layer (3 weeks)

1. CQRS Implementation

   • Create command/query handlers
   • Implement use cases
   • Add event handlers

2. Service Layer Refactor

   • Extract business logic into domain services
   • Implement calculation orchestrator
   • Add validation services

3. Event-Driven Architecture

   • Implement event publishing
   • Create event handlers
   • Setup async processing

Phase 3: Interface Layer (2 weeks)

1. HTTP API

   • RESTful endpoints for calculations
   • GraphQL API for complex queries
   • Proper error responses

2. CLI Refactor

   • Use DI container in CLI
   • Implement command handlers
   • Add interactive mode

3. Event Processing

   • Background job processing
   • Event sourcing for audit trails
   • Real-time notifications

Phase 4: Testing & Documentation (1 week)

1. Integration Tests

   • End-to-end test scenarios
   • Performance testing
   • Load testing

2. Documentation

   • API documentation
   • Architecture decision records
   • Deployment guides

---

🔄 Migration Strategy

Incremental Migration

1. Week 1-2: Create new architecture alongside existing code
2. Week 3-4: Migrate leaf packages (params, validator) to new structure
3. Week 5-6: Migrate calculation logic with adapters
4. Week 7-8: Replace CLI and add HTTP API
5. Week 9-10: Full integration and testing

Backward Compatibility

• Keep existing CLI working during migration
• Create adapter layer for gradual migration
• Maintain existing JSON schemas and outputs

Testing Strategy

• Unit Tests: Test each layer in isolation with mocks
• Integration Tests: Test layer interactions
• E2E Tests: Test complete workflows
• Performance Tests: Ensure no regression in calculation speed

---

🎯 Benefits of New Architecture

Technical Benefits

• ✅ Testability: Each component can be tested in isolation
• ✅ Maintainability: Clear separation of concerns
• ✅ Extensibility: Easy to add new calculation methods or data sources
• ✅ Scalability: Event-driven architecture supports high throughput
• ✅ Reliability: Proper error handling and monitoring

Business Benefits

• ✅ Time-to-Market: Faster feature development with modular design
• ✅ Quality: Better testing coverage and error handling
• ✅ Scalability: Support for 1000+ businesses with event-driven processing
• ✅ Maintainability: Easier to modify and extend without breaking changes

Development Benefits

• ✅ Developer Experience: Clear interfaces and dependency injection
• ✅ Code Reuse: Domain services can be used across different interfaces
• ✅ Debugging: Better error tracing and logging
• ✅ Onboarding: Clear architectural boundaries for new developers

---

📊 Risk Assessment & Mitigation

Risk	Probability	Impact	Mitigation
Migration Complexity	Medium	High	Incremental migration with adapters
Performance Regression	Low	Medium	Comprehensive performance testing
Breaking Changes	Medium	High	Maintain backward compatibility layer
Team Learning Curve	Medium	Low	Training sessions and documentation
Increased Complexity	High	Low	Start simple, add complexity gradually

---

🏁 Success Metrics

• Code Coverage: >90% unit test coverage
• Performance: No regression in calculation speed (<10% impact)
• Maintainability: Cyclomatic complexity <10 per function
• Scalability: Support 10x current load with same infrastructure
• Reliability: 99.9% uptime with proper error handling

---

This architectural refactoring transforms the mathematical model into a production-ready backend foundation that can scale to support thousands of businesses while maintaining code quality and developer productivity.