turash/docs/implementation/ARCHITECTURAL_REFACTORING_PLAN.md

# Architectural Refactoring Plan: From Math Model to Backend Foundation

**Date:** November 1, 2025
**Status:** Analysis Complete - Implementation Plan Ready
**Goal:** Transform mathematical calculation engine into scalable backend foundation

## Executive Summary

The current `models` package has evolved from a simple mathematical calculator into the core computational engine of the Turash platform. However, its architecture reflects its origins as a CLI math tool rather than a backend service foundation. This document outlines a comprehensive refactoring plan to introduce proper layered architecture, dependency injection, interfaces, and scalability patterns required for a production backend system.

## Current Architecture Analysis

### 🏗️ **Current Structure**
```
models/
├── calc.go              # Core orchestration (monolithic)
├── params/              # Configuration structs (no DI)
├── customer/            # Business logic (direct calls)
├── revenue/             # Business logic (direct calls)
├── cost/                # Business logic (direct calls)
├── impact/              # Business logic (direct calls)
├── unit/                # Business logic (direct calls)
├── profitability/       # Business logic (direct calls)
├── transport/           # Business logic (direct calls)
├── match/               # Business logic (direct calls)
├── validator/           # Cross-cutting (mixed concerns)
├── cli/                 # Presentation layer (tightly coupled)
└── scenarios/           # Batch processing (direct calls)
```

### 🔍 **Current Issues**

#### 1. **Tight Coupling & Direct Dependencies**
```go
// Current: Direct function calls everywhere
custMetrics := customer.CalculateCustomerMetrics(year, p)
tierDist := customer.CalculateTierDistribution(year, custMetrics.PayingOrgs, p)
revBreakdown := revenue.CalculateRevenue(year, custMetrics, tierDist, p)
```
- **Problem**: Impossible to test in isolation, swap implementations, or mock dependencies

#### 2. **Mixed Concerns & Responsibilities**
- Business logic, data access, validation, and presentation all mixed
- No separation between domain models and infrastructure
- Error handling inconsistent across packages

#### 3. **No Abstraction Layers**
- No interfaces for different implementations
- Hard-coded algorithms and data sources
- Impossible to extend or modify without breaking changes

#### 4. **Configuration Management**
- Parameters passed as structs everywhere
- No dependency injection container
- Configuration scattered across multiple places

#### 5. **Synchronous Request-Response Only**
- No event-driven architecture
- No background processing capabilities
- No scalability patterns for high-throughput scenarios

---

## 🏛️ **Proposed Architecture: Clean Architecture + DDD**

### **Layered Architecture Overview**

```
┌─────────────────────────────────────────────────────────┐
│                    PRESENTATION LAYER                   │
│  ┌─────────────────────────────────────────────────┐   │
│  │ HTTP/GraphQL API │ CLI │ Event Handlers │ Jobs │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                   APPLICATION LAYER                     │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Use Cases │ Commands │ Queries │ Event Handlers │  │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                    DOMAIN LAYER                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │ Entities │ Value Objects │ Domain Services │    │   │
│  │ Events │ Repositories │ Specifications │       │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
                                │
┌─────────────────────────────────────────────────────────┐
│                 INFRASTRUCTURE LAYER                    │
│  ┌─────────────────────────────────────────────────┐   │
│  │  Repositories │ External APIs │ Message Queues │   │
│  │  Cache │ Files │ Databases │ Event Store │     │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘
```

### **Package Structure Refactor**

```
internal/
├── domain/
│   ├── model/              # Domain entities and value objects
│   │   ├── calculation.go  # Core domain model
│   │   ├── customer.go     # Customer aggregate
│   │   ├── resource.go     # Resource flow aggregate
│   │   ├── match.go        # Match aggregate
│   │   └── events.go       # Domain events
│   ├── service/            # Domain services
│   │   ├── calculation_service.go
│   │   ├── matching_service.go
│   │   └── validation_service.go
│   ├── repository/         # Repository interfaces
│   │   ├── calculation_repository.go
│   │   ├── customer_repository.go
│   │   └── match_repository.go
│   └── specification/      # Domain specifications
│       ├── customer_specs.go
│       └── match_specs.go
├── application/
│   ├── command/            # CQRS Commands
│   │   ├── calculate_model_command.go
│   │   ├── create_match_command.go
│   │   └── update_calculation_command.go
│   ├── query/              # CQRS Queries
│   │   ├── get_calculation_query.go
│   │   ├── list_matches_query.go
│   │   └── get_model_summary_query.go
│   ├── event/              # Event handlers
│   │   ├── calculation_completed_handler.go
│   │   └── match_created_handler.go
│   └── dto/                # Data transfer objects
│       ├── calculation_dto.go
│       └── match_dto.go
├── infrastructure/
│   ├── repository/         # Repository implementations
│   │   ├── postgres/
│   │   │   ├── calculation_repository.go
│   │   │   └── match_repository.go
│   │   ├── neo4j/
│   │   │   ├── graph_repository.go
│   │   │   └── matching_repository.go
│   │   └── in_memory/      # For testing
│   ├── messaging/          # Message queue implementations
│   │   ├── nats/
│   │   │   ├── publisher.go
│   │   │   └── consumer.go
│   │   └── kafka/          # For scale phase
│   ├── cache/              # Cache implementations
│   │   ├── redis/
│   │   └── in_memory/
│   ├── config/             # Configuration management
│   │   ├── config.go
│   │   └── environment.go
│   └── logging/            # Logging infrastructure
├── interfaces/
│   ├── http/               # HTTP API layer
│   │   ├── handlers/
│   │   ├── middleware/
│   │   └── router.go
│   ├── cli/                # CLI interface (refactored)
│   ├── graphql/            # GraphQL API (future)
│   └── events/             # Event handling
└── shared/
    ├── kernel/             # Dependency injection container
    ├── errors/             # Error handling
    ├── validation/         # Validation framework
    └── types/              # Shared types and utilities
```

---

## 🔧 **Key Architectural Improvements**

### **1. Dependency Injection & IoC Container**

#### **Current Problem:**
```go
// Tight coupling everywhere
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    custMetrics := customer.CalculateCustomerMetrics(year, p)
    // Direct dependency - can't mock, test, or swap
}
```

#### **Proposed Solution:**
```go
// interfaces/calculation_service.go
type CalculationService interface {
    CalculateYear(ctx context.Context, year int) (*domain.CalculationResult, error)
    CalculateModel(ctx context.Context, req *application.CalculateModelRequest) (*domain.ModelResult, error)
}

// infrastructure/kernel/container.go
type Container struct {
    CalculationService application.CalculationService
    CustomerRepository  domain.CustomerRepository
    MatchRepository     domain.MatchRepository
    EventPublisher      messaging.EventPublisher
    Cache               cache.Cache
    Logger              logging.Logger
}

// Constructor with proper DI
func NewCalculationService(
    customerRepo domain.CustomerRepository,
    revenueSvc domain.RevenueService,
    costSvc domain.CostService,
    validator domain.ValidationService,
    publisher messaging.EventPublisher,
    cache cache.Cache,
    logger logging.Logger,
) CalculationService {
    return &calculationService{
        customerRepo: customerRepo,
        revenueSvc:   revenueSvc,
        costSvc:      costSvc,
        validator:    validator,
        publisher:   publisher,
        cache:       cache,
        logger:      logger,
    }
}
```

### **2. Domain-Driven Design (DDD) Entities**

#### **Current Problem:**
```go
// Flat structs with no behavior
type YearResult struct {
    Year       int                       `json:"year"`
    Customer   customer.CustomerMetrics  `json:"customer"`
    Revenue    revenue.RevenueBreakdown  `json:"revenue"`
    // No business logic, just data containers
}
```

#### **Proposed Solution:**
```go
// domain/model/calculation.go
type Calculation struct {
    id          uuid.UUID
    year        int
    customer    *Customer
    revenue     *Revenue
    costs       *Costs
    impact      *Impact
    status      CalculationStatus
    createdAt   time.Time
    completedAt *time.Time
    events      []domain.Event
}

func (c *Calculation) CalculateProfit() (float64, error) {
    if c.revenue == nil || c.costs == nil {
        return 0, errors.New("revenue and costs must be calculated first")
    }
    return c.revenue.Total() - c.costs.Total(), nil
}

func (c *Calculation) MarkCompleted() error {
    if c.status != CalculationStatusInProgress {
        return errors.New("calculation not in progress")
    }
    c.status = CalculationStatusCompleted
    c.completedAt = &time.Time{}
    c.events = append(c.events, CalculationCompletedEvent{
        CalculationID: c.id,
        CompletedAt:   *c.completedAt,
    })
    return nil
}
```

### **3. Repository Pattern for Data Access**

#### **Current Problem:**
```go
// Direct data manipulation - no abstraction
func CalculateCustomerMetrics(year int, p *params.Params) CustomerMetrics {
    totalOrgs := p.Adoption.TotalOrgs.GetYear(year)
    // Direct parameter access - tightly coupled to data structure
}
```

#### **Proposed Solution:**
```go
// domain/repository/customer_repository.go
type CustomerRepository interface {
    GetByYear(ctx context.Context, year int) (*Customer, error)
    GetAdoptionMetrics(ctx context.Context, year int) (*AdoptionMetrics, error)
    Save(ctx context.Context, customer *Customer) error
}

// infrastructure/repository/postgres/customer_repository.go
type postgresCustomerRepository struct {
	db     *gorm.DB
	logger logging.Logger
}

func (r *postgresCustomerRepository) GetByYear(ctx context.Context, year int) (*domain.Customer, error) {
	var customer domain.Customer
	
	err := r.db.WithContext(ctx).
		Where("year = ?", year).
		First(&customer).Error
	
	if err != nil {
		if errors.Is(err, gorm.ErrRecordNotFound) {
			return nil, domain.ErrCustomerNotFound
		}
		r.logger.Error("Failed to get customer by year", "year", year, "error", err)
		return nil, fmt.Errorf("failed to get customer: %w", err)
	}

	return &customer, nil
}
```

### **4. CQRS Pattern for Complex Operations**

#### **Current Problem:**
```go
// Single method does everything - hard to test, extend, monitor
func Calculate(p *params.Params) (*ModelResult, error) {
    // 50+ lines of mixed logic
    // Hard to debug, test, or extend
}
```

#### **Proposed Solution:**
```go
// application/command/calculate_model_command.go
type CalculateModelCommand struct {
    Years          []int  `json:"years"`
    Scenario       string `json:"scenario,omitempty"`
    IncludeMatches bool   `json:"include_matches"`
}

type CalculateModelHandler struct {
    calculationSvc domain.CalculationService
    matchSvc       domain.MatchService
    eventPublisher messaging.EventPublisher
    cache          cache.Cache
}

func (h *CalculateModelHandler) Handle(ctx context.Context, cmd *CalculateModelCommand) (*application.CalculationResponse, error) {
    // Command validation
    if len(cmd.Years) == 0 {
        return nil, errors.New("years cannot be empty")
    }

    // Check cache first
    cacheKey := fmt.Sprintf("calculation:%v", cmd.Years)
    if cached, err := h.cache.Get(cacheKey); err == nil {
        return cached.(*application.CalculationResponse), nil
    }

    // Execute business logic
    result, err := h.calculationSvc.CalculateModel(ctx, cmd.Years)
    if err != nil {
        return nil, fmt.Errorf("calculation failed: %w", err)
    }

    // Include matches if requested
    if cmd.IncludeMatches {
        matches, err := h.matchSvc.FindMatches(ctx, cmd.Years)
        if err != nil {
            h.eventPublisher.Publish(ctx, MatchCalculationFailedEvent{
                Years:  cmd.Years,
                Reason: err.Error(),
            })
            return nil, fmt.Errorf("match calculation failed: %w", err)
        }
        result.Matches = matches
    }

    // Publish success event
    h.eventPublisher.Publish(ctx, ModelCalculationCompletedEvent{
        Years:       cmd.Years,
        Result:      result,
        CompletedAt: time.Now(),
    })

    // Cache result
    h.cache.Set(cacheKey, result, 1*time.Hour)

    return result, nil
}
```

### **5. Event-Driven Architecture**

#### **Current Problem:**
```go
// Everything synchronous - no background processing
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    // Synchronous calculation only
}
```

#### **Proposed Solution:**
```go
// domain/events.go
type CalculationCompletedEvent struct {
    CalculationID uuid.UUID `json:"calculation_id"`
    Year         int        `json:"year"`
    Result       *CalculationResult `json:"result"`
    CompletedAt  time.Time  `json:"completed_at"`
}

// interfaces/events/calculation_handler.go
type CalculationCompletedHandler struct {
    notificationSvc notification.Service
    analyticsSvc    analytics.Service
}

func (h *CalculationCompletedHandler) Handle(ctx context.Context, event *domain.CalculationCompletedEvent) error {
    // Send notifications asynchronously
    go h.notificationSvc.NotifyCalculationComplete(ctx, event.CalculationID)

    // Update analytics asynchronously
    go h.analyticsSvc.RecordCalculationMetrics(ctx, event.Result)

    // Trigger dependent calculations
    if event.Result.RequiresMatchAnalysis {
        h.eventPublisher.Publish(ctx, TriggerMatchAnalysisEvent{
            CalculationID: event.CalculationID,
            Year:         event.Year,
        })
    }

    return nil
}
```

### **6. Configuration Management**

#### **Current Problem:**
```go
// Parameters scattered and passed everywhere
func CalculateYear(year int, p *params.Params) (*YearResult, error) {
    // p is passed through every function
}
```

#### **Proposed Solution:**
```go
// infrastructure/config/config.go
type Config struct {
    Database    DatabaseConfig    `yaml:"database"`
    Cache       CacheConfig       `yaml:"cache"`
    Messaging   MessagingConfig   `yaml:"messaging"`
    Calculation CalculationConfig `yaml:"calculation"`
    Matching    MatchingConfig    `yaml:"matching"`
}

type CalculationConfig struct {
    DefaultDiscountRate     float64 `yaml:"default_discount_rate"`
    MaxYears               int     `yaml:"max_years"`
    CacheEnabled           bool    `yaml:"cache_enabled"`
    AsyncProcessingEnabled bool    `yaml:"async_processing_enabled"`
}

// infrastructure/kernel/container.go
func NewContainer(cfg *config.Config) (*Container, error) {
    // Initialize all dependencies based on configuration
    db, err := setupDatabase(cfg.Database)
    if err != nil {
        return nil, fmt.Errorf("database setup failed: %w", err)
    }

    cache, err := setupCache(cfg.Cache)
    if err != nil {
        return nil, fmt.Errorf("cache setup failed: %w", err)
    }

    // Wire all services with proper dependencies
    customerRepo := postgres.NewCustomerRepository(db, logger)
    calculationSvc := application.NewCalculationService(
        customerRepo,
        cache,
        cfg.Calculation,
        logger,
    )

    return &Container{
        CalculationService: calculationSvc,
        CustomerRepository: customerRepo,
        // ... other services
    }, nil
}
```

---

## 📦 **Dependency Stack & Technology Choices**

### **Final Dependency List (2025 Recommendations)**

Based on documentation requirements and 2025 Go best practices, here is the comprehensive dependency stack:

**Key Technology Decisions:**
- ✅ **Echo v4** - HTTP framework (clean API, excellent middleware)
- ✅ **GORM** - ORM for PostgreSQL (associations, hooks, query builder)
- ✅ **golang-migrate/v4** - Database migrations (SQL-based, version control, rollback support)
- ⚠️ **DO NOT use GORM AutoMigrate** - Use golang-migrate for production migrations

#### **1. Core Framework & HTTP**

```go
// HTTP Framework
github.com/labstack/echo/v4 v4.14.0     // ✅ FINAL CHOICE: Echo - Clean API, excellent middleware, good performance
// Decision: Echo chosen for clean API design and middleware support
// Alternative considered: Gin (more mature ecosystem), Fiber (lower latency)
// Rationale: Echo provides better middleware chaining and cleaner handler signatures

// Dependency Injection (2025 Recommended)
go.uber.org/fx v1.24.0                   // ✅ FINAL CHOICE: Uber's fx - Clean, functional DI
// Alternatives considered:
//   - github.com/google/wire (code generation) - Too verbose for MVP
//   - github.com/samber/do (minimal) - Too simple for complex needs
// Rationale: fx provides lifecycle hooks, graceful shutdown, and clean functional composition

// Configuration Management
github.com/spf13/viper v1.21.0          // ✅ FINAL CHOICE: YAML/JSON/ENV support
github.com/spf13/cobra v1.10.1          // Already in use - CLI framework
```

#### **2. Database & Data Access**

```go
// Graph Database (Primary)
github.com/neo4j/neo4j-go-driver/v5 v5.23.0  // ✅ Neo4j driver with connection pooling

// ORM (Relational Database)
gorm.io/gorm v1.25.12                 // ✅ FINAL CHOICE: GORM - Full-featured ORM with migrations, associations, hooks
gorm.io/driver/postgres v1.5.9        // ✅ GORM PostgreSQL driver (uses pgx/v5 under the hood)
// Decision: GORM chosen for ActiveRecord-style ORM, automatic migrations, associations
// Rationale: GORM provides excellent developer experience with hooks, associations, and migrations
// Alternative considered: pgx/v5 direct (more control, less convenience) - GORM chosen for productivity

// PostgreSQL Driver (Used by GORM)
github.com/jackc/pgx/v5 v5.7.2        // ✅ Used by GORM driver - Best performance PostgreSQL driver
// Note: GORM uses pgx/v5 as the underlying driver for optimal performance

// Spatial/Geographic Support
github.com/twpayne/go-geom v1.6.0      // PostGIS support for geographic queries

// Database Migrations
github.com/golang-migrate/migrate/v4 v4.18.1 // ✅ FINAL CHOICE: golang-migrate - Industry standard, SQL-based migrations
// Decision: Use golang-migrate instead of GORM AutoMigrate for production-grade migrations
// Rationale: SQL-based migrations provide version control, rollback support, and full control over schema changes
// Features: Supports PostgreSQL, Neo4j (via Cypher), version control, up/down migrations, CLI tool
// Alternative considered: pressly/goose (good DX but golang-migrate is more widely adopted)
// Note: GORM AutoMigrate should NOT be used in production - only for rapid prototyping
```

#### **3. Caching & Message Queue**

```go
// Cache (MVP)
github.com/redis/go-redis/v9 v9.7.0     // ✅ FINAL CHOICE: Official Redis client, fast and reliable

// Message Queue (MVP)
github.com/nats-io/nats.go v1.35.0     // ✅ FINAL CHOICE: Go-native, 60% simpler than Kafka
// Migration path: github.com/segmentio/kafka-go v0.4.48 (at 1000+ business scale)

// Distributed Task Queue
github.com/hibiken/asynq v0.24.1        // ✅ Redis-based task queue for background jobs
```

#### **4. Validation & Error Handling**

```go
// Validation
github.com/go-playground/validator/v10 v10.22.1  // ✅ FINAL CHOICE: Struct tags, widely used
// Go 1.25 generics can supplement for type-safe validation

// Error Handling
github.com/pkg/errors v0.9.1            // ✅ Enhanced error wrapping and stack traces
// Standard errors package for Go 1.25 error handling features
```

#### **5. Logging & Observability**

```go
// Logging (2025 Recommended)
github.com/rs/zerolog v1.33.0           // ✅ FINAL CHOICE: Fast structured logging, zero allocations
// Alternative considered: logrus - zerolog is 10x faster

// Observability
go.opentelemetry.io/otel v1.33.0        // ✅ FINAL CHOICE: Industry standard observability
go.opentelemetry.io/otel/trace v1.33.0
go.opentelemetry.io/otel/metric v1.33.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0

// Metrics
github.com/prometheus/client_golang v1.20.5    // ✅ Prometheus metrics integration
```

#### **6. Testing Framework**

```go
// Testing
github.com/stretchr/testify v1.9.0       // ✅ Already in use - assertions and mocks
github.com/golang/mock v1.6.0            // ✅ Mock generation from interfaces
// Alternative: github.com/vektra/mockery/v2 - testify/mock is simpler

// Test Containers (Integration Testing)
github.com/testcontainers/testcontainers-go v0.34.0  // ✅ Docker-based integration tests
github.com/testcontainers/testcontainers-go/modules/postgres v0.34.0
github.com/testcontainers/testcontainers-go/modules/neo4j v0.34.0
```

#### **7. API & GraphQL**

```go
// GraphQL (Future)
github.com/99designs/gqlgen v0.17.47     // ✅ FINAL CHOICE: Schema-first, code generation
// Alternative considered: graphql-go (runtime-first) - gqlgen is more type-safe

// WebSocket
nhooyr.io/websocket v1.8.11              // ✅ FINAL CHOICE: Fast, minimal dependencies
// Alternative considered: gorilla/websocket - nhooyr is more modern and faster

// HTTP Client
github.com/go-resty/resty/v2 v2.14.0    // ✅ Convenient HTTP client (optional, for external APIs)
```

#### **8. Event-Driven & Background Processing**

```go
// Event Sourcing (Future)
github.com/EventStore/EventStore-Client-Go v1.4.1  // Optional: For event sourcing

// Background Jobs
github.com/hibiken/asynq v0.24.1        // Already listed above - Redis-based task queue

// Rate Limiting
golang.org/x/time v0.10.0               // Rate limiting for API endpoints
```

#### **9. Security & Authentication**

```go
// JWT Authentication
github.com/golang-jwt/jwt/v5 v5.2.1     // ✅ FINAL CHOICE: Official JWT library (v5 is latest)
// Alternative: github.com/form3tech-oss/jwt-go - JWT v5 is official and maintained

// Password Hashing
golang.org/x/crypto v0.35.0              // ✅ bcrypt and other crypto functions

// OAuth2
golang.org/x/oauth2 v0.28.0             // ✅ OAuth2 client library
```

#### **10. Configuration & Serialization**

```go
// YAML (Already in use)
gopkg.in/yaml.v3 v3.0.1                 // ✅ Already in use

// JSON (Go 1.25 Experimental)
// encoding/json/v2                       // ✅ Use if Go 1.25 experimental features stable
// Fallback: standard encoding/json

// UUID Generation
github.com/google/uuid v1.6.0            // ✅ UUID generation for entities
```

#### **11. Utilities & Helpers**

```go
// Time Utilities
github.com/jonboulle/clockwork v0.4.0    // ✅ Time mocking for tests

// String Utilities
github.com/google/go-cmp v0.6.0         // ✅ Better diffing for tests

// Context & Timeouts
golang.org/x/sync v0.11.0               // ✅ errgroup, semaphore for concurrency
```

#### **12. Development Tools**

```go
// Code Generation
github.com/google/wire v0.6.0           // Optional: For advanced DI code generation

// Linting
// golangci-lint (external tool)          // ✅ Recommended: Use in CI/CD

// Documentation
github.com/swaggo/swag v1.16.3          // ✅ Swagger documentation generation
github.com/swaggo/echo-swagger v1.4.1    // ✅ Echo Swagger integration
```

### **📋 Complete go.mod (MVP Phase)**

```go
module github.com/damirmukimov/city_resource_graph

go 1.25.3

require (
    // Core Framework
    github.com/labstack/echo/v4 v4.14.0
    go.uber.org/fx v1.24.0
    
    // Database
    github.com/neo4j/neo4j-go-driver/v5 v5.23.0
    gorm.io/gorm v1.25.12
    gorm.io/driver/postgres v1.5.9
    github.com/jackc/pgx/v5 v5.7.2        // Used by GORM driver
    github.com/twpayne/go-geom v1.6.0
    
    // Database Migrations (REQUIRED - Do NOT use GORM AutoMigrate)
    github.com/golang-migrate/migrate/v4 v4.18.1  // ✅ SQL-based migrations with version control
    
    // Cache & Messaging
    github.com/redis/go-redis/v9 v9.7.0
    github.com/nats-io/nats.go v1.35.0
    github.com/hibiken/asynq v0.24.1
    
    // Validation & Errors
    github.com/go-playground/validator/v10 v10.22.1
    github.com/pkg/errors v0.9.1
    
    // Logging & Observability
    github.com/rs/zerolog v1.33.0
    go.opentelemetry.io/otel v1.33.0
    go.opentelemetry.io/otel/trace v1.33.0
    go.opentelemetry.io/otel/metric v1.33.0
    github.com/prometheus/client_golang v1.20.5
    
    // Testing
    github.com/stretchr/testify v1.9.0
    github.com/golang/mock v1.6.0
    github.com/testcontainers/testcontainers-go v0.34.0
    
    // API & WebSocket
    nhooyr.io/websocket v1.8.11
    github.com/go-resty/resty/v2 v2.14.0
    
    // Security
    github.com/golang-jwt/jwt/v5 v5.2.1
    golang.org/x/crypto v0.35.0
    golang.org/x/oauth2 v0.28.0
    
    // Configuration
    github.com/spf13/viper v1.21.0
    github.com/spf13/cobra v1.10.1
    gopkg.in/yaml.v3 v3.0.1
    
    // Utilities
    github.com/google/uuid v1.6.0
    github.com/jonboulle/clockwork v0.4.0
    golang.org/x/sync v0.11.0
    golang.org/x/time v0.10.0
)

// Development-only dependencies
require (
    github.com/swaggo/swag v1.16.3
    github.com/swaggo/echo-swagger v1.4.1
)
```

### **🎯 Technology Decision Matrix**

| Category | Choice | Rationale | Alternatives Considered |
|----------|--------|-----------|------------------------|
| **HTTP Framework** | Echo v4 | ✅ FINAL: Clean API, excellent middleware chaining, good performance | Gin (mature ecosystem), Fiber (lower latency) - Echo chosen for cleaner API |
| **DI Container** | fx (Uber) | Functional composition, lifecycle hooks, graceful shutdown | wire (too verbose), dig (deprecated) |
| **Graph DB Driver** | neo4j-go-driver/v5 | Official, connection pooling, transaction support | Official driver only viable option |
| **ORM** | GORM | ✅ FINAL: Full-featured ORM with associations, hooks, query builder - Excellent developer experience | pgx/v5 direct (more control, less convenience) - GORM chosen for productivity |
| **Migration Tool** | golang-migrate/v4 | ✅ FINAL: Industry standard, SQL-based migrations, version control, rollback support | GORM AutoMigrate (NOT for production), pressly/goose (good DX but less adopted) |
| **PostgreSQL Driver** | pgx/v5 | Used by GORM driver - 30% faster than database/sql, PostGIS support | database/sql + pq (standard but slower) |
| **Cache** | go-redis/v9 | Official client, fastest, reliable | redigo (older), rueidis (newer but less stable) |
| **Message Queue (MVP)** | NATS | Go-native, 60% simpler than Kafka, perfect for MVP | Kafka (too complex for MVP), Redis Streams (limited features) |
| **Logging** | zerolog | Zero allocations, 10x faster than logrus | logrus (slower), zap (more complex) |
| **Validation** | validator/v10 | Struct tags, widely adopted, comprehensive | custom generics (future enhancement) |
| **Testing** | testify + golang/mock | Simple, widely used, good mocking | mockery (more features but more complex) |
| **WebSocket** | nhooyr.io/websocket | Fast, minimal, modern | gorilla/websocket (older, more dependencies) |
| **Observability** | OpenTelemetry | Industry standard, vendor-agnostic | Prometheus-only (less flexible) |

### **🔧 Echo & GORM Integration Notes**

#### **Echo Framework Benefits**
- **Clean Handler Signatures**: `func Handler(c echo.Context) error` - consistent error handling
- **Middleware Chaining**: Built-in support for CORS, JWT, rate limiting, logging
- **Context Propagation**: Echo's context extends Go's `context.Context` for request-scoped values
- **Performance**: Excellent performance with minimal overhead
- **Validation Integration**: Works seamlessly with `validator/v10` for struct validation

#### **GORM Benefits**
- **Associations**: Built-in support for has-one, has-many, many-to-many relationships
- **Hooks**: BeforeSave, AfterCreate, etc. for business logic triggers
- **Query Builder**: Fluent API for complex queries: `db.Where().Joins().Preload()`
- **Performance**: Uses pgx/v5 under the hood, so no performance penalty
- **Transaction Support**: Built-in transaction management with rollback support
- **⚠️ AutoMigrate Note**: AutoMigrate should NOT be used in production - use golang-migrate for schema management

#### **golang-migrate Benefits (Migration Management)**
- **SQL-Based Migrations**: Write explicit SQL migrations for full control
- **Version Control**: Track migration versions and status
- **Rollback Support**: Automatic down migrations for safe rollbacks
- **Multiple Databases**: Supports PostgreSQL, Neo4j (Cypher), MySQL, SQLite, etc.
- **CLI Tool**: `migrate` command-line tool for easy migration management
- **Library API**: Programmatic migration support for automated deployments
- **Developer Experience**: 
  - Simple file naming: `000001_create_customers.up.sql` and `000001_create_customers.down.sql`
  - Migration verification before execution
  - Force version support for fixing migration issues
  - Dry-run mode for testing migrations

#### **Migration Workflow with golang-migrate**
```go
// Example: Migration setup and execution
import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

// Migration directory structure:
// migrations/
//   ├── 000001_create_customers.up.sql
//   ├── 000001_create_customers.down.sql
//   ├── 000002_add_customer_indexes.up.sql
//   ├── 000002_add_customer_indexes.down.sql
//   └── ...

func RunMigrations(databaseURL string, migrationsPath string) error {
    m, err := migrate.New(
        "file://"+migrationsPath,
        databaseURL,
    )
    if err != nil {
        return fmt.Errorf("failed to initialize migrations: %w", err)
    }
    defer m.Close()

    // Run all pending migrations
    if err := m.Up(); err != nil {
        if err == migrate.ErrNoChange {
            // No pending migrations - this is OK
            return nil
        }
        return fmt.Errorf("failed to run migrations: %w", err)
    }

    return nil
}

// Example SQL migration files:
// 000001_create_customers.up.sql:
// CREATE TABLE customers (
//     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
//     year INTEGER NOT NULL,
//     total_orgs INTEGER NOT NULL,
//     paying_orgs INTEGER NOT NULL,
//     created_at TIMESTAMP DEFAULT NOW(),
//     updated_at TIMESTAMP DEFAULT NOW()
// );
// CREATE INDEX idx_customers_year ON customers(year);

// 000001_create_customers.down.sql:
// DROP INDEX IF EXISTS idx_customers_year;
// DROP TABLE IF EXISTS customers;
```

#### **golang-migrate CLI Usage**
```bash
# Install golang-migrate CLI
go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

# Create a new migration
migrate create -ext sql -dir migrations -seq create_customers

# Run all pending migrations
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" up

# Rollback one migration
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" down 1

# Check migration status
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" version

# Force version (for fixing migration issues)
migrate -path ./migrations -database "postgres://user:pass@localhost/dbname?sslmode=disable" force 1
```

#### **Migration Best Practices**
- ✅ **Always create up AND down migrations** - Enables safe rollbacks
- ✅ **Version migrations sequentially** - Use sequential numbering: `000001`, `000002`, etc.
- ✅ **Keep migrations idempotent** - Use `IF NOT EXISTS` and `IF EXISTS` clauses
- ✅ **Test migrations** - Test both up and down migrations before deploying
- ✅ **Review SQL before committing** - Never auto-generate migrations blindly
- ✅ **Keep migrations small** - One logical change per migration
- ✅ **Use transactions** - Wrap migrations in transactions when possible (PostgreSQL)
- ⚠️ **Never modify existing migrations** - Create new migrations to fix issues

#### **Architecture Integration**
```go
// Example: Repository pattern with GORM
import (
    "context"
    "errors"
    "fmt"
    
    "gorm.io/gorm"
)

type customerRepository struct {
    db *gorm.DB
    logger logging.Logger
}

func (r *customerRepository) GetByYear(ctx context.Context, year int) (*domain.Customer, error) {
    var customer domain.Customer
    err := r.db.WithContext(ctx).
        Where("year = ?", year).
        First(&customer).Error
    
    if err != nil {
        if errors.Is(err, gorm.ErrRecordNotFound) {
            return nil, domain.ErrCustomerNotFound
        }
        r.logger.Error("Failed to get customer", "year", year, "error", err)
        return nil, fmt.Errorf("failed to get customer: %w", err)
    }
    
    return &customer, nil
}

// Echo handler integration
import (
    "net/http"
    "strconv"
    
    "github.com/labstack/echo/v4"
)

func (h *customerHandler) GetCustomer(c echo.Context) error {
    year, err := strconv.Atoi(c.Param("year"))
    if err != nil {
        return echo.NewHTTPError(http.StatusBadRequest, "invalid year")
    }
    
    customer, err := h.customerRepo.GetByYear(c.Request().Context(), year)
    if err != nil {
        if errors.Is(err, domain.ErrCustomerNotFound) {
            return echo.NewHTTPError(http.StatusNotFound, "customer not found")
        }
        return echo.NewHTTPError(http.StatusInternalServerError, "failed to get customer")
    }
    
    return c.JSON(http.StatusOK, customer)
}

// Echo router setup with middleware
func setupEchoRouter(h *customerHandler) *echo.Echo {
    e := echo.New()
    
    // Middleware
    e.Use(middleware.Logger())
    e.Use(middleware.Recover())
    e.Use(middleware.CORS())
    
    // Routes
    api := e.Group("/api/v1")
    api.GET("/customers/:year", h.GetCustomer)
    
    return e
}
```

### **📈 Migration Path for Scale Phase**

**When reaching 1000+ businesses:**
- **Kafka**: Replace NATS with `github.com/segmentio/kafka-go v0.4.48`
- **TimescaleDB**: Add `github.com/influxdata/influxdb-client-go/v2` for time-series
- **Service Mesh**: Consider `istio` or `linkerd` for inter-service communication
- **GORM**: Continue using GORM - it scales well with connection pooling and query optimization

---

## 📋 **Implementation Roadmap**

### **Phase 1: Foundation (2 weeks)**
1. **Create Domain Layer**
   - Define domain entities (`Customer`, `Calculation`, `Match`)
   - Create value objects and domain events
   - Define repository interfaces

2. **Setup Infrastructure**
   - Create DI container structure with `fx`
   - Implement configuration management with `viper`
   - Setup logging with `zerolog` and error handling
   - **Setup Database Migrations** with `golang-migrate`
     - Create migrations directory structure
     - Setup initial PostgreSQL migrations (customers, calculations, matches)
     - Setup Neo4j migrations (graph schema, constraints, indexes)
     - Create migration runner service with DI integration

3. **Create Repository Pattern**
   - Implement in-memory repositories for current data
   - Define repository interfaces
   - Migrate parameter loading to repositories
   - **⚠️ DO NOT use GORM AutoMigrate** - Use golang-migrate for all schema changes

### **Phase 2: Application Layer (3 weeks)**
1. **CQRS Implementation**
   - Create command/query handlers
   - Implement use cases
   - Add event handlers

2. **Service Layer Refactor**
   - Extract business logic into domain services
   - Implement calculation orchestrator
   - Add validation services

3. **Event-Driven Architecture**
   - Implement event publishing
   - Create event handlers
   - Setup async processing

### **Phase 3: Interface Layer (2 weeks)**
1. **HTTP API**
   - RESTful endpoints for calculations
   - GraphQL API for complex queries
   - Proper error responses

2. **CLI Refactor**
   - Use DI container in CLI
   - Implement command handlers
   - Add interactive mode

3. **Event Processing**
   - Background job processing
   - Event sourcing for audit trails
   - Real-time notifications

### **Phase 4: Testing & Documentation (1 week)**
1. **Integration Tests**
   - End-to-end test scenarios
   - Performance testing
   - Load testing

2. **Documentation**
   - API documentation
   - Architecture decision records
   - Deployment guides

---

## 🔄 **Migration Strategy**

### **Incremental Migration**
1. **Week 1-2**: Create new architecture alongside existing code
2. **Week 3-4**: Migrate leaf packages (params, validator) to new structure
3. **Week 5-6**: Migrate calculation logic with adapters
4. **Week 7-8**: Replace CLI and add HTTP API
5. **Week 9-10**: Full integration and testing

### **Backward Compatibility**
- Keep existing CLI working during migration
- Create adapter layer for gradual migration
- Maintain existing JSON schemas and outputs

### **Testing Strategy**
- **Unit Tests**: Test each layer in isolation with mocks
- **Integration Tests**: Test layer interactions
- **E2E Tests**: Test complete workflows
- **Performance Tests**: Ensure no regression in calculation speed

---

## 🎯 **Benefits of New Architecture**

### **Technical Benefits**
- ✅ **Testability**: Each component can be tested in isolation
- ✅ **Maintainability**: Clear separation of concerns
- ✅ **Extensibility**: Easy to add new calculation methods or data sources
- ✅ **Scalability**: Event-driven architecture supports high throughput
- ✅ **Reliability**: Proper error handling and monitoring

### **Business Benefits**
- ✅ **Time-to-Market**: Faster feature development with modular design
- ✅ **Quality**: Better testing coverage and error handling
- ✅ **Scalability**: Support for 1000+ businesses with event-driven processing
- ✅ **Maintainability**: Easier to modify and extend without breaking changes

### **Development Benefits**
- ✅ **Developer Experience**: Clear interfaces and dependency injection
- ✅ **Code Reuse**: Domain services can be used across different interfaces
- ✅ **Debugging**: Better error tracing and logging
- ✅ **Onboarding**: Clear architectural boundaries for new developers

---

## 📊 **Risk Assessment & Mitigation**

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| Migration Complexity | Medium | High | Incremental migration with adapters |
| Performance Regression | Low | Medium | Comprehensive performance testing |
| Breaking Changes | Medium | High | Maintain backward compatibility layer |
| Team Learning Curve | Medium | Low | Training sessions and documentation |
| Increased Complexity | High | Low | Start simple, add complexity gradually |

---

## 🏁 **Success Metrics**

- **Code Coverage**: >90% unit test coverage
- **Performance**: No regression in calculation speed (<10% impact)
- **Maintainability**: Cyclomatic complexity <10 per function
- **Scalability**: Support 10x current load with same infrastructure
- **Reliability**: 99.9% uptime with proper error handling

---

*This architectural refactoring transforms the mathematical model into a production-ready backend foundation that can scale to support thousands of businesses while maintaining code quality and developer productivity.*