turash/docs/concept/29_technical_architecture_diagrams.md
Damir Mukimov 000eab4740
Major repository reorganization and missing backend endpoints implementation
Repository Structure:
- Move files from cluttered root directory into organized structure
- Create archive/ for archived data and scraper results
- Create bugulma/ for the complete application (frontend + backend)
- Create data/ for sample datasets and reference materials
- Create docs/ for comprehensive documentation structure
- Create scripts/ for utility scripts and API tools

Backend Implementation:
- Implement 3 missing backend endpoints identified in gap analysis:
  * GET /api/v1/organizations/{id}/matching/direct - Direct symbiosis matches
  * GET /api/v1/users/me/organizations - User organizations
  * POST /api/v1/proposals/{id}/status - Update proposal status
- Add complete proposal domain model, repository, and service layers
- Create database migration for proposals table
- Fix CLI server command registration issue

API Documentation:
- Add comprehensive proposals.md API documentation
- Update README.md with Users and Proposals API sections
- Document all request/response formats, error codes, and business rules

Code Quality:
- Follow existing Go backend architecture patterns
- Add proper error handling and validation
- Match frontend expected response schemas
- Maintain clean separation of concerns (handler -> service -> repository)
2025-11-25 06:01:16 +01:00

22 KiB
Raw Blame History

29. Technical Architecture Diagrams

System Context Diagram

graph TB
    subgraph "External Systems"
        ERP[ERP Systems<br/>SAP, Oracle]
        SCADA[SCADA Systems<br/>OPC UA, Modbus]
        IoT[IoT Devices<br/>Sensors, Meters]
        UTIL[Utility APIs<br/>Energy, Water]
        GOV[Government Data<br/>Building Permits, NACE]
    end

    subgraph "Turash Platform"
        API[API Gateway<br/>Kong/Traefik]
        AUTH[Authentication Service<br/>JWT, OAuth2]

        subgraph "Application Layer"
            WEB[Web Frontend<br/>Next.js + React]
            MOBILE[Mobile PWA<br/>Progressive Web App]
            ADMIN[Admin Dashboard<br/>Enterprise Features]
        end

        subgraph "Service Layer"
            MATCH[Matching Engine<br/>Graph Algorithms]
            ECON[Economic Calculator<br/>NPV, ROI Models]
            NOTIF[Notification Service<br/>WebSocket, Email]
            SEARCH[Search Service<br/>Full-text, Spatial]
        end

        subgraph "Data Layer"
            NEO4J[(Neo4j Graph DB<br/>Relationships, Entities)]
            POSTGRES[(PostgreSQL + PostGIS<br/>Spatial, Time-series)]
            REDIS[(Redis Cache<br/>Sessions, Matches)]
        end

    subgraph "Infrastructure"
        K8S[Kubernetes Cluster<br/>AWS EKS/GCP GKE]
        MQ["Message Queue<br/>NATS/Redis Streams (MVP)<br/>Kafka (Scale)"]
        CDN[CDN<br/>CloudFront/Cloudflare]
        MONITOR[Monitoring Stack<br/>Prometheus + Grafana]
    end
    end

    subgraph "Users & Partners"
        SME[SME Businesses<br/>Resource Providers]
        ENTERPRISE[Enterprise Companies<br/>Multi-site Operations]
        MUNI[Municipalities<br/>City Governments]
        UTIL_PART[Utility Partners<br/>Data & Distribution]
        FACIL[Facilitators<br/>Engineers, Consultants]
    end

    %% Connections
    ERP --> API
    SCADA --> API
    IoT --> API
    UTIL --> API
    GOV --> API

    API --> AUTH
    AUTH --> WEB
    AUTH --> MOBILE
    AUTH --> ADMIN

    WEB --> MATCH
    MOBILE --> MATCH
    ADMIN --> MATCH

    MATCH --> ECON
    MATCH --> NOTIF
    MATCH --> SEARCH

    MATCH --> NEO4J
    ECON --> POSTGRES
    SEARCH --> REDIS

    NEO4J --> K8S
    POSTGRES --> K8S
    REDIS --> K8S

    NOTIF --> MQ
    MQ --> MONITOR

    SME --> WEB
    ENTERPRISE --> ADMIN
    MUNI --> API
    UTIL_PART --> API
    FACIL --> WEB

Application Architecture (C4 Container Diagram)

graph TB
    subgraph "User Interfaces"
        WEB_BROWSER[Web Browser<br/>React + Next.js]
        MOBILE_APP[Mobile PWA<br/>React + Service Worker]
        ADMIN_PORTAL[Admin Portal<br/>Enterprise Dashboard]
    end

    subgraph "API Gateway"
        KONG["API Gateway<br/>Kong/Traefik<br/>Rate Limiting, Auth"]
    end

    subgraph "Microservices"
        AUTH_SERVICE["Authentication Service<br/>Go 1.25 + JWT<br/>User Management"]
        BUSINESS_SERVICE["Business Service<br/>Go 1.25 + Neo4j<br/>CRUD Operations"]
        MATCHING_SERVICE["Matching Service<br/>Go 1.25 + Graph Algorithms<br/>Core Engine + Hybrid Automation"]
        ECONOMIC_SERVICE["Economic Service<br/>Go 1.25 + PostgreSQL<br/>Calculations"]
        NOTIFICATION_SERVICE["Notification Service<br/>Go 1.25 + Redis<br/>WebSocket/Email"]
        INGESTION_SERVICE["Ingestion Service<br/>Go 1.25 + NATS/Redis Streams<br/>Data Processing"]
        SEARCH_SERVICE["Search Service<br/>Go 1.25 + Elasticsearch<br/>Full-text Search"]
        ANALYTICS_SERVICE["Analytics Service<br/>Go 1.25 + TimescaleDB<br/>Reporting"]
        DATA_QUALITY_SERVICE["Data Quality Service<br/>Go 1.25 + PostgreSQL<br/>Scoring & Incentives"]
        FACILITATOR_SERVICE["Facilitator Service<br/>Go 1.25 + Neo4j<br/>Marketplace & Routing"]
    end

    subgraph "Data Stores"
        NEO4J["(Neo4j Cluster<br/>Graph Database<br/>Relationships)"]
        POSTGRES["(PostgreSQL Cluster<br/>Spatial Database<br/>Business Data)"]
        REDIS["(Redis Cluster<br/>Cache + Sessions<br/>Performance)"]
        NATS["(NATS/Redis Streams<br/>Event Streaming (MVP)<br/>Async Processing)"]
        KAFKA["(Kafka Cluster<br/>Event Streaming (Scale)<br/>Migration at 1000+ businesses)"]
        ELASTIC["(Elasticsearch<br/>Search Index<br/>Full-text)"]
        MINIO["(MinIO/S3<br/>Object Storage<br/>Files & Assets)"]
    end

    subgraph "Infrastructure Services"
        PROMETHEUS[Prometheus<br/>Metrics Collection]
        GRAFANA[Grafana<br/>Visualization]
        ALERTMANAGER[AlertManager<br/>Incident Management]
        LOGGING[Loki + ELK<br/>Log Aggregation]
        VAULT[HashiCorp Vault<br/>Secrets Management]
    end

    %% Internal connections
    WEB_BROWSER --> KONG
    MOBILE_APP --> KONG
    ADMIN_PORTAL --> KONG

    KONG --> AUTH_SERVICE
    AUTH_SERVICE --> BUSINESS_SERVICE
    BUSINESS_SERVICE --> MATCHING_SERVICE
    BUSINESS_SERVICE --> DATA_QUALITY_SERVICE
    MATCHING_SERVICE --> ECONOMIC_SERVICE
    MATCHING_SERVICE --> NOTIFICATION_SERVICE
    MATCHING_SERVICE --> FACILITATOR_SERVICE
    BUSINESS_SERVICE --> INGESTION_SERVICE
    BUSINESS_SERVICE --> SEARCH_SERVICE
    BUSINESS_SERVICE --> ANALYTICS_SERVICE

    BUSINESS_SERVICE --> NEO4J
    MATCHING_SERVICE --> NEO4J
    FACILITATOR_SERVICE --> NEO4J
    ECONOMIC_SERVICE --> POSTGRES
    DATA_QUALITY_SERVICE --> POSTGRES
    NOTIFICATION_SERVICE --> REDIS
    INGESTION_SERVICE --> NATS
    SEARCH_SERVICE --> ELASTIC
    ANALYTICS_SERVICE --> MINIO

    MATCHING_SERVICE --> PROMETHEUS
    PROMETHEUS --> GRAFANA
    PROMETHEUS --> ALERTMANAGER

    ALL_SERVICES --> LOGGING
    ALL_SERVICES --> VAULT

Data Architecture Diagram

graph TD
    subgraph "Data Sources"
        ERP["ERP Systems<br/>SAP, Oracle<br/>Transactional Data"]
        SCADA["SCADA Systems<br/>OPC UA, Modbus<br/>Real-time Process Data"]
        IOT_SENSORS["IoT Sensors<br/>Temperature, Flow, Pressure<br/>Time-series Data"]
        MANUAL["Manual Entry<br/>Web Forms, CSV Upload<br/>User-Provided Data"]
        PUBLIC_DATA["Public Datasets<br/>Government, Utilities<br/>Reference Data"]
    end

    subgraph "Ingestion Layer"
        API_INGEST["API Ingestion<br/>REST/WebSocket<br/>Real-time"]
        BATCH_INGEST["Batch Ingestion<br/>CSV, XML, JSON<br/>Scheduled"]
        STREAM_INGEST["Stream Processing<br/>NATS/Redis Streams/MQTT<br/>Event-driven"]
        WEBHOOKS["Webhook Ingestion<br/>ERP/SCADA Push<br/>Event-triggered"]
    end

    subgraph "Processing Layer"
        VALIDATION["Data Validation<br/>Schema, Business Rules<br/>Quality Gates"]
        NORMALIZATION["Data Normalization<br/>Unit Conversion, Standardization<br/>Data Harmonization"]
        ENRICHMENT["Data Enrichment<br/>Geocoding, Classification<br/>Context Addition"]
        QUALITY_SCORING["Data Quality Scoring<br/>Precision Levels: Rough/Estimated/Measured<br/>Trust Scoring & Incentives"]
        DEDUPLICATION["Deduplication<br/>Entity Resolution<br/>Data Quality"]
    end

    subgraph "Storage Layer"
        subgraph "Graph Database (Neo4j)"
            BUSINESS_NODES["Business Nodes<br/>Companies, Legal Entities<br/>Properties: name, industry, size"]
            SITE_NODES["Site Nodes<br/>Physical Locations<br/>Properties: address, coordinates, type"]
            RESOURCE_NODES["Resource Flow Nodes<br/>Energy, Water, Waste<br/>Properties: type, amount, quality"]
            SERVICE_NODES["Service Nodes<br/>Maintenance, Consulting<br/>Properties: domain, capacity, rate"]
        end

        subgraph "Relational Database (PostgreSQL)"
            BUSINESS_TABLES["Business Tables<br/>Normalized Business Data<br/>Indexes: name, industry, location"]
            SITE_TABLES["Site Tables<br/>Spatial Site Data<br/>PostGIS: coordinates, boundaries"]
            RESOURCE_TABLES["Resource Tables<br/>Time-series Flows<br/>Partitioned by time, indexed by location"]
            TRANSACTION_TABLES["Transaction Tables<br/>Match Records, Contracts<br/>Audit trail, compliance"]
        end

        subgraph "Cache Layer (Redis)"
            MATCH_CACHE["Match Results Cache<br/>TTL: 15 minutes<br/>Key: location + resource_type"]
            SESSION_CACHE["Session Cache<br/>JWT tokens, user state<br/>TTL: 24 hours"]
            COMPUTATION_CACHE["Computation Cache<br/>Economic calculations<br/>TTL: 1 hour"]
        end
    end

    subgraph "Access Layer"
        GRAPH_API["Graph API<br/>Cypher Queries<br/>Relationship Traversal"]
        REST_API["REST API<br/>CRUD Operations<br/>Standard HTTP"]
        GRAPHQL_API["GraphQL API<br/>Flexible Queries<br/>Client-driven"]
        WEBSOCKET_API["WebSocket API<br/>Real-time Updates<br/>Live Notifications"]
        BULK_API["Bulk API<br/>Data Export/Import<br/>Batch Operations"]
    end

    subgraph "Consumption Layer"
        WEB_APP["Web Application<br/>React + Mapbox<br/>User Interface"]
        MOBILE_APP["Mobile PWA<br/>Progressive Web App<br/>Field Operations"]
        API_CLIENTS["API Clients<br/>ERP Integration<br/>Third-party Systems"]
        ANALYTICS["Analytics Platform<br/>Business Intelligence<br/>Reporting & Insights"]
        ML_MODELS["ML Models<br/>Recommendation Engine<br/>Predictive Analytics"]
    end

    %% Data Flow
    ERP --> API_INGEST
    SCADA --> STREAM_INGEST
    IOT_SENSORS --> STREAM_INGEST
    MANUAL --> BATCH_INGEST
    PUBLIC_DATA --> WEBHOOKS

    API_INGEST --> VALIDATION
    BATCH_INGEST --> VALIDATION
    STREAM_INGEST --> VALIDATION
    WEBHOOKS --> VALIDATION

    VALIDATION --> NORMALIZATION
    NORMALIZATION --> ENRICHMENT
    ENRICHMENT --> QUALITY_SCORING
    QUALITY_SCORING --> DEDUPLICATION

    DEDUPLICATION --> BUSINESS_NODES
    DEDUPLICATION --> SITE_NODES
    DEDUPLICATION --> RESOURCE_NODES
    DEDUPLICATION --> SERVICE_NODES

    DEDUPLICATION --> BUSINESS_TABLES
    DEDUPLICATION --> SITE_TABLES
    DEDUPLICATION --> RESOURCE_TABLES
    DEDUPLICATION --> TRANSACTION_TABLES

    BUSINESS_NODES --> GRAPH_API
    SITE_NODES --> GRAPH_API
    RESOURCE_NODES --> GRAPH_API
    SERVICE_NODES --> GRAPH_API

    BUSINESS_TABLES --> REST_API
    SITE_TABLES --> REST_API
    RESOURCE_TABLES --> REST_API
    TRANSACTION_TABLES --> REST_API

    GRAPH_API --> GRAPHQL_API
    REST_API --> GRAPHQL_API

    GRAPHQL_API --> WEBSOCKET_API
    GRAPHQL_API --> BULK_API

    WEBSOCKET_API --> WEB_APP
    WEBSOCKET_API --> MOBILE_APP
    GRAPHQL_API --> API_CLIENTS
    BULK_API --> ANALYTICS
    REST_API --> ML_MODELS

    %% Cache connections
    MATCH_CACHE -.-> GRAPHQL_API
    SESSION_CACHE -.-> WEBSOCKET_API
    COMPUTATION_CACHE -.-> REST_API

Matching Engine Architecture

graph TD
    subgraph "Input Processing"
        QUERY_IN[Resource Query<br/>Location, Type, Preferences]
        CONTEXT_IN[Business Context<br/>Industry, Size, Capabilities]
        CONSTRAINTS_IN[Matching Constraints<br/>Distance, Quality, Cost]
    end

    subgraph "Pre-filtering Stage"
        SPATIAL_FILTER["Spatial Filter<br/>PostGIS Query<br/>Radius Search"]
        BASIC_FILTER["Basic Filter<br/>Resource Type Match<br/>Availability Windows"]
        QUALITY_FILTER["Quality Filter<br/>Grade/Purity Match<br/>Hazard Compatibility"]
    end

    subgraph "Candidate Selection"
        GEO_INDEX["Geospatial Index<br/>R-tree/B-tree<br/>Location-based"]
        RESOURCE_INDEX["Resource Index<br/>Composite Index<br/>Type + Direction + Location"]
        TEMPORAL_INDEX["Temporal Index<br/>Time-window Index<br/>Availability Overlap"]
    end

    subgraph "Compatibility Scoring"
        QUALITY_SCORER["Quality Scorer<br/>Temperature, Pressure, Purity<br/>Weighted Compatibility"]
        TEMPORAL_SCORER["Temporal Scorer<br/>Availability Overlap<br/>Seasonal Factors"]
        QUANTITY_SCORER["Quantity Scorer<br/>Supply/Demand Match<br/>Variability Adjustment"]
        ECONOMIC_SCORER["Economic Scorer<br/>Cost Arbitrage<br/>Transport Costs"]
        TRUST_SCORER["Trust Scorer<br/>Data Precision<br/>Source Validation"]
    end

    subgraph "Optimization Engine"
        WEIGHTED_SUM["Weighted Sum<br/>Multi-criteria Scoring<br/>Configurable Weights"]
        RANKING_ENGINE["Ranking Engine<br/>Top-N Selection<br/>Diversity Consideration"]
        HYBRID_ROUTER["Hybrid Automation Router<br/>80% Automated, 20% Facilitated<br/>Match Complexity Scoring"]
        CONSTRAINT_CHECKER["Constraint Checker<br/>Business Rules<br/>Regulatory Compliance"]
        FALLBACK_ENGINE["Fallback Engine<br/>Relaxation Strategies<br/>Alternative Matches"]
    end

    subgraph "Result Processing"
        ECONOMIC_CALCULATION["Economic Calculation<br/>NPV, Payback, ROI<br/>Detailed Projections"]
        RISK_ASSESSMENT["Risk Assessment<br/>Technical, Market, Regulatory<br/>Probability Scoring"]
        PARTNER_PACKETS["Partner Packet Generation<br/>Technical Specs, Contacts<br/>Legal Templates"]
        NOTIFICATION_ENGINE["Notification Engine<br/>WebSocket Push<br/>Email/SMS Alerts"]
    end

    subgraph "Caching & Persistence"
        RESULT_CACHE["Result Cache<br/>Redis TTL Cache<br/>15-minute expiry"]
        MATCH_STORAGE["Match Storage<br/>PostgreSQL<br/>Audit Trail"]
        ANALYTICS_STORAGE["Analytics Storage<br/>ClickHouse/Timescale<br/>Performance Metrics"]
    end

    %% Data flow
    QUERY_IN --> SPATIAL_FILTER
    CONTEXT_IN --> BASIC_FILTER
    CONSTRAINTS_IN --> QUALITY_FILTER

    SPATIAL_FILTER --> GEO_INDEX
    BASIC_FILTER --> RESOURCE_INDEX
    QUALITY_FILTER --> TEMPORAL_INDEX

    GEO_INDEX --> QUALITY_SCORER
    RESOURCE_INDEX --> TEMPORAL_SCORER
    TEMPORAL_INDEX --> QUANTITY_SCORER

    QUALITY_SCORER --> WEIGHTED_SUM
    TEMPORAL_SCORER --> WEIGHTED_SUM
    QUANTITY_SCORER --> WEIGHTED_SUM
    ECONOMIC_SCORER --> WEIGHTED_SUM
    TRUST_SCORER --> WEIGHTED_SUM

    WEIGHTED_SUM --> RANKING_ENGINE
    RANKING_ENGINE --> HYBRID_ROUTER
    HYBRID_ROUTER --> CONSTRAINT_CHECKER
    CONSTRAINT_CHECKER --> FALLBACK_ENGINE

    FALLBACK_ENGINE --> ECONOMIC_CALCULATION
    ECONOMIC_CALCULATION --> RISK_ASSESSMENT
    RISK_ASSESSMENT --> PARTNER_PACKETS
    PARTNER_PACKETS --> NOTIFICATION_ENGINE

    NOTIFICATION_ENGINE --> RESULT_CACHE
    RESULT_CACHE --> MATCH_STORAGE
    MATCH_STORAGE --> ANALYTICS_STORAGE

Deployment Architecture (AWS)

graph TB
    subgraph "Internet"
        INTERNET[Global Users<br/>Web, Mobile, API]
    end

    subgraph "AWS CloudFront (CDN)"
        CLOUDFRONT["CloudFront Distribution<br/>Global Edge Locations<br/>SSL Termination"]
    end

    subgraph "AWS Global Accelerator"
        ACCELERATOR["Global Accelerator<br/>TCP/UDP Optimization<br/>Failover Routing"]
    end

    subgraph "AWS WAF & Shield"
        WAF["Web Application Firewall<br/>OWASP Rules<br/>Rate Limiting"]
        SHIELD["Shield Advanced<br/>DDoS Protection<br/>Auto Mitigation"]
    end

    subgraph "AWS Route 53"
        ROUTE53["Route 53<br/>DNS Management<br/>Health Checks<br/>Failover"]
    end

    subgraph "AWS Application Load Balancer"
        ALB["Application Load Balancer<br/>Layer 7 Routing<br/>SSL Offloading<br/>Health Checks"]
    end

    subgraph "AWS Elastic Kubernetes Service (EKS)"
        EKS_CONTROL["Control Plane<br/>Managed Kubernetes<br/>Auto-scaling"]

        subgraph "Worker Nodes (Auto-scaling Group)"
            ON_DEMAND["On-Demand Nodes<br/>General Workloads<br/>t3.large × 3-10"]
            SPOT["Spot Nodes<br/>Batch Processing<br/>c6i.xlarge × 2-20"]
        end

        subgraph "Kubernetes Services"
            INGRESS["Ingress Controller<br/>Traefik/Nginx<br/>External Traffic"]
            API_PODS["API Service Pods<br/>Go Applications<br/>Horizontal Scaling"]
            WORKER_PODS["Worker Pods<br/>Matching Engine<br/>CPU Optimized"]
            WEB_PODS["Web Pods<br/>Next.js SSR<br/>Memory Optimized"]
        end
    end

    subgraph "AWS RDS (PostgreSQL)"
        RDS_CLUSTER["RDS Aurora Cluster<br/>Multi-AZ<br/>PostGIS Enabled<br/>Read Replicas"]
        RDS_PROXY["RDS Proxy<br/>Connection Pooling<br/>Failover Handling"]
    end

    subgraph "AWS ElastiCache (Redis)"
        REDIS_CLUSTER["Redis Cluster<br/>Multi-AZ<br/>Backup Enabled<br/>Encryption"]
    end

    subgraph "AWS Neptune (Neo4j-compatible)"
        NEPTUNE_CLUSTER["Neptune Cluster<br/>Graph Database<br/>Gremlin/Cypher<br/>Multi-AZ"]
    end

    subgraph "AWS MSK (Kafka)"
        KAFKA_CLUSTER["MSK Cluster<br/>Event Streaming (Scale Phase)<br/>Multi-AZ, Migration at 1000+ businesses<br/>Auto-scaling"]
    end

    subgraph "Message Queue (MVP)"
        NATS_CLUSTER["NATS/Redis Streams<br/>Event Streaming (MVP)<br/>Simple, Cost-effective<br/>Kubernetes-native"]
    end

    subgraph "AWS OpenSearch"
        OPENSEARCH["OpenSearch Domain<br/>Full-text Search<br/>Multi-AZ<br/>KNN for ML"]
    end

    subgraph "AWS S3 & Glacier"
        S3_BUCKETS["S3 Buckets<br/>Data Lake<br/>Backup Storage<br/>CDN Origin"]
        GLACIER["Glacier Deep Archive<br/>Long-term Backup<br/>Compliance Storage"]
    end

    subgraph "AWS CloudWatch & X-Ray"
        CLOUDWATCH["CloudWatch<br/>Metrics & Logs<br/>Alarms & Dashboards"]
        XRAY["X-Ray<br/>Distributed Tracing<br/>Performance Insights"]
    end

    subgraph "AWS Systems Manager"
        SSM["Parameter Store<br/>Configuration Management<br/>Secrets (non-sensitive)"]
    end

    subgraph "AWS Secrets Manager"
        SECRETS_MANAGER["Secrets Manager<br/>Database Passwords<br/>API Keys<br/>Auto-rotation"]
    end

    subgraph "AWS Backup"
        BACKUP_SERVICE["Backup Service<br/>Automated Backups<br/>Cross-region Replication<br/>Retention Policies"]
    end

    %% Traffic flow
    INTERNET --> CLOUDFRONT
    CLOUDFRONT --> ACCELERATOR
    ACCELERATOR --> WAF
    WAF --> SHIELD
    SHIELD --> ROUTE53
    ROUTE53 --> ALB
    ALB --> INGRESS

    INGRESS --> API_PODS
    INGRESS --> WEB_PODS

    API_PODS --> WORKER_PODS

    API_PODS --> RDS_PROXY
    RDS_PROXY --> RDS_CLUSTER

    API_PODS --> REDIS_CLUSTER
    WORKER_PODS --> REDIS_CLUSTER

    API_PODS --> NEPTUNE_CLUSTER
    WORKER_PODS --> NEPTUNE_CLUSTER

    API_PODS --> NATS_CLUSTER
    WORKER_PODS --> NATS_CLUSTER
    API_PODS --> KAFKA_CLUSTER
    WORKER_PODS --> KAFKA_CLUSTER

    API_PODS --> OPENSEARCH

    ALL_SERVICES --> S3_BUCKETS
    S3_BUCKETS --> GLACIER

    ALL_SERVICES --> CLOUDWATCH
    ALL_SERVICES --> XRAY

    ALL_SERVICES --> SSM
    ALL_SERVICES --> SECRETS_MANAGER

    RDS_CLUSTER --> BACKUP_SERVICE
    NEPTUNE_CLUSTER --> BACKUP_SERVICE
    REDIS_CLUSTER --> BACKUP_SERVICE

CI/CD Pipeline Architecture

graph LR
    subgraph "Developer Workstations"
        LOCAL_DEV["Local Development<br/>Docker Compose<br/>Kind Cluster"]
    end

    subgraph "GitHub Repository"
        MAIN_BRANCH[main Branch<br/>Production Code]
        FEATURE_BRANCHES[feature/* Branches<br/>Development]
        RELEASE_BRANCHES[release/* Branches<br/>Staging]
    end

    subgraph "GitHub Actions"
        LINT_JOB["Lint & Format<br/>golangci-lint<br/>ESLint"]
        TEST_JOB["Unit & Integration Tests<br/>Go 1.25 test (with feature flags)<br/>Jest, Coverage Reports"]
        BUILD_JOB["Build & Containerize<br/>Go 1.25 build (fallback to 1.23)<br/>Docker Build, Multi-arch Images"]
        SECURITY_JOB["Security Scan<br/>Trivy, Snyk<br/>Vulnerability Check"]
        DEPLOY_DEV_JOB["Deploy to Development<br/>Kind Cluster<br/>Integration Tests"]
        DEPLOY_STAGING_JOB["Deploy to Staging<br/>EKS Staging<br/>E2E Tests"]
        DEPLOY_PROD_JOB["Deploy to Production<br/>EKS Production<br/>Blue-Green Deployment"]
    end

    subgraph "Development Environment"
        DEV_CLUSTER["Kind Cluster<br/>Local Kubernetes<br/>Full Stack"]
        DEV_MONITORING["Development Monitoring<br/>Prometheus + Grafana<br/>Local Stack"]
    end

    subgraph "Staging Environment"
        STAGING_EKS["EKS Staging Cluster<br/>Full Infrastructure<br/>Production-like"]
        STAGING_DB["Staging Databases<br/>Neo4j, PostgreSQL, Redis<br/>Test Data"]
        STAGING_MONITORING["Staging Monitoring<br/>CloudWatch, X-Ray<br/>Alert Testing"]
    end

    subgraph "Production Environment"
        PROD_EKS["EKS Production Cluster<br/>High Availability<br/>Auto-scaling"]
        PROD_DB["Production Databases<br/>Multi-AZ, Backups<br/>Real Data"]
        PROD_CDN["CloudFront CDN<br/>Global Distribution<br/>Edge Locations"]
        PROD_MONITORING["Production Monitoring<br/>24/7 Alerting<br/>Incident Response"]
    end

    subgraph "Quality Gates"
        CODE_REVIEW["Code Review<br/>Pull Request Review<br/>Architecture Review"]
        SECURITY_REVIEW["Security Review<br/>Dependency Check<br/>Secrets Scanning"]
        PERFORMANCE_TEST["Performance Test<br/>Load Testing<br/>Scalability Check"]
        COMPLIANCE_CHECK["Compliance Check<br/>GDPR, SOC2<br/>Audit Logging"]
    end

    subgraph "Artifact Repository"
        GHCR["GitHub Container Registry<br/>Docker Images<br/>Vulnerability Scanning"]
        ARTIFACTS["Test Artifacts<br/>Coverage Reports<br/>Performance Metrics"]
    end

    %% Development flow
    LOCAL_DEV --> FEATURE_BRANCHES
    FEATURE_BRANCHES --> CODE_REVIEW
    CODE_REVIEW --> MAIN_BRANCH

    MAIN_BRANCH --> LINT_JOB
    LINT_JOB --> TEST_JOB
    TEST_JOB --> BUILD_JOB
    BUILD_JOB --> SECURITY_JOB

    SECURITY_JOB --> DEPLOY_DEV_JOB
    DEPLOY_DEV_JOB --> DEV_CLUSTER

    DEV_CLUSTER --> SECURITY_REVIEW
    SECURITY_REVIEW --> RELEASE_BRANCHES

    RELEASE_BRANCHES --> PERFORMANCE_TEST
    PERFORMANCE_TEST --> DEPLOY_STAGING_JOB
    DEPLOY_STAGING_JOB --> STAGING_EKS

    STAGING_EKS --> COMPLIANCE_CHECK
    COMPLIANCE_CHECK --> DEPLOY_PROD_JOB
    DEPLOY_PROD_JOB --> PROD_EKS

    %% Artifact flow
    BUILD_JOB --> GHCR
    TEST_JOB --> ARTIFACTS
    SECURITY_JOB --> ARTIFACTS

    %% Monitoring flow
    DEV_CLUSTER --> DEV_MONITORING
    STAGING_EKS --> STAGING_MONITORING
    PROD_EKS --> PROD_MONITORING

    %% Database flow
    DEPLOY_DEV_JOB -.-> DEV_CLUSTER
    DEPLOY_STAGING_JOB --> STAGING_DB
    DEPLOY_PROD_JOB --> PROD_DB
    PROD_EKS --> PROD_CDN