mirror of
https://github.com/SamyRai/turash.git
synced 2025-12-26 23:01:33 +00:00
Repository Structure:
- Move files from cluttered root directory into organized structure
- Create archive/ for archived data and scraper results
- Create bugulma/ for the complete application (frontend + backend)
- Create data/ for sample datasets and reference materials
- Create docs/ for comprehensive documentation structure
- Create scripts/ for utility scripts and API tools
Backend Implementation:
- Implement 3 missing backend endpoints identified in gap analysis:
* GET /api/v1/organizations/{id}/matching/direct - Direct symbiosis matches
* GET /api/v1/users/me/organizations - User organizations
* POST /api/v1/proposals/{id}/status - Update proposal status
- Add complete proposal domain model, repository, and service layers
- Create database migration for proposals table
- Fix CLI server command registration issue
API Documentation:
- Add comprehensive proposals.md API documentation
- Update README.md with Users and Proposals API sections
- Document all request/response formats, error codes, and business rules
Code Quality:
- Follow existing Go backend architecture patterns
- Add proper error handling and validation
- Match frontend expected response schemas
- Maintain clean separation of concerns (handler -> service -> repository)
606 lines
22 KiB
Markdown
606 lines
22 KiB
Markdown
# 29. Technical Architecture Diagrams
|
||
|
||
## System Context Diagram
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "External Systems"
|
||
ERP[ERP Systems<br/>SAP, Oracle]
|
||
SCADA[SCADA Systems<br/>OPC UA, Modbus]
|
||
IoT[IoT Devices<br/>Sensors, Meters]
|
||
UTIL[Utility APIs<br/>Energy, Water]
|
||
GOV[Government Data<br/>Building Permits, NACE]
|
||
end
|
||
|
||
subgraph "Turash Platform"
|
||
API[API Gateway<br/>Kong/Traefik]
|
||
AUTH[Authentication Service<br/>JWT, OAuth2]
|
||
|
||
subgraph "Application Layer"
|
||
WEB[Web Frontend<br/>Next.js + React]
|
||
MOBILE[Mobile PWA<br/>Progressive Web App]
|
||
ADMIN[Admin Dashboard<br/>Enterprise Features]
|
||
end
|
||
|
||
subgraph "Service Layer"
|
||
MATCH[Matching Engine<br/>Graph Algorithms]
|
||
ECON[Economic Calculator<br/>NPV, ROI Models]
|
||
NOTIF[Notification Service<br/>WebSocket, Email]
|
||
SEARCH[Search Service<br/>Full-text, Spatial]
|
||
end
|
||
|
||
subgraph "Data Layer"
|
||
NEO4J[(Neo4j Graph DB<br/>Relationships, Entities)]
|
||
POSTGRES[(PostgreSQL + PostGIS<br/>Spatial, Time-series)]
|
||
REDIS[(Redis Cache<br/>Sessions, Matches)]
|
||
end
|
||
|
||
subgraph "Infrastructure"
|
||
K8S[Kubernetes Cluster<br/>AWS EKS/GCP GKE]
|
||
MQ["Message Queue<br/>NATS/Redis Streams (MVP)<br/>Kafka (Scale)"]
|
||
CDN[CDN<br/>CloudFront/Cloudflare]
|
||
MONITOR[Monitoring Stack<br/>Prometheus + Grafana]
|
||
end
|
||
end
|
||
|
||
subgraph "Users & Partners"
|
||
SME[SME Businesses<br/>Resource Providers]
|
||
ENTERPRISE[Enterprise Companies<br/>Multi-site Operations]
|
||
MUNI[Municipalities<br/>City Governments]
|
||
UTIL_PART[Utility Partners<br/>Data & Distribution]
|
||
FACIL[Facilitators<br/>Engineers, Consultants]
|
||
end
|
||
|
||
%% Connections
|
||
ERP --> API
|
||
SCADA --> API
|
||
IoT --> API
|
||
UTIL --> API
|
||
GOV --> API
|
||
|
||
API --> AUTH
|
||
AUTH --> WEB
|
||
AUTH --> MOBILE
|
||
AUTH --> ADMIN
|
||
|
||
WEB --> MATCH
|
||
MOBILE --> MATCH
|
||
ADMIN --> MATCH
|
||
|
||
MATCH --> ECON
|
||
MATCH --> NOTIF
|
||
MATCH --> SEARCH
|
||
|
||
MATCH --> NEO4J
|
||
ECON --> POSTGRES
|
||
SEARCH --> REDIS
|
||
|
||
NEO4J --> K8S
|
||
POSTGRES --> K8S
|
||
REDIS --> K8S
|
||
|
||
NOTIF --> MQ
|
||
MQ --> MONITOR
|
||
|
||
SME --> WEB
|
||
ENTERPRISE --> ADMIN
|
||
MUNI --> API
|
||
UTIL_PART --> API
|
||
FACIL --> WEB
|
||
```
|
||
|
||
## Application Architecture (C4 Container Diagram)
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "User Interfaces"
|
||
WEB_BROWSER[Web Browser<br/>React + Next.js]
|
||
MOBILE_APP[Mobile PWA<br/>React + Service Worker]
|
||
ADMIN_PORTAL[Admin Portal<br/>Enterprise Dashboard]
|
||
end
|
||
|
||
subgraph "API Gateway"
|
||
KONG["API Gateway<br/>Kong/Traefik<br/>Rate Limiting, Auth"]
|
||
end
|
||
|
||
subgraph "Microservices"
|
||
AUTH_SERVICE["Authentication Service<br/>Go 1.25 + JWT<br/>User Management"]
|
||
BUSINESS_SERVICE["Business Service<br/>Go 1.25 + Neo4j<br/>CRUD Operations"]
|
||
MATCHING_SERVICE["Matching Service<br/>Go 1.25 + Graph Algorithms<br/>Core Engine + Hybrid Automation"]
|
||
ECONOMIC_SERVICE["Economic Service<br/>Go 1.25 + PostgreSQL<br/>Calculations"]
|
||
NOTIFICATION_SERVICE["Notification Service<br/>Go 1.25 + Redis<br/>WebSocket/Email"]
|
||
INGESTION_SERVICE["Ingestion Service<br/>Go 1.25 + NATS/Redis Streams<br/>Data Processing"]
|
||
SEARCH_SERVICE["Search Service<br/>Go 1.25 + Elasticsearch<br/>Full-text Search"]
|
||
ANALYTICS_SERVICE["Analytics Service<br/>Go 1.25 + TimescaleDB<br/>Reporting"]
|
||
DATA_QUALITY_SERVICE["Data Quality Service<br/>Go 1.25 + PostgreSQL<br/>Scoring & Incentives"]
|
||
FACILITATOR_SERVICE["Facilitator Service<br/>Go 1.25 + Neo4j<br/>Marketplace & Routing"]
|
||
end
|
||
|
||
subgraph "Data Stores"
|
||
NEO4J["(Neo4j Cluster<br/>Graph Database<br/>Relationships)"]
|
||
POSTGRES["(PostgreSQL Cluster<br/>Spatial Database<br/>Business Data)"]
|
||
REDIS["(Redis Cluster<br/>Cache + Sessions<br/>Performance)"]
|
||
NATS["(NATS/Redis Streams<br/>Event Streaming (MVP)<br/>Async Processing)"]
|
||
KAFKA["(Kafka Cluster<br/>Event Streaming (Scale)<br/>Migration at 1000+ businesses)"]
|
||
ELASTIC["(Elasticsearch<br/>Search Index<br/>Full-text)"]
|
||
MINIO["(MinIO/S3<br/>Object Storage<br/>Files & Assets)"]
|
||
end
|
||
|
||
subgraph "Infrastructure Services"
|
||
PROMETHEUS[Prometheus<br/>Metrics Collection]
|
||
GRAFANA[Grafana<br/>Visualization]
|
||
ALERTMANAGER[AlertManager<br/>Incident Management]
|
||
LOGGING[Loki + ELK<br/>Log Aggregation]
|
||
VAULT[HashiCorp Vault<br/>Secrets Management]
|
||
end
|
||
|
||
%% Internal connections
|
||
WEB_BROWSER --> KONG
|
||
MOBILE_APP --> KONG
|
||
ADMIN_PORTAL --> KONG
|
||
|
||
KONG --> AUTH_SERVICE
|
||
AUTH_SERVICE --> BUSINESS_SERVICE
|
||
BUSINESS_SERVICE --> MATCHING_SERVICE
|
||
BUSINESS_SERVICE --> DATA_QUALITY_SERVICE
|
||
MATCHING_SERVICE --> ECONOMIC_SERVICE
|
||
MATCHING_SERVICE --> NOTIFICATION_SERVICE
|
||
MATCHING_SERVICE --> FACILITATOR_SERVICE
|
||
BUSINESS_SERVICE --> INGESTION_SERVICE
|
||
BUSINESS_SERVICE --> SEARCH_SERVICE
|
||
BUSINESS_SERVICE --> ANALYTICS_SERVICE
|
||
|
||
BUSINESS_SERVICE --> NEO4J
|
||
MATCHING_SERVICE --> NEO4J
|
||
FACILITATOR_SERVICE --> NEO4J
|
||
ECONOMIC_SERVICE --> POSTGRES
|
||
DATA_QUALITY_SERVICE --> POSTGRES
|
||
NOTIFICATION_SERVICE --> REDIS
|
||
INGESTION_SERVICE --> NATS
|
||
SEARCH_SERVICE --> ELASTIC
|
||
ANALYTICS_SERVICE --> MINIO
|
||
|
||
MATCHING_SERVICE --> PROMETHEUS
|
||
PROMETHEUS --> GRAFANA
|
||
PROMETHEUS --> ALERTMANAGER
|
||
|
||
ALL_SERVICES --> LOGGING
|
||
ALL_SERVICES --> VAULT
|
||
```
|
||
|
||
## Data Architecture Diagram
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph "Data Sources"
|
||
ERP["ERP Systems<br/>SAP, Oracle<br/>Transactional Data"]
|
||
SCADA["SCADA Systems<br/>OPC UA, Modbus<br/>Real-time Process Data"]
|
||
IOT_SENSORS["IoT Sensors<br/>Temperature, Flow, Pressure<br/>Time-series Data"]
|
||
MANUAL["Manual Entry<br/>Web Forms, CSV Upload<br/>User-Provided Data"]
|
||
PUBLIC_DATA["Public Datasets<br/>Government, Utilities<br/>Reference Data"]
|
||
end
|
||
|
||
subgraph "Ingestion Layer"
|
||
API_INGEST["API Ingestion<br/>REST/WebSocket<br/>Real-time"]
|
||
BATCH_INGEST["Batch Ingestion<br/>CSV, XML, JSON<br/>Scheduled"]
|
||
STREAM_INGEST["Stream Processing<br/>NATS/Redis Streams/MQTT<br/>Event-driven"]
|
||
WEBHOOKS["Webhook Ingestion<br/>ERP/SCADA Push<br/>Event-triggered"]
|
||
end
|
||
|
||
subgraph "Processing Layer"
|
||
VALIDATION["Data Validation<br/>Schema, Business Rules<br/>Quality Gates"]
|
||
NORMALIZATION["Data Normalization<br/>Unit Conversion, Standardization<br/>Data Harmonization"]
|
||
ENRICHMENT["Data Enrichment<br/>Geocoding, Classification<br/>Context Addition"]
|
||
QUALITY_SCORING["Data Quality Scoring<br/>Precision Levels: Rough/Estimated/Measured<br/>Trust Scoring & Incentives"]
|
||
DEDUPLICATION["Deduplication<br/>Entity Resolution<br/>Data Quality"]
|
||
end
|
||
|
||
subgraph "Storage Layer"
|
||
subgraph "Graph Database (Neo4j)"
|
||
BUSINESS_NODES["Business Nodes<br/>Companies, Legal Entities<br/>Properties: name, industry, size"]
|
||
SITE_NODES["Site Nodes<br/>Physical Locations<br/>Properties: address, coordinates, type"]
|
||
RESOURCE_NODES["Resource Flow Nodes<br/>Energy, Water, Waste<br/>Properties: type, amount, quality"]
|
||
SERVICE_NODES["Service Nodes<br/>Maintenance, Consulting<br/>Properties: domain, capacity, rate"]
|
||
end
|
||
|
||
subgraph "Relational Database (PostgreSQL)"
|
||
BUSINESS_TABLES["Business Tables<br/>Normalized Business Data<br/>Indexes: name, industry, location"]
|
||
SITE_TABLES["Site Tables<br/>Spatial Site Data<br/>PostGIS: coordinates, boundaries"]
|
||
RESOURCE_TABLES["Resource Tables<br/>Time-series Flows<br/>Partitioned by time, indexed by location"]
|
||
TRANSACTION_TABLES["Transaction Tables<br/>Match Records, Contracts<br/>Audit trail, compliance"]
|
||
end
|
||
|
||
subgraph "Cache Layer (Redis)"
|
||
MATCH_CACHE["Match Results Cache<br/>TTL: 15 minutes<br/>Key: location + resource_type"]
|
||
SESSION_CACHE["Session Cache<br/>JWT tokens, user state<br/>TTL: 24 hours"]
|
||
COMPUTATION_CACHE["Computation Cache<br/>Economic calculations<br/>TTL: 1 hour"]
|
||
end
|
||
end
|
||
|
||
subgraph "Access Layer"
|
||
GRAPH_API["Graph API<br/>Cypher Queries<br/>Relationship Traversal"]
|
||
REST_API["REST API<br/>CRUD Operations<br/>Standard HTTP"]
|
||
GRAPHQL_API["GraphQL API<br/>Flexible Queries<br/>Client-driven"]
|
||
WEBSOCKET_API["WebSocket API<br/>Real-time Updates<br/>Live Notifications"]
|
||
BULK_API["Bulk API<br/>Data Export/Import<br/>Batch Operations"]
|
||
end
|
||
|
||
subgraph "Consumption Layer"
|
||
WEB_APP["Web Application<br/>React + Mapbox<br/>User Interface"]
|
||
MOBILE_APP["Mobile PWA<br/>Progressive Web App<br/>Field Operations"]
|
||
API_CLIENTS["API Clients<br/>ERP Integration<br/>Third-party Systems"]
|
||
ANALYTICS["Analytics Platform<br/>Business Intelligence<br/>Reporting & Insights"]
|
||
ML_MODELS["ML Models<br/>Recommendation Engine<br/>Predictive Analytics"]
|
||
end
|
||
|
||
%% Data Flow
|
||
ERP --> API_INGEST
|
||
SCADA --> STREAM_INGEST
|
||
IOT_SENSORS --> STREAM_INGEST
|
||
MANUAL --> BATCH_INGEST
|
||
PUBLIC_DATA --> WEBHOOKS
|
||
|
||
API_INGEST --> VALIDATION
|
||
BATCH_INGEST --> VALIDATION
|
||
STREAM_INGEST --> VALIDATION
|
||
WEBHOOKS --> VALIDATION
|
||
|
||
VALIDATION --> NORMALIZATION
|
||
NORMALIZATION --> ENRICHMENT
|
||
ENRICHMENT --> QUALITY_SCORING
|
||
QUALITY_SCORING --> DEDUPLICATION
|
||
|
||
DEDUPLICATION --> BUSINESS_NODES
|
||
DEDUPLICATION --> SITE_NODES
|
||
DEDUPLICATION --> RESOURCE_NODES
|
||
DEDUPLICATION --> SERVICE_NODES
|
||
|
||
DEDUPLICATION --> BUSINESS_TABLES
|
||
DEDUPLICATION --> SITE_TABLES
|
||
DEDUPLICATION --> RESOURCE_TABLES
|
||
DEDUPLICATION --> TRANSACTION_TABLES
|
||
|
||
BUSINESS_NODES --> GRAPH_API
|
||
SITE_NODES --> GRAPH_API
|
||
RESOURCE_NODES --> GRAPH_API
|
||
SERVICE_NODES --> GRAPH_API
|
||
|
||
BUSINESS_TABLES --> REST_API
|
||
SITE_TABLES --> REST_API
|
||
RESOURCE_TABLES --> REST_API
|
||
TRANSACTION_TABLES --> REST_API
|
||
|
||
GRAPH_API --> GRAPHQL_API
|
||
REST_API --> GRAPHQL_API
|
||
|
||
GRAPHQL_API --> WEBSOCKET_API
|
||
GRAPHQL_API --> BULK_API
|
||
|
||
WEBSOCKET_API --> WEB_APP
|
||
WEBSOCKET_API --> MOBILE_APP
|
||
GRAPHQL_API --> API_CLIENTS
|
||
BULK_API --> ANALYTICS
|
||
REST_API --> ML_MODELS
|
||
|
||
%% Cache connections
|
||
MATCH_CACHE -.-> GRAPHQL_API
|
||
SESSION_CACHE -.-> WEBSOCKET_API
|
||
COMPUTATION_CACHE -.-> REST_API
|
||
```
|
||
|
||
## Matching Engine Architecture
|
||
|
||
```mermaid
|
||
graph TD
|
||
subgraph "Input Processing"
|
||
QUERY_IN[Resource Query<br/>Location, Type, Preferences]
|
||
CONTEXT_IN[Business Context<br/>Industry, Size, Capabilities]
|
||
CONSTRAINTS_IN[Matching Constraints<br/>Distance, Quality, Cost]
|
||
end
|
||
|
||
subgraph "Pre-filtering Stage"
|
||
SPATIAL_FILTER["Spatial Filter<br/>PostGIS Query<br/>Radius Search"]
|
||
BASIC_FILTER["Basic Filter<br/>Resource Type Match<br/>Availability Windows"]
|
||
QUALITY_FILTER["Quality Filter<br/>Grade/Purity Match<br/>Hazard Compatibility"]
|
||
end
|
||
|
||
subgraph "Candidate Selection"
|
||
GEO_INDEX["Geospatial Index<br/>R-tree/B-tree<br/>Location-based"]
|
||
RESOURCE_INDEX["Resource Index<br/>Composite Index<br/>Type + Direction + Location"]
|
||
TEMPORAL_INDEX["Temporal Index<br/>Time-window Index<br/>Availability Overlap"]
|
||
end
|
||
|
||
subgraph "Compatibility Scoring"
|
||
QUALITY_SCORER["Quality Scorer<br/>Temperature, Pressure, Purity<br/>Weighted Compatibility"]
|
||
TEMPORAL_SCORER["Temporal Scorer<br/>Availability Overlap<br/>Seasonal Factors"]
|
||
QUANTITY_SCORER["Quantity Scorer<br/>Supply/Demand Match<br/>Variability Adjustment"]
|
||
ECONOMIC_SCORER["Economic Scorer<br/>Cost Arbitrage<br/>Transport Costs"]
|
||
TRUST_SCORER["Trust Scorer<br/>Data Precision<br/>Source Validation"]
|
||
end
|
||
|
||
subgraph "Optimization Engine"
|
||
WEIGHTED_SUM["Weighted Sum<br/>Multi-criteria Scoring<br/>Configurable Weights"]
|
||
RANKING_ENGINE["Ranking Engine<br/>Top-N Selection<br/>Diversity Consideration"]
|
||
HYBRID_ROUTER["Hybrid Automation Router<br/>80% Automated, 20% Facilitated<br/>Match Complexity Scoring"]
|
||
CONSTRAINT_CHECKER["Constraint Checker<br/>Business Rules<br/>Regulatory Compliance"]
|
||
FALLBACK_ENGINE["Fallback Engine<br/>Relaxation Strategies<br/>Alternative Matches"]
|
||
end
|
||
|
||
subgraph "Result Processing"
|
||
ECONOMIC_CALCULATION["Economic Calculation<br/>NPV, Payback, ROI<br/>Detailed Projections"]
|
||
RISK_ASSESSMENT["Risk Assessment<br/>Technical, Market, Regulatory<br/>Probability Scoring"]
|
||
PARTNER_PACKETS["Partner Packet Generation<br/>Technical Specs, Contacts<br/>Legal Templates"]
|
||
NOTIFICATION_ENGINE["Notification Engine<br/>WebSocket Push<br/>Email/SMS Alerts"]
|
||
end
|
||
|
||
subgraph "Caching & Persistence"
|
||
RESULT_CACHE["Result Cache<br/>Redis TTL Cache<br/>15-minute expiry"]
|
||
MATCH_STORAGE["Match Storage<br/>PostgreSQL<br/>Audit Trail"]
|
||
ANALYTICS_STORAGE["Analytics Storage<br/>ClickHouse/Timescale<br/>Performance Metrics"]
|
||
end
|
||
|
||
%% Data flow
|
||
QUERY_IN --> SPATIAL_FILTER
|
||
CONTEXT_IN --> BASIC_FILTER
|
||
CONSTRAINTS_IN --> QUALITY_FILTER
|
||
|
||
SPATIAL_FILTER --> GEO_INDEX
|
||
BASIC_FILTER --> RESOURCE_INDEX
|
||
QUALITY_FILTER --> TEMPORAL_INDEX
|
||
|
||
GEO_INDEX --> QUALITY_SCORER
|
||
RESOURCE_INDEX --> TEMPORAL_SCORER
|
||
TEMPORAL_INDEX --> QUANTITY_SCORER
|
||
|
||
QUALITY_SCORER --> WEIGHTED_SUM
|
||
TEMPORAL_SCORER --> WEIGHTED_SUM
|
||
QUANTITY_SCORER --> WEIGHTED_SUM
|
||
ECONOMIC_SCORER --> WEIGHTED_SUM
|
||
TRUST_SCORER --> WEIGHTED_SUM
|
||
|
||
WEIGHTED_SUM --> RANKING_ENGINE
|
||
RANKING_ENGINE --> HYBRID_ROUTER
|
||
HYBRID_ROUTER --> CONSTRAINT_CHECKER
|
||
CONSTRAINT_CHECKER --> FALLBACK_ENGINE
|
||
|
||
FALLBACK_ENGINE --> ECONOMIC_CALCULATION
|
||
ECONOMIC_CALCULATION --> RISK_ASSESSMENT
|
||
RISK_ASSESSMENT --> PARTNER_PACKETS
|
||
PARTNER_PACKETS --> NOTIFICATION_ENGINE
|
||
|
||
NOTIFICATION_ENGINE --> RESULT_CACHE
|
||
RESULT_CACHE --> MATCH_STORAGE
|
||
MATCH_STORAGE --> ANALYTICS_STORAGE
|
||
```
|
||
|
||
## Deployment Architecture (AWS)
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph "Internet"
|
||
INTERNET[Global Users<br/>Web, Mobile, API]
|
||
end
|
||
|
||
subgraph "AWS CloudFront (CDN)"
|
||
CLOUDFRONT["CloudFront Distribution<br/>Global Edge Locations<br/>SSL Termination"]
|
||
end
|
||
|
||
subgraph "AWS Global Accelerator"
|
||
ACCELERATOR["Global Accelerator<br/>TCP/UDP Optimization<br/>Failover Routing"]
|
||
end
|
||
|
||
subgraph "AWS WAF & Shield"
|
||
WAF["Web Application Firewall<br/>OWASP Rules<br/>Rate Limiting"]
|
||
SHIELD["Shield Advanced<br/>DDoS Protection<br/>Auto Mitigation"]
|
||
end
|
||
|
||
subgraph "AWS Route 53"
|
||
ROUTE53["Route 53<br/>DNS Management<br/>Health Checks<br/>Failover"]
|
||
end
|
||
|
||
subgraph "AWS Application Load Balancer"
|
||
ALB["Application Load Balancer<br/>Layer 7 Routing<br/>SSL Offloading<br/>Health Checks"]
|
||
end
|
||
|
||
subgraph "AWS Elastic Kubernetes Service (EKS)"
|
||
EKS_CONTROL["Control Plane<br/>Managed Kubernetes<br/>Auto-scaling"]
|
||
|
||
subgraph "Worker Nodes (Auto-scaling Group)"
|
||
ON_DEMAND["On-Demand Nodes<br/>General Workloads<br/>t3.large × 3-10"]
|
||
SPOT["Spot Nodes<br/>Batch Processing<br/>c6i.xlarge × 2-20"]
|
||
end
|
||
|
||
subgraph "Kubernetes Services"
|
||
INGRESS["Ingress Controller<br/>Traefik/Nginx<br/>External Traffic"]
|
||
API_PODS["API Service Pods<br/>Go Applications<br/>Horizontal Scaling"]
|
||
WORKER_PODS["Worker Pods<br/>Matching Engine<br/>CPU Optimized"]
|
||
WEB_PODS["Web Pods<br/>Next.js SSR<br/>Memory Optimized"]
|
||
end
|
||
end
|
||
|
||
subgraph "AWS RDS (PostgreSQL)"
|
||
RDS_CLUSTER["RDS Aurora Cluster<br/>Multi-AZ<br/>PostGIS Enabled<br/>Read Replicas"]
|
||
RDS_PROXY["RDS Proxy<br/>Connection Pooling<br/>Failover Handling"]
|
||
end
|
||
|
||
subgraph "AWS ElastiCache (Redis)"
|
||
REDIS_CLUSTER["Redis Cluster<br/>Multi-AZ<br/>Backup Enabled<br/>Encryption"]
|
||
end
|
||
|
||
subgraph "AWS Neptune (Neo4j-compatible)"
|
||
NEPTUNE_CLUSTER["Neptune Cluster<br/>Graph Database<br/>Gremlin/Cypher<br/>Multi-AZ"]
|
||
end
|
||
|
||
subgraph "AWS MSK (Kafka)"
|
||
KAFKA_CLUSTER["MSK Cluster<br/>Event Streaming (Scale Phase)<br/>Multi-AZ, Migration at 1000+ businesses<br/>Auto-scaling"]
|
||
end
|
||
|
||
subgraph "Message Queue (MVP)"
|
||
NATS_CLUSTER["NATS/Redis Streams<br/>Event Streaming (MVP)<br/>Simple, Cost-effective<br/>Kubernetes-native"]
|
||
end
|
||
|
||
subgraph "AWS OpenSearch"
|
||
OPENSEARCH["OpenSearch Domain<br/>Full-text Search<br/>Multi-AZ<br/>KNN for ML"]
|
||
end
|
||
|
||
subgraph "AWS S3 & Glacier"
|
||
S3_BUCKETS["S3 Buckets<br/>Data Lake<br/>Backup Storage<br/>CDN Origin"]
|
||
GLACIER["Glacier Deep Archive<br/>Long-term Backup<br/>Compliance Storage"]
|
||
end
|
||
|
||
subgraph "AWS CloudWatch & X-Ray"
|
||
CLOUDWATCH["CloudWatch<br/>Metrics & Logs<br/>Alarms & Dashboards"]
|
||
XRAY["X-Ray<br/>Distributed Tracing<br/>Performance Insights"]
|
||
end
|
||
|
||
subgraph "AWS Systems Manager"
|
||
SSM["Parameter Store<br/>Configuration Management<br/>Secrets (non-sensitive)"]
|
||
end
|
||
|
||
subgraph "AWS Secrets Manager"
|
||
SECRETS_MANAGER["Secrets Manager<br/>Database Passwords<br/>API Keys<br/>Auto-rotation"]
|
||
end
|
||
|
||
subgraph "AWS Backup"
|
||
BACKUP_SERVICE["Backup Service<br/>Automated Backups<br/>Cross-region Replication<br/>Retention Policies"]
|
||
end
|
||
|
||
%% Traffic flow
|
||
INTERNET --> CLOUDFRONT
|
||
CLOUDFRONT --> ACCELERATOR
|
||
ACCELERATOR --> WAF
|
||
WAF --> SHIELD
|
||
SHIELD --> ROUTE53
|
||
ROUTE53 --> ALB
|
||
ALB --> INGRESS
|
||
|
||
INGRESS --> API_PODS
|
||
INGRESS --> WEB_PODS
|
||
|
||
API_PODS --> WORKER_PODS
|
||
|
||
API_PODS --> RDS_PROXY
|
||
RDS_PROXY --> RDS_CLUSTER
|
||
|
||
API_PODS --> REDIS_CLUSTER
|
||
WORKER_PODS --> REDIS_CLUSTER
|
||
|
||
API_PODS --> NEPTUNE_CLUSTER
|
||
WORKER_PODS --> NEPTUNE_CLUSTER
|
||
|
||
API_PODS --> NATS_CLUSTER
|
||
WORKER_PODS --> NATS_CLUSTER
|
||
API_PODS --> KAFKA_CLUSTER
|
||
WORKER_PODS --> KAFKA_CLUSTER
|
||
|
||
API_PODS --> OPENSEARCH
|
||
|
||
ALL_SERVICES --> S3_BUCKETS
|
||
S3_BUCKETS --> GLACIER
|
||
|
||
ALL_SERVICES --> CLOUDWATCH
|
||
ALL_SERVICES --> XRAY
|
||
|
||
ALL_SERVICES --> SSM
|
||
ALL_SERVICES --> SECRETS_MANAGER
|
||
|
||
RDS_CLUSTER --> BACKUP_SERVICE
|
||
NEPTUNE_CLUSTER --> BACKUP_SERVICE
|
||
REDIS_CLUSTER --> BACKUP_SERVICE
|
||
```
|
||
|
||
## CI/CD Pipeline Architecture
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph "Developer Workstations"
|
||
LOCAL_DEV["Local Development<br/>Docker Compose<br/>Kind Cluster"]
|
||
end
|
||
|
||
subgraph "GitHub Repository"
|
||
MAIN_BRANCH[main Branch<br/>Production Code]
|
||
FEATURE_BRANCHES[feature/* Branches<br/>Development]
|
||
RELEASE_BRANCHES[release/* Branches<br/>Staging]
|
||
end
|
||
|
||
subgraph "GitHub Actions"
|
||
LINT_JOB["Lint & Format<br/>golangci-lint<br/>ESLint"]
|
||
TEST_JOB["Unit & Integration Tests<br/>Go 1.25 test (with feature flags)<br/>Jest, Coverage Reports"]
|
||
BUILD_JOB["Build & Containerize<br/>Go 1.25 build (fallback to 1.23)<br/>Docker Build, Multi-arch Images"]
|
||
SECURITY_JOB["Security Scan<br/>Trivy, Snyk<br/>Vulnerability Check"]
|
||
DEPLOY_DEV_JOB["Deploy to Development<br/>Kind Cluster<br/>Integration Tests"]
|
||
DEPLOY_STAGING_JOB["Deploy to Staging<br/>EKS Staging<br/>E2E Tests"]
|
||
DEPLOY_PROD_JOB["Deploy to Production<br/>EKS Production<br/>Blue-Green Deployment"]
|
||
end
|
||
|
||
subgraph "Development Environment"
|
||
DEV_CLUSTER["Kind Cluster<br/>Local Kubernetes<br/>Full Stack"]
|
||
DEV_MONITORING["Development Monitoring<br/>Prometheus + Grafana<br/>Local Stack"]
|
||
end
|
||
|
||
subgraph "Staging Environment"
|
||
STAGING_EKS["EKS Staging Cluster<br/>Full Infrastructure<br/>Production-like"]
|
||
STAGING_DB["Staging Databases<br/>Neo4j, PostgreSQL, Redis<br/>Test Data"]
|
||
STAGING_MONITORING["Staging Monitoring<br/>CloudWatch, X-Ray<br/>Alert Testing"]
|
||
end
|
||
|
||
subgraph "Production Environment"
|
||
PROD_EKS["EKS Production Cluster<br/>High Availability<br/>Auto-scaling"]
|
||
PROD_DB["Production Databases<br/>Multi-AZ, Backups<br/>Real Data"]
|
||
PROD_CDN["CloudFront CDN<br/>Global Distribution<br/>Edge Locations"]
|
||
PROD_MONITORING["Production Monitoring<br/>24/7 Alerting<br/>Incident Response"]
|
||
end
|
||
|
||
subgraph "Quality Gates"
|
||
CODE_REVIEW["Code Review<br/>Pull Request Review<br/>Architecture Review"]
|
||
SECURITY_REVIEW["Security Review<br/>Dependency Check<br/>Secrets Scanning"]
|
||
PERFORMANCE_TEST["Performance Test<br/>Load Testing<br/>Scalability Check"]
|
||
COMPLIANCE_CHECK["Compliance Check<br/>GDPR, SOC2<br/>Audit Logging"]
|
||
end
|
||
|
||
subgraph "Artifact Repository"
|
||
GHCR["GitHub Container Registry<br/>Docker Images<br/>Vulnerability Scanning"]
|
||
ARTIFACTS["Test Artifacts<br/>Coverage Reports<br/>Performance Metrics"]
|
||
end
|
||
|
||
%% Development flow
|
||
LOCAL_DEV --> FEATURE_BRANCHES
|
||
FEATURE_BRANCHES --> CODE_REVIEW
|
||
CODE_REVIEW --> MAIN_BRANCH
|
||
|
||
MAIN_BRANCH --> LINT_JOB
|
||
LINT_JOB --> TEST_JOB
|
||
TEST_JOB --> BUILD_JOB
|
||
BUILD_JOB --> SECURITY_JOB
|
||
|
||
SECURITY_JOB --> DEPLOY_DEV_JOB
|
||
DEPLOY_DEV_JOB --> DEV_CLUSTER
|
||
|
||
DEV_CLUSTER --> SECURITY_REVIEW
|
||
SECURITY_REVIEW --> RELEASE_BRANCHES
|
||
|
||
RELEASE_BRANCHES --> PERFORMANCE_TEST
|
||
PERFORMANCE_TEST --> DEPLOY_STAGING_JOB
|
||
DEPLOY_STAGING_JOB --> STAGING_EKS
|
||
|
||
STAGING_EKS --> COMPLIANCE_CHECK
|
||
COMPLIANCE_CHECK --> DEPLOY_PROD_JOB
|
||
DEPLOY_PROD_JOB --> PROD_EKS
|
||
|
||
%% Artifact flow
|
||
BUILD_JOB --> GHCR
|
||
TEST_JOB --> ARTIFACTS
|
||
SECURITY_JOB --> ARTIFACTS
|
||
|
||
%% Monitoring flow
|
||
DEV_CLUSTER --> DEV_MONITORING
|
||
STAGING_EKS --> STAGING_MONITORING
|
||
PROD_EKS --> PROD_MONITORING
|
||
|
||
%% Database flow
|
||
DEPLOY_DEV_JOB -.-> DEV_CLUSTER
|
||
DEPLOY_STAGING_JOB --> STAGING_DB
|
||
DEPLOY_PROD_JOB --> PROD_DB
|
||
PROD_EKS --> PROD_CDN
|
||
```
|