turash/concept/13_apis_and_ingestion.md

## 11. APIs and Ingestion

### External Data Feeds

**Ingestion Performance Targets**:
- **Throughput**: 1,000+ data sources processed/hour
- **Latency**: <5 minutes from source to searchable matches
- **Reliability**: 99.5% successful ingestion rate
- **Data Quality**: 90%+ automated validation pass rate

| Data Source | Format | Volume | Frequency | Processing Time |
|-------------|--------|--------|-----------|----------------|
| Smart meters | Modbus → MQTT → Kafka | 10k devices | Real-time | <30s |
| ERP / SCADA | CSV, JSON, API | 500 systems | Daily | <10min |
| Manual forms | Web forms | 100 entries/day | Real-time | <5s |
| Public datasets | Bulk files, APIs | 1M records | Weekly | <2h |

### Internal APIs

**Performance Targets**:
- **Throughput**: 10,000+ requests/second sustained load
- **Latency**: p95 <100ms, p99 <500ms for all endpoints
- **Availability**: 99.9% uptime, <1 second MTTR
- **Data Volume**: 1M+ resource flows, 100k+ businesses supported

| Endpoint              | Function                    | Target Response | Daily Volume |
| --------------------- | --------------------------- | -------------- | ----------- |
| `/facilities`         | CRUD on nodes               | <50ms         | 10k requests |
| `/resources`          | CRUD on flows               | <100ms        | 50k requests |
| `/match`              | returns candidate matches   | <2s           | 5k requests  |
| `/economics/simulate` | compute ROI and CO₂ savings | <500ms        | 2k requests  |
| `/map/geojson`        | render data for front-end   | <200ms        | 20k requests |

### REST API Enhancements

**Recommendations**:

1. **API Versioning**:
   - URL versioning: `/api/v1/resources`, `/api/v2/resources`
   - Header versioning: `Accept: application/vnd.api+json;version=1`
   - Version strategy documented in ADR

2. **Pagination**:
   - Cursor-based pagination for graph queries (more efficient than offset)
   - Consistent pagination format across all list endpoints

3. **Filtering & Sorting**:
   - Query parameters: `?filter[type]=heat&filter[direction]=output&sort=-created_at`
   - GraphQL alternative for complex filtering

4. **Error Handling**:
   - Consistent error format (RFC 7807 Problem Details)
   - Proper HTTP status codes
   - Error codes for programmatic handling

5. **Rate Limiting**:
   - Per-user rate limits
   - Tiered limits (free, paid, enterprise)
   - Rate limit headers (`X-RateLimit-*`)

### GraphQL Consideration

**Recommendation**: Consider GraphQL for frontend flexibility.

**Benefits**:
- Frontend requests exactly needed data
- Reduces over-fetching
- Single endpoint for complex queries
- Strong typing with schema

**Implementation**:
- **GraphQL Server**: `github.com/99designs/gqlgen` (schema-first, code generation)
  - Type-safe resolvers
  - Built-in DataLoader support for N+1 prevention
  - Subscriptions for real-time updates
- **Alternative**: `github.com/graphql-go/graphql` (runtime-first)

### Open Standards Integration (Critical for Smart City Interoperability)

**Recommendation**: Adopt NGSI-LD API and OGC SensorThings API from day one.

**NGSI-LD API** (FIWARE Standard):
- **Purpose**: Graph-based context information model for smart city interoperability
- **Benefits**: 
  - Enables integration with 40%+ of smart city deployments using FIWARE-compliant infrastructure
  - Reduces integration costs by 50-60% through standardized APIs
  - Graph-based model aligns perfectly with platform architecture
- **Implementation**: 
  - Graph-based context information modeling
  - Standardized API endpoints following FIWARE NGSI-LD specification
  - Estimated development overhead: 2-3 weeks for schema mapping and API layer
- **Use Cases**: 
  - Municipal platform integration
  - Utility system interoperability
  - Public-private partnership deployments (40% of market)

**OGC SensorThings API**:
- **Purpose**: Standardized sensor data exposure for IoT integration
- **Benefits**:
  - Critical for IoT device integration (Modbus, MQTT data ingestion)
  - Widely adopted in smart city deployments
  - Enables standardized sensor data access
- **Implementation**:
  - Standard RESTful API following OGC SensorThings specification
  - Supports MQTT, Modbus, SCADA data ingestion
  - Integration with smart meter infrastructure

**Strategic Value**:
- **Market Access**: 40% of smart city platform deployments in 2024 leveraged public-private partnerships using open standards
- **Competitive Advantage**: Platform differentiation through interoperability
- **Revenue Opportunity**: Municipal/utility licensing enabled through standards compliance

### WebSocket API

**Recommendation**: Real-time API for match updates.

**Endpoints**:
- `/ws/matches`: Subscribe to match updates for user's resources
- `/ws/notifications`: Real-time notifications (new matches, messages)
- `/ws/analytics`: Live analytics dashboard updates

**Implementation**:
- **Go WebSocket Libraries**:
  - `github.com/gorilla/websocket`: Mature, widely used
  - `nhooyr.io/websocket`: Modern, performant alternative
  - `github.com/gobwas/ws`: Low-level, high-performance option
- Authentication via JWT in handshake
- Room-based subscriptions (per business, per resource)
- **Go Pattern**: One goroutine per connection, channel-based message broadcasting

### Authentication & Authorization

**Recommendation**: Comprehensive auth strategy.

**Components**:
1. **Identity Provider**: Keycloak (self-hosted) or Auth0 (SaaS)
   - Support: OAuth2, OIDC, SAML for enterprise SSO
   - Multi-factor authentication (MFA)
   - Passwordless options (WebAuthn)

2. **API Authentication**:
   - JWT tokens for stateless API access
   - API keys for service-to-service communication
   - Refresh token rotation

3. **Authorization** (RBAC - Role-Based Access Control):
   - **Roles**: Admin, Business Owner, Site Manager, Viewer, Analyst
   - **Permissions**: 
     - Read: Public matches, own business data
     - Write: Own business/site/resource flows
     - Delete: Own data only
     - Admin: All operations

4. **Graph-Level Security**:
   - Use Neo4j security rules or application-level filtering
   - Row-level security based on tenant_id and ownership

---