* docs: Update TASKS.md and PRODUCTION-TASKS.md to reflect current codebase state (December 2024 audit)
* refactor: Unify all commands into a single Cobra CLI
- Refactor cmd/api/main.go into 'tercul serve' command
- Refactor cmd/worker/main.go into 'tercul worker' command
- Refactor cmd/tools/enrich/main.go into 'tercul enrich' command
- Add 'tercul bleve-migrate' command for Bleve index migration
- Extract common initialization logic into cmd/cli/internal/bootstrap
- Update Dockerfile to build unified CLI
- Update README with new CLI usage
This consolidates all entry points into a single, maintainable CLI structure.
* fix: Fix CodeQL workflow and add comprehensive test coverage
- Fix Go version mismatch by setting up Go before CodeQL init
- Add Go version verification step
- Improve error handling for code scanning upload
- Add comprehensive test suite for CLI commands:
- Bleve migration tests with in-memory indexes
- Edge case tests (empty data, large batches, errors)
- Command-level integration tests
- Bootstrap initialization tests
- Optimize tests to use in-memory Bleve indexes for speed
- Add test tags for skipping slow tests in short mode
- Update workflow documentation
Test coverage: 18.1% with 806 lines of test code
All tests passing in short mode
* fix: Fix test workflow and Bleve test double-close panic
- Add POSTGRES_USER to PostgreSQL service configuration in test workflow
- Fix TestInitBleveIndex double-close panic by removing defer before explicit close
- Test now passes successfully
Fixes failing Unit Tests workflow in PR #64
Add configurable SearchAlpha parameter for hybrid search tuning
- Added SearchAlpha config parameter (default: 0.7) for tuning BM25 vs vector search balance
- Updated NewWeaviateWrapper to accept host and searchAlpha parameters
- Enhanced hybrid search with configurable alpha parameter via WithAlpha()
- Fixed all type mismatches in mocks and tests to use domainsearch.SearchResults
- Updated GraphQL resolver to use new SearchResults structure with SearchResultItem
- All tests and vet checks passing
Closes#30
* Fix workflow triggers to use 'main' branch instead of 'master'
* Switch to semantic version tags for GitHub Actions instead of SHAs for better maintainability
* Fix golangci-lint by adding go mod tidy and specifying paths ./... for linting
* feat: Restructure workflows following Single Responsibility Principle
- Remove old monolithic workflows (ci.yml, ci-cd.yml, cd.yml)
- Add focused workflows: lint.yml, test.yml, build.yml, security.yml, docker-build.yml, deploy.yml
- Each workflow has a single, clear responsibility
- Follow 2025 best practices with semantic versioning, OIDC auth, build attestations
- Add comprehensive README.md with workflow documentation
- Configure Dependabot for automated dependency updates
Workflows now run independently and can be triggered separately for better CI/CD control.
* fix: Resolve CI/CD workflow failures and GraphQL integration test issues
- Fix Application struct mismatch in application_builder.go
- Add global config.Cfg variable and BleveIndexPath field
- Regenerate GraphQL code to fix ProcessArgField errors
- Add search.InitBleve() call in main.go
- Fix all errcheck issues (12 total) in main.go files and test files
- Fix staticcheck issues (deprecated handler.NewDefaultServer, tagged switch)
- Remove all unused code (50 unused items including mock implementations)
- Fix GraphQL 'transport not supported' error in integration tests
- Add comprehensive database cleanup for integration tests
- Update GraphQL server setup with proper error presenter
* feat: Complete backend CI/CD workflow setup
- Add comprehensive GitHub Actions workflows for Go backend
- Build workflow with binary compilation and attestation
- Test workflow with coverage reporting and race detection
- Lint workflow with golangci-lint and security scanning
- Docker build workflow with multi-architecture support
- Deploy workflow for production deployment
- Security workflow with vulnerability scanning
- All workflows follow Single Responsibility Principle
- Use semantic versioning and latest action versions
- Enable security features: OIDC auth, attestations, minimal permissions
* fix: correct Go build path to ./cmd/api
- Fix build workflow to target ./cmd/api instead of ./cmd
- The main.go file is located in cmd/api/ subdirectory
* fix: correct Dockerfile build path to ./cmd/api
- Fix Docker build to target ./cmd/api instead of root directory
- The main.go file is located in cmd/api/ subdirectory
This commit significantly increases the test coverage across the application and fixes several underlying bugs that were discovered while writing the new tests.
The key changes include:
- **New Tests:** Added extensive integration and unit tests for GraphQL resolvers, application services, and data repositories, substantially increasing the test coverage for packages like `graphql`, `user`, `translation`, and `analytics`.
- **Authorization Bug Fixes:**
- Fixed a critical bug where a user creating a `Work` was not correctly associated as its author, causing subsequent permission failures.
- Corrected the authorization logic in `authz.Service` to properly check for entity ownership by non-admin users.
- **Test Refactoring:**
- Refactored numerous test suites to use `testify/mock` instead of manual mocks, improving test clarity and maintainability.
- Isolated integration tests by creating a fresh admin user and token for each test run, eliminating test pollution.
- Centralized domain errors into `internal/domain/errors.go` and updated repositories to use them, making error handling more consistent.
- **Code Quality Improvements:**
- Replaced manual mock implementations with `testify/mock` for better consistency.
- Cleaned up redundant and outdated test files.
These changes stabilize the test suite, improve the overall quality of the codebase, and move the project closer to the goal of 80% test coverage.
This commit improves the test coverage for the `internal/data/sql`
package by adding comprehensive tests for several repositories and
refactoring the testing strategy to be more robust.
The following changes were made:
- Refactored the `analytics_repository_test.go` to use an in-memory
SQLite database instead of `sqlmock`. This makes the tests more
reliable and less brittle against GORM's SQL generation.
- Added new tests for `auth_repository.go` and
`copyright_claim_repository.go` using the same in-memory SQLite
database strategy.
- Added the missing `WorkID` field to the `CopyrightClaim` domain
entity to align it with the repository's logic.
This effort increased the test coverage for the `internal/data/sql`
package from 30.5% to 37.1%.
This commit introduces a comprehensive test suite for the `RedisCache`
implementation in `internal/platform/cache/redis_cache.go`.
The following changes were made:
- Added `go-redis/redismock/v9` as a dev dependency to mock the Redis
client.
- Created `internal/platform/cache/redis_cache_test.go` with tests for
all public methods of `RedisCache`.
- Covered various scenarios, including success cases, cache misses,
and Redis errors.
- Tested basic operations (Get, Set, Delete, Clear), multi-key
operations (GetMulti, SetMulti), entity-specific helpers, and the
`InvalidateEntityType` method.
This effort increased the test coverage for the `internal/platform/cache`
package from 10.3% to 89.7%, contributing to the overall goal of
achieving >80% test coverage for the project.
This commit increases the test coverage of the `internal/app/work` package from 73.1% to over 80% by adding new tests and fixing a bug discovered during testing.
The following changes were made:
- Added tests for the `ListByCollectionID` query in `queries_test.go`.
- Added a unit test for the `NewService` constructor in `service_test.go`.
- Added tests for authorization, unauthorized access, and other edge cases in the `UpdateWork`, `DeleteWork`, and `MergeWork` commands in `commands_test.go`.
- Fixed a bug in the `mergeWorkStats` function where it was not correctly creating stats for a target work that had no prior stats. This was discovered and fixed as part of writing the new tests.
- Updated the `analytics.Service` interface and its mock implementation to support the bug fix.
This commit introduces a new test suite for the `ChangePassword` GraphQL mutation in `internal/adapters/graphql/auth_mutations_test.go`.
The new tests cover the following scenarios:
- A user successfully changes their password.
- The mutation fails when the provided current password is incorrect.
- The mutation fails when the request is made by an unauthenticated user.
This commit introduces a new test suite for the `DeleteUser` GraphQL mutation in `internal/adapters/graphql/user_mutations_test.go`. The tests cover successful deletion by an admin and by the user themselves, as well as failure cases for invalid permissions, non-existent users, and invalid input.
During the implementation of these tests, an inconsistency in error handling was discovered. The `internal/data/sql` repositories were using a mix of local and domain-level errors. This has been refactored to consistently use the centralized errors defined in the `internal/domain` package. This change improves the robustness and predictability of the data layer.
The following files were modified to standardize error handling:
- internal/data/sql/base_repository.go
- internal/data/sql/book_repository.go
- internal/data/sql/category_repository.go
- internal/data/sql/copyright_repository.go
- internal/data/sql/country_repository.go
- internal/data/sql/edition_repository.go
- internal/data/sql/email_verification_repository.go
- internal/data/sql/password_reset_repository.go
- internal/data/sql/source_repository.go
- internal/data/sql/tag_repository.go
- internal/data/sql/user_profile_repository.go
- internal/data/sql/user_repository.go
- internal/data/sql/user_session_repository.go
- internal/data/sql/work_repository.go
- internal/data/sql/base_repository_test.go
- internal/data/sql/copyright_repository_test.go
Key changes include:
- **Architectural Refactoring (CQRS/DTOs):** Refactored the `work` and `translation` application services to use Data Transfer Objects (DTOs) for query responses. This separates the domain layer from the API layer, improving maintainability and performance.
- **Implemented Core Business Logic:** Implemented the `AnalyzeWork` command, which was previously a stub. This command now performs linguistic analysis on works and translations by calling the analytics service.
- **Dependency Injection Improvements:**
- Refactored the configuration loading in `internal/platform/config/config.go` to use a local `viper` instance, removing the reliance on a global singleton.
- Injected the `analytics.Service` into the `work.Service` to support the `AnalyzeWork` command.
- **Comprehensive Documentation:**
- Created a new root `README.md` with a project overview, setup instructions, and architectural principles.
- Added detailed `README.md` files to key packages (`api`, `analytics`, `auth`, `work`, `db`) to document their purpose and usage.
- **Improved Test Coverage:**
- Added new unit tests for the refactored `work` and `translation` query handlers.
- Added a new test suite for the `translation` queries, which were previously untested.
- Added tests for the new `AnalyzeWork` command.
- Fixed numerous compilation errors in the test suites caused by the refactoring.
Refactored the application's dependency injection and server setup to improve modularity and adhere to the Dependency Inversion Principle.
- Moved the instantiation of all application services from `internal/app/app.go` to the composition root in `cmd/api/main.go`.
- The `app.NewApplication` function now accepts pre-built service interfaces, making the `app` package a simple container.
- Updated `internal/testutil/integration_test_utils.go` to reflect the new DI pattern, ensuring tests align with the refactored structure.
- Corrected build errors that arose from the refactoring, including import conflicts and incorrect function calls.
- Updated `TASKS.md` to mark the 'Refactor Dependency Injection' task as complete.
Refactored the API server setup to improve modularity and simplify the main application entry point.
- Created a new `NewAPIServer` function in `cmd/api/server.go` that encapsulates the creation and configuration of the `http.ServeMux`.
- This new function now handles the registration of all API routes, including the GraphQL endpoint (`/query`), the GraphQL Playground (`/playground`), and the Prometheus metrics endpoint (`/metrics`).
- Simplified `cmd/api/main.go` by removing the manual `ServeMux` creation and instead calling the new `NewAPIServer` function.
- This change makes the `main` function cleaner and more focused on its core responsibilities of application initialization and graceful shutdown.
This commit addresses numerous linting errors, improves overall code quality, and updates the main task list to reflect the current state of the project.
- Fixed dozens of 'errcheck' violations by adding error handling and logging for ignored errors, particularly in analytics goroutines and test setup.
- Resolved 'ineffassign' and 'staticcheck' warnings by refactoring variable scopes and suppressing intentional-but-flagged test patterns.
- Removed dead code identified by the 'unused' linter, including helper functions and mock services.
- Refactored test suites to fix inheritance issues, consolidating GraphQL integration tests and correcting test setup logic.
- Corrected invalid logging calls that were causing type check failures.
- Updated `TASKS.md` to mark completed items and add notes where investigation revealed that listed issues were already resolved.
The codebase now passes 'make lint-test' cleanly.
This commit addresses numerous linting errors and improves overall code quality.
- Fixed dozens of 'errcheck' violations by adding error handling and logging for ignored errors, particularly in analytics goroutines and test setup.
- Resolved 'ineffassign' and 'staticcheck' warnings by refactoring variable scopes and suppressing intentional-but-flagged test patterns.
- Removed dead code identified by the 'unused' linter, including helper functions and mock services.
- Refactored test suites to fix inheritance issues, consolidating GraphQL integration tests and correcting test setup logic.
- Corrected invalid logging calls that were causing type check failures.
The codebase now passes 'make lint-test' cleanly.
This commit marks the completion of a major refactoring effort to stabilize the codebase, improve its structure, and prepare it for production.
The key changes include:
- **Domain Layer Consolidation:** The `Work` entity and its related types, along with all other domain entities and repository interfaces, have been consolidated into the main `internal/domain` package. This eliminates import cycles and provides a single, coherent source of truth for the domain model.
- **Data Access Layer Refactoring:** The repository implementations in `internal/data/sql` have been updated to align with the new domain layer. The `BaseRepositoryImpl` has been corrected to use pointer receivers, and all concrete repositories now correctly embed it, ensuring consistent and correct behavior.
- **Application Layer Stabilization:** All application services in `internal/app` have been updated to use the new domain types and repository interfaces. Dependency injection has been corrected throughout the application, ensuring that all services are initialized with the correct dependencies.
- **GraphQL Adapter Fixes:** The GraphQL resolver implementation in `internal/adapters/graphql` has been updated to correctly handle the new domain types and service methods. The auto-generated GraphQL code has been regenerated to ensure it is in sync with the schema and runtime.
- **Test Suite Overhaul:** All test suites have been fixed to correctly implement their respective interfaces and use the updated domain model. Mock repositories and test suites have been corrected to properly embed the `testify` base types, resolving numerous build and linter errors.
- **Dependency Management:** The Go modules have been tidied, and the module cache has been cleaned to ensure a consistent and correct dependency graph.
- **Code Quality and Verification:** The entire codebase now passes all builds, tests, and linter checks, ensuring a high level of quality and stability.
This comprehensive effort has resulted in a more robust, maintainable, and production-ready application.
This commit addresses several high-priority tasks from the TASKS.md file, including:
- **Fix Background Job Panic:** Replaced `log.Fatalf` with `log.Printf` in the `asynq` server to prevent crashes.
- **Refactor API Server Setup:** Consolidated the GraphQL Playground and Prometheus metrics endpoints into the main API server.
- **Implement `DeleteUser` Mutation:** Implemented the `DeleteUser` resolver.
- **Implement `CreateContribution` Mutation:** Implemented the `CreateContribution` resolver and its required application service.
Additionally, this commit includes a major refactoring of the configuration management system to fix a broken build. The global `config.Cfg` variable has been removed and replaced with a dependency injection approach, where the configuration object is passed to all components that require it. This change has been applied across the entire codebase, including the test suite, to ensure a stable and testable application.
The goal of this refactoring is to eliminate the global configuration singleton (`config.Cfg`) and replace it with explicit dependency injection of a `Config` struct.
This commit includes the following partial changes:
- The `Config` struct in `internal/platform/config/config.go` has been updated with all necessary fields.
- Several platform packages (`db`, `cache`, `auth`, `http`, `jobs/sync`) have been modified to accept the `*config.Config` struct.
- The API server entry point (`cmd/api/main.go`) has been updated to load and provide the configuration.
- A new worker entry point (`cmd/worker/main.go`) has been created to house the background job runner, as per the architecture defined in `refactor.md`.
NOTE: The build is currently broken as this refactoring is incomplete. This commit is for saving progress as requested.
- `Me`: Fetches the details of the currently authenticated user from the request context.
- `User`: Fetches the public details of a user by their ID.
- `Author`: Fetches the details of an author by their ID, including their biography from the localization service.
These changes are part of the larger effort to complete the unimplemented resolvers in the GraphQL API.
- Fixes a potential panic in the background job queue by changing `log.Fatalf` to `log.Printf`, allowing for more graceful error handling.
- Implements all `panic("not implemented")` methods in the mock repositories for `Like`, `Work`, and `User`, enabling more robust testing.
- Consolidates duplicated `WorkAnalytics` and `TranslationAnalytics` structs into a central `internal/domain/analytics` package to reduce code duplication and improve maintainability.
- Corrects build errors that arose during testing, including an unused import and an incorrect struct field name in a mock repository.
This commit refactors the API server startup logic in `cmd/api/main.go` to simplify the application's architecture.
Key changes:
- Consolidates the three separate HTTP servers (GraphQL API, GraphQL Playground, and Prometheus metrics) into a single `http.Server` instance.
- Uses a single `http.ServeMux` to route requests to the appropriate handlers on distinct paths (`/query`, `/playground`, `/metrics`).
- Removes the now-redundant `PlaygroundPort` from the application's configuration.
This change simplifies the server startup and shutdown logic, reduces resource usage, and makes the application's entry point cleaner and easier to maintain.
This commit represents the first major batch of foundational refactoring work.
It accomplishes two main goals:
1. **Restores Project State:**
* Restores the comprehensive technical debt backlog in `TASKS.md` after it was accidentally deleted.
* Restores the detailed GraphQL API documentation in `api/README.md`.
* Re-applies the critical stability fix to the background job worker, replacing a `panic` with `log.Fatalf` for graceful termination on startup failure.
2. **Establishes CI Pipeline:**
* Creates a `Makefile` with a `lint-test` target to standardize running linters and tests.
* Adds a GitHub Actions workflow (`.github/workflows/ci.yml`) to automatically run the `lint-test` target on every push and pull request to the `main` branch.
This provides a stable, documented, and automatically-verified baseline for all future development and refactoring work.
Replaces a raw `panic` in the `asynq` server startup with a call to `log.Fatalf`.
If the job server fails to start, it is an unrecoverable, fatal error. Using `log.Fatalf` ensures the application exits with a clear error message and a non-zero status code, which is a more conventional and robust way to handle fatal startup errors than panicking.
Creates a new, comprehensive guide for the GraphQL API in `api/README.md`. This documentation details all available queries, mutations, and types, providing clear examples and explanations for developers.
This addresses the most critical gap in the project's documentation, which was previously sparse and only covered a single query. The new documentation is structured for clarity and will serve as a vital resource for both internal and external developers.
This commit introduces a significant refactoring to improve the application's quality, test coverage, and production readiness, focusing on core localization and business logic features.
Key changes include:
- Consolidated the `CreateTranslation` and `UpdateTranslation` commands into a single, more robust `CreateOrUpdateTranslation` command. This uses a database-level `Upsert` for atomicity.
- Centralized authorization for translatable entities into a new `CanEditEntity` check within the application service layer.
- Fixed a critical bug in the `MergeWork` command that caused a UNIQUE constraint violation when merging works with conflicting translations. The logic now intelligently handles language conflicts.
- Implemented decrementing for "like" counts in the analytics service when a like is deleted, ensuring accurate statistics.
- Stabilized the test suite by switching to a file-based database for integration tests, fixing test data isolation issues, and adding a unique index to the `Translation` model to enforce data integrity.
- Refactored manual mocks to use the `testify/mock` library for better consistency and maintainability.
This commit addresses all the high-priority tasks outlined in the TASKS.md file, significantly improving the application's observability, completing key features, and refactoring critical parts of the codebase.
### Observability
- **Centralized Logging:** Implemented a new structured, context-aware logging system using `zerolog`. A new logging middleware injects request-specific information (request ID, user ID, trace ID) into the logger, and all application logging has been refactored to use this new system.
- **Prometheus Metrics:** Added Prometheus metrics for database query performance by creating a GORM plugin that automatically records query latency and totals.
- **OpenTelemetry Tracing:** Fully instrumented all application services in `internal/app` and data repositories in `internal/data/sql` with OpenTelemetry tracing, providing deep visibility into application performance.
### Features
- **Analytics:** Implemented like, comment, and bookmark counting. The respective command handlers now call the analytics service to increment counters when these actions are performed.
- **Enrichment Tool:** Built a new, extensible `enrich` command-line tool to fetch data from external sources. The initial implementation enriches author data using the Open Library API.
### Refactoring & Fixes
- **Decoupled Testing:** Refactored the testing utilities in `internal/testutil` to be database-agnostic, promoting the use of mock-based unit tests and improving test speed and reliability.
- **Build Fixes:** Resolved numerous build errors, including a critical import cycle between the logging, observability, and authentication packages.
- **Search Service:** Fixed the search service integration by implementing the `GetWorkContent` method in the localization service, allowing the search indexer to correctly fetch and index work content.
- Implemented view counting for works and translations by adding calls to the analytics service in the `Work` and `Translation` resolvers.
- Implemented translation counting for works by adding a call to the analytics service in the `CreateTranslation` resolver.
- Fixed the `Translation` resolver, which was previously unimplemented and caused a panic.
- Updated `TASKS.md` to mark the implemented analytics features as complete.
The 'Urgent: Build Failures' section in TASKS.md was outdated. The issues listed have already been resolved, and the build is stable. This commit removes the section to reflect the current state of the project.
This commit introduces the `MergeWork` command, a new feature to merge two `Work` entities, their associations, and their statistics. The entire operation is performed atomically within a database transaction to ensure data integrity.
Key changes include:
- A new `MergeWork` method in `internal/app/work/commands.go`.
- An `Add` method on the `WorkStats` entity for combining statistics.
- A new `GetWithAssociationsInTx` method on the `WorkRepository` to fetch entities within a transaction.
- A comprehensive integration test using an in-memory SQLite database to validate the merge logic.
This commit also consolidates all scattered TODOs and build issues from `TODO.md` and `BUILD_ISSUES.md` into a single, actionable `TASKS.md` file. The legacy documentation files have been removed to create a single source of truth for pending work.
This commit consolidates all outstanding tasks from `TODO.md`, `BUILD_ISSUES.md`, and inline `TODO` comments into a single `TASKS.md` file. This new file serves as the single source of truth for all pending development work, categorized by priority.
The task list has been enhanced with additional context, linking tasks to the architectural vision in `refactor.md` and making them more actionable for developers.
The following legacy files have been removed to avoid confusion:
- `TODO.md`
- `BUILD_ISSUES.md`
The `AGENTS.md` file has also been updated to remove a temporary debug log and now directs agents to the new `TASKS.md` file for guidance.
This commit consolidates all outstanding tasks from `TODO.md`, `BUILD_ISSUES.md`, and inline `TODO` comments into a single `TASKS.md` file. This new file serves as the single source of truth for all pending development work, categorized by priority.
The following legacy files have been removed to avoid confusion:
- `TODO.md`
- `BUILD_ISSUES.md`
The `AGENTS.md` file has also been updated to remove a temporary debug log and now directs agents to the new `TASKS.md` file for guidance.
