# BrokeCrew System Design Specification - **Project Name:** BrokeCrew - **Date:** 2026-09-26 - **Status:** Approved Design - **Architecture Style:** Async-first, modular CLI & Core Library (Approach A: Dual-Target Crate) --- ## 1. Executive Summary BrokeCrew is an open-source, production-grade Rust CLI and library that enables developers to orchestrate a coordinated team of AI personas powered by OpenRouter models. BrokeCrew treats free-tier models as first-class citizens while eliminating their notorious instability through live model catalog discovery, multi-candidate fallback chains, and automatic rate-limit/budget guarding. When tasks demand advanced capabilities, a two-stage routing engine seamlessly escalates to paid model candidates. Multi-agent handoff pipelines allow personas to collaborate sequentially (e.g. coder -> reviewer) with visually distinct terminal rendering and real-time Server-Sent Events (SSE) streaming. --- ## 2. System Architecture & Boundaries To preserve strict separation of concerns and allow future frontend expansions (such as a GUI binary or daemon service) without refactoring core logic, BrokeCrew adopts a dual-target architecture: 1. **Core Library (`brokecrew` / `src/lib.rs`)**: - Contains all domain types, OpenRouter HTTP client, live model registry, rate limiter, routing engine, persona managers, and pipeline executors. - Completely agnostic of terminal I/O, UI styling, and CLI argument parsing. - Exposes clean async APIs and channels (`tokio::sync::mpsc`) for streaming tokens and pipeline lifecycle events. 2. **CLI Executable (`src/main.rs`)**: - Thin entrypoint that handles `clap` argument parsing, signal trapping, terminal spinner management (`indicatif`), colored text formatting (`owo-colors`), and OS keyring integration. ``` brokecrew/ ├── Cargo.toml ├── src/ │ ├── lib.rs # Library root: re-exports domain modules │ ├── main.rs # Thin binary CLI entrypoint │ ├── cli.rs # clap command definitions & interactive prompts │ ├── config.rs # Configuration loader, paths & credential resolution │ ├── client.rs # ModelClient trait, OpenRouter HTTP client, SSE parser │ ├── registry.rs # Live OpenRouter model catalog cache & sync │ ├── persona.rs # Persona & ModelCandidate types, fallback cascade │ ├── pipeline.rs # Multi-agent typed pipeline execution │ ├── budget.rs # Account-wide rate limiter & local budget persistence │ └── router/ │ ├── mod.rs # Router coordinator (Stage A -> Stage B -> Stage C) │ ├── keyword.rs # Stage A: regex/keyword pre-filter │ └── llm.rs # Stage B: compact LLM classifier; Stage C: complexity escalation ├── config/ │ └── personas.example.toml # Default production personas & pipelines ├── tests/ │ ├── router_tests.rs # Table-driven unit tests for classification │ ├── fallback_tests.rs # Fault-injection candidate cascade tests │ └── wiremock_client.rs # WireMock integration tests for SSE & error handling ├── .github/workflows/ci.yml ├── CODEOWNERS ├── CONTRIBUTING.md ├── CHANGELOG.md ├── LICENSE-MIT ├── LICENSE-APACHE └── README.md ``` --- ## 3. Detailed Component Specifications ### 3.1 Persona & Candidate Fallback System (`persona.rs`) Personas are defined declaratively in `personas.toml`: - `name`: Unique identifier (e.g., `"coder"`, `"reviewer"`, `"architect"`). - `system_prompt`: Instructions governing persona behavior and tone. - `trigger_words`: High-confidence keywords/phrases triggering Stage A routing. - `candidates`: Ordered list of `ModelCandidate` structs: - `id`: OpenRouter model identifier (e.g. `"qwen/qwen-2.5-coder-32b-instruct:free"`). - `tier`: `Free` or `Paid`. - `is_stealth`: Boolean indicating experimental or unlisted prerelease models. #### Candidate Selection & Fallback Walk When dispatching a prompt to a persona: 1. Filter candidates: - Verify model presence in the cached `ModelRegistry` (unless in offline bypass mode). - If `allow_stealth` is `false`, exclude candidates marked `is_stealth: true`. - If escalation to `Tier::Paid` was triggered, filter or prioritize `Tier::Paid` candidates; otherwise prefer `Tier::Free`. 2. Sequentially attempt completion on each candidate: - On success: Return `DispatchResult` containing the reply, the model used, and whether fallback occurred. - On `ClientError::RateLimited` (429) or `ClientError::ServerError` (5xx) or missing model: Log structured audit event and immediately attempt the next candidate in the chain. - On `ClientError::Unauthorized` (401) or `ClientError::PaymentRequired` (402): Abort candidate walking immediately and surface the error. 3. If all candidates fail: Return an aggregated `AllCandidatesFailed` error detailing each attempt. --- ### 3.2 Two-Stage Router & Complexity Escalation (`router/`) Routing determines which persona handles an unassigned user message and whether the request requires higher-tier execution. #### Stage A: Zero-Cost Keyword Pre-Filter (`keyword.rs`) - Evaluates prompt against persona `trigger_words` using word-boundary matching and case-insensitive scanning. - If high-confidence matches point unambiguously to a persona (e.g. "write a python script" -> `coder`), routing completes instantly with zero API calls and zero latency. #### Stage B: Low-Token LLM Classifier (`llm.rs`) - Invoked only when Stage A is ambiguous or inconclusive. - Calls a lightweight, fast model (e.g. `meta-llama/llama-3.2-3b-instruct:free` or user-configured router model). - Strict system prompt: `"You are a message router. Respond with ONLY the exact persona name from this list: [coder, reviewer, general]. Do not include explanations, punctuation, or any other words."` - Request configured with `max_tokens = 10` and `temperature = 0.0`. - Output is trimmed and validated against active personas. If unparseable or unexpected, defaults cleanly to `default_persona` (e.g. `"general"`). #### Stage C: Complexity Escalation (`llm.rs`) - Assesses prompt complexity heuristics: - Estimated token volume and instruction density. - Multi-file or architectural keywords (`"refactor the entire repo"`, `"implement full backend with database migrations and tests"`). - Explicit user flag `--escalate`. - When complexity exceeds `complexity_threshold` (configurable in `config.toml`, default 7/10), the router instructs the persona candidate walker to prefer paid-tier candidates. --- ### 3.3 Model Registry & Resilience (`registry.rs`) - Maintains a cached snapshot of OpenRouter's live catalog: - Fetches `GET https://openrouter.ai/api/v1/models` on startup. - Periodic background sync every 30 minutes via `tokio::spawn`. - Caches `id`, `pricing` (prompt, completion), `context_length`, and `supported_parameters`. - Safe resilience: - If a free model is removed from OpenRouter or marked offline, BrokeCrew skips it proactively before wasting a network roundtrip. - If registry fetch fails (e.g. initial offline state), BrokeCrew operates in degraded mode using static fallback configurations with warning logs. --- ### 3.4 Rate Limiting & Account Budget Awareness (`budget.rs`) OpenRouter enforces strict account-wide rate limits: - **20 requests / minute** rolling window. - **Daily Quotas**: - 50 requests/day for accounts with < $10 lifetime credit purchase. - 1,000 requests/day for accounts with ≥ $10 lifetime credit purchase. #### Tracking Mechanism: - Local state persisted in `~/.config/brokecrew/budget.json` (or platform config directory) using atomic file writes (`tempfile` + rename). - Every outbound call—including Stage B router queries—is accounted for. - State tracks: - Recent call timestamps for the rolling 60-second window. - Cumulative daily counter reset at UTC midnight. - Prior to firing any request: - If rolling minute count >= 20: Pause and wait for window decay, or warn the user. - If daily count reaches 80% of quota: Emit warning in CLI. - If daily count hits quota ceiling: Halt execution with clear explanation rather than triggering a silent 429. --- ### 3.5 Multi-Agent Handoff Pipeline (`pipeline.rs`) Defined in `personas.toml`: ```toml [[pipeline]] name = "build-and-review" steps = [ { persona = "coder" }, { persona = "reviewer", pass_previous_output = true } ] ``` - Steps execute sequentially; step $N$ receives the accumulated output of step $N-1$ when `pass_previous_output = true`. - Output is emitted as streaming events over a channel: - `PipelineEvent::StepStarted { persona, model, tier }` - `PipelineEvent::Chunk(String)` - `PipelineEvent::StepCompleted { persona, total_tokens }` - The CLI formats each step distinctly with colored headers and persona avatars, ensuring personas never blend into an anonymous voice. --- ### 3.6 Security & Keyring Integration (`config.rs`) - API keys are resolved with the following priority: 1. CLI parameter `--api-key ` 2. Environment variable `OPENROUTER_API_KEY` 3. OS Credential Manager via `keyring` crate (`keyring::Entry::new("brokecrew", "openrouter_api_key")`) 4. Local configuration file (`config.toml`) - Key sanitization: - Struct fields containing API keys implement a custom `Debug` that redacts values to `"[REDACTED]"`. - `tracing` spans and error formatters never output raw tokens or authorization headers. --- ## 4. Acceptance Criteria & Verification 1. **Cargo Install & Zero-Config Execution**: - `cargo build --release` succeeds without warnings (`-D warnings`). - Default `personas.toml` provides `coder`, `reviewer`, `architect`, and `general` personas with working free OpenRouter candidates. 2. **Resilience & Fallback Walk**: - Simulating 503 or 429 on Candidate 1 routes seamlessly to Candidate 2 with visible fallback logs. 3. **Budget Guarding**: - `brokecrew status` accurately displays remaining quota and rolling minute usage. 4. **Pipeline Separation**: - `brokecrew chat --pipeline build-and-review "Create a LRU cache"` displays `[coder]` output followed by `[reviewer]` output in distinct colored blocks. 5. **Security & Privacy**: - No API keys leaked across any logging level (`--json-logs` or stdout). 6. **Cross-Platform & CI**: - Full test suite passes on Linux, macOS, and Windows runners in GitHub Actions.