192 lines
10 KiB
Markdown
192 lines
10 KiB
Markdown
# BrokeCrew System Design Specification
|
|
|
|
- **Project Name:** BrokeCrew
|
|
- **Date:** 2026-09-26
|
|
- **Status:** Approved Design
|
|
- **Architecture Style:** Async-first, modular CLI & Core Library (Approach A: Dual-Target Crate)
|
|
|
|
---
|
|
|
|
## 1. Executive Summary
|
|
|
|
BrokeCrew is an open-source, production-grade Rust CLI and library that enables developers to orchestrate a coordinated team of AI personas powered by OpenRouter models. BrokeCrew treats free-tier models as first-class citizens while eliminating their notorious instability through live model catalog discovery, multi-candidate fallback chains, and automatic rate-limit/budget guarding. When tasks demand advanced capabilities, a two-stage routing engine seamlessly escalates to paid model candidates. Multi-agent handoff pipelines allow personas to collaborate sequentially (e.g. coder -> reviewer) with visually distinct terminal rendering and real-time Server-Sent Events (SSE) streaming.
|
|
|
|
---
|
|
|
|
## 2. System Architecture & Boundaries
|
|
|
|
To preserve strict separation of concerns and allow future frontend expansions (such as a GUI binary or daemon service) without refactoring core logic, BrokeCrew adopts a dual-target architecture:
|
|
1. **Core Library (`brokecrew` / `src/lib.rs`)**:
|
|
- Contains all domain types, OpenRouter HTTP client, live model registry, rate limiter, routing engine, persona managers, and pipeline executors.
|
|
- Completely agnostic of terminal I/O, UI styling, and CLI argument parsing.
|
|
- Exposes clean async APIs and channels (`tokio::sync::mpsc`) for streaming tokens and pipeline lifecycle events.
|
|
2. **CLI Executable (`src/main.rs`)**:
|
|
- Thin entrypoint that handles `clap` argument parsing, signal trapping, terminal spinner management (`indicatif`), colored text formatting (`owo-colors`), and OS keyring integration.
|
|
|
|
```
|
|
brokecrew/
|
|
├── Cargo.toml
|
|
├── src/
|
|
│ ├── lib.rs # Library root: re-exports domain modules
|
|
│ ├── main.rs # Thin binary CLI entrypoint
|
|
│ ├── cli.rs # clap command definitions & interactive prompts
|
|
│ ├── config.rs # Configuration loader, paths & credential resolution
|
|
│ ├── client.rs # ModelClient trait, OpenRouter HTTP client, SSE parser
|
|
│ ├── registry.rs # Live OpenRouter model catalog cache & sync
|
|
│ ├── persona.rs # Persona & ModelCandidate types, fallback cascade
|
|
│ ├── pipeline.rs # Multi-agent typed pipeline execution
|
|
│ ├── budget.rs # Account-wide rate limiter & local budget persistence
|
|
│ └── router/
|
|
│ ├── mod.rs # Router coordinator (Stage A -> Stage B -> Stage C)
|
|
│ ├── keyword.rs # Stage A: regex/keyword pre-filter
|
|
│ └── llm.rs # Stage B: compact LLM classifier; Stage C: complexity escalation
|
|
├── config/
|
|
│ └── personas.example.toml # Default production personas & pipelines
|
|
├── tests/
|
|
│ ├── router_tests.rs # Table-driven unit tests for classification
|
|
│ ├── fallback_tests.rs # Fault-injection candidate cascade tests
|
|
│ └── wiremock_client.rs # WireMock integration tests for SSE & error handling
|
|
├── .github/workflows/ci.yml
|
|
├── CODEOWNERS
|
|
├── CONTRIBUTING.md
|
|
├── CHANGELOG.md
|
|
├── LICENSE-MIT
|
|
├── LICENSE-APACHE
|
|
└── README.md
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Detailed Component Specifications
|
|
|
|
### 3.1 Persona & Candidate Fallback System (`persona.rs`)
|
|
|
|
Personas are defined declaratively in `personas.toml`:
|
|
- `name`: Unique identifier (e.g., `"coder"`, `"reviewer"`, `"architect"`).
|
|
- `system_prompt`: Instructions governing persona behavior and tone.
|
|
- `trigger_words`: High-confidence keywords/phrases triggering Stage A routing.
|
|
- `candidates`: Ordered list of `ModelCandidate` structs:
|
|
- `id`: OpenRouter model identifier (e.g. `"qwen/qwen-2.5-coder-32b-instruct:free"`).
|
|
- `tier`: `Free` or `Paid`.
|
|
- `is_stealth`: Boolean indicating experimental or unlisted prerelease models.
|
|
|
|
#### Candidate Selection & Fallback Walk
|
|
When dispatching a prompt to a persona:
|
|
1. Filter candidates:
|
|
- Verify model presence in the cached `ModelRegistry` (unless in offline bypass mode).
|
|
- If `allow_stealth` is `false`, exclude candidates marked `is_stealth: true`.
|
|
- If escalation to `Tier::Paid` was triggered, filter or prioritize `Tier::Paid` candidates; otherwise prefer `Tier::Free`.
|
|
2. Sequentially attempt completion on each candidate:
|
|
- On success: Return `DispatchResult` containing the reply, the model used, and whether fallback occurred.
|
|
- On `ClientError::RateLimited` (429) or `ClientError::ServerError` (5xx) or missing model: Log structured audit event and immediately attempt the next candidate in the chain.
|
|
- On `ClientError::Unauthorized` (401) or `ClientError::PaymentRequired` (402): Abort candidate walking immediately and surface the error.
|
|
3. If all candidates fail: Return an aggregated `AllCandidatesFailed` error detailing each attempt.
|
|
|
|
---
|
|
|
|
### 3.2 Two-Stage Router & Complexity Escalation (`router/`)
|
|
|
|
Routing determines which persona handles an unassigned user message and whether the request requires higher-tier execution.
|
|
|
|
#### Stage A: Zero-Cost Keyword Pre-Filter (`keyword.rs`)
|
|
- Evaluates prompt against persona `trigger_words` using word-boundary matching and case-insensitive scanning.
|
|
- If high-confidence matches point unambiguously to a persona (e.g. "write a python script" -> `coder`), routing completes instantly with zero API calls and zero latency.
|
|
|
|
#### Stage B: Low-Token LLM Classifier (`llm.rs`)
|
|
- Invoked only when Stage A is ambiguous or inconclusive.
|
|
- Calls a lightweight, fast model (e.g. `meta-llama/llama-3.2-3b-instruct:free` or user-configured router model).
|
|
- Strict system prompt: `"You are a message router. Respond with ONLY the exact persona name from this list: [coder, reviewer, general]. Do not include explanations, punctuation, or any other words."`
|
|
- Request configured with `max_tokens = 10` and `temperature = 0.0`.
|
|
- Output is trimmed and validated against active personas. If unparseable or unexpected, defaults cleanly to `default_persona` (e.g. `"general"`).
|
|
|
|
#### Stage C: Complexity Escalation (`llm.rs`)
|
|
- Assesses prompt complexity heuristics:
|
|
- Estimated token volume and instruction density.
|
|
- Multi-file or architectural keywords (`"refactor the entire repo"`, `"implement full backend with database migrations and tests"`).
|
|
- Explicit user flag `--escalate`.
|
|
- When complexity exceeds `complexity_threshold` (configurable in `config.toml`, default 7/10), the router instructs the persona candidate walker to prefer paid-tier candidates.
|
|
|
|
---
|
|
|
|
### 3.3 Model Registry & Resilience (`registry.rs`)
|
|
|
|
- Maintains a cached snapshot of OpenRouter's live catalog:
|
|
- Fetches `GET https://openrouter.ai/api/v1/models` on startup.
|
|
- Periodic background sync every 30 minutes via `tokio::spawn`.
|
|
- Caches `id`, `pricing` (prompt, completion), `context_length`, and `supported_parameters`.
|
|
- Safe resilience:
|
|
- If a free model is removed from OpenRouter or marked offline, BrokeCrew skips it proactively before wasting a network roundtrip.
|
|
- If registry fetch fails (e.g. initial offline state), BrokeCrew operates in degraded mode using static fallback configurations with warning logs.
|
|
|
|
---
|
|
|
|
### 3.4 Rate Limiting & Account Budget Awareness (`budget.rs`)
|
|
|
|
OpenRouter enforces strict account-wide rate limits:
|
|
- **20 requests / minute** rolling window.
|
|
- **Daily Quotas**:
|
|
- 50 requests/day for accounts with < $10 lifetime credit purchase.
|
|
- 1,000 requests/day for accounts with ≥ $10 lifetime credit purchase.
|
|
|
|
#### Tracking Mechanism:
|
|
- Local state persisted in `~/.config/brokecrew/budget.json` (or platform config directory) using atomic file writes (`tempfile` + rename).
|
|
- Every outbound call—including Stage B router queries—is accounted for.
|
|
- State tracks:
|
|
- Recent call timestamps for the rolling 60-second window.
|
|
- Cumulative daily counter reset at UTC midnight.
|
|
- Prior to firing any request:
|
|
- If rolling minute count >= 20: Pause and wait for window decay, or warn the user.
|
|
- If daily count reaches 80% of quota: Emit warning in CLI.
|
|
- If daily count hits quota ceiling: Halt execution with clear explanation rather than triggering a silent 429.
|
|
|
|
---
|
|
|
|
### 3.5 Multi-Agent Handoff Pipeline (`pipeline.rs`)
|
|
|
|
Defined in `personas.toml`:
|
|
```toml
|
|
[[pipeline]]
|
|
name = "build-and-review"
|
|
steps = [
|
|
{ persona = "coder" },
|
|
{ persona = "reviewer", pass_previous_output = true }
|
|
]
|
|
```
|
|
|
|
- Steps execute sequentially; step $N$ receives the accumulated output of step $N-1$ when `pass_previous_output = true`.
|
|
- Output is emitted as streaming events over a channel:
|
|
- `PipelineEvent::StepStarted { persona, model, tier }`
|
|
- `PipelineEvent::Chunk(String)`
|
|
- `PipelineEvent::StepCompleted { persona, total_tokens }`
|
|
- The CLI formats each step distinctly with colored headers and persona avatars, ensuring personas never blend into an anonymous voice.
|
|
|
|
---
|
|
|
|
### 3.6 Security & Keyring Integration (`config.rs`)
|
|
|
|
- API keys are resolved with the following priority:
|
|
1. CLI parameter `--api-key <KEY>`
|
|
2. Environment variable `OPENROUTER_API_KEY`
|
|
3. OS Credential Manager via `keyring` crate (`keyring::Entry::new("brokecrew", "openrouter_api_key")`)
|
|
4. Local configuration file (`config.toml`)
|
|
- Key sanitization:
|
|
- Struct fields containing API keys implement a custom `Debug` that redacts values to `"[REDACTED]"`.
|
|
- `tracing` spans and error formatters never output raw tokens or authorization headers.
|
|
|
|
---
|
|
|
|
## 4. Acceptance Criteria & Verification
|
|
|
|
1. **Cargo Install & Zero-Config Execution**:
|
|
- `cargo build --release` succeeds without warnings (`-D warnings`).
|
|
- Default `personas.toml` provides `coder`, `reviewer`, `architect`, and `general` personas with working free OpenRouter candidates.
|
|
2. **Resilience & Fallback Walk**:
|
|
- Simulating 503 or 429 on Candidate 1 routes seamlessly to Candidate 2 with visible fallback logs.
|
|
3. **Budget Guarding**:
|
|
- `brokecrew status` accurately displays remaining quota and rolling minute usage.
|
|
4. **Pipeline Separation**:
|
|
- `brokecrew chat --pipeline build-and-review "Create a LRU cache"` displays `[coder]` output followed by `[reviewer]` output in distinct colored blocks.
|
|
5. **Security & Privacy**:
|
|
- No API keys leaked across any logging level (`--json-logs` or stdout).
|
|
6. **Cross-Platform & CI**:
|
|
- Full test suite passes on Linux, macOS, and Windows runners in GitHub Actions.
|