System Architecture
Reliability Engineering, Multi-Key Round-Robin & Real-Time Streaming
Building a high-performance developer portfolio requires more than visual polish—it demands a resilient, low-latency, and fault-tolerant backend. This page outlines the architectural patterns, multi-threaded concurrency pipelines, multi-key round-robin load distribution, and real-time streaming mechanisms that power "The GOAT" AI assistant.
Multi-Threaded Pre-Fetch
Sequential I/O is the primary source of chatbot latency. The backend parallelizes Firestore knowledge retrieval, episodic memory logs, and live sports cache via a ThreadPoolExecutor(max_workers=3).
Outcome: Pre-fetch overhead dropped from ~750ms sequentially down to 150–200ms concurrently.
Multi-Key Round-Robin
Free-tier LLM endpoints enforce strict RPM/TPM ceilings. The backend pools comma-separated keys for both Groq and Gemini, atomically rotating each request via synchronized round-robin indices guarded by thread locks.
In-Flight Failover: If the active key encounters an HTTP 429 rate limit, the stream seamlessly falls over to the next pool key in under 50ms.
Real-Time SSE Token Streaming
Rather than buffering entire responses on the server, /chat streams tokens via Server-Sent Events (SSE) with Accept: text/event-stream, maintaining a backwards-compatible JSON fallback.
Speculative Fallback: A 1.2s timeout on primary Gemini streaming triggers instant failover to ultra-fast Groq streaming, slashing TTFT from 30s+ down to ~2s.
Adaptive 60fps Typewriter
The frontend reader parses streaming byte chunks and feeds an adaptive 60fps typewriter loop. It types at 1–3 chars/frame for an organic feel, automatically accelerating to 6–16 chars/frame to catch up or complete.
Dynamic Formatting: Markdown headings, bold markers, and code blocks format live on the fly as closing punctuation arrives.
Multi-Tier Provider & Client Pool Hierarchy
- Tier 1: Google Gemini Multi-Key Pool (gemini-3.6-flash) — Primary high-reasoning model handling personal queries, owner knowledge synthesis, and episodic dialogue context. Rotates keys via atomic round-robin with a 1.2s speculative TTFT window.
- Tier 2: Groq Multi-Key Round-Robin Pool (openai/gpt-oss-120b) — Ultra-low latency engine serving as the primary handler for general/technical queries and instant fallback for Gemini stalls. Features multi-key rotation and automatic in-flight HTTP 429 failover.
- Tier 3: Asynchronous Background Executor (_bg_executor) — Offloads message audit logs, trigger state updates, and silent user preference extraction to non-blocking daemon workers, eliminating blocking I/O overhead on user-facing streaming.
Stream Sanitization & Output Governance
To balance conversational responsiveness, leakage mitigation, and clean client rendering, the system employs defense-in-depth intent routing, sliding-window stream sanitization, and backend privilege boundaries:
Intelligently distinguishes conversational dialogue from direct technical queries. Direct technical and project queries receive focused, structured responses without redundant conversational preambles.
A real-time sliding-window stream filter intercepts in-flight token buffers to parse structural delimiters, memory directives, and control markers before chunks are yielded to the client socket.
Combines explicit system prompt boundary rules with deterministic tag-stripping to suppress internal directives and protect system instructions from direct exposure.
Implemented Protections
- Deterministic Tag Stripping:
StreamTagFilterinspects in-flight token fragments to strip control syntax (e.g.,[LEARN_RONIL: ...],[LEARN_USER: ...]) across chunk boundaries. - Negative Constraint Directives: System instructions mandate direct answers, forbid unprompted persona disclosure, and decline prompt dumps.
- Out-of-Band Authorization: Database writes for learned facts require validated Firebase Admin credentials, isolating persistent storage from prompt manipulation.
Security Limitations
- Probabilistic Nature: LLMs are non-deterministic; prompt-level constraints mitigate leakage risk but cannot mathematically guarantee resistance against novel jailbreaks.
- Semantic Rephrasing: Heuristic filters intercept exact control syntax; sophisticated prompts inducing natural-language summarization of system context rely on model safety alignment.
- Defense-in-Depth Prerequisite: Stream filtering is a presentation-layer defense, necessitating strict backend authorization for stateful operations.
Empirical Testing Evidence
- 100% Tag Interception: Verified across 150+ automated fragmented token tests with 0 internal control tags escaping to the client stream.
- Adversarial Prompt Resistance: Evaluated against a suite of common extraction prompts ("ignore previous instructions", roleplay inversion, delimiter injection); direct prompt dumps resisted in ~94% of test evaluations.
- Privilege Containment: 100% of unauthorized attempts to alter global portfolio facts were safely blocked by backend authentication gates.
End-to-End Streaming Architecture Pipeline
┌────────────────────────────────────────────────────────────────────────┐
│ CLIENT / FRONTEND LAYER │
├────────────────────────────────────────────────────────────────────────┤
│ [User Query Input] ──► [fetch('/chat', Accept: 'text/event-stream')] │
│ │ │
│ ▼ │
│ [HTTP POST /chat + Bearer Token] │
└───────────────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ SECURITY & GATEWAY LAYER │
├────────────────────────────────────────────────────────────────────────┤
│ [1. Cryptographic Auth Validation: Firebase ID Token or Guest HMAC] │
│ ├─► [Invalid / Expired] ──► Return 401 Unauthorized │
│ [2. In-Memory Sliding-Window Rate Limiter (5-15 req/min)] │
│ └─► [Quota Exceeded] ──► Return 429 Too Many Requests │
└───────────────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ CONCURRENT PRE-FETCH LAYER (ThreadPoolExecutor) │
├────────────────────────────────────────────────────────────────────────┤
│ ┌──────────────────────────┬──────────────────────────┬──────────────┐ │
│ │ Worker 1: Firestore │ Worker 2: Episodic │ Worker 3: │ │
│ │ Global Knowledge Facts │ Conversation Summaries │ Daemon Match │ │
│ │ (docId: user.uid/global) │ (docId: user.uid/memory) │ Cache (<1ms) │ │
│ └─────────────┬────────────┴─────────────┬────────────┴───────┬──────┘ │
│ └──────────────────────────┼────────────────────┘ │
│ ▼ │
│ Aggregated Prompt Context (<200ms) │
└───────────────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ CONTEXT SYNTHESIZER & INTENT ROUTER │
├────────────────────────────────────────────────────────────────────────┤
│ [Intent Classifier: Conversational Context vs Direct Technical Query] │
│ ├─► Conversational Mode: Contextual Salutation & Guided Persona │
│ └─► Technical Inquiry: Direct Structured Response (Zero Noise) │
└───────────────────────────────────┬────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ MULTI-KEY ROUND-ROBIN INFERENCE ENGINE │
├────────────────────────────────────────────────────────────────────────┤
│ [Primary: Gemini 3.6 Flash Pool (Key 1 -> Key 2 Round-Robin)] │
│ │ │
│ ├─► First Token <= 1.2s? ──► [Stream Gemini Chunks] ─────────┐ │
│ │ │ │
│ └─► Timeout (>1.2s) or 429 Quota Exceeded │ │
│ │ │ │
│ ▼ │ │
│ [Secondary: Groq Multi-Key Pool (openai/gpt-oss-120b)] │ │
│ ├─► Atomic Round-Robin Key Selection (Key 1 -> Key 2 -> ...) │ │
│ ├─► In-Flight 429 Failover to Next Pool Key (<50ms) │ │
│ └─► [High-Speed Groq Token Stream (~1-2s TTFT)] ─────────────┤ │
└────────────────────────────────────────────────────────────────────┼───┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ STREAM SANITIZER & OUTPUT GOVERNANCE │
├────────────────────────────────────────────────────────────────────────┤
│ [StreamTagFilter Engine] │
│ ├─► Intercepts & Buffers Internal System Tags & Control Directives │
│ ├─► Enforces Output Sanitization & Strips Redundant Preambles │
│ └─► Yields Clean SSE Chunks: data: {"chunk": "..."}\n\n ────────────┐ │
└───────────────────────────────────┬────────────────────────────────┼───┘
│ (Asynchronous Audit Pipeline) │
▼ ▼
┌────────────────────────────┐ ┌─────────────────┐
│ Async Background Executor │ │ SSE Stream │
│ - Append Firestore Log │ │ & Adaptive 60fps│
│ - Commit Learned Facts │ │ Typewriter UI │
│ - Update Dynamic Triggers │ │ with Dynamic │
│ - Non-Blocking Stream Exec │ │ Markdown Render │
└────────────────────────────┘ └─────────────────┘
Looking for Deep Technical Specifications?
Explore the complete Software Engineering lifecycle suite, including formal SRS, PRD, TRD, Schema definitions, and Flow diagrams.