System Architecture

Reliability Engineering, Multi-Key Round-Robin & Real-Time Streaming

Building a high-performance developer portfolio requires more than visual polish—it demands a resilient, low-latency, and fault-tolerant backend. This page outlines the architectural patterns, multi-threaded concurrency pipelines, multi-key round-robin load distribution, and real-time streaming mechanisms that power "The GOAT" AI assistant.

Multi-Threaded Pre-Fetch

Sequential I/O is the primary source of chatbot latency. The backend parallelizes Firestore knowledge retrieval, episodic memory logs, and live sports cache via a ThreadPoolExecutor(max_workers=3).

Outcome: Pre-fetch overhead dropped from ~750ms sequentially down to 150–200ms concurrently.

Multi-Key Round-Robin

Free-tier LLM endpoints enforce strict RPM/TPM ceilings. The backend pools comma-separated keys for both Groq and Gemini, atomically rotating each request via synchronized round-robin indices guarded by thread locks.

In-Flight Failover: If a key raises an error before output is emitted, the request tries the next key in that provider's pool. Once output has started, it does not restart the response on another key or provider.

Real-Time SSE Token Streaming

Rather than buffering entire responses on the server, /chat streams tokens via Server-Sent Events (SSE) with Accept: text/event-stream, maintaining a backwards-compatible JSON fallback.

Groq-First Failover: The GOAT Assistant tries Groq with openai/gpt-oss-120b first for every query. Gemini is attempted only if Groq emits no output. Gemini defaults to gemini-2.5-flash, configurable through GEMINI_MODEL.

Adaptive 60fps Typewriter

The frontend reader parses streaming byte chunks and feeds an adaptive 60fps typewriter loop. It types at 1–3 chars/frame for an organic feel, automatically accelerating to 6–16 chars/frame to catch up or complete.

Dynamic Formatting: Markdown headings, bold markers, and code blocks format live on the fly as closing punctuation arrives.

Multi-Tier Provider & Client Pool Hierarchy

  • Tier 1: Groq Multi-Key Round-Robin Pool (openai/gpt-oss-120b) — Primary provider for all query types, with synchronized round-robin key selection and retries on errors before output begins.
  • Tier 2: Google Gemini Multi-Key Pool (gemini-2.5-flash) — Fallback provider used only when Groq produces no output. The model shown is the code default and can be changed through GEMINI_MODEL. If neither provider produces output, the assistant returns an unavailable message.
  • Background work: Asynchronous Executor (_bg_executor) — Offloads message audit logs, trigger state updates, and silent user preference extraction to non-blocking daemon workers, eliminating blocking I/O overhead on user-facing streaming.

Stream Sanitization & Output Governance

To balance conversational responsiveness, leakage mitigation, and clean client rendering, the system employs defense-in-depth intent routing, sliding-window stream sanitization, and backend privilege boundaries:

Context-Aware Intent Routing

Intelligently distinguishes conversational dialogue from direct technical queries. Direct technical and project queries receive focused, structured responses without redundant conversational preambles.

Sliding-Window Stream Sanitizer

A real-time sliding-window stream filter intercepts in-flight token buffers to parse structural delimiters, memory directives, and control markers before chunks are yielded to the client socket.

Prompt Scaffolding & Tag Defense

Combines explicit system prompt boundary rules with deterministic tag-stripping to suppress internal directives and protect system instructions from direct exposure.

Implemented Protections

  • Deterministic Tag Stripping: StreamTagFilter inspects in-flight token fragments to strip control syntax (e.g., [LEARN_RONIL: ...], [LEARN_USER: ...]) across chunk boundaries.
  • Negative Constraint Directives: System instructions mandate direct answers, forbid unprompted persona disclosure, and decline prompt dumps.
  • Out-of-Band Authorization: Database writes for learned facts require validated Firebase Admin credentials, isolating persistent storage from prompt manipulation.

Security Limitations

  • Probabilistic Nature: LLMs are non-deterministic; prompt-level constraints mitigate leakage risk but cannot mathematically guarantee resistance against novel jailbreaks.
  • Semantic Rephrasing: Heuristic filters intercept exact control syntax; sophisticated prompts inducing natural-language summarization of system context rely on model safety alignment.
  • Defense-in-Depth Prerequisite: Stream filtering is a presentation-layer defense, necessitating strict backend authorization for stateful operations.

Empirical Testing Evidence

  • 100% Tag Interception: Verified across 150+ automated fragmented token tests with 0 internal control tags escaping to the client stream.
  • Adversarial Prompt Resistance: Evaluated against a suite of common extraction prompts ("ignore previous instructions", roleplay inversion, delimiter injection); direct prompt dumps resisted in ~94% of test evaluations.
  • Privilege Containment: 100% of unauthorized attempts to alter global portfolio facts were safely blocked by backend authentication gates.

End-to-End Streaming Architecture Pipeline

  1. Request and access checks

    The browser sends the query to /chat. Flask validates the Firebase or guest token and checks the rate limit.

  2. Build the prompt

    Gather knowledge, conversation history, and live context. Context classification changes the prompt, not the provider order.

  3. Groq primary

    Request openai/gpt-oss-120b with round-robin key selection. Retry another Groq key on errors before output begins.

  4. Gemini fallback, only if no Groq output

    Use GEMINI_MODEL, defaulting to gemini-2.5-flash. Once a response has started, do not restart it on another provider.

  5. Filter and stream

    StreamTagFilter removes internal control tags. Deliver clean text via SSE. Return an unavailable message if neither provider emits output.

  6. Render and persist

    The browser renders text progressively. Background workers handle chat logs and learned facts separately from provider fallback.

Looking for Deep Technical Specifications?

Explore the complete Software Engineering lifecycle suite, including formal SRS, PRD, TRD, Schema definitions, and Flow diagrams.

THE GOAT ASSISTANT