This is the depth of analysis behind every project we build. We publish it as confirmation of how we actually work — the real options we weighed, the numbers that ruled them out, the one architecture that survived contact with the requirements — so our process stays as transparent to clients as the results are. Think of it as the engineering equivalent of a design studio showing five logo sketches before committing to one — except here, the chisel marks are benchmark numbers, not pencil lines.
The client ran a growing D2C storefront with a support team of 6 agents handling roughly 3,000 conversations/week across chat and email. During peak season (Black Friday through New Year), first-response time regularly blew past 20 minutes, and a large share of that volume was the same handful of questions: "Where’s my order," "I want to return this," "Can I get a refund."
The ask was not "build us a chatbot." It was: cut response time without adding headcount, and don’t let the agent make a wrong call on money (refunds, in particular, are the one place where an overconfident bot is worse than no bot at all).
That last constraint — a wrong tool call has a financial consequence — ended up driving more architecture decisions than any performance requirement did. Keep it in mind; it comes up again and again below.
This agent’s core loop is: hold a chat connection open, stream tokens back as GPT-4o generates them, and — mid-stream — pause to execute a tool call (look up an order, check a return eligibility rule) before resuming the response. That’s not a CPU-bound workload; it’s dozens to hundreds of concurrent connections that are mostly waiting — on OpenAI’s API, on our own Postgres, on the client’s order-management webhook.
Flask and classic Django both default to a synchronous, one-request-per-worker model (WSGI). A request that’s waiting 5–10 seconds on a GPT-4o call blocks that entire worker process for the duration — the classic thread-pool-exhaustion problem under concurrent I/O-bound load. You can bolt on gevent/eventlet to fake concurrency, but that’s a workaround, not a design.
Django 5.x does have async views now, and that closed part of the gap — but per Django’s own async documentation, async ORM support is still explicitly partial: async transactions aren’t supported yet, and the project’s own docs describe async ORM work as ongoing, not finished. For a support agent constantly reading/writing order and conversation state inside the same request that’s streaming tokens, that’s a real gap, not a theoretical one.
We deliberately didn’t lean on raw throughput benchmarks to make this call — the public FastAPI-vs-Django-vs-Flask numbers floating around online are inconsistent enough between sources that we don’t trust them enough to put in front of a client. The architectural fact is the one that’s actually load-bearing: an ASGI event loop serves hundreds of waiting LLM sessions on one process; a sync WSGI worker serves one.
The detail that locked it in: FastAPI’s request/response models are Pydantic models, and Pydantic generates JSON Schema natively. OpenAI’s function-calling API is JSON Schema. That means the same Pydantic class that validates a create_return tool call from GPT-4o also validates the FastAPI endpoint that actually executes it — one schema, not two definitions to keep in sync by hand. Given the brief’s hard constraint, removing an entire class of schema-drift bugs was worth more than any raw RPS number.
FastAPI. Django stays exactly where it already is in our stack — the right call for content-heavy, admin-driven, "get to market fast" builds (like the ISTRALLEN site itself). This was never "FastAPI is better than Django"; it’s "this workload’s shape doesn’t match Django’s sync-first ORM story yet."
Why self-hosting got eliminated first, before we even compared model quality: a 6-week build window with a 6-person support team as the only ops backstop is not the window to also stand up GPU inference infrastructure, model-serving monitoring, and a fine-tuning pipeline for tool-calling reliability. That’s not a quality judgment on open models — it’s a timeline judgment. Self-hosting becomes the right call again the moment token volume is high enough that per-token API cost outweighs the infra + ops overhead; we track that crossover, we didn’t hit it at this client’s volume.
Why GPT-4o over GPT-4-Turbo, specifically: OpenAI’s own August 2024 announcement of Structured Outputs is the deciding data point. With strict:true schema enforcement, gpt-4o-2024-08-06 hit 100% schema-compliant function calls on their internal complex-schema eval, versus under 40% for gpt-4-0613 without Structured Outputs. Even without constrained decoding, GPT-4o’s natural schema compliance rate was already 93%. For a refund tool call, "the model tries to call the function correctly most of the time" isn’t good enough — 100% engineered schema compliance directly answers the brief’s one hard constraint.
The open-model comparison, with an honest caveat: on the Berkeley Function-Calling Leaderboard (the standard public benchmark for exactly this capability), GPT-4o-class models have scored materially ahead of Llama-3-8B-Instruct on overall tool-calling accuracy (roughly 83% vs. 59% on the snapshot we reviewed). We’re flagging this with a confidence caveat on purpose: BFCL is a live, monthly-updated leaderboard, the numbers above are from a 2024 snapshot, and by the time you’re reading this the ranking has almost certainly shifted — newer open models close this gap fast. The honest version of this decision isn’t "GPT-4o always wins," it’s "re-run this comparison against the current leaderboard before assuming the conclusion still holds."
GPT-4o with Structured Outputs, tool schemas defined once as Pydantic models and shared verbatim between the OpenAI function spec and the FastAPI endpoint that executes it (see §1).
Why we didn’t reach for a dedicated vector DB: the knowledge base here is return policy, shipping FAQs, and product-category rules — realistically a few thousand chunks, not millions. Supabase’s published pgvector benchmark (HNSW indexing, September 2023) tested this at real scale: on a 1-million-vector, 1536-dimension dataset, HNSW indexing gave over 6x the queries/sec of the older IVFFlat index at 98% accuracy, and at 99% accuracy the same benchmark reported pgvector outperforming Qdrant on equivalent hardware. Our corpus is roughly three orders of magnitude smaller than that benchmark’s dataset. Standing up a second database, a second set of ops runbooks, and a second failure domain to search a few thousand FAQ chunks isn’t rigor — it’s over-engineering a problem Postgres already solves at this scale.
Why keeping it all in one Postgres instance mattered beyond performance: the brief’s real risk wasn’t search speed, it was consistency between "what the agent said about policy" and "what’s actually true about this specific order." Cross-referencing a return eligibility rule (in the vector-searched policy corpus) against an actual order’s return window (in the transactional orders table) inside a single transaction is trivial when it’s one database. Across two systems, that becomes an eventual-consistency problem with a customer’s money on the line — exactly the failure mode the brief told us to avoid.
Postgres with the pgvector extension, one instance, one connection pool, one backup/restore story. We’re tracking corpus growth as a trigger to revisit — the same research suggests the crossover point where a dedicated vector engine starts winning on raw throughput is well past 10 million vectors. We are not close.
Why Kafka was eliminated in about five minutes: Confluent — the company behind Kafka — publishes its own "when you shouldn’t use Kafka" guidance, and it’s blunt. Kafka earns its operational overhead when you have durable, ordered event streams with multiple independent consumers replaying the same event log, sustained at meaningfully high throughput (their own rule of thumb sits around 10,000+ events/sec sustained). Our actual background workload is escalation notifications, CSAT-survey triggers, and retrying a webhook call to the client’s order system when it hiccups — a few hundred jobs a day, not thousands a second, with no replay or fan-out requirement. Standing up a Kafka cluster for that workload is buying a freight train to deliver one parcel.
Redis queue vs. RabbitMQ/Celery: Celery’s own documentation is candid about the trade-off — Redis as a broker is well suited to fast delivery of small messages, while RabbitMQ handles larger messages and very high message rates more gracefully at real scale. Our jobs are small (a notification payload, a webhook retry, a report trigger), and we were already running Redis for caching and rate-limiting — adding a second broker alongside it would mean two systems to operate for a workload that fits comfortably inside one.
A Redis-backed queue. This is the one decision on this list we’ll almost certainly revisit without a major requirements change — if support volume grows 10–20x, or we start needing durable replay of every customer event for analytics, Kafka’s calculus flips from "overkill" to "correct." We’re not there, and building for a future scale we haven’t hit yet just adds operational surface area today for no benefit today.
A widget built on a full framework runtime is the wrong shape for embedding regardless of which framework it is — the entire embeddable-widget category (support chat widgets, in particular) ships compiled, framework-agnostic runtimes for exactly this reason. You don’t get to inherit the host page’s React (or lack of one), and you don’t get to assume the host page wants your framework’s runtime loaded alongside its own.
React earns its place on the internal console for the same reason we’d apply to any internal ops dashboard: mature real-time-UI patterns, the deepest hiring pool for whoever the client’s own team eventually hands this off to, and zero bundle-size constraint because it’s never sitting on someone else’s storefront. The point of separating these two answers explicitly: "we use React" is not an architecture decision, it’s a habit. The actual decision is "what does each surface need," and here the two surfaces needed opposite things.
A compiled, framework-agnostic Web Component for the storefront widget (drops into Shopify, a custom React storefront, or legacy jQuery identically) + React for the authenticated internal escalation console.
The published headline result isn’t a coincidence of these five decisions; each one removes a specific way the system would otherwise have failed:
Auto-resolution rates in the 68% range sit toward the upper end of what’s realistically achievable for a mature deployment — publicly reported industry figures commonly cluster in a 40–70% range, with higher headline numbers from vendors usually reflecting favorable conditions rather than a typical baseline. We’re not claiming an outlier result; we’re explaining why landing at the credible high end of that range required all five decisions above, not just the model choice.
Worth stating plainly, because no architecture is permanent:
None of these are failures of the original decision — they’re the conditions under which the same reasoning process, run again, would produce a different answer.
We’re publishing this confidence table on purpose. A client is better served by "here’s what we’re sure of and here’s what we’d re-verify" than by a document that reads clean because the uncertainty got quietly edited out.