A narrow job with sharp constraints.
A chat terminal that answers questions about Gazzali Zayn — role, experience, skills, projects — grounded in a one-page profile, and politely refuses anything off-topic. The constraints are what make the decision interesting:
The "knowledge" is a handful of facts (AUM, funnel rates, timeline). It changes only when the résumé changes.
Recruiters click occasionally. There's no steady high-QPS load to amortise a dedicated server against.
A résumé that takes 10s to answer — or costs real money to run — is a worse résumé.
Zero hallucinated facts; reliable off-topic and prompt-injection refusal matters more than eloquence.
Success = accurate on the hard numbers · near-zero hallucination · reliable refusal · sub-second feel · ~$0 marginal cost · updates in minutes (edit a prompt), not a retraining run.
Three ways to power it.
Hosted small-model API
A thin FastAPI proxy to gemini-2.5-flash-lite (or Groq / OpenRouter). The status quo.
shippedSelf-hosted open LLM
A small, quantized instruct model served from my own container on Render. The path this study explores.
prototypedNon-generative retrieval
Embedding / keyword match → return pre-written facts. No generation at all. Cheapest, least flexible.
consideredIf I built it — how.
The constraint that drives everything: Render is CPU-only. No GPU on standard plans. So any self-hosted model runs on CPU — capping size at ~3B params (quantized), pushing latency to seconds, and ruling out the free tier (512 MB RAM can't hold a usable model).
Model selection
Small, permissively-licensed instruct models suited to a fixed-domain FAQ:
| Model | Params | License | Note |
|---|---|---|---|
| Qwen2.5-1.5B-Instruct ⭐ | 1.5B | Apache-2.0 | Best instruction-following per byte; clean license. |
| Llama-3.2-1B / 3B | 1–3B | Llama | Strong; heavier license terms. |
| Gemma-2-2b-it | 2B | Gemma | Good quality, Google license. |
| SmolLM2-1.7B | 1.7B | Apache-2.0 | Built for lightweight / on-device. |
| Phi-3.5-mini | 3.8B | MIT | Highest quality of the set; most RAM. |
Pick → Qwen2.5-1.5B-Instruct · GGUF Q4_K_M (~1.1 GB) · served in-process via llama-cpp-python · needs a ~2 GB instance.
Key design call: ground with RAG, don't fine-tune the facts. For a KB this small and volatile, I'd retrieve the relevant profile facts into context at inference time (a 22M-param embedder is plenty) rather than baking them into weights. That keeps updates to "edit the text" — fine-tuning would mean retraining every time a number changes, and risks reciting stale facts. A QLoRA tune stays a "only if measured necessary" lever for format & refusal behaviour, never knowledge.
The part that's actually engineering.
A golden set of ~80 questions across the categories that matter — single-fact lookups, multi-fact synthesis, off-topic refusals, and prompt-injection — scored on factual accuracy (regex checks on hard numbers like ₹400 Cr, 48%→90%), faithfulness (LLM-as-judge vs. the source), refusal precision/recall, format adherence, latency, and cost.
Projected scorecard — factual accuracy
$0 today — but that's a free-tier number.
The hosted option costs nothing right now only because traffic sits inside Gemini's free tier. That's a property of volume, not architecture. Honest unit economics: a hosted API scales with usage, while a self-hosted box is a flat monthly fee. Past some volume the lines cross — and self-hosting becomes the rational call.
What trips the switch — sooner than raw cost. Two things move first: (1) free-tier rate / daily quotas — at higher traffic you blow past them and start paying per call; (2) reliability and latency under load. At ~150 tokens/answer, flash-lite is fractions of a cent per question, so the pure-cost crossover with a ~$25/mo CPU box sits in the hundreds-of-thousands of questions per month range — but the quota ceiling bites long before that.
So the decision is explicitly volume-dependent: at a résumé's traffic, buy. If "Ask Gazzali.ai" ever became a high-traffic product, the same analysis flips to build — and because the proxy isolates the model behind a single _generate() function, switching to a local Llama (or any other provider) is a one-function change, not a rewrite.
Scoring the trade-offs.
| Criterion | A · Hosted | B · Self-host | C · Retrieval |
|---|---|---|---|
| Answer quality | |||
| Accuracy / low hallucination | |||
| Latency | |||
| Cost — at current volume | |||
| Ops / maintenance | |||
| Update effort | |||
| "Run my own model" wow-factor |
▸ Cost & ops ratings assume résumé-scale traffic. Past the crossover (§05), the cost row flips toward self-hosting.
Buy, not build — at this volume.
At a résumé's traffic, self-hosting loses on every axis that matters — ~10× slower, less accurate, and (today, on the free tier) it would add cost — to gain only "I own the weights," meaningless for public data with no privacy driver. So: ship hosted now, but treat it as a volume-dependent call. The cost crossover (§05) is the documented trigger to flip to a local model — and because the proxy isolates the model behind one _generate() function, that flip is a one-function change, not a rewrite. The discipline is knowing which regime you're in.
Not "never" — "not yet."
Self-hosting flips positive when the trade-offs change: