Free Checklist
The AI MVP launch checklist
The 40-point list we run before every client launch. Most AI products that fail post-launch fail on something in here.
Product & scope
- The core workflow is validated with at least 5 real target users, not just friends
- The MVP scope fits on one page — everything else is explicitly deferred
- There is one primary success metric the launch will be judged on
- The 'AI-optional' test is done: you know exactly what the AI adds over a non-AI version
- Manual fallback exists for every AI-powered step
Quality & evaluation
- A golden evaluation set exists — at minimum 50 real (not synthetic) examples
- The quality bar is written down with concrete examples of good and bad outputs
- Evals run automatically on every prompt or model change
- Current eval scores are known and above the launch threshold
- Hallucination / faithfulness rate is measured, not assumed
- Edge cases are tested: empty input, huge input, wrong language, adversarial prompts
- A human has reviewed at least 100 real outputs end-to-end
Cost & performance
- Cost per user action is calculated at current model pricing
- Cost at 10x and 100x current usage is projected — and survivable
- Caching is in place for repeated or similar requests
- Cheaper model routing is configured for low-stakes steps
- P95 latency is measured and acceptable for the user's context
- Rate limits and spending caps are set on every model API account
- Streaming is enabled anywhere users wait more than 2 seconds
Failure handling
- Model API outages degrade gracefully — users see a useful message, not a spinner
- A fallback model or provider is configured and actually tested
- Timeouts and retries have sane limits (no infinite retry loops burning budget)
- Low-confidence outputs are flagged or routed to human review
- Users can report a bad output in one click, and reports go somewhere you read
Security & data
- API keys live in a secrets manager, not in code or client bundles
- User data sent to model providers is inventoried and documented
- Data retention settings on model providers are configured (training opt-outs, retention windows)
- Prompt injection is tested against your system prompts and tools
- PII is stripped or masked before it reaches third-party APIs where required
- Terms of service and privacy policy reflect what the AI actually does with data
Observability
- Every model call is logged: input, output, latency, cost, model version
- A dashboard shows daily volume, cost, error rate, and eval score trends
- Alerts fire on cost spikes, error spikes, and eval regressions
- You can trace any user-reported bad output back to its full context in one lookup
Go-live
- Prompts, model versions, and eval sets are version-controlled like code
- A rollback path exists for every prompt and model change
- Launch traffic is gated: soft launch to a subset before full exposure
- Support macros exist for the three most likely AI-related complaints
- The first-week review is scheduled: eval scores, costs, and user feedback in one meeting
- Someone owns the AI quality metric — by name, not by committee
Tip: print this page (Ctrl/Cmd+P) for a physical checklist.
Get the full checklist
Enter your email and the complete 40-point checklist unlocks instantly — printable and free to share with your team.
No spam — occasional practical AI-building notes, unsubscribe anytime.
Failing items on this list? We fix exactly that.
A free scoping call tells you which gaps matter for your launch — and what closing them costs.