How we built a food-expiry AI that scans your fridge in 2 seconds
An engineering walkthrough of FreshCheck's vision pipeline: prompt design, on-device fallbacks, and our planned migration from Gemini 2.0 Flash to Claude Sonnet 4.5.
The two-second budget
A grocery scan needs to feel like a Polaroid — point, shoot, done. We gave ourselves a hard 2-second budget from shutter press to "tracked" toast. Anything slower and users go back to typing dates into Notes.
The vision pipeline
The current production pipeline runs on Gemini 2.0 Flash through a Supabase Edge Function. The model returns a structured JSON envelope: `name`, `brand`, `category`, `expiry`, `allergens`, `storage_instructions`. We deliberately ask for a tight schema rather than free-form text — the latency saved on smaller outputs more than pays for the prompt rigidity.
The flow:
1. Camera shutter fires; we compress to 1024px max edge before upload. 2. Edge Function streams the image to the model with a system prompt that pins the JSON schema. 3. Response is validated against a Zod schema; failures fall back to the user-edit screen rather than crashing.
What we learned
- Date OCR is the hardest part. "BB 04/26" is ambiguous (April 2026? US M/D/Y? UK D/M/Y?) so we ask the model to return all candidates and let the app disambiguate based on the user's locale.
- Brand + category > name alone. "Yogurt" tells you nothing about shelf life. "Greek yogurt, plain, Fage" tells you 14 days from open.
- Allergens are a feature, not a label. Users with kids check this constantly.
Where we're going
We're migrating to Claude Sonnet 4.5 for the next major version. Two reasons: better calibration on date OCR (fewer wrong-century picks) and structured-output stability that lets us drop a layer of validation. The Supabase Edge Function abstraction means the swap is a single environment variable.
The 2-second budget holds. Most of our time is now in network, not inference — which is exactly where we want it.
Start tracking your fridge in 60 seconds
The Free plan covers 10 AI scans a month. No card required.
- The $1,500 question: how much food is your family actually wasting?We pull the numbers behind the average household food-waste bill — and show you how to estimate yours in under a minute.
- 3-day, 1-day, day-of: the science of food-waste remindersWhy FreshCheck fires three notifications per item — and why the timing matters more than the count.