Menuto

You could send a model a photo of the menu and ask “what should I order?” You’d get a generic answer: no memory of what you’ve liked, no awareness of what reviewers say about this restaurant, no way to learn that you loved the cacio e pepe but hated the carbonara. Every conversation starts from zero. I wanted a system with state: one that tracks your favorites across restaurants, extracts taste signals from your ratings, and runs an 8-component scoring algorithm with Bayesian weight learning that adapts to how you decide.

Rather than rigid scoring formulas, an LLM agent receives every signal about the user and reasons about what to recommend. An earlier version used 10 hand-tuned components (personal taste: 0.30, sentiment: 0.17, etc.), but the weights were identical for everyone and couldn't reason about context.

Informed by Microsoft’s RecAI framework (Zhao et al., 2024): the “LLM-as-brain, models-as-tools” pattern where traditional signals generate candidates and the LLM handles final reasoning. The serendipity slot draws on SERAL (Chen et al., 2025) for filter-bubble mitigation, and the implicit negative feedback model follows Hu, Koren & Volinsky (2008) on collaborative filtering for implicit feedback.

8 signal sources per dish: parsed menu items, Google Places reviews (cached 14 days), review-based popularity, cross-user order counts, past ratings, behavioral signals (views/orders/favorites), LLM-extracted taste keywords, and embedding-based taste similarity (2 batch API calls).

The only rigid step. LLM-generated dietary flags per dish, prompted to catch hidden ingredients (anchovy in Caesar dressing, fish sauce in Pad Thai). Falls back to a 30+ term keyword list for older menus.

Each candidate gets readable flags: MATCHES YOUR TASTE, POPULAR (60%), WELL-REVIEWED, LOOKED AT BUT NEVER ORDERED. No numerical scoring, just facts the agent can reason about.

The agent receives the full user narrative: taste profile, spice tolerance, hunger, cravings, the adventure-vs-safe slider, dining occasion, free-text mood, and history here. It reasons about meal composition, honors cravings, and writes a personal explanation per dish.

After ordering, the user rates dishes with quick-tap tags and optional notes. The LLM extracts taste signals from the text: “Loved the cream sauce” becomes liked: [“cream”, “rich sauce”], boosting similar dishes next time.

The 8-component algorithm doesn't use fixed weights. Each user has Bayesian priors (alpha/beta per component) that update on every rating, so the system learns whether they respond more to popularity, taste matching, or craving alignment, with no cold-start dataset. After ~10 ratings the weights diverge meaningfully from the uniform prior.

The user's taste profile and each dish description are embedded into the same vector space, then scored by cosine similarity. Someone who likes "creamy burrata" scores well on "stracciatella with olive oil" even with no keyword overlap. Two API calls total: one for the profile, one batch for all candidates.

The LLM extracts per-dish sentiment from Google Places reviews. “The cacio e pepe was transcendent but the tiramisu was dry” decomposes into dish-level praise and criticism scores feeding the customer_praise component. Cached 14 days to stay within the Places API free tier.

Three input paths (URL/HTML scraping, PDF via PyMuPDF, camera photo via LLM vision) normalize into the same ParsedDish schema. Content type is auto-detected from response headers with a byte-sniffing fallback.

Personal taste (embedding similarity), craving match, hunger fit, popularity/sentiment, dietary compliance, cuisine affinity, price fit, and friend boost. The system explains each recommendation by surfacing which components dominated.

dish_views, dish_ratings, dish_orders, dish_favorites are separate normalized tables rather than a single interactions table. Enables efficient per-signal queries and signal-specific columns (hunger_level_when_ordered on orders, taste_signals JSONB on ratings).

New users get recommendations weighted toward what others ordered and review sentiment. Free-text mood ("celebrating tonight") gives the agent context even without rating history.