The entire pitch of the LairsFlow Advisor is that it reasons over a student's actual logged data instead of handing out generic productivity tips. That is a claim worth testing directly, not assuming, so this post documents exactly how we tested it, what came back, and where it fell short.
The constraint that shaped the test
The Advisor needs the student's own free Gemini API key, and the app itself caps usage at ten prompts a day per device on top of whatever the free API tier allows. A real stress test - a dozen or more pointed questions, back to back, checking for hallucination and generic non-answers - would blow through that quota in a single sitting before getting anywhere close to a real picture.
So instead of calling the Gemini API at all, we used the app's own context-building code, buildAiContext(), completely unmodified, to generate the exact system prompt the Advisor would send for a given student, and pasted that snapshot straight into Gemini's free web chat as the first message of a conversation. No API key, no quota, and it's the real prompt-construction logic doing the work, not a hand-written approximation of it.
This is not identical to the production call. The live app sends that text through a separate systemInstruction field with a specific model (gemini-2.5-flash, thinking disabled) via the API; Gemini web has no equivalent channel, so the pasted block just becomes the first conversational turn, and the underlying model/thinking behavior may differ slightly. That gap means this test measures grounding and reasoning quality, not byte-for-byte production wording - good enough to catch the thing we actually cared about: is it reading the data, or just sounding plausible.
The fabricated student
We seeded a large, entirely fictional dataset through the real code - no real student's data was used. The profile: a JEE aspirant, 587 logged questions across Maths/Physics/Chemistry, a syllabus-pacing deadline 44 days out, one chapter sitting at a genuinely bad 35% solve rate, a handful of chapters gone quiet for over a week, and a recurring [phone, social media] tag on the days where a daily target got badly missed. Every number in that snapshot is fixed and known in advance, which turns "did it give a good answer" into something checkable: does the reply cite the real 35%, the real chapter name, the real dates, or does it default to something vaguer.
Ten questions were run in a single thread, covering four areas: assessing weak spots, planning a day's target, adjusting pace against the deadline, and two deliberate honesty checks (asking for data that was switched off, and asking about a subject the student never studies).
What it got right
The strongest result was on cross-referencing. Asked what was actually driving its mistakes, it correctly separated the aggregate picture (conceptual errors edge out calculation and silly mistakes overall) from the per-chapter reality (Chemistry and Physics chapters were consistently failing on calculation, while the weak Maths chapter's problem was specifically silly/execution errors) - two different diagnoses that require reading down to the chapter level, not just quoting one aggregate number.
Asked to build a six-week plan for the remaining 44 days, it produced a week-by-week breakdown that named a different, correct set of real chapters every week - the weakest chapter first, then the two chapters gone quietest, then the two with zero logged questions at all, then the two mid-tier chapters, closing with the three strongest ones as timed sprints. Every single solve rate and staleness figure it cited in that plan matched the source data exactly.
On the deadline question, it correctly reported the actual pacing numbers (44 days left, required pace 0.05 chapters/day, actual pace 0.43) and, more importantly, added a distinction the raw numbers alone don't make explicit: being "on pace" here only measures first-pass syllabus coverage, not mastery, and a chapter can be technically "covered" while still sitting at a 35% solve rate. Asked directly for the single biggest risk to the deadline, it picked the phone-distraction pattern and defended it by citing the two exact dates where a target got badly missed, both correctly tagged.
On the two honesty checks, it declined to invent a specific note or timestamp it did not have access to, and correctly said no Biology data existed for a JEE-focused profile rather than fabricating a subject that was never part of the dataset.
What it got wrong
Under a deliberately terse, time-pressured prompt ("I only have 90 minutes, not my usual 6 hours - what should I actually spend it on?"), the reply skipped the natural-language explanation entirely and came back as a bare, unfenced JSON blob with no surrounding sentence. In the live app, the structured juzflow-target block is only ever stripped out of what the student sees when it is wrapped in a specific fenced format; an unfenced block like this one would not be recognized by that parser and would render as a raw, ugly JSON string directly inside the chat bubble instead of a clean plan. That is a real formatting failure mode under a specific kind of prompt, not just a cosmetic quirk of testing outside the app.
One question in the batch - a direct count comparison between two subjects - produced no visible answer at all in the transcript. It is not yet clear whether that was a dropped reply, a copy-paste gap while collecting the transcript, or something else; it is flagged as open rather than explained away.
Neither of these is fixed yet. They are logged as follow-ups, and this post is being published with the gap named rather than quietly patched out of the record.
Try it yourself
Everything needed to reproduce this is attached below: the exact context snapshot pasted into Gemini web, the ten test prompts with an answer key for each, and the full raw transcript of the run this post is based on.
- context-snapshot.txt - the real, unmodified output of
buildAiContext()for the fabricated student, ready to paste as the first message in a fresh Gemini web chat. - test-prompts.md - the ten prompts, in order, each with what a grounded answer should contain versus what a generic one looks like.
- raw-transcript.txt - the unedited replies this post is analyzing.
Paste the snapshot into a new chat at gemini.google.com, work through the prompts, and compare notes. If something looks off, that is exactly the kind of signal this whole exercise exists to surface.