The problem
Groww receives roughly 1,500 app store reviews a week across iOS and Android. Nobody reads them. The ones that matter — a ₹50 fee to change an email ID, MTF positions silently drawing from regular holdings, WhatsApp messages continuing after opt-out — are buried under thousands of reviews that say "good".
I built a pipeline that ingests both stores weekly, classifies each review into one of five themes, and produces a one-page note: top three themes, three real user quotes, three action ideas, under 250 words, delivered as an email.
What I found
88.3% of reviews carry no actionable signal. Across 5,791 reviews, only 11.7% contained an identifiable issue or request — about 130 a week. That reframed the problem from "summarise reviews" to "find the small fraction worth a PM's attention."
A quarter of the signal is feature requests, not complaints. In one week, 20 of the 44 "App Experience" reviews were people asking for features. Without separating those, a PM reads 44 problems and prioritises the wrong work. I added a second axis — theme plus complaint/request — specifically for this.
Apple's public feed only reaches back 9 days. At ~62 iOS reviews/day, the 500-review cap is exhausted in a week and a half, so the 8–12 weeks of history the brief asked for is unreachable from it. I documented the ceiling and sourced trend history from Play instead of claiming coverage I didn't have.
How I worked
Four phases, each with pass/fail exit criteria and committed evidence — no phase closed on a claim, only on an artifact. I hand-labeled 100 reviews blind to validate the classifier rather than trusting its own report.
What failed
One exit criterion failed and I shipped it failing. The classifier mislabels a dashboard-regression review as no-signal. I traced it to batch-composition sensitivity — a review's label isn't fully independent of its neighbours — and measured the effect at ~4% of labels. I didn't tune the prompt to fix it, because the five test cases were what I derived the classification rules from, so passing them would have been tuning on the test set.
Stack
Python, Groq (gpt-oss-120b) for classification only. The note itself is deterministic — counts, trends, quote selection and actions are all code, so it can't hallucinate a statistic or invent a recommendation.