10 Domains, 120 Checks: Why fluent AI fails at business math — and how Creeklytics verifies it

01

The fluency trap

Ask a general-purpose AI assistant for revenue and it will usually write a confident, well-structured answer. The problem is underneath. A model writes the formula the way it writes prose: by predicting what looks plausible. “Price times quantity” can turn into “total price times total quantity” — a formula that reads almost the same and produces a number thousands of times too large.

We saw exactly this in early testing. An analyst asked for revised revenue under a discount change; the first attempt multiplied summed columns and produced a figure larger than the economy of a small country. Our own guardrails caught it and refused to present it. That is a good outcome for a guardrail, and a bad experience for a user. The fix belonged in the engine, not in the model’s instructions.

“The sentence was perfect. The number was off by a factor of a thousand.”

02

The same mistakes, in every industry

Sales: a win rate quietly drops open deals from the denominator and looks healthier than it is. Manufacturing: the share of defects from two quality grades is answered by adding two rounded shares, drifting from the true combined figure. HR: an average salary ignores part-time weighting.

Healthcare: a ward with no admissions reports a 0% readmission rate instead of “no rate”. Ecommerce: revenue becomes sum of prices × sum of quantities. Finance: a what-if is estimated from averages instead of recalculating each row. Logistics: “shipments over $20 per parcel” is answered by comparing total cost ÷ total parcels to the threshold, which can only ever return all shipments or none.

Support: a weekday-vs-weekend comparison invents a column that doesn’t exist. Marketing: channels are ranked by raw conversions instead of conversion rate. SaaS: month-over-month growth is computed mentally from rounded totals. None of these are exotic. They are the questions teams ask every week.

03

Let the model choose the question, not do the math

In Creeklytics the Analyst’s job is to understand what you asked and describe the calculation: which columns, which filters, which grouping, which formula. A deterministic calculation engine then executes it over every row — in your browser for files, or inside your own data system for connected sources. The model never supplies the number.

The engine understands the shapes business questions take: row-level arithmetic inside totals, conditional what-if rules, per-row thresholds, weighted averages, calendar groupings and rankings. If the model writes a formula the engine can’t execute, it fails loudly rather than guessing.

04

Checks after the calculation

Every rate in an answer is re-run as an explicit numerator and denominator. An empty denominator produces no rate rather than zero. Filters that quietly exclude blanks are flagged. Any percentage in the answer that isn’t backed by an executed result is marked Not verified. Comparisons between groups are not described as causes.

These checks don’t make the Analyst infallible. An ambiguous question or the wrong business definition can still mislead. That is why saved business metrics and a visible result table sit next to every answer.

05

120 checks before every change

We maintain ten synthetic industry datasets — sales, manufacturing, HR, subscriptions, healthcare, ecommerce, finance, logistics, support and marketing — each with its own column names, categories and quirks. For each one we test twelve kinds of question, from simple totals to what-if recalculation and zero denominators. Every expected answer is computed independently with plain loops, never by the engine being tested.

That makes 120 checks, alongside a few hundred more for formulas, imports and exports. They run on every change. The datasets are synthetic on purpose: we don’t test on customer data, and customer data doesn’t train our models.

You can try the same industries yourself. Every new account has starter workbooks for all ten — open one, ask a question you can check by hand, and see the calculation behind the answer.

“A change that fixes one question and breaks two others never reaches you.”