On this page
Key takeaways
- Test any AI analyst on your own data before you trust it, using questions you already know the answers to.
- Most errors are not random. They fall into eight repeatable failure types.
- Score three outcomes: correct, asked for clarification, and wrong. Clarification counts as safe.
- Rerun the same test after every change to your data, definitions or tool.
Why test before you trust
AI data analysis tools are quick and usually right. "Usually" is the problem: one confident wrong number in a board deck costs more than the time the tool saved.
The fix is simple. Before you rely on any tool, including DashboardGPT, run it through a fixed set of questions on a dataset you know well, and score the answers. This article gives you that set.
Build your answer key first
- Pick one dataset you know well, such as last year's orders or subscriptions.
- Work out the true answers for the questions below, with SQL, a spreadsheet or your existing BI reports.
- Write them down before you start, so the tool's answer cannot anchor you.
The 50 questions
Replace the column names with your own. The questions get harder as you go.
Level 1: Single numbers (1–10)
- Total revenue last year
- Number of orders last month
- Average order value this quarter
- Number of unique customers this year
- Largest single order ever
- Revenue on a specific date
- Number of orders with a discount
- Median order value last month
- Total refunds last quarter
- Number of active subscriptions today
Level 2: Breakdowns (11–20)
- Revenue by region last year
- Orders by product category this quarter
- Average order value by channel
- Customers by country, top 10
- Revenue by sales rep, top 5
- Refund rate by category
- Orders by day of week
- Revenue by plan
- Share of revenue from new vs returning customers
- Discount rate by region
Level 3: Time and comparisons (21–30)
- Revenue by month for the last 12 months
- Week-over-week order growth for the last 8 weeks
- This quarter vs same quarter last year, by region
- Month with the highest revenue this year
- Year-to-date revenue vs last year to the same date
- Rolling 30-day revenue
- Days between first and second order, median
- Cumulative revenue this year by month
- Change in average order value from Q2 to Q3
- Revenue by month, split by channel
Level 4: Definitions and edge cases (31–40)
- Monthly churn rate by plan
- Net revenue after refunds and discounts by month
- Active customers (ordered in the last 90 days) by month
- Repeat purchase rate by acquisition channel
- Revenue per customer by signup cohort
- Orders excluding test and internal accounts
- MRR with annual plans normalised to monthly
- Customers who ordered in Q2 but not in Q3
- Share of revenue from the top 10 customers
- Conversion rate from trial to paid by month
Level 5: Why and what-if (41–50)
- Why did revenue drop in the weakest month?
- Which region explains most of this quarter's growth?
- What changed in the week with the most refunds?
- Which channel's cost per order rose the most?
- Which customer segment churns fastest?
- If the worst channel's orders were removed, how would growth change?
- What is unusual about the largest orders?
- Which product category is shrinking in share?
- Is there a day of the week with unusually high refunds?
- What would revenue be if refund rates matched the best category?
The eight ways answers go wrong
| Failure type | What it looks like | How to catch it |
|---|---|---|
| Wrong column | "Revenue" used gross when you meant net | Ask which column was used |
| Double counting | A join to line items inflates totals | Compare the total with your answer key |
| Wrong grain | An average of daily averages instead of a monthly average | Check how the average was computed |
| Wrong time window | "Last month" read as the last 30 days | State the window explicitly |
| Time zone | Daily counts shifted by midnight UTC | Compare one day's count |
| Missing filter | Test accounts or cancelled orders included | Say what to exclude |
| Definition drift | "Active" or "churn" defined differently from your team | State your definition in the question |
| Over-confident explanation | A "why" answer that names a cause the data does not show | Check the insight against the chart's own numbers |
Most failures at levels 1 to 3 are the first five types. Level 4 is mostly definitions. Level 5 is where over-confident explanations appear.
How to score the test
Give each answer one of three marks:
- Correct: matches your answer key.
- Clarified: the tool asked a sensible question before answering, such as "gross or net?". Count this as safe.
- Wrong: a confident answer that does not match.
Then look at two numbers:
- Wrong rate at levels 1 to 3. For everyday use this should be very low. If not, look at column names and the data profile before blaming the tool.
- Wrong rate at level 4. This usually drops sharply once you state your definitions in the question.
Treat level 5 differently. "Why" answers are starting points for investigation. Judge them on whether they point you to the right place, and always check the chart underneath.
Rerun it regularly
Keep the 50 questions and your answer key. Rerun them when you change your data source, rename columns, adopt new definitions or switch tools. It takes about an hour and it is the most reliable way to know whether you can trust the numbers.
Frequently asked questions
It depends on the tool, the data and the question. Clear column names, stated definitions and simple questions give the best results. Test on your own data to know for sure.
Wrong columns, double counting from joins and wrong time windows account for most errors on everyday questions.
Twenty questions give a quick signal. The full 50 cover definitions and "why" questions, where tools differ most.
No. A clarifying question is the safest outcome for an ambiguous question.
Yes. Use it on the playground's demo dataset to learn the tool, then on your own data with your own answer key.