WHITEBOXX FIELD GUIDE
Run an AI research pilot
against a business task.
Evaluate an AI research pilot by asking it to complete one representative business task. Check the source records, reproduce the important calculations, test a missing-input case and record the human work needed to use and repeat the result.
How should you evaluate an AI research platform?
Use a question your team already needs to answer, an agreed input set and a written acceptance record. Compare the result with a manual or existing-system baseline. Judge whether the output is correct, traceable and usable in the intended process before judging how polished or fluent the explanation sounds.
An effective pilot separates three decisions: whether the available data can answer the question, whether the analysis performs the required work, and whether the surrounding application fits the organisation. A successful report does not by itself demonstrate scheduled operation, enterprise identity, downstream system actions or an operating service.
EXAMPLE / A REGIONAL REVIEW
Give the pilot a clear assignment.
For a three-country market review, agree the countries, measures, common period and intended reader. Ask for a comparison table, findings with sources and questions for further commercial research.
Record the public data, internal measures and licensed material the pilot can access. If a required source is unavailable, ask it to identify the gap.
A REPEATABLE EVALUATION
How to run the pilot review.
- 01
Fix the scope
Write down the decision, user, entities, countries, period, measures and expected deliverable. Keep the same brief for each approach you evaluate.
- 02
Keep a reference result
Retain an accepted manual calculation or source-system report. Compare totals, definitions and time spent preparing the result, rather than comparing writing style alone.
- 03
Trace material findings
Choose important claims and work back to their records, source dates, transformations and calculations. Investigate discrepancies before accepting the narrative.
- 04
Test an incomplete input
Remove or delay a required source in the pilot. Check that the result identifies the gap and that the user can see what remains unresolved.
- 05
Repeat the task
Use a second period or capture and check what changed. Record checking time, corrections, access requirements and the implementation needed to repeat delivery.
THE ACCEPTANCE RECORD
Review the output with the team that will use it.
| Question for the reviewer | Evidence to retain |
|---|---|
| Did it use the agreed scope? | The brief, selected entities or geographies, measures and periods. |
| Can the material findings be checked? | Original references, returned observations and the calculations. |
| Did it leave useful research questions? | The unresolved inputs and proposed follow-up investigation. |
| How much work remains for the team? | Time spent checking, corrections and the steps needed to use the deliverable. |
| What is required to repeat or integrate it? | Source access, workflow setup, output format and ownership. |
READING THE RESULT
What should cause you to change the pilot?
| Observation | Interpretation | Next step |
|---|---|---|
| Totals differ from the reference | The disagreement may come from scope, units, duplicates or calculation logic. | Reconcile the inputs and calculation before evaluating the recommendation. |
| A required source is absent | The available evidence cannot support the full assignment. | Narrow the question or obtain the missing source; retain the gap in the result. |
| The analysis works but delivery is manual | The analytical task is useful; the application and operating process remain to be built. | Scope the interface, scheduling, review and system integrations separately. |
| The result cannot be reproduced | A fluent answer does not establish a repeatable research process. | Retain the source capture and intermediate calculation, then rerun the review. |
THE DECISION
Use the findings to choose the next step.
The pilot may show that the available platform tools fit the task, that another source or calculation is needed, or that the team needs a custom application around the research. Keep the output and review comments with that decision.
Inspect an example evidence pack