When Product Research and Analytics Disagree

Six research participants call bulk export essential. Product analytics show that only three per cent of active accounts used it last month.

One stakeholder says the interviews are anecdotal. Another says the tracking misses what users really need. Both want the evidence to declare a winner.

It cannot, because the statements may not contradict each other at all.

One describes the importance of a task to selected people. The other describes a recorded action among a defined population during a particular period.

Before prioritising the feature, the team needs to ask whether both claims refer to the same people, behaviour, opportunity and meaning.

The numbers in this example are invented to make the diagnosis concrete. They are not a benchmark or product result.

Open a discrepancy case, not a debate

Write each claim in full. Ban “users say” and “the data says” until the speaker can complete the sentence.

Research claim

Six of eight interviewed finance administrators described exporting transaction rows as necessary for month-end reconciliation with an external accounting process.

Analytics claim

During the previous calendar month, three per cent of accounts classified as active recorded at least one bulk_export_completed event.

Already, the edges show. People are not accounts. Necessity is not frequency. Month-end work may be concentrated in a short window. A completed event may omit failed attempts.

This claim discipline follows the spirit of data-driven product management: evidence supports a decision only when its definition is inspectable.

Check the population

The research recruited finance administrators who reconcile transactions. The analytics denominator contains every active account, including teams that never perform reconciliation.

Ask of the research:

• Who was eligible and how were they recruited?
• Which roles, plans, regions and levels of experience are represented?
• Were people selected because they already used export?
• Which relevant users could not participate?

Ask of the analytics:

• What makes an account active?
• Does the population include accounts without export permission?
• Can finance administrators be identified reliably?
• Are trials, internal accounts or test traffic present?
• Are plan and region differences hidden by aggregation?

Neither population is automatically superior. They answer different questions until the team aligns or deliberately compares them.

Check the unit

Qualitative research usually observes people, tasks and situations. Product analytics often counts users, sessions, devices, accounts or events.

The unit can reverse the story.

One export by a finance administrator may serve an account with hundreds of members. Account-level use can look modest while the workflow remains operationally critical.

Conversely, a retry bug can produce many events from one struggling person. Event volume can look healthy while successful use is narrow.

Write the unit beside every measure. If the decision concerns retained accounts, individual click frequency may be the wrong outcome.

Check time and cadence

The analytics window is one month. The task happens at month end. Did the window include a complete cycle for every region and accounting calendar?

Ask whether the research concerns a daily habit, a quarterly obligation, an annual exception or a one-off migration.

Low frequency does not imply low value when the event prevents a costly failure. High frequency does not imply value when people repeat an avoidable step.

Look at cohorts across appropriate cycles. Do not widen the date range blindly; product changes, tracking changes and seasonality can make older periods incomparable.

Check opportunity and exposure

People cannot use a control they cannot find, access or understand.

The analytics event records completion, not opportunity. The team must reconstruct the funnel:

1. Was the person in a role allowed to export?
2. Did the account’s plan include the feature?
3. Did the person visit a page where export was available?
4. Was the control rendered and visible?
5. Did an attempt begin?
6. Did it fail, cancel or complete?

An exposure event should mean the feature was genuinely available, not merely that a page loaded somewhere below the fold.

This is where instrumentation review becomes product discovery. The event model reveals which opportunities the team has chosen to see.

Check the task, not only the label

Participants may say “export” while solving different jobs:

• moving data into an accounting package;
• creating an audit trail;
• checking anomalies in a spreadsheet;
• sending a client report;
• escaping a product they do not trust.

The interface event bulk_export_completed may cover only CSV export from one screen. Users may copy a table, use an API, request a support extract or abandon the task.

During analysis, return to observations and context. The GOV.UK guidance on research analysis recommends separating raw observations from findings and involving people who observed the sessions.

This guards against turning a memorable quote into a universal demand.

Check the construct

“Essential” and “used” are different constructs.

Research may reveal perceived importance, workaround cost, confidence, comprehension or a hidden constraint. Analytics may reveal recorded behaviour, sequence, frequency or correlation.

Neither directly measures the causal value of building a better export.

A participant may call export essential because the product lacks an integration. Improving export could entrench the workaround rather than solve the underlying need.

Equally, low use may be caused by poor discoverability. Observed struggle can explain the pattern without estimating how common it is.

Good qualitative research and good analytics are complementary because their blind spots differ.

Check the instruments

Now audit how each signal was produced.

For research:

• Was the question leading or hypothetical?
• Did participants describe recent behaviour or future preference?
• Was the task observed, demonstrated or merely recalled?
• Were contradictory cases retained in the analysis?
• Does the evidence trail support the finding?

For analytics:

• Does the event fire only after a successful export?
• Did a release rename, duplicate or suppress it?
• Are ad blockers, consent choices or offline clients relevant?
• Can server logs corroborate client events?
• Does identity stitching merge the right people and accounts?

Instrument failure is not a philosophical dispute between methods. It is a defect in the evidence pipeline.

List explanations that could make both signals true

The team produces competing explanations before choosing more research:

Explanation	What it predicts	Cheapest useful check
The task matters to a small, valuable role	Use concentrates among finance administrators and correlates with reconciliation periods	Segment eligible roles and inspect account context
People need the outcome but cannot find the feature	Eligible page visits are common, control exposure is low, and observed sessions show discovery trouble	Add validated exposure and attempt events; run task observation
People use another route	API, copy or support requests appear in accounts reporting the need	Review adjacent events and support cases
Research recruitment over-selected export users	Need weakens in a broader purposive sample	Recruit eligible non-users and recent adopters
Completion tracking is incomplete	Server-side export jobs exceed client completion events	Reconcile client events with job logs
“Export” is shorthand for interoperability	Participants value transfer into another system, not a file	Map the reconciliation journey and test integration concepts

The table prevents an expensive “do more research” response. Each follow-up should distinguish explanations, not merely collect another pile of opinions.

Choose the smallest discriminating follow-up

For the hypothetical case, the first decision is not to redesign export. It is to repair the comparison.

The team chooses a two-part follow-up:

1. Validate export completion against server-side job records, then segment eligible accounts by role, plan and month-end exposure.
2. Observe a small purposive set of eligible people attempting a recent reconciliation task, deliberately recruiting both recorded users and non-users.

The sample size is not presented as statistically representative. The sessions are intended to expose task routes and explain behavioural patterns.

The quantitative check is not expected to explain motives. It is intended to establish opportunity, completion and concentration with a trustworthy denominator.

Before collecting data, write decision branches:

• If use concentrates in a valuable role and the task is costly, improve the role-specific workflow.
• If exposure is the main break, test navigation and language before adding capability.
• If another route dominates, decide whether export should be maintained, integrated or retired.
• If event coverage is poor, repair instrumentation before using frequency to prioritise.
• If the underlying need is interoperability, test that outcome rather than polishing a CSV button.

This is a discriminating evidence plan, not a controlled experiment.

The experiment design guide becomes relevant if the team deliberately changes exposure or compares assigned alternatives.

Decide what would count as resolution

Triangulation does not require every method to produce the same result. It asks the team to interpret convergence, complementarity and discrepancy deliberately.

A published mixed-methods study by Tonkin-Crine and colleagues compared four datasets through a triangulation protocol.

Its analysts classified pairwise findings as agreement, partial agreement, dissonance or silence.

Its healthcare context is not a product-playbook proof. The transferable lesson is methodological: disagreement can reveal differences in constructs, samples and interpretation.

The case is resolved when the team can explain which population and task each signal represents, which explanation best fits the combined evidence, and what decision follows.

It is not resolved when one discipline wins the meeting.

Keep a short discrepancy record

For consequential conflicts, preserve:

• the two full claims;
• population, unit, window and exposure definitions;
• research recruitment and task context;
• event definitions and instrumentation checks;
• competing explanations;
• chosen follow-up and decision branches;
• unresolved limits and owner.

This record stops the same argument returning with different screenshots.

It also improves how teams include user feedback. Feedback is evidence with provenance, not a vote that bypasses observed behaviour.

When research and analytics appear to disagree, resist the urge to average them or choose the more impressive chart.

Make both claims precise. Find the mismatch. Then buy only the next piece of evidence that can change the decision.

Sources

• GOV.UK: Learning about users and their needs
• GOV.UK: Using performance data to improve your service
• GOV.UK: Analyse a research session
• Tonkin-Crine et al.: Discrepancies between qualitative and quantitative evaluation of randomised controlled trial results

Read next

• Making Smart Decisions: Data-Driven Product Management
• Best Practices for Including User Feedback in Product Management
• Better Products Start with Qualitative Research
• From Data to Decisions: Experiment Design