Product Analytics Instrumentation QA: An Event-Taxonomy Audit Scorecard

The audit starts before code ships

A dashboard can be numerically correct yet drive a wrong decision: an event trigger changed, a property uses a display label instead of a stable ID, or anonymous activity disappears at login. The chart renders, but the data contract failed.

A product analytics tracking-plan QA checklist controls the path from spreadsheet to data product that growth and engineering teams can trust. It inspects a defined taxonomy before release and verifies shipped payloads match it; it does not replace product strategy, warehouse modelling, or generic analytics review.

Review the unit supporting a decision: which events and properties answer “did onboarding reduce time to first value?” or “which plan drives expansion?” Ambiguous names, missing identities, and incomparable timestamps are defects even when an event fires. This scorecard gives product managers, analytics engineers, analysts, and engineers a release standard: tracking as a testable interface with owners, evidence, and a path for defects that cannot be fixed before launch.

A tracking plan behaves like a data contract

An event taxonomy is controlled behavioral-data vocabulary: events record business actions and properties provide context. The plan defines event name, trigger, required properties, allowed values, identity rules, timestamp convention, consent constraints, owner, and destination.

Separate user actions from interface interactions. checkout_started can mean a customer entered a purchase flow; button_clicked means an element received a click. Both matter but answer different questions. Interface-only plans force analysts to reconstruct business states from brittle click sequences, while broad business-only events can hide friction that blocked progress.

Required properties are part of an event. plan_selected without plan_id, billing_interval, and surface cannot distinguish a monthly pricing-page choice from an annual in-product upgrade. Define types and value rules: “string” is insufficient for revenue reporting; specify a canonical ID, currency code, enum, or decimal amount in minor units.

The contract also has time. A client timestamp records when an action occurred on-device; a received timestamp records endpoint acceptance. Both help with offline queues and mobile clients, but mixing them without a convention corrupts funnels, daily active user counts, and experiment exposure windows.

If the team is still deciding what deserves instrumentation, define instrumentation scope and deliverables before writing events. QA cannot rescue a plan unconnected to a product decision.

Audit a complete user journey

Row review catches spelling drift but misses broken joins between steps. Audit at least one end-to-end journey across failure-prone transitions: anonymous visitor to signed-in user, trial to paid customer, web to mobile, or online action to queued delivery.

In SaaS onboarding, a campaign visitor creates a workspace, invites a teammate, connects a data source, and sees the first usable report. Events may include workspace_created, member_invited, source_connected, and report_viewed. Activation might be report_viewed only when the report contains live source data. Counting an empty-state view as activation overstates acquisition and starts retention analysis with the wrong cohort.

Follow data through consumer paths: a product analyst calculates activation, a lifecycle marketer suppresses a reminder email, and the data team joins a subscription table. These reveal different defects: missing workspace_id prevents account reporting, nullable user_id may be valid before login, and mutable source_name breaks joins after a customer renames a source.

Journey review also exposes sequencing assumptions. An invite can precede verified email, payment confirmation can follow product access, and a mobile event can arrive hours late offline. State whether downstream logic uses occurrence time, receipt time, or both; otherwise teams make inconsistent choices and call the disagreement attribution.

Matching names can hide broken meaning

Comparing implementation only with spreadsheet event names is the weakest audit habit. Name parity proves little; these failures can pass review until a report is challenged.

More properties do not necessarily improve data. Extra fields add privacy-review work, schema drift, storage cost, and uncertainty about authority. Capture minimum context for a named decision; a property without an owner, allowed values, or known consumer is a removal candidate.

Likewise, an identify call does not prove identity merging works. It may reach one destination but not another, merge after business events emit, or create valid but surprising shared-device relationships. Inspect resulting person or account history, not just the outbound request.

Score defects by decision damage

Without severity, release review becomes a contest over who speaks loudest. Score defects by the decision they can distort, not repair annoyance. A missing cosmetic property is not equivalent to duplicate purchase events, even if each takes an engineer an hour to fix.

Use the sheet below as a downloadable audit sheet: copy rows into a spreadsheet, add one per rule, and export CSV or XLSX with the release packet. Severity weight is 1 for low impact, 3 for material reporting risk, and 5 for defects that can corrupt revenue, experimentation, identity, or compliance-sensitive data. Impact scope is 1 for a bounded feature, 2 for a core journey, and 3 for data reused across teams. Risk score = severity weight × impact scope × recurrence factor; recurrence is 1 for rare, 2 for intermittent, and 3 for every affected action.

A gate need not reject every issue; that encourages hiding defects. Block when an open defect changes a primary KPI, double-counts conversion, breaks legally required consent, or makes identity resolution unsafe. A warning may proceed only with documented temporary treatment, a named owner, and a retest date. The sheet makes this visible.

The score prioritizes rather than replaces judgment. A low-frequency payment event can still block release because it affects revenue recognition. A high-volume UI diagnostic event can accept post-launch naming cleanup if it has no decision consumer. Record the rationale rather than pretending the formula decides every case.

Test payloads at system boundaries

Unit tests can show an application calls its tracking library, not that a destination receives the expected schema, a transformation preserves types, or warehouse rows support analysis. Test where context is lost: client to collector, collector to destination, identity merge, destination to warehouse, and warehouse to dashboard.

Adapt these generic SQL queries to warehouse field names and JSON functions. Run them on controlled pre-production data, then during a short production window after release. A nonzero result is not always defective, but each needs an explanation in the audit sheet.

-- Required-property failures in a chosen release window
SELECT event_name, COUNT(*) AS invalid_events
FROM raw_events
WHERE event_name = 'plan_selected'
  AND (properties.plan_id IS NULL OR properties.billing_interval NOT IN ('monthly', 'annual'))
GROUP BY event_name;

-- Potential duplicate logical conversions; inspect retries before deleting anything
SELECT order_id, COUNT(*) AS event_count
FROM raw_events
WHERE event_name = 'order_completed'
GROUP BY order_id
HAVING COUNT(*) > 1;

-- Identity merge check: anonymous activity with no later signed-in relationship
SELECT e.anonymous_id, COUNT(*) AS pre_login_events
FROM raw_events e
LEFT JOIN identity_map m ON e.anonymous_id = m.anonymous_id
WHERE e.user_id IS NULL
  AND e.anonymous_id IS NOT NULL
GROUP BY e.anonymous_id
HAVING MAX(m.user_id) IS NULL;

-- Timezone and late-arrival checks
SELECT
  COUNTIF(EXTRACT(TIMEZONE FROM occurred_at) != 0) AS non_utc_events,
  COUNTIF(received_at > occurred_at + INTERVAL 24 HOUR) AS late_events
FROM raw_events
WHERE occurred_at IS NOT NULL AND received_at IS NOT NULL;

Treat duplicate queries carefully. Network retries, SDK retries, and event replay can create repeated transport records. The deduplication key may be event_id, order_id, or a compound key such as account, action, and idempotency token. Do not collapse repeats solely by timestamp proximity: two valid purchases can occur close together.

Timezone defects also resist quick patches: client clocks can be wrong, daylight-saving rules applied twice, or events backfilled from older systems. Preserve raw timestamps when possible and create a documented normalized analysis field. Rewriting historical source time without evidence makes diagnosis harder.

Late arrival needs a reporting rule. A daily funnel can reopen prior days as delayed events arrive, or freeze a cutoff and track late records separately. Both are defensible; trouble starts when dashboards use one rule and experiment analysis another without telling readers.

Release only with accountable fixes

The audit ends in a decision, not screenshots. Assign one person to certify the taxonomy change, one to approve analytic meaning, and an owner for every open remediation. An engineering owner is not automatically accountable for the activation definition, and an analyst should not repair a mobile SDK queue.

Use this procedure for every tracked change:

• Freeze the event contract for the release candidate, including versioned property enums and identity rules.
• Send controlled test actions through each platform; retain event IDs, raw payloads, and query results as evidence.
• Review findings against block or warn gates, and record the release decision and approving roles.
• Create dated remediation for accepted warnings, with the exact failing rule and retest condition.
• Re-run payload and warehouse checks after deployment, because production consent settings, routing, and queues can differ from staging.

Do not gate only technical delivery: valid JSON can still violate business semantics. Nor should governance debate every optional property and become a bottleneck. Be strict where decisions depend on the contract and lighter where data is exploratory.

Versioning keeps this workable. Add a schema version or release label to materially changed events, retain prior definitions for a stated transition period, and annotate dashboard breaks. Renaming activation_completed to mean a different product state without a version creates a false trend line no retrospective can fully repair.

A release gate creates trustworthy evidence

A tracking plan earns trust when claims can be checked against payloads, identities, timestamps, and a named decision owner. Start with one important journey, score defects by decision damage, and block only failures that would mislead a real decision. An initial audit sheet may expose uncomfortable gaps; that is useful evidence caught before a dashboard turns it into false certainty.