Contact

GlossaryPublic records, privacy, and data quality

Deduplication

Understand deduplication, why repeated incident numbers are not always errors, and how matching rules affect public-safety counts.

Deduplication is the process of identifying repeated records and deciding how to handle them so the same item is not counted more than intended. In public-safety data, that decision depends on what a row represents: an incident, an offense, a responding unit, or another part of an event.

Start with what is being counted

Two rows can share an incident number without being duplicates. An incident may involve two units, and a unit-level file may correctly contain one row for each response. Counting those rows answers a different question from counting distinct incidents.

The government's data-quality framework treats uniqueness as a distinct quality concern. Applying that principle requires defining the item that should appear only once. UK Government Data Quality Framework.

For a public count, a useful method states which fields establish a match and which record is retained when versions differ. An agency identifier may be needed alongside an incident number because two departments can use the same number for unrelated events.

Example

Hypothetical: A file contains three rows:

  • Incident A, unit 1.
  • Incident A, unit 2.
  • Incident A, unit 2, imported a second time by mistake.

Removing the repeated import leaves two legitimate unit-response rows. Counting distinct incident identifiers produces one incident. Both results can be correct, but they measure different things. Removing every row with a repeated incident number would discard a real unit response.

Common confusion

Matching text is not always enough to establish a duplicate. Two separate calls may occur at the same address within a short time. Conversely, two versions of the same report may use slightly different text or corrected coordinates.

Deduplication also differs from aggregation. Deduplication handles unintended repetition; aggregation summarizes the records that remain. A summary can still be inflated if its input contains duplicate imports.

What a documented method should explain

Look for the intended counting unit, the matching fields, and the treatment of updates. When one row is an older version of another, the method should explain how the current version is selected. When a match is uncertain, treating the records as unquestionably identical can erase real activity.

If a published total falls after cleanup, the explanation should identify duplicate removal as the reason. A reduction in repeated records is a change in the data, not evidence that fewer events occurred in the original period.

Sources