SQL Reconciliation Checks: Catch Missing, Duplicate and Mismatched Records Before Reporting
A successful refresh tells you that a process finished. Reconciliation tells you whether the reporting data agrees with the source under the rules you actually care about.
13 October 2026 · UK business reporting · SQL Server / Azure SQL Database
- Scope the batchSame cut-off, same grain, same currency.
- Classify the differencesDuplicates, absent keys and changed values.
- Release with evidenceHold this illustrative batch for investigation.
Why this matters
“Five rows loaded” is not the same as “five correct records”
A missing invoice can disappear behind a duplicate. An incorrect amount can hide inside a plausible total. Build separate controls for separate failure modes.
Duplicate business keys
A repeated load can create two rows for one invoice. Joining before checking uniqueness can multiply records and make the comparison itself unreliable.
Different populations
Different extraction times, filters or status rules create false alarms. A source snapshot and a reporting batch must represent the same agreed population.
Values that drift
A transformation may change the amount, currency or business date while leaving the row count unchanged. Compare the fields that determine the report's meaning.
Define the comparison
Write the contract before writing the join
For this tutorial, each record is one invoice in a completed GBP batch. InvoiceID is unique across that population. All example data is invented.
Agree the business definition
- Grain: one row per invoice, not per invoice line.
- Key: InvoiceID; use a composite key in production if IDs repeat across companies.
- Value: GBP net amount with two decimal places; no tolerance in this example.
- Null rule: a missing amount is invalid, never silently zero.
- Scope: the same completed batch on both sides.
Make the inputs comparable
Use aligned, immutable extracts identified by batch ID or watermark. Reading two changing live tables at different times is not a stable baseline.
For date windows, use the same time zone and an inclusive start with an exclusive end. Document cancellations, credits, late arrivals and any expected transformation before calling a difference an error.
If the target intentionally aggregates invoice lines, transform the source to the same agreed grain first and reconcile that transformation separately.
Hands-on build
Run a complete reconciliation in one SQL batch
Paste the entire block into one query window in SQL Server or Azure SQL Database. It uses local table variables and does not modify business tables. Run it as one batch; do not insert GO between its sections.
The example deliberately contains a duplicated invoice, two missing invoices, an unexpected invoice and an amount mismatch. Both inputs have five rows. The code first isolates duplicate keys, then compares the remaining one-to-one records.
SET NOCOUNT ON;
-- 1. Stage the illustrative completed batch.
DECLARE @Source TABLE (
InvoiceID int NOT NULL,
Amount decimal(19, 2) NULL
);
DECLARE @Target TABLE (
InvoiceID int NOT NULL,
Amount decimal(19, 2) NULL
);
INSERT INTO @Source (InvoiceID, Amount)
VALUES (101, 100.00), (102, 200.00), (103, 120.00),
(104, 80.00), (105, 50.00);
INSERT INTO @Target (InvoiceID, Amount)
VALUES (101, 100.00), (102, 200.00), (102, 200.00),
(103, 125.00), (106, 50.00);
-- 2. Baseline: equal counts do not prove equality.
SELECT 'Source' AS Side,
COUNT(*) AS RowCount,
COUNT(DISTINCT InvoiceID) AS UniqueKeys,
SUM(Amount) AS TotalAmount
FROM @Source
UNION ALL
SELECT 'Target', COUNT(*), COUNT(DISTINCT InvoiceID),
SUM(Amount)
FROM @Target;
-- 3. Quarantine keys duplicated on either side.
DECLARE @DuplicateKeys TABLE (
InvoiceID int NOT NULL PRIMARY KEY
);
INSERT INTO @DuplicateKeys (InvoiceID)
SELECT InvoiceID
FROM @Source
GROUP BY InvoiceID
HAVING COUNT(*) > 1
UNION
SELECT InvoiceID
FROM @Target
GROUP BY InvoiceID
HAVING COUNT(*) > 1;
DECLARE @Result TABLE (
InvoiceID int NOT NULL PRIMARY KEY,
Status varchar(30) NOT NULL,
SourceAmount decimal(19, 2) NULL,
TargetAmount decimal(19, 2) NULL
);
-- 4. Compare only keys that are unique on both sides.
INSERT INTO @Result (
InvoiceID, Status, SourceAmount, TargetAmount
)
SELECT
COALESCE(s.InvoiceID, t.InvoiceID),
CASE
WHEN s.InvoiceID IS NULL THEN 'Unexpected in target'
WHEN t.InvoiceID IS NULL THEN 'Missing from target'
WHEN s.Amount IS NULL OR t.Amount IS NULL
THEN 'Invalid amount'
WHEN EXISTS (
SELECT s.Amount
EXCEPT
SELECT t.Amount
) THEN 'Amount mismatch'
ELSE 'Matched'
END,
s.Amount,
t.Amount
FROM @Source AS s
FULL OUTER JOIN @Target AS t
ON t.InvoiceID = s.InvoiceID
WHERE NOT EXISTS (
SELECT 1
FROM @DuplicateKeys AS d
WHERE d.InvoiceID = COALESCE(s.InvoiceID, t.InvoiceID)
);
-- A duplicated key receives one blocking result.
-- Its amounts are deliberately not collapsed or summed.
INSERT INTO @Result (
InvoiceID, Status, SourceAmount, TargetAmount
)
SELECT InvoiceID, 'Duplicate key', NULL, NULL
FROM @DuplicateKeys;
-- 5. Detailed evidence, then a status summary.
SELECT InvoiceID, Status, SourceAmount, TargetAmount
FROM @Result
ORDER BY InvoiceID;
SELECT Status, COUNT(*) AS KeyCount
FROM @Result
GROUP BY Status
ORDER BY Status;
-- 6. Fail closed for exceptions or an empty population.
-- Replace the empty-batch rule only with an agreed policy.
SELECT CASE
WHEN NOT EXISTS (SELECT 1 FROM @Source)
OR NOT EXISTS (SELECT 1 FROM @Target)
THEN 'HOLD: empty population'
WHEN EXISTS (
SELECT 1 FROM @Result WHERE Status <> 'Matched'
) THEN 'HOLD: investigate exceptions'
ELSE 'PASS: defined checks passed'
END AS ReleaseDecision;
Why isolate duplicate keys first?
If both tables contain repeated keys, a join can generate several comparisons for one business entity. The quarantine makes the remaining comparison one-to-one. It also avoids hiding a defect with an arbitrary first row or a blanket DISTINCT.
Why handle nulls explicitly?
The tutorial's business rule requires an amount, so either missing amount blocks the record before value comparison. Two null amounts would be equal under EXCEPT's set comparison, but they are still invalid under this contract.
Read the evidence
One matched key. Five keys to investigate.
The result contains six business keys across the two inputs. A duplicate key takes priority over field-level classification; fix it and rerun to expose any remaining difference for that key.
| InvoiceID | Status | Source | Target | Next action |
|---|---|---|---|---|
| 101 | Matched | 100.00 | 100.00 | No exception for the checked fields. |
| 102 | Duplicate key | Not compared | Not compared | Investigate repeated target loading. |
| 103 | Amount mismatch | 120.00 | 125.00 | Trace the £5 transformation difference. |
| 104 | Missing from target | 80.00 | Absent | Check source selection and load exceptions. |
| 105 | Missing from target | 50.00 | Absent | Check cut-off and rejected records. |
| 106 | Unexpected in target | Absent | 50.00 | Check stale data or incorrect batch scope. |
From tutorial to production
Make the result control the reporting release
The example returns a decision string. It does not stop a scheduler, refresh or publication by itself. Your orchestration must enforce that decision.
Stage and validate
Load both completed populations into controlled staging tables. Validate non-null keys, required fields, types and extract completion. Keep incomplete or failed extracts out of the normal comparison path.
Do not interpret “the query failed” as “zero exceptions”. Record execution failure as a hold.
Persist the evidence
Store RunID, source and target batch IDs, rule version, start/end timestamps, counts, exception keys and final status. Record the currency and comparison grain alongside the run.
Keep access and retention appropriate to the business data. Large batches need indexed staging and query-plan review.
Gate and resolve
Have the pipeline branch on the saved result or fail through a controlled error path. Publish only the batch that passed; retain the last approved report with an explicit freshness label when a new batch is held.
Fix the cause, reload where needed and rerun. An approved exception needs an owner, reason and expiry.
| Situation | Suitable approach | Avoid |
|---|---|---|
| Same-grain replication | Unique-key, presence and exact field checks. | Count-only approval. |
| Aggregated reporting target | Aggregate source to the agreed grain; check upstream detail separately. | Joining invoice headers directly to invoice lines. |
| Legitimate rounding differences | Document per-record tolerance and separate aggregate limits. | Increasing tolerance until every exception disappears. |
| Multiple currencies | Reconcile per currency or compare an explicitly governed conversion. | Adding GBP and EUR into one unexplained total. |
| Late-arriving data | Use a documented completion watermark and rerun policy. | Comparing a finished target to a moving source. |
Ownership and business impact
An exception needs a person and a decision
A reconciliation dashboard becomes useful when its results change what happens next. Assign responsibility before the first failed run.
Business owner
Approves the grain, material fields, cut-off, tolerance and release policy. Decides whether an exception can be accepted and documents its business impact.
Data engineering owner
Maintains staging, comparison logic, pipeline gates and regression cases. Investigates extraction and transformation failures and retains run evidence.
Reporting owner
Communicates freshness and held releases. Tracks correction effort, recurring defects and time to resolution, then checks whether changes reduce reporting rework.
Measure blocked batches, exception age, repeat causes, resolution time and post-release corrections. Define the denominator for any exception rate: affected business keys and raw duplicate rows are different measures.
Readiness assessment
Would your reporting release survive these checks?
Tick the controls you have implemented. This local checklist does not submit or store data, and its score is a planning aid rather than an audit opinion.
Test before release
Prove the failure paths, not just the happy path
Data cases
Test a perfect match, a source-only key, a target-only key, duplicates on either side, a null amount, a changed amount, equal totals with offsetting errors and an empty batch. For a composite key, test the same invoice number in two companies.
Operational cases
Test incomplete extracts, different batch IDs, comparison failure, late arrivals and an unsuccessful rerun. Check that the release stays held, the responsible owner receives the result and the previous approved report remains clearly dated.
Frequently asked questions
SQL reconciliation checks FAQs
Are equal row counts enough to prove that two tables match?
No. A missing record and a duplicate record can cancel out in the count. Compare key uniqueness, record presence and the business fields as separate checks.
Why not use EXCEPT as the only reconciliation check?
EXCEPT compares distinct rows, so it does not preserve duplicate multiplicity. Use a duplicate-key check as well, and compare both directions when you use EXCEPT to detect missing rows.
Should a missing amount be treated as zero?
Only when the business definition explicitly says so. This tutorial treats a missing amount as invalid, because an unknown value and a confirmed zero have different meanings.
Can this script be used unchanged in every SQL platform?
No. The runnable example targets SQL Server and Azure SQL Database. Other engines may require different staging, variable and exception-handling syntax.
Does a PASS result prove the report is correct?
No. PASS means that the defined checks passed for the selected batch and fields. It does not prove that the original business data, transformation rules or report measures are correct.
Technical references
Microsoft documentation behind the SQL
Checked on 13 October 2026. The reconciliation workflow is an implementation pattern; all sample records, totals and dashboard figures are illustrative.
- COUNT (Transact-SQL) — row counts, distinct counts and null handling.
- EXCEPT and INTERSECT (Transact-SQL) — distinct comparisons, compatible data types and null equality.
- decimal and numeric (Transact-SQL) — precision, scale and conversion behaviour.