Streamlining your tech stack for maximum efficiency

Learn, explore, and grow with our knowledge hub.

SQL Reconciliation Checks: A Practical Guide | Smart Statistics
Smart Statistics · Practical SQL tutorial

SQL Reconciliation Checks: Catch Missing, Duplicate and Mismatched Records Before Reporting

A successful refresh tells you that a process finished. Reconciliation tells you whether the reporting data agrees with the source under the rules you actually care about.

13 October 2026 · UK business reporting · SQL Server / Azure SQL Database

Reporting release control / invoice batch Illustrative example · HOLD
Source rows5Illustrative example
Reporting rows5Illustrative example
Reporting unique keys4Illustrative example
Target minus source+£125Illustrative example
Equal row counts conceal different totals Illustrative example. Source total is 550 pounds. Reporting total is 675 pounds. Both have five rows. Batch totals · Illustrative example Source £550 Reporting £675 · Equal counts are not a release decision.
  • Scope the batchSame cut-off, same grain, same currency.
  • Classify the differencesDuplicates, absent keys and changed values.
  • Release with evidenceHold this illustrative batch for investigation.

Why this matters

“Five rows loaded” is not the same as “five correct records”

A missing invoice can disappear behind a duplicate. An incorrect amount can hide inside a plausible total. Build separate controls for separate failure modes.

Duplicate business keys

A repeated load can create two rows for one invoice. Joining before checking uniqueness can multiply records and make the comparison itself unreliable.

Different populations

Different extraction times, filters or status rules create false alarms. A source snapshot and a reporting batch must represent the same agreed population.

Values that drift

A transformation may change the amount, currency or business date while leaving the row count unchanged. Compare the fields that determine the report's meaning.

Define the comparison

Write the contract before writing the join

For this tutorial, each record is one invoice in a completed GBP batch. InvoiceID is unique across that population. All example data is invented.

Agree the business definition

  • Grain: one row per invoice, not per invoice line.
  • Key: InvoiceID; use a composite key in production if IDs repeat across companies.
  • Value: GBP net amount with two decimal places; no tolerance in this example.
  • Null rule: a missing amount is invalid, never silently zero.
  • Scope: the same completed batch on both sides.

Make the inputs comparable

Use aligned, immutable extracts identified by batch ID or watermark. Reading two changing live tables at different times is not a stable baseline.

For date windows, use the same time zone and an inclusive start with an exclusive end. Document cancellations, credits, late arrivals and any expected transformation before calling a difference an error.

If the target intentionally aggregates invoice lines, transform the source to the same agreed grain first and reconcile that transformation separately.

Hands-on build

Run a complete reconciliation in one SQL batch

Paste the entire block into one query window in SQL Server or Azure SQL Database. It uses local table variables and does not modify business tables. Run it as one batch; do not insert GO between its sections.

The example deliberately contains a duplicated invoice, two missing invoices, an unexpected invoice and an amount mismatch. Both inputs have five rows. The code first isolates duplicate keys, then compares the remaining one-to-one records.

SET NOCOUNT ON;

-- 1. Stage the illustrative completed batch.
DECLARE @Source TABLE (
    InvoiceID int NOT NULL,
    Amount decimal(19, 2) NULL
);
DECLARE @Target TABLE (
    InvoiceID int NOT NULL,
    Amount decimal(19, 2) NULL
);

INSERT INTO @Source (InvoiceID, Amount)
VALUES (101, 100.00), (102, 200.00), (103, 120.00),
       (104, 80.00), (105, 50.00);

INSERT INTO @Target (InvoiceID, Amount)
VALUES (101, 100.00), (102, 200.00), (102, 200.00),
       (103, 125.00), (106, 50.00);

-- 2. Baseline: equal counts do not prove equality.
SELECT 'Source' AS Side,
       COUNT(*) AS RowCount,
       COUNT(DISTINCT InvoiceID) AS UniqueKeys,
       SUM(Amount) AS TotalAmount
FROM @Source
UNION ALL
SELECT 'Target', COUNT(*), COUNT(DISTINCT InvoiceID),
       SUM(Amount)
FROM @Target;

-- 3. Quarantine keys duplicated on either side.
DECLARE @DuplicateKeys TABLE (
    InvoiceID int NOT NULL PRIMARY KEY
);

INSERT INTO @DuplicateKeys (InvoiceID)
SELECT InvoiceID
FROM @Source
GROUP BY InvoiceID
HAVING COUNT(*) > 1
UNION
SELECT InvoiceID
FROM @Target
GROUP BY InvoiceID
HAVING COUNT(*) > 1;

DECLARE @Result TABLE (
    InvoiceID int NOT NULL PRIMARY KEY,
    Status varchar(30) NOT NULL,
    SourceAmount decimal(19, 2) NULL,
    TargetAmount decimal(19, 2) NULL
);

-- 4. Compare only keys that are unique on both sides.
INSERT INTO @Result (
    InvoiceID, Status, SourceAmount, TargetAmount
)
SELECT
    COALESCE(s.InvoiceID, t.InvoiceID),
    CASE
        WHEN s.InvoiceID IS NULL THEN 'Unexpected in target'
        WHEN t.InvoiceID IS NULL THEN 'Missing from target'
        WHEN s.Amount IS NULL OR t.Amount IS NULL
            THEN 'Invalid amount'
        WHEN EXISTS (
            SELECT s.Amount
            EXCEPT
            SELECT t.Amount
        ) THEN 'Amount mismatch'
        ELSE 'Matched'
    END,
    s.Amount,
    t.Amount
FROM @Source AS s
FULL OUTER JOIN @Target AS t
    ON t.InvoiceID = s.InvoiceID
WHERE NOT EXISTS (
    SELECT 1
    FROM @DuplicateKeys AS d
    WHERE d.InvoiceID = COALESCE(s.InvoiceID, t.InvoiceID)
);

-- A duplicated key receives one blocking result.
-- Its amounts are deliberately not collapsed or summed.
INSERT INTO @Result (
    InvoiceID, Status, SourceAmount, TargetAmount
)
SELECT InvoiceID, 'Duplicate key', NULL, NULL
FROM @DuplicateKeys;

-- 5. Detailed evidence, then a status summary.
SELECT InvoiceID, Status, SourceAmount, TargetAmount
FROM @Result
ORDER BY InvoiceID;

SELECT Status, COUNT(*) AS KeyCount
FROM @Result
GROUP BY Status
ORDER BY Status;

-- 6. Fail closed for exceptions or an empty population.
-- Replace the empty-batch rule only with an agreed policy.
SELECT CASE
    WHEN NOT EXISTS (SELECT 1 FROM @Source)
      OR NOT EXISTS (SELECT 1 FROM @Target)
        THEN 'HOLD: empty population'
    WHEN EXISTS (
        SELECT 1 FROM @Result WHERE Status <> 'Matched'
    ) THEN 'HOLD: investigate exceptions'
    ELSE 'PASS: defined checks passed'
END AS ReleaseDecision;

Why isolate duplicate keys first?

If both tables contain repeated keys, a join can generate several comparisons for one business entity. The quarantine makes the remaining comparison one-to-one. It also avoids hiding a defect with an arbitrary first row or a blanket DISTINCT.

Why handle nulls explicitly?

The tutorial's business rule requires an amount, so either missing amount blocks the record before value comparison. Two null amounts would be equal under EXCEPT's set comparison, but they are still invalid under this contract.

Read the evidence

One matched key. Five keys to investigate.

The result contains six business keys across the two inputs. A duplicate key takes priority over field-level classification; fix it and rerun to expose any remaining difference for that key.

Expected results · Illustrative example · GBP
InvoiceIDStatusSourceTargetNext action
101Matched100.00100.00No exception for the checked fields.
102Duplicate keyNot comparedNot comparedInvestigate repeated target loading.
103Amount mismatch120.00125.00Trace the £5 transformation difference.
104Missing from target80.00AbsentCheck source selection and load exceptions.
105Missing from target50.00AbsentCheck cut-off and rejected records.
106Unexpected in targetAbsent50.00Check stale data or incorrect batch scope.

From tutorial to production

Make the result control the reporting release

The example returns a decision string. It does not stop a scheduler, refresh or publication by itself. Your orchestration must enforce that decision.

Stage and validate

Load both completed populations into controlled staging tables. Validate non-null keys, required fields, types and extract completion. Keep incomplete or failed extracts out of the normal comparison path.

Do not interpret “the query failed” as “zero exceptions”. Record execution failure as a hold.

Persist the evidence

Store RunID, source and target batch IDs, rule version, start/end timestamps, counts, exception keys and final status. Record the currency and comparison grain alongside the run.

Keep access and retention appropriate to the business data. Large batches need indexed staging and query-plan review.

Gate and resolve

Have the pipeline branch on the saved result or fail through a controlled error path. Publish only the batch that passed; retain the last approved report with an explicit freshness label when a new batch is held.

Fix the cause, reload where needed and rerun. An approved exception needs an owner, reason and expiry.

Choose the rule that matches the business meaning
SituationSuitable approachAvoid
Same-grain replicationUnique-key, presence and exact field checks.Count-only approval.
Aggregated reporting targetAggregate source to the agreed grain; check upstream detail separately.Joining invoice headers directly to invoice lines.
Legitimate rounding differencesDocument per-record tolerance and separate aggregate limits.Increasing tolerance until every exception disappears.
Multiple currenciesReconcile per currency or compare an explicitly governed conversion.Adding GBP and EUR into one unexplained total.
Late-arriving dataUse a documented completion watermark and rerun policy.Comparing a finished target to a moving source.

Ownership and business impact

An exception needs a person and a decision

A reconciliation dashboard becomes useful when its results change what happens next. Assign responsibility before the first failed run.

Business owner

Approves the grain, material fields, cut-off, tolerance and release policy. Decides whether an exception can be accepted and documents its business impact.

Data engineering owner

Maintains staging, comparison logic, pipeline gates and regression cases. Investigates extraction and transformation failures and retains run evidence.

Reporting owner

Communicates freshness and held releases. Tracks correction effort, recurring defects and time to resolution, then checks whether changes reduce reporting rework.

Measure blocked batches, exception age, repeat causes, resolution time and post-release corrections. Define the denominator for any exception rate: affected business keys and raw duplicate rows are different measures.

Readiness assessment

Would your reporting release survive these checks?

Tick the controls you have implemented. This local checklist does not submit or store data, and its score is a planning aid rather than an audit opinion.

Implemented reconciliation controls

Test before release

Prove the failure paths, not just the happy path

Data cases

Test a perfect match, a source-only key, a target-only key, duplicates on either side, a null amount, a changed amount, equal totals with offsetting errors and an empty batch. For a composite key, test the same invoice number in two companies.

Operational cases

Test incomplete extracts, different batch IDs, comparison failure, late arrivals and an unsuccessful rerun. Check that the release stays held, the responsible owner receives the result and the previous approved report remains clearly dated.

Frequently asked questions

SQL reconciliation checks FAQs

Are equal row counts enough to prove that two tables match?

No. A missing record and a duplicate record can cancel out in the count. Compare key uniqueness, record presence and the business fields as separate checks.

Why not use EXCEPT as the only reconciliation check?

EXCEPT compares distinct rows, so it does not preserve duplicate multiplicity. Use a duplicate-key check as well, and compare both directions when you use EXCEPT to detect missing rows.

Should a missing amount be treated as zero?

Only when the business definition explicitly says so. This tutorial treats a missing amount as invalid, because an unknown value and a confirmed zero have different meanings.

Can this script be used unchanged in every SQL platform?

No. The runnable example targets SQL Server and Azure SQL Database. Other engines may require different staging, variable and exception-handling syntax.

Does a PASS result prove the report is correct?

No. PASS means that the defined checks passed for the selected batch and fields. It does not prove that the original business data, transformation rules or report measures are correct.

Technical references

Microsoft documentation behind the SQL

Checked on 13 October 2026. The reconciliation workflow is an implementation pattern; all sample records, totals and dashboard figures are illustrative.