Streamlining your tech stack for maximum efficiency

Learn, explore, and grow with our knowledge hub.

SQL Data Quality Checks: Build a Power BI Refresh Gate
SMART STATISTICS · HANDS-ON SQL TUTORIAL

SQL data quality checks: build a pre-refresh gate for Power BI

A successful refresh only proves that data moved. Build a reusable T-SQL control that tests each batch, preserves evidence and prevents a known critical failure from becoming tomorrow’s dashboard.

1 October 2026 · For UK data, finance, operations and reporting teams

ILLUSTRATIVE EXAMPLE · ALL DASHBOARD FIGURES

Pre-refresh quality gate

48,216Batch rows
8Rules executed
7Rules passed
1Blocking failure
STAGED
DATA
→
SQL
GATE
→
REFRESH
HELD
Freshness
0
Completeness
0
Uniqueness
0
Control totals
1

Decision: hold refresh and assign the totals exception to Finance Data Operations.

Illustrative example variance: £12,480.00.

Invented batch, rule counts and variance for teaching only; not client evidence or a guaranteed outcome.

A green refresh icon is not a data-quality certificate

A pipeline can load every row it receives while those rows are late, duplicated, incomplete, orphaned or financially inconsistent. The gate must test the business contract before the reporting model is released.

Technically complete, commercially wrong

The load finishes without an SQL error, but a source extract is missing a region or day. The dashboard publishes a plausible partial result.

Rules live only in people’s heads

An analyst notices unusual totals manually. When that person is absent, the same warning signs do not delay the refresh.

Every defect becomes an outage

Without severity levels, even advisory warnings stop delivery—or critical failures are ignored to keep the report on time.

Design principle: test the smallest trusted release unit—usually a batch, business date or partition—and make the downstream refresh depend on a recorded pass, not on somebody remembering to inspect a report.

Define the data contract before writing SQL

Every rule needs a business reason, threshold, severity, accountable owner and response. “Looks unusual” is not executable acceptance logic.

Example rule catalogue—adapt thresholds to the real process
RuleQuestionExample thresholdOwnerSeverity
FreshnessDid the expected batch arrive recently enough?Extract age no more than 180 minutes.Source-system ownerBlocking
CompletenessAre mandatory business fields populated?No missing invoice, customer or transaction date.Process ownerBlocking
UniquenessIs the declared transaction key unique?Zero excess rows per invoice number.Data-product ownerBlocking
Referential integrityDoes every customer exist in the governed master?Zero orphan customer keys.Master-data ownerBlocking
Control totalsDoes loaded value equal the signed source control?Absolute variance no more than £0.01.Finance data ownerBlocking
ReasonablenessIs movement outside an agreed operating range?Flag, but do not stop, change above 35%.Reporting ownerWarning

Create persistent evidence tables

Store one row for each gate run and one row for each rule result. This separates operational evidence from the source table being tested.

STEP 1

Create the run and result tables

The following T-SQL targets SQL Server or Azure SQL Database. Review data types, schemas, permissions, indexes and platform compatibility before production use.

CREATE TABLE dbo.QualityRun
(
    RunId                bigint IDENTITY(1,1) PRIMARY KEY,
    BatchId              uniqueidentifier NOT NULL,
    StartedAtUtc         datetime2(0) NOT NULL
        CONSTRAINT DF_QualityRun_StartedAtUtc DEFAULT SYSUTCDATETIME(),
    CompletedAtUtc       datetime2(0) NULL,
    GateStatus           varchar(12) NOT NULL,
    BlockingFailureCount int NOT NULL CONSTRAINT DF_QualityRun_Failures DEFAULT 0,
    ErrorMessage         nvarchar(2048) NULL
);

CREATE TABLE dbo.QualityResult
(
    ResultId       bigint IDENTITY(1,1) PRIMARY KEY,
    RunId          bigint NOT NULL,
    RuleCode       varchar(50) NOT NULL,
    RuleName       nvarchar(200) NOT NULL,
    Severity       varchar(10) NOT NULL,
    Passed         bit NOT NULL,
    FailedRowCount bigint NOT NULL,
    ObservedValue  nvarchar(200) NULL,
    TestedAtUtc    datetime2(0) NOT NULL
        CONSTRAINT DF_QualityResult_TestedAtUtc DEFAULT SYSUTCDATETIME(),
    CONSTRAINT FK_QualityResult_QualityRun
        FOREIGN KEY (RunId) REFERENCES dbo.QualityRun(RunId)
);

CREATE INDEX IX_QualityResult_RunId
    ON dbo.QualityResult(RunId, Severity, Passed);

Give each batch a control record

The gate needs an independent expectation. Otherwise, comparing a loaded table with itself can confirm an incomplete batch.

STEP 2

Capture source-supplied control information

CREATE TABLE dbo.SalesBatchControl
(
    BatchId          uniqueidentifier PRIMARY KEY,
    ExtractedAtUtc   datetime2(0) NOT NULL,
    ExpectedRowCount bigint NOT NULL,
    ExpectedAmount   decimal(19,4) NOT NULL,
    ApprovedBy       nvarchar(150) NULL
);

-- Simplified staging structure used by the tutorial.
CREATE TABLE dbo.SalesStage
(
    BatchId       uniqueidentifier NOT NULL,
    InvoiceNumber nvarchar(50) NULL,
    CustomerId    int NULL,
    InvoiceDate   date NULL,
    NetAmount     decimal(19,4) NULL,
    LoadedAtUtc   datetime2(0) NOT NULL
        CONSTRAINT DF_SalesStage_LoadedAtUtc DEFAULT SYSUTCDATETIME()
);

CREATE INDEX IX_SalesStage_BatchInvoice
    ON dbo.SalesStage(BatchId, InvoiceNumber);

Populate SalesBatchControl from a trusted manifest, source-system control table or independently signed extract—not from an aggregation of SalesStage.

The later referential-integrity rule assumes an existing governed dbo.Customer master table with a unique CustomerId.

Execute the rules and persist every result

The procedure below calculates five blocking checks, stores their results and raises an exception when any blocking rule fails.

STEP 3

Create the quality-gate procedure

CREATE OR ALTER PROCEDURE dbo.usp_RunSalesQualityGate
    @BatchId uniqueidentifier,
    @MaxAgeMinutes int = 180,
    @AmountTolerance decimal(19,4) = 0.01
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    DECLARE
        @RunId bigint,
        @ExpectedRows bigint,
        @ActualRows bigint,
        @ExpectedAmount decimal(19,4),
        @ActualAmount decimal(19,4),
        @ExtractedAtUtc datetime2(0),
        @MissingMandatory bigint,
        @DuplicateRows bigint,
        @OrphanCustomers bigint,
        @BlockingFailures int;

    DECLARE @Results TABLE
    (
        RuleCode varchar(50),
        RuleName nvarchar(200),
        Severity varchar(10),
        Passed bit,
        FailedRowCount bigint,
        ObservedValue nvarchar(200)
    );

    BEGIN TRY
        INSERT dbo.QualityRun (BatchId, GateStatus)
        VALUES (@BatchId, 'Running');

        SET @RunId = SCOPE_IDENTITY();

        SELECT
            @ExpectedRows = ExpectedRowCount,
            @ExpectedAmount = ExpectedAmount,
            @ExtractedAtUtc = ExtractedAtUtc
        FROM dbo.SalesBatchControl
        WHERE BatchId = @BatchId;

        IF @ExpectedRows IS NULL
            THROW 51000, 'No trusted control record exists for this batch.', 1;

        SELECT
            @ActualRows = COUNT_BIG(*),
            @ActualAmount = COALESCE(SUM(NetAmount), 0)
        FROM dbo.SalesStage
        WHERE BatchId = @BatchId;

        SELECT @MissingMandatory = COUNT_BIG(*)
        FROM dbo.SalesStage
        WHERE BatchId = @BatchId
          AND (
              NULLIF(LTRIM(RTRIM(InvoiceNumber)), '') IS NULL
              OR CustomerId IS NULL
              OR InvoiceDate IS NULL
              OR NetAmount IS NULL
          );

        SELECT @DuplicateRows = COALESCE(SUM(ExcessRows), 0)
        FROM
        (
            SELECT COUNT_BIG(*) - 1 AS ExcessRows
            FROM dbo.SalesStage
            WHERE BatchId = @BatchId
              AND InvoiceNumber IS NOT NULL
            GROUP BY InvoiceNumber
            HAVING COUNT_BIG(*) > 1
        ) AS Duplicates;

        SELECT @OrphanCustomers = COUNT_BIG(*)
        FROM dbo.SalesStage AS S
        LEFT JOIN dbo.Customer AS C
          ON C.CustomerId = S.CustomerId
        WHERE S.BatchId = @BatchId
          AND S.CustomerId IS NOT NULL
          AND C.CustomerId IS NULL;

        INSERT @Results VALUES
        (
            'FRESHNESS', 'Source extract is within the agreed age', 'Blocking',
            IIF(
                DATEDIFF(minute, @ExtractedAtUtc, SYSUTCDATETIME())
                    BETWEEN 0 AND @MaxAgeMinutes,
                1, 0
            ),
            IIF(
                DATEDIFF(minute, @ExtractedAtUtc, SYSUTCDATETIME())
                    BETWEEN 0 AND @MaxAgeMinutes,
                0, 1
            ),
            CONCAT(DATEDIFF(minute, @ExtractedAtUtc, SYSUTCDATETIME()), ' minutes old')
        ),
        (
            'COMPLETENESS', 'Mandatory sales fields are populated', 'Blocking',
            IIF(@MissingMandatory = 0, 1, 0), @MissingMandatory,
            CONCAT(@MissingMandatory, ' incomplete rows')
        ),
        (
            'UNIQUENESS', 'Invoice number is unique within the batch', 'Blocking',
            IIF(@DuplicateRows = 0, 1, 0), @DuplicateRows,
            CONCAT(@DuplicateRows, ' excess duplicate rows')
        ),
        (
            'CUSTOMER_KEY', 'Every customer exists in the master table', 'Blocking',
            IIF(@OrphanCustomers = 0, 1, 0), @OrphanCustomers,
            CONCAT(@OrphanCustomers, ' orphan customer rows')
        ),
        (
            'CONTROL_TOTAL', 'Row count and amount agree to source controls', 'Blocking',
            IIF(
                @ActualRows = @ExpectedRows
                AND ABS(@ActualAmount - @ExpectedAmount) <= @AmountTolerance,
                1, 0
            ),
            IIF(@ActualRows = @ExpectedRows, 0, ABS(@ActualRows - @ExpectedRows)),
            CONCAT(
                'Rows ', @ActualRows, '/', @ExpectedRows,
                '; amount variance ',
                CONVERT(varchar(50), @ActualAmount - @ExpectedAmount)
            )
        );

        INSERT dbo.QualityResult
            (RunId, RuleCode, RuleName, Severity, Passed, FailedRowCount, ObservedValue)
        SELECT
            @RunId, RuleCode, RuleName, Severity, Passed, FailedRowCount, ObservedValue
        FROM @Results;

        SELECT @BlockingFailures = COUNT(*)
        FROM @Results
        WHERE Severity = 'Blocking'
          AND Passed = 0;

        UPDATE dbo.QualityRun
        SET CompletedAtUtc = SYSUTCDATETIME(),
            GateStatus = IIF(@BlockingFailures = 0, 'Passed', 'Blocked'),
            BlockingFailureCount = @BlockingFailures
        WHERE RunId = @RunId;

        IF @BlockingFailures > 0
            THROW 51001, 'Data quality gate blocked the reporting refresh.', 1;

        SELECT RunId, BatchId, GateStatus, BlockingFailureCount
        FROM dbo.QualityRun
        WHERE RunId = @RunId;
    END TRY
    BEGIN CATCH
        IF @RunId IS NOT NULL
        BEGIN
            UPDATE dbo.QualityRun
            SET CompletedAtUtc = COALESCE(CompletedAtUtc, SYSUTCDATETIME()),
                GateStatus = CASE WHEN GateStatus = 'Blocked' THEN GateStatus ELSE 'Error' END,
                ErrorMessage = ERROR_MESSAGE()
            WHERE RunId = @RunId;
        END;

        THROW;
    END CATCH;
END;
Why the exception matters: Microsoft documents that THROW raises an exception and transfers execution to a CATCH block when used inside TRY...CATCH. Your orchestrator can therefore treat a blocked gate as a failed activity while the result tables retain the business evidence.

Test the gate with a known bad batch

Do not wait for production failure. Prove that each rule passes clean data and blocks its intended defect without suppressing unrelated warnings.

STEP 4

Run the procedure

DECLARE @BatchId uniqueidentifier =
    'A8D9F557-7175-4D4A-96B6-19D71D097CF4';

EXEC dbo.usp_RunSalesQualityGate
    @BatchId = @BatchId,
    @MaxAgeMinutes = 180,
    @AmountTolerance = 0.01;

Test separate fixtures for a late extract, missing invoice number, duplicate key, orphan customer and total mismatch. Include a clean control batch.

STEP 5

Read the evidence

SELECT
    R.RunId,
    R.BatchId,
    R.GateStatus,
    R.StartedAtUtc,
    Q.RuleCode,
    Q.Severity,
    Q.Passed,
    Q.FailedRowCount,
    Q.ObservedValue
FROM dbo.QualityRun AS R
JOIN dbo.QualityResult AS Q
  ON Q.RunId = R.RunId
WHERE R.BatchId =
    'A8D9F557-7175-4D4A-96B6-19D71D097CF4'
ORDER BY R.RunId DESC, Q.ResultId;

Expose this history to operational monitoring, but restrict sensitive row-level exceptions to authorised investigators.

Put the gate before the Power BI refresh

The workflow should have an explicit success route, a controlled failure route and no path that refreshes the semantic model after an unresolved blocking result.

Run and evaluate

After staging completes, execute the quality procedure for the exact batch or partition intended for release. Pass the batch identifier explicitly.

Block and notify

On SQL failure, stop the release path. Notify the named owner with batch, rule, observed value, evidence link and agreed response deadline.

Refresh only on success

On gate success, invoke the semantic-model refresh. In Fabric Data Factory, the semantic model refresh activity can form the next success-dependent activity; other orchestrators can use the Power BI REST API with approved authentication and permissions.

Separate warnings, blockers and overrides

Not every unusual value is wrong. Severity should reflect the decision risk and whether the organisation can safely publish while an issue is investigated.

Recommended response model
ClassEffectTypical useEvidence
BlockingRefresh does not start.Missing batch, duplicate transaction keys, failed financial controls, unauthorised scope.Rule result, affected count or value, owner and resolution.
WarningRefresh may proceed while an alert is raised.Unusual but plausible movement, optional attributes, emerging trend breaches.Observed value, reviewer, monitoring window and escalation point.
OverrideNamed authority releases a blocked batch exceptionally.Time-critical reporting where impact is understood and disclosed.Approver, reason, expiry, affected reports, user communication and remediation ticket.
RetiredRule no longer executes.Source or process changed and the control is no longer relevant.Decision date, owner, replacement control and retained history.
Never implement an invisible override. If the business can release a failed batch, make that a governed action with a named approver, clear expiry and visible caveat—not an engineer editing the procedure at 7 am.

Assign ownership across the full control

The SQL developer owns implementation quality, but the business must own what “acceptable data” means.

BUSINESS

Data owner

Approves critical fields, totals, thresholds, severity and exceptional release decisions.

SOURCE

System owner

Explains missing or late extracts, fixes upstream defects and confirms restoration.

DATA

Product owner

Owns the gate, evidence model, release workflow, regression tests and support boundary.

REPORTING

Report owner

Understands affected measures, communicates impact and confirms whether users need a caveat.

Choose the right control location

Place the rule where it can prevent the defect most reliably and where an accountable team can maintain it.

Practical decision criteria
SituationBest starting pointReason
Invalid entry can be preventedSource application or database constraint.Stop the defect closest to creation rather than repeatedly detecting it downstream.
Several reports reuse the same curated tableSQL or governed platform layer.One shared gate protects every downstream consumer and retains central evidence.
Rule is specific to one semantic definitionTransformation or semantic-model test.Keep report-specific acceptance logic with the product it protects.
Small analyst-owned prototypePower Query profiling plus documented manual review.Prove value before building operational infrastructure, while retaining release checks.
Real-time or high-volume processingStreaming or pipeline-native validation.A batch stored procedure may not meet latency, scale or quarantine requirements.

Measure whether the gate improves reporting

A quality gate creates value when it reduces user exposure and speeds resolution—not simply because more rules exist.

Control measures

Track pass rate, blocking failures by rule, warning rate, aged failures, overrides, repeat failures, test execution time, false positives and batches without an independent control record.

Segment by source and business criticality so a high pass rate in low-risk feeds does not hide instability in an executive dataset.

Business measures

Track incorrect reports prevented, time from detection to resolution, refresh delay, post-publication corrections, analyst investigation time and users affected by disclosed defects.

Compare the cost of controlled delay with the impact of publishing a known wrong number.

Illustrative impact example: if a weekly reporting error previously reached 40 managers and required six hours to diagnose and republish, blocking it before refresh may avoid that user exposure and rework. Measure actual incidents and time; this is not a client result or guaranteed saving.

Is your pre-refresh gate production-ready?

Select only controls supported by current evidence. Any missing critical item keeps the result at high priority.

Frequently asked questions

Should every data-quality failure stop Power BI refresh?

No. Block failures that make the intended decision unsafe or breach an agreed data contract. Route unusual but usable values as warnings, with an owner and escalation threshold.

Why store quality results instead of only throwing an SQL error?

An exception controls workflow, but persistent results explain which rule failed, what was observed and whether the problem repeats. That evidence supports investigation, governance and improvement.

Can this pattern be used with Microsoft Fabric?

The pattern can be adapted to Fabric, but confirm each T-SQL feature against the current endpoint and use the platform’s supported orchestration activities. Fabric Data Factory includes a semantic model refresh activity for pipeline-driven refresh.

What happens when the quality gate itself errors?

Treat an unhandled gate error as a blocked release, retain the error message, alert the technical owner and do not assume the data passed. A control that did not complete cannot provide positive assurance.

Stop unreliable data before it becomes a trusted dashboard

Smart Statistics helps UK organisations design practical data-quality controls, governed reporting pipelines and Power BI solutions that make failures visible before users act on them.

Sources and publication notes

Microsoft documentation checked on 1 October 2026: THROW, TRY...CATCH, Power BI semantic-model refresh REST API, refresh history API and Fabric semantic model refresh activity.

The rule catalogue, table design, stored procedure, severity model, ownership model, decision guide, assessment and impact example are original Smart Statistics guidance. Validate the code, permissions, indexing, transaction design, platform support, source controls and business thresholds before production use.

Review draft: confirm canonical and social-image URLs before publication. In WordPress, place metadata and schema in the SEO layer and use an approved mechanism for CSS and JavaScript if the editor strips them.