The correct answer is C — @dlt.expect_or_fail.

Lakeflow Spark Declarative Pipelines (SDP) offers three expectation decorators, each with a different violation response:

@dlt.expect — logs the violation as a metric but writes all records, including bad ones, to the table. Suitable for monitoring only.

@dlt.expect_or_drop — drops violating records and continues the pipeline. The table receives only clean rows, but the pipeline update commits successfully.

@dlt.expect_or_fail — fails the entire pipeline update when a violation is detected. The table update is never committed. This is the correct choice when data integrity is non-negotiable: ' Prevent an update to Table1 from being committed when data that violates rule1 is detected. '

@dlt.expect_all_or_drop takes a dictionary of rules and drops violating rows but still commits — it doesn ' t halt the pipeline.

A foreign catalog mirrors the external ERP database and provides read-only access through Lakehouse Federation. Consequently, it cannot host managed volumes or locally created Delta tables, and its foreign tables cannot be altered to add analytics columns. The appropriate design is to create governed views in the managed analytics catalog and have those views reference the foreign tables through fully qualified catalog.schema.table names. This preserves the ERP data in its source system, avoids duplication, and exposes standardized curated objects through Unity Catalog. Permissions can then be granted on the managed views while access to underlying foreign objects remains controlled. Unity Catalog uses a three-level namespace, and foreign catalogs are specifically intended to make external database data queryable without copying it into Databricks-managed storage. Microsoft Learn

Two actions are needed to reclaim storage while keeping tables queryable:

Set delta.deletedFileRetentionDuration and delta.logRetentionDuration to 90 days on each table. These properties define the retention floor — VACUUM will not touch anything newer than this threshold, so no data needed for time travel within 90 days can be accidentally removed.

Run VACUUM on each table. VACUUM is the Delta Lake command that physically removes data files and transaction log entries older than the retention duration. Importantly, VACUUM runs as a background operation — it uses Delta Lake ' s MVCC (multi-version concurrency control) to ensure that concurrent reads against the table continue uninterrupted while cleanup happens. Tables are fully available throughout.

OPTIMIZE compacts small files for query performance but doesn ' t delete anything. Manually deleting files outside the Delta protocol would corrupt the table.