Essential session storage is required to sign in. Optional analytics remain off unless you allow them.
Build a production-shaped data platform. Requirements: (1) INGESTION from a source — CDC or a batch connector (optionally a streaming path with Kafka + a consumer); (2) a LAKEHOUSE or warehouse layered bronze -> silver -> gold (or staging -> marts), using an open table format (Delta Lake / Apache Iceberg) OR a warehouse with ELT/dbt; (3) DATA-QUALITY gates (dbt tests / Great Expectations) plus a data-OBSERVABILITY check (freshness / volume / schema) that alerts; (4) ORCHESTRATION with retries and backfills over idempotent tasks; (5) INFRASTRUCTURE AS CODE (Terraform) or containerized jobs, with CI running the tests on pull requests; (6) governance basics — documented lineage/catalog and a note on PII/access control. Document the modeling, data-quality, and cost choices.
A real ingestion path (CDC/batch, optionally streaming); bronze->silver->gold or staging->marts via an open table format or warehouse ELT.
Data-quality gates (dbt/Great Expectations) + an observability/alert check; orchestrated with retries/backfills and idempotent tasks.
IaC (Terraform) or containers + CI running tests on PRs; documented lineage/catalog and a PII/access note; cost choices documented.
Use this README structure to make the work reviewable.
DevPath fetches a read-only, size-capped snapshot. Submitted code is never executed.
# A production-shaped data platform ## Goal Build a production-shaped data platform. Requirements: (1) INGESTION from a source — CDC or a batch connector (optionally a streaming path with Kafka + a consumer); (2) a LAKEHOUSE or warehouse layered bronze -> silver -> gold (or staging -> marts), using an open table format (Delta Lake / Apache Iceberg) OR a warehouse with ELT/dbt; (3) DATA-QUALITY gates (dbt tests / Great Expectations) plus a data-OBSERVABILITY check (freshness / volume / schema) that alerts; (4) ORCHESTRATION with retries and backfills over idempotent tasks; (5) INFRASTRUCTURE AS CODE (Terraform) or containerized jobs, with CI running the tests on pull requests; (6) governance basics — documented lineage/catalog and a note on PII/access control. Document the modeling, data-quality, and cost choices. ## Acceptance criteria - [ ] Ingestion & layering: A real ingestion path (CDC/batch, optionally streaming); bronze->silver->gold or staging->marts via an open table format or warehouse ELT. - [ ] Quality & orchestration: Data-quality gates (dbt/Great Expectations) + an observability/alert check; orchestrated with retries/backfills and idempotent tasks. - [ ] Platform & governance: IaC (Terraform) or containers + CI running tests on PRs; documented lineage/catalog and a PII/access note; cost choices documented. ## Verification - [ ] Document installation and run commands. - [ ] Record automated checks and their exact commands. - [ ] Add representative output, screenshots, or a short demo where useful. - [ ] Confirm failure paths and known constraints. ## Decision log ### Decision title - Context: - Choice: - Alternatives considered: - Tradeoffs: ## Evidence - Link each rubric criterion to the file, test, or artifact that demonstrates it. ## Known limitations - List what is intentionally out of scope and what should be improved next.