Layered structure
Raw, cleansed and serving layers stay separate. The raw layer is a copy of the source and is never modified; corrections happen in the cleansed layer so the origin of an error remains traceable.
- The raw layer is append-only, never edited
- Transformation rules are defined in code and versioned
- The serving layer is shaped around business definitions
- Movement between layers passes a quality gate
Catalog, ownership and quality
Every dataset needs an owner, a definition and a quality expectation. The catalog gives that information to whoever is searching, and quality rules turn the expectation into something measurable.
- Owner and definition are recorded per dataset
- Quality rules run as data arrives
- A rule breach stops or flags the data
- Lineage shows where each field came from
Access and privacy
A lake holds the broadest collection of data in the organization, so access design is part of the architecture. Fields containing personal data are marked, and masking and aggregation are applied through a rule set.
- Personal data fields are marked through classification
- Masking applies at query time rather than by copying
- Access is role-based and logged
- Analysis environments do not receive production data as-is
Cost and lifecycle
Storage is cheap; querying is not. Unless partitioning, file format and retention tiers are planned early, cost grows faster than data volume — and unused datasets live forever.
- Partitioning follows the query pattern
- Hot, warm and cold tiers are defined
- Unused datasets are flagged and archived
- Cost is reported back to the dataset owner