Datom.World

The Data Lake Is Just One Point in the Moduli Space

Published Aug 16, 2026

Modern data infrastructure suffers from a persistent schism. On one side lies the Data Lake (GCS, S3, Apache Parquet, Apache Iceberg): exabytes of immutable, columnar data optimized for bulk throughput and analytical scans. On the other side lie Operational Databases (Bigtable, Cassandra, Postgres): specialized key-value and relational stores built for low-latency point lookups.

Because traditional query engines cannot serve interactive point queries directly from object-store files, organizations spend enormous resources running batch pipelines to duplicate exabytes of lake data into operational serving tiers. Recently, Spotify addressed this exact dilemma by introducing Random Access Parquet (RAP)—building external indices and covering layouts directly over immutable Parquet files in Google Cloud Storage to enable low-latency point lookups without replication.

Spotify’s move points toward a deeper architectural truth that Datom.world formalizes from first principles: A Data Lake is not a separate storage silo. It is just one point in an open moduli space of database interpreters.

DaoJing: Storage Without Domain Semantics

To understand why a data lake is merely an interpreter, we have to look beneath the database level to the storage boundary. In Datom.world, that boundary is DaoJing (井, "the well").

DaoJing observes an explicit pool of intake DaoStreams and performs a single, representation-level operation for every arriving payload:

bytes = canonical-encode(x)
key   = content-hash(bytes)       ; :segment/sha256-
KV[key] = x                       ; insert-if-absent

DaoJing is itself an interpreter over DaoStream, but strictly at the representation level. It assigns content identity and idempotent retrieval semantics, but remains completely devoid of domain semantics. DaoJing does not know whether an element is a datom [e a v t m], a covered B-Tree node, an AST node, an embedding vector, or an Apache Parquet column chunk. All are opaque, immutable bytes at rest.

Interpretation is external to the stream and external to storage. Semantics lives in the interpreter that observes the bytes, not in the bytes themselves.

S3 as the Native Physical Substrate

Where does DaoJing actually persist these segments? In Datom.world, storage handles are plain data: {:put-content-fn f, :get-content-fn g}. Because DaoJing requires no mutable roots, no CAS registers, and no delete operations, cloud object stores like Amazon S3, Google Cloud Storage, or Cloudflare R2 are its most natural physical backends.

S3 maps onto DaoJing’s contract with zero impedance mismatch:

  • Content Address as Object Key: Each segment is written under its content address: s3://bucket/segments/a7/3f/sha256-a73f.... Partitioning by hash prefix automatically spreads I/O across S3’s internal physical partitions, preventing hot spots.
  • Lock-Free Immutability: Because writes are idempotent insert-if-absent operations under cryptographic hashes, storage requires no coordination, locks, or transactor consensus. Segments can be cached indefinitely across edge CDNs with Cache-Control: immutable.
  • HTTP Range Reads for Sub-File Access: S3 natively supports Range: bytes=start-end requests. When an interpreter faults a single B-Tree node or executes a Spotify RAP point query, it retrieves only the precise byte range needed rather than downloading entire files.

Because DaoJing coordinates are serializable data ({:dao.jing/type :dao.jing/s3, :bucket "datom-lake", :region "us-east-1"}), any reader—a local Datalog engine, an analytics cluster, or an ML worker—can open the storage directly over S3 without a database server in the middle.

The Moduli Space: Sibling Interpreters Over One Substrate

Because DaoJing stores opaque, content-addressed payloads in S3, Datom.world admits an open moduli space of database interpreters (a metaphor borrowed from mathematics). Different interpreters observe the exact same immutable substrate and project distinct semantic structures:

  • DaoSpace (The Tuple Space): Interprets segments as covered B-Tree indices (EAVT/AEVT) and evaluates discrete associative queries via Datalog unification.
  • DaoField (The Vector Field): Interprets segments as continuous d_k coordinate tuples and evaluates metric proximity (cosine similarity, gradients) in ℓ²(E).
  • DaoLake (The Data Lakehouse): Interprets segments as columnar row groups, partition manifests, and external lookup trees, evaluating vectorized OLAP scans alongside index-assisted point lookups.

DaoSpace, DaoField, and DaoLake are siblings, not layers. None sits on top of the other; none owns a private storage silo. They are simply different mathematical shadows projected from the same underlying facts.

Spotify RAP as Morphism and Dimensional Compression

In What Is Computation?, we established that all computation is a structural transformation governed by three primitive moves: Expansion (generating richer semantics), Compression (extracting basis-free invariants), and Morphism Construction (building bridges between spaces).

The traditional data lake and Spotify’s RAP architecture illustrate this triad with striking clarity:

1. Compression into Invariants

Columnar formats (Parquet, Eve flat segments) achieve extreme compression by stripping away redundant row headers and encoding basis-free columnar invariants (run-length, dictionary, bit-packing). In DaoJing, content-addressing pushes this further: identical content across independent streams collapses into a single canonical hash address on S3.

2. Dimensional Expansion

A Data Lake query engine (like Trino or DuckDB) expands flat columnar bytes into high-dimensional relational tables, aggregations, and join trees during analytical scans.

3. Morphism Construction (RAP)

Full scans are high-entropy operations. When an online service needs a single record, scanning exabytes of Parquet files is unfeasible. Replicating the data into a key-value database was the old, brute-force answer.

Spotify’s RAP instead constructs a morphic bridge: an external index mapping Lookup Key → (Segment Hash, Byte/Row Offset). Combined with covering layouts and S3 HTTP Range requests, this morphism allows an online point query (O(1)) and an analytical scan (O(N)) to traverse the exact same immutable S3 objects.

Zero-ETL Multi-Model Unification

When a data lake is recognized as an interpreter in Datom.world's moduli space rather than a monolithic silo, the need for complex ETL architectures disappears:

                dao.stream (append-only event log)
                     │
                     │ [Representation Interpreter: Canonical Hashing]
                     ▼
          dao.jing on S3 (immutable content-addressed segments in object store)
                     │
         ┌───────────┼──────────────────────────┐
         ▼           ▼                          ▼
     dao.space    dao.field                  dao.lake
     (Datalog)   (Vector Metric)         (Columnar Lakehouse)
                                                │
                                                │ [External Index / RAP Range Reads]
                                                ▼
                                      Low-Latency Point Queries

Because all three interpreters read from the same content-addressed S3 storage:

  • An agent can write datoms to its local stream once.
  • The lake interpreter indexes those datoms into columnar segments for analytical queries.
  • The vector interpreter computes coordinate embeddings for semantic retrieval.
  • The tuple space allows Datalog rules to join across operational, analytical, and metric views simultaneously.

Conclusion

A Data Lake is not a specialized empire requiring dedicated storage formats and duplicate database tiers. It is a specific way of observing data: columnar projection, partition bounds, and vectorized aggregation.

By placing storage in a dumb, content-addressed well (dao.jing) backed by cloud object storage like S3, and moving all semantics to embeddable interpreters, Datom.world collapses the artificial barrier between operational databases, AI vector fields, and analytical data lakes. They are all just points in the moduli space, unified over immutable streams.

Learn more: