Unified Macros: Metaprogramming as a Stream of Immutable Facts

Published Mar 9, 2026

Metaprogramming, code that writes code, is the defining feature of the Lisp family. It allows developers to mold the language to their domain rather than molding their domain to the language. But historically, macro expansion has been a "black box" compiler phase. It destroys source provenance, complicates debugging, and makes secure runtime execution nearly impossible.

In Datom.world, we treat the Universal AST as a canonical stream of immutable facts (datoms). Because of this, we can model macros differently. A macro is not a compiler phase; it is an observable event in a bounded data stream. This is the keystone of our architecture: the deliberate fusion of compile-time and runtime capabilities enabled by the underlying datom structure.

The Problem with Traditional Macros

When a Lisp compiler encounters a macro like (when test body...), it passes the raw syntax to the macro function, takes the resulting new syntax (e.g., (if test (do body...))), and silently overwrites the old AST. The original when form is gone. If an error occurs during execution, the stack trace points to the expanded if form, confusing the developer.

Furthermore, because macro expansion requires invoking the compiler dynamically, Lisp's eval is notoriously slow and completely prohibited in Ahead-Of-Time (AOT) environments like Dart, GraalVM native images, or WebAssembly. You cannot have runtime self-modifying code without dragging a heavy JVM classloader or JIT compiler along with you. As the article "Data > Functions > Macros" correctly points out, in traditional systems, macros are simply not available at runtime and are opaque after expansion.

The Yin.vm Solution: Event Sourcing for ASTs

Yin.vm unifies compile-time and runtime macros by modeling them as explicit facts appended to the program-datoms stream. The expansion contract is straightforward:

In the canonical model, a macro is not a separate AST species. It is a lambda with metadata: :yin/macro? true and :yin/phase-policy set to :compile, :runtime, or :both. This lets the same macro run at compile time or runtime under explicit phase rules.

1. Interception: When the VM executes a call to a function tagged with :yin/macro? true, it passes the literal AST entity references to the macro, unevaluated.
2. Expansion: The macro runs, generating new AST datoms.
3. Provenance: Instead of destroying the original call, the VM asserts a new :macro-expand-event entity. This event has a :yin/source-call pointer back to the original AST, and a :yin/expansion-root pointer to the newly generated AST.
4. Metadata Linkage: Every new datom produced by the macro gets its m (metadata) component set to the ID of the expansion event.
5. Boundary Recompile: The newly appended datoms mark the execution stream as dirty. At the next safe execution boundary (e.g., a function call or scheduler yield), the VM dynamically updates its bytecode cache (ast-datoms->asm) and continues execution safely.

The Mechanism: Portable cljc and Persistent Datoms

The technical foundation for this dual-phase capability is the portability of the system. The yang.clojure compiler and the Yin.vm interpreter are written in portable cljc (Clojure Common) code. This means the same codebase can run on the JVM for traditional compile-time, on JavaScript, and crucially, on static runtimes like Dart/Flutter.

• Compile-Time Expansion: When the compiler encounters a macro during its compilation phase, it expands it, generating the final, optimized AST datoms that represent the program's initial state. This satisfies the need for static, ahead-of-time code generation.
• Preserving the Macro: The key innovation is that the compiler doesn't throw the macro definition away. Because everything is data, the macro itself is also persisted as datoms in the program stream. Its definition, its metadata, and its code are all stored alongside the code it generated.
• Runtime Availability: Later, when the Yin.vm interpreter (running on Dart, for example) encounters a new macro invocation, it can load the macro definition datoms still present in the stream and run the macro right then to generate new code.

This creates a continuum of metaprogramming that shatters the traditional boundary:

Predictable Power

This design addresses performance directly. Because the macro definition is just data, and its runtime expansion uses the same lightweight, sandboxed Semantic VM interpreter that is always present, there is no need to spin up a heavy, separate compiler at runtime. The boundary recompile and subsequent JIT optimization by the Register or Stack VMs can then happen efficiently, making runtime code generation a first-class, performant citizen.

Crucially, this expansion process is execution-model agnostic. While the Register and Stack VMs are optimized for high-performance execution, they both delegate the evaluation of the macro lambda itself to the Semantic VM, the system's canonical reference interpreter. This ensures that macro expansion remains perfectly consistent across all environments, with the Semantic VM providing a safe, isolated sandbox for code generation.

Because the Semantic VM is a lightweight, pure-functional interpreter for the datom stream, it is easily embedded into any distribution target, from a Flutter app to a WebAssembly module. However, in highly constrained edge environments, a node can even choose to delegate expansion entirely. Since a macro call and its result are simply facts in a synchronized datom stream, a resource-limited "edge" VM can park its execution, wait for a more powerful "leader" node to perform the expansion, and then resume execution once the new datoms arrive. Metaprogramming becomes a distributed, asynchronous capability of the network.

This architecture provides perfect provenance. If you ask the database, "Where did this AST node come from?", a single Datalog query can traverse the m field back to the expansion event, and from there to the exact line of code the user typed. The compiler doesn't need to build a complex sourcemap side-channel; the sourcemap is intrinsically encoded in the datoms themselves.

How It Compares to Existing Systems

1. The Lisp Family (Clojure, Common Lisp, Scheme)

While Lisp pioneered the concept, it restricts macros to compile-time. Invoking eval at runtime spins up the entire compiler pipeline, generating ephemeral JVM bytecode or host classes. Yin.vm's boundary recompile never allocates host machine code. It simply translates datoms into a data array (Yin bytecode) that the static VM loop iterates over, making it perfectly safe for AOT targets like Flutter/Dart and WASM.

2. Fexprs / Vau Calculi (Kernel, PicoLisp)

Yin's unevaluated argument passing is conceptually identical to an fexpr. Historically, fexprs were abandoned because compilers couldn't optimize them; they never knew if an argument would be evaluated or treated as data. Yin solves this by making macro-expansion an explicit event that triggers a bounded recompile, allowing the Register and Stack VMs to aggressively optimize the resulting stream.

3. Racket

Racket operates on "Syntax Objects", AST nodes decorated with rich metadata (lexical scope, source location). While Racket uses this metadata primarily for complex macro hygiene during compilation, Yin tracks its metadata (m) for causality and provenance across the entire lifecycle: storage, distribution, and runtime execution.

4. Unison

Like Unison, Yin treats ASTs as immutable and hashes them by their structural content (Merkle hashing). Unison focuses on using these hashes to decouple code from names, enabling seamless refactoring. Yin takes this further by introducing computational time (t) and causality (m) via streams. Furthermore, while Unison enforces strict alpha-equivalence (throwing away variable names in the hash), Yin preserves the literal syntax in the primary :yin/content-hash and treats alpha-equivalence as a separate derived datom (:yin/alpha-hash).

5. Object-Capability Security (E, Agoric)

Perhaps the most significant departure is security. Allowing code to rewrite itself at runtime is inherently dangerous in a multi-tenant system. Yin.vm requires a ShiBi capability token to execute a runtime :yin/macro-expand event. A rogue continuation cannot inject new code unless the host explicitly granted it a capability token scoped to the target program stream and the specific macro namespace.

6. Cross-Language Metaprogramming

Traditional macros are deeply tied to the syntax of their host language. A Clojure macro cannot expand Python code. In datom.world, macro expansion happens purely on the Universal AST datom stream, making it completely language-agnostic.

This enables a unique architecture for non-homoiconic languages like Python or JavaScript. Writing AST-manipulating macros in these languages is notoriously painful. Instead of forcing it, Yin.vm treats macros as a cross-language database service. A macro can be seamlessly authored in yang.clojure and persisted in DaoDB. When the yang.python compiler encounters a macro invocation, it doesn't parse text; it queries DaoDB for the macro's canonical lambda EID and emits a :yin/macro-expand node. The burden of metaprogramming is gracefully shifted to the Lisp layer, while the benefits are universally shared across all frontends.

Conclusion

By treating macro expansion not as a compiler phase, but as a capability-gated, auditable database transaction, Yin.vm achieves Lisp's expressive power, Unison's structural purity, and Datomic's temporal observability. The ability for the same macro definition to be usable at both compile-time and runtime is the keystone that makes Yin.vm's approach a new foundation for dynamic metaprogramming.