The Macro System That Isn't There: yin.vm Macros
Macros are the defining superpower of Lisp. They allow developers to bend the language to their will, creating domain-specific abstractions that feel native. But this power comes at a steep architectural cost: compile-time environments bleed into runtime, infinite loops crash compilers, and tracking the provenance of generated code is notoriously difficult.
In datom.world, the yin.vm runtime takes a radical approach to metaprogramming. It achieves the full power of homoiconic Lisp macros by doing something unthinkable: it hides macros completely from the evaluator.
Here is a look at what makes the yin.vm macro system utterly unique, what it can do that other systems cannot, and how it compares to the macro engines of Clojure, Erlang, Rust, Racket, and Unison.
1. The Evaluator is Macro-Blind
In almost every Lisp (like Clojure or Common Lisp), macroexpansion is deeply interleaved with the compiler and evaluator. The engine looks at a symbol, checks if it has a special macro flag, and halts evaluation to execute the macro transformation before continuing.
In yin.vm, the evaluator knows absolutely nothing about macros. It has no macro instruction, no expansion ledger, and no macro flag on closures. A lambda can be marked as a macro on the syntax side, but that mark is dropped before code reaches the evaluator. Instead, yin.vm models macroexpansion strictly as a stream topology:
Frontend -> (program-in) -> Expander -> (program-out) -> Evaluator
The Expander is simply a pure stream transducer. It wakes up, consumes batches of unexpanded AST packets from the program-in stream, rewrites them to a fixpoint, and appends the resulting canonical tuples to the program-out stream. By the time the Evaluator wakes up to read program-out, macros have ceased to exist.
2. The Opposite of Racket (Abolishing Phases)
Racket is widely considered the undisputed king of Lisp macro systems. It solved the problem of macro hygiene and environment leakage by inventing an intricate, mathematically rigorous tower of Phases (Phase 0 for runtime, Phase 1 for compile-time, Phase -1 for templating, etc.). In Racket, different phases have strictly isolated environments that are instantiated and destroyed as the compiler climbs the tower.
Racket serves as a brilliant foil to yin.vm because it solves the same problem from the exact opposite direction. Racket looked at the chaos of macros and said: "We must build complex internal walls (Phases) to keep compile-time and runtime from contaminating each other within the same memory space."
yin.vm looks at Phase theory and says: "Phases are just a symptom of running two different systems in the same process."
By pushing expansion out of the compiler and into a dedicated stream transducer (whether a network pipe or a simple in-memory ringbuffer), yin.vm abolishes phases entirely. There is no "compile time" vs "runtime." There is only data flowing down a pipe.
3. What yin.vm Can Do That Others Can't
Because yin.vm treats macroexpansion as a pure stream transformation over content-addressed tuples (rows in a database, rather than raw text), it unlocks capabilities that are structurally impossible in other languages:
Decoupled Compilers and Capability-Gated Macros
By stripping away the text parser and moving the compiler boundary to the data stream, the role of a frontend is radically simplified. A compiler can literally be a remote network service. If a program holds a writer handle to the program-in stream, it is effectively a compiler. Today that handle is the capability; gating it with ShiBi capability tokens is the planned next step. A Python script, a visual block editor, or an AI agent can synthesize Universal AST tuples and append them to the stream. Because they output pure data to the standard topology, they instantly inherit the entire backend ecosystem (the Macro Expander, the AST-walking evaluator, the bytecode VM, and whatever backends come later, such as the planned register VM and JIT) completely for free. Metaprogramming is no longer a language-specific feature; it is a capability-gated service on the stream.
Guaranteed Safety via Deterministic Fuel Guards
If you write an infinite loop in a Clojure or Rust macro, you will hang the compiler or crash the build process with a Stack Overflow. Because yin.vm is an autonomous system, a macro cannot be allowed to crash a node. Expansion execution is deterministically bounded by fuel guards. The Expander counts body steps, result rows, and expansion depth. If a macro spins out of control, the Expander safely halts it, records the fuel-guard error as data (and as an event row when a log stream is attached), and keeps processing. A macro body that tries to touch a stream is stopped the same way, by an effect guard.
Distributed, Cross-Machine Expansion
Because the Expander and the Evaluator are just two observers connected by streams, they do not have to live on the same machine. The Expander speaks only the stream protocol, so nothing prevents running the heavy Macro Expander on a massive cloud cluster, which streams the resulting lightweight, macro-free program-out tuples over WebSockets to a low-power IoT device running the yin.vm Evaluator.
Pure Dynamic Evaluation Without an Eval Function
Dynamic evaluation needs no magical eval function. New code containing macros is simply appended as AST packets to the program-in stream, and the exact same stream topology takes over: the Expander wakes up, crunches the macros, and feeds the Evaluator. Today the REPL shell is the writer that does this. A running yin.vm program handed the same writer could do it too, and the runtime would then self-modify by participating in its own plumbing.
Comparing the Competition
How does this stack up against the broader landscape of metaprogramming?
Clojure
Clojure macros transform S-expressions (lists and symbols) and have full access to the host JVM. A Clojure macro can read from the filesystem, make HTTP requests, or mutate global state. yin.vm macros are strictly pure: they consume AST tree packets and return AST tree packets, with zero access to host I/O or streams.
Rust (Procedural Macros)
Rust is conceptually the closest. A procedural macro is compiled as a separate binary that takes a TokenStream and returns a TokenStream. However, Rust macros are still arbitrary host code (meaning they lack yin.vm's deterministic fuel-guards and pure isolation).
Erlang
Erlang offers simple C-style textual macros (which are extremely limited) and parse-transforms (which allow arbitrary AST mutation). However, parse-transforms are notoriously difficult to write, globally mutating, and heavily discouraged by the community because they break tooling. yin.vm's macro system is local, deterministic, and explicitly designed as the primary abstraction mechanism (though it consciously eschews structural hygiene, opting instead for explicit, opt-in hygiene via deterministic gensyms).
Unison
While Unison doesn't have a macro system, it shares yin.vm's fundamental philosophy of content-addressed code. Unison stores ASTs in a database by their cryptographic hash rather than in text files, just as yin.vm operates natively on content-addressed row tuples. Unison evaluation never parses text; it evaluates canonical AST structures directly from the store. However, Unison eschews macros entirely, relying on Algebraic Effects for abstraction. yin.vm takes the Unison data-model and proves that you can build a safe, robust macro system on top of it by keeping the transformations confined to the stream layer.
Conclusion
The brilliance of the yin.vm macro system lies in what it refuses to do. By refusing to let the evaluator know about macros, refusing to let macros access I/O, and refusing to invent complex compiler phases, it distills metaprogramming down to its purest essence: a deterministic, safe, and easily distributed stream of data.