Yin.vm on Dart: Portable cljc Meets Static Compilation
The ClojureDart Constraint
ClojureDart is a remarkable achievement: it compiles Clojure to Dart, enabling Clojure developers to build Flutter applications. But Dart's compilation model imposes fundamental constraints:
- No runtime eval: All code must be known at compile time. You cannot construct and execute code dynamically.
- No dynamic code loading: Tree shaking eliminates unused code paths. Hot reload exists during development, but production builds are sealed.
- No traditional macros: Clojure's macro system operates at compile time, but the compiled Dart output is static. Runtime macros that transform executing code are impossible.
- Limited reflection: Dart's mirrors are restricted or unavailable in Flutter. Introspection of runtime structures is constrained.
These limitations stem from Dart's design goals: ahead-of-time compilation enables fast startup, small binaries, and predictable performance. But they conflict with Lisp's fundamental promise of code as data and runtime malleability.
The Insight: Move Dynamism into Data
The solution is conceptually simple but profound: implement an interpreter that runs on Dart, and make the interpreted language dynamic.
This is exactly what Yin.vm provides. Instead of trying to make Dart dynamic (impossible) or accepting static constraints (limiting), we build a CESK continuation machine that interprets datom ASTs. The interpreter code is fixed and AOT-compiled. The datom ASTs are dynamic, loadable at runtime, and fully interpretable.
The key insight: Yin.vm and the Yang compiler are written in portable cljc code. This means the same codebase runs on:
- JVM (via Clojure)
- JavaScript (via ClojureScript)
- Dart/Flutter (via ClojureDart)
Write once, interpret everywhere. The interpreter itself leverages Clojure's cross-platform story while providing dynamic semantics on top of static runtimes.
┌─────────────────────────────────────────────────────┐
│ Dart Runtime (static, AOT compiled) │
│ ┌───────────────────────────────────────────────┐ │
│ │ Yin.vm Interpreter (CESK Machine) │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ C: Control (current AST node) │ │ │
│ │ │ E: Env (variable bindings) │ │ │
│ │ │ S: Store (heap/memory) │ │ │
│ │ │ K: Kont (what happens next) │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────┘ │
│ ↑ │
│ │ interprets │
│ ↓ │
│ ┌───────────────────────────────────────────────┐ │
│ │ Datom AST Stream (dynamic, loadable) │ │
│ │ [e :yin/op :apply t m] │ │
│ │ [e :yin/args [...] t m] │ │
│ │ [e :yin/body body-ref t m] │ │
│ └───────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘Why CESK?
The CESK machine (Control, Environment, Store, Kontinuation) is a canonical model for implementing interpreters with explicit control flow. Each component serves a specific purpose:
- Control: The current expression being evaluated (an AST node, represented as datoms).
- Environment: A mapping from variable names to addresses in the store (lexical bindings).
- Store: A heap that maps addresses to values (enables mutation and sharing).
- Kontinuation: A reified representation of "what to do next" after the current expression completes.
The continuation is the key innovation. By making the continuation explicit and first-class, Yin.vm gains capabilities impossible in Dart alone:
- Pause execution at any point and serialize the entire state
- Resume execution later, potentially on a different device
- Implement cooperative multitasking without threads
- Enable time-travel debugging by storing continuation snapshots
Datom AST: Code as Queryable Data
In Yin.vm, the abstract syntax tree is not a transient compiler artifact. It persists as datoms:
;; A simple function: (fn [x] (+ x 1))
[fn-1 :yin/op :fn t m]
[fn-1 :yin/params [param-1] t m]
[param-1 :yin/name "x" t m]
[fn-1 :yin/body body-1 t m]
[body-1 :yin/op :apply t m]
[body-1 :yin/fn :+ t m]
[body-1 :yin/args [arg-1 arg-2] t m]
[arg-1 :yin/ref param-1 t m]
[arg-2 :yin/literal 1 t m]Because the AST is data (datoms), it can be:
- Loaded dynamically: Fetch new AST datoms from a stream, network, or database at runtime.
- Queried: Use Datalog to find all functions that reference a given variable.
- Transformed: Runtime macros that rewrite AST datoms before or during execution.
- Versioned: The
t(transaction) component tracks when each node was created or modified. - Provenance-tracked: The
m(metadata) component records where each node came from.
What This Enables
Implementing Yin.vm in Dart unlocks capabilities that would otherwise be impossible:
Dynamic Code Loading
New functionality can arrive as datom streams at runtime. A mobile app can download new features, workflows, or bug fixes without going through app store review:
;; Receive new AST datoms over the network
(defn load-feature [stream]
(let [ast-datoms (stream/take-until stream :eof)]
(yin/register-fn! vm ast-datoms)))The Dart code (Yin.vm interpreter) stays fixed. Only the interpreted AST changes.
Runtime Macros
Because ASTs are datoms, code can transform itself using the same structures the VM executes. A macro is just a function that takes AST datoms and returns AST datoms:
;; A logging macro that wraps function bodies
(defn wrap-with-logging [fn-datoms]
(let [body (find-body fn-datoms)
logged-body (wrap-log body)]
(replace-body fn-datoms logged-body)))These transformations happen at interpretation time, not compile time. True runtime metaprogramming.
Serializable Continuations
The CESK machine's continuation is an explicit data structure. It can be serialized to datoms:
;; Serialize current execution state
(let [k (yin/capture-continuation vm)]
(datom/write-to-stream k output-stream))
;; Later, possibly on a different device
(let [k (datom/read-from-stream input-stream)]
(yin/resume-continuation vm k))This enables mobile agents, distributed computation, and fault-tolerant workflows that can checkpoint and resume across crashes.
Hot Patching in Production
Update running applications without restart. Push new AST datoms, and the interpreter picks them up. Functions can be replaced atomically while the system continues running.
The Tradeoff: Interpretation vs. Compilation
Interpretation is slower than AOT compilation. A function interpreted by Yin.vm will execute more slowly than the same function compiled directly to Dart. This is the fundamental tradeoff.
But the performance gap can be managed:
- Hot paths in Dart: Performance-critical primitives (arithmetic, collections, I/O) are implemented as native Dart functions that the interpreter calls directly.
- Selective interpretation: Only dynamic, changeable logic runs through Yin.vm. UI rendering, networking, and other framework interactions stay in compiled Dart.
- JIT opportunities: Frequently executed AST paths can be identified and specialized.
- Acceptable for coordination: Business logic, workflows, and rules often don't need nanosecond performance. The flexibility of runtime modification outweighs the speed cost.
Architecture in Practice
A Flutter app using Yin.vm might be structured like this:
┌─────────────────────────────────────────────┐
│ Flutter Framework (Dart, compiled) │
│ - Widgets, rendering, platform channels │
└─────────────────────────────────────────────┘
↑
│ calls
↓
┌─────────────────────────────────────────────┐
│ Application Shell (Dart, compiled) │
│ - Yin.vm interpreter │
│ - Native primitives (I/O, crypto, etc.) │
│ - Stream infrastructure │
└─────────────────────────────────────────────┘
↑
│ interprets
↓
┌─────────────────────────────────────────────┐
│ Business Logic (Datom AST, dynamic) │
│ - Workflows, rules, transformations │
│ - User-defined functions │
│ - Remotely loaded features │
└─────────────────────────────────────────────┘
↑
│ streams from
↓
┌─────────────────────────────────────────────┐
│ DaoDB / Datom Streams │
│ - Persistent AST storage │
│ - Continuation checkpoints │
│ - Distributed synchronization │
└─────────────────────────────────────────────┘The compiled Dart layer handles performance-critical rendering and platform integration. The interpreted Yin.vm layer handles dynamic, changeable business logic. The datom stream layer provides persistence and distribution.
Comparison: Other Approaches
This pattern is not unique to Yin.vm. Similar architectures exist elsewhere:
- Lua in games: Game engines (Unity, Unreal, custom) embed Lua for scriptable game logic. The engine is compiled C++; the scripts are interpreted Lua.
- JavaScript in React Native: The native shell is compiled; the JS business logic is interpreted/JIT'd.
- WebAssembly runtimes: Wasmer, Wasmtime embed a Wasm interpreter in native applications.
What distinguishes Yin.vm is the datom representation. ASTs are not opaque bytecode but queryable, versionable, provenance-tracked data. The code itself participates in the same data model as everything else in the system.
Native Dart Implementation
For the native Dart benchmark implementation, several areas require attention. These patterns also inform optimizations that can be applied to hot paths in the cljc version:
Efficient Datom Representation
Dart's type system allows defining efficient datom structures. A datom might be:
class Datom {
final int e; // Entity ID
final Symbol a; // Attribute (interned)
final Object v; // Value
final int t; // Transaction ID
final int m; // Metadata entity
const Datom(this.e, this.a, this.v, this.t, this.m);
}Symbol interning keeps attribute comparisons fast. Immutability enables structural sharing.
Continuation Representation
Continuations must be serializable. A continuation frame might look like:
sealed class Kont {}
class KontHalt extends Kont {}
class KontArg extends Kont {
final List<Object> evaluatedArgs;
final List<Datom> remainingArgs;
final Env env;
final Kont next;
// ...
}
class KontFn extends Kont {
final List<Datom> args;
final Env env;
final Kont next;
// ...
}Dart's sealed classes ensure exhaustive pattern matching on continuation types.
Primitive Operations
Core operations (arithmetic, comparison, collection manipulation) should be implemented as native Dart functions, not interpreted AST:
final primitives = <Symbol, Function>{
Symbol('+'): (a, b) => a + b,
Symbol('-'): (a, b) => a - b,
Symbol('*'): (a, b) => a * b,
Symbol('cons'): (h, t) => [h, ...t],
Symbol('first'): (l) => l.first,
// ...
};Two Paths: Portable cljc and Native Dart
The implementation strategy follows two parallel paths:
Primary: Portable cljc
Yin.vm and the Yang compiler are written in portable cljc code. This leverages the existing Clojure compiler ecosystem:
- Clojure compiles cljc to JVM bytecode
- ClojureScript compiles cljc to JavaScript
- ClojureDart compiles cljc to Dart
One codebase, three platforms. The interpreter runs wherever Clojure runs. ClojureDart handles the compilation to Dart; Yin.vm handles the dynamic interpretation on top.
Secondary: Native Dart Implementation
A separate Yin.vm implementation written directly in native Dart provides a performance baseline. This allows us to measure:
- How much overhead ClojureDart compilation introduces
- Where native Dart optimizations might be beneficial
- Whether hot paths should be rewritten in native Dart
The native Dart version serves as a benchmark, not a replacement. The portable cljc version remains the primary implementation for maintainability and cross-platform consistency.
The Resulting Architecture
This dual approach yields a layered architecture:
- cljc for writing the interpreter and compiler (portable)
- ClojureDart for compiling cljc to Dart
- Native Dart for performance-critical primitives and benchmarking
- Yin.vm for interpreting dynamic AST datoms
- DaoDB for persisting and distributing code and continuations
- Flutter for cross-platform UI rendering
Conclusion
ClojureDart's static compilation constraints are not bugs but consequences of Dart's design. Rather than fighting those constraints, Yin.vm embraces them by making Dart the substrate rather than the language.
The dynamism moves into data. ASTs become datom streams, loadable and interpretable at runtime. Continuations become serializable checkpoints, pausable and resumable across devices. Code becomes queryable, versionable, transformable while the system runs.
By writing Yin.vm and the Yang compiler in portable cljc, we get the best of both worlds: write once in Clojure, run on JVM, JavaScript, and Dart. The native Dart implementation serves as a performance benchmark, guiding optimizations without sacrificing the maintainability of the portable codebase.
This is not a workaround but a design principle: interpretation over compilation when flexibility matters more than raw speed. The fixed interpreter provides a stable foundation. The dynamic AST provides unbounded extensibility.
Yin.vm on Dart: where static compilation meets dynamic semantics, and portability meets performance.
Related Reading: