Beyond FFI with Datom Streams

Published Jan 25, 2026

How Datom.world replaces traditional Foreign Function Interfaces (FFI) with high-performance reactive streams and zero-copy ring buffers.

In the landscape of virtual machine design, the boundary between the host environment and the execution engine is often the most significant bottleneck. For Yin.VM, the computation core of the Datom.world ecosystem, we have moved away from the decades-old standard of function-based Foreign Function Interfaces (FFI).

Instead of treating the VM as a library to be commanded through imperative calls, Yin.VM treats the execution boundary as a continuous stream of Datoms. This shift from "doing" to "telling" represents a fundamental rethink of how decentralized systems should handle cross-language execution.

The Traditional Bottleneck: The "Marshaling Tax"

Historically, embedding a VM (like Lua or Python) into a host language (like Clojure) follows a Request/Response pattern. To execute a single instruction or process a transaction, the host must pause its own execution, translate its internal data structures into C-compatible structs, and pass control across the FFI border.

This "Traditional Way" introduces several points of friction:

• Synchronous Blocking: The host language sits idle while the VM processes the call.
• Impedance Mismatch: Rich, functional data in Clojure must be "shredded" into low-level C pointers and rebuilt on the other side.
• The Marshaling Tax: A significant percentage of CPU cycles is spent on serialization and deserialization rather than actual computation.

In a distributed or decentralized context where state changes occur rapidly across a network, this constant "stopping and starting" of the execution flow creates unacceptable latency.

The Yin.VM Solution: Streaming the Truth

Yin.VM operates on a different philosophy. It views the world not as a series of commands, but as a sequence of immutable facts. By utilizing a Datom Stream (Entity, Attribute, Value, Transaction, Metadata), the VM transforms the execution boundary into a reactive data pipe.

How It Works

The host application (e.g., a Clojure backend) no longer calls specific functions within the VM. Instead, it "pushes" datoms into an input stream. The VM, written in a systems language like Rust, independently consumes these facts. As the VM processes the stream, it triggers internal continuations (captured execution states) that react to the incoming data.

Crucially, communication flows in both directions via datom streams. When Yin.VM needs to invoke native capabilities (file I/O, cryptography, network), it emits datoms to an output stream. The Bridge Dispatcher consumes these datoms and triggers the appropriate native functions. The result? A symmetric architecture where both "calling into" and "calling out of" the VM use the same unified interface.

This approach offers three primary advantages:

• Temporal Decoupling: The host can produce facts at its own speed, and the VM can consume them as resources allow. They are no longer locked in a synchronous "handshake."
• Native Replayability: Since the stream is a log of facts, the entire state of the VM can be reconstructed at any time by simply replaying the stream. This makes debugging and decentralized consensus inherently easier.
• Unified Interface: Whether the VM is running in the same process via shared memory or across the globe via a socket, the interface remains a simple stream of bytes.

Engineering Efficiency with Ring Buffers

To ensure this streaming model outperforms traditional FFI, Yin.VM utilizes Shared Memory Ring Buffers. These are lock-free data structures that allow the host and the VM to communicate bidirectionally with virtually zero overhead. Two ring buffers handle the two directions: one for datoms flowing into the VM, one for datoms flowing out.

The Zero-Copy Workflow

By mapping a shared region of memory between the host (Clojure/JVM) and the guest (Rust/Yin.VM), the system achieves the "holy grail" of performance: Zero-Copy.

1. The host writes the raw binary representation of a Datom (e.g., five 64-bit integers) into a slot in the ring buffer.
2. The host increments a write cursor.
3. The Yin.VM engine, polling the cursor, sees the new data and reads it directly from the same memory address.

Because both sides agree on the binary layout of a Datom, there is no "translation" step. The data is written once and read once, bypassing the expensive overhead of traditional FFI calls.

┌────────────────────────────────────────────────────────────────┐
│                     Shared Memory Region                       │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Input Ring Buffer (Host → VM)                           │  │
│  │  ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐       │  │
│  │  │ D0  │ D1  │ D2  │ ... │     │     │     │     │       │  │
│  │  └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘       │  │
│  └──────────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Output Ring Buffer (VM → Host)                          │  │
│  │  ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐       │  │
│  │  │ D0  │ D1  │ D2  │ ... │     │     │     │     │       │  │
│  │  └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘       │  │
│  └──────────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────┘
      ↑          │                     │          ↑
reads │          │ writes        writes│          │ reads
      │          ↓                     ↓          │
 ┌────────────────────┐         ┌────────────────────┐
 │    Yin.VM (Rust)   │         │   Host (Clojure)   │
 │                    │         │  Bridge Dispatcher │
 └────────────────────┘         └────────────────────┘

The Bridge Dispatcher: Connecting the Brain to its Limbs

For a stream-based FFI to function, the Bridge Dispatcher must be "bolted onto" the native host environment before the VM starts processing.

In the world of Yin.VM, the VM is essentially a "brain" in a jar. It can think (execute logic) and remember (store datoms), but it has no "limbs" unless the native host provides them via the dispatcher.

Installation as Capability Registration

When you set up the native host (whether in Rust, C, or Clojure), you are performing a Capability Registration. You are telling the dispatcher: "If the VM asks for 'CalculateHash', run this specific native function."

1. Define Native Functions: You have your high-performance C/Rust functions ready.
2. Register with Dispatcher: You map a string or an ID (e.g., :op/sha256) to a native function pointer.
3. Launch VM and Dispatcher: Both start polling their respective ends of the ring buffers.

The Host as an Operating System

Think of the native host as an Operating System and Yin.VM as a User-space Process. A process cannot talk to the hardware directly; it must make a System Call.

• In a traditional OS, a system call triggers an interrupt that jumps to the kernel stack.
• In Yin.VM, a "system call" is simply writing a datom to the output stream.

The Bridge Dispatcher is essentially the Kernel's Syscall Handler. If you don't "install" the handler for a specific call, the VM can scream into the stream all it wants, but nothing will happen.

Dynamic vs. Static Dispatchers

Because Datom.world is designed to be decentralized, there are two ways to "install" these dispatchers:

Shibi Tokens: Capability-Gated Registration

In a decentralized network, who gets to install a dispatcher? You cannot simply allow any node to register arbitrary native functions. This is where Shibi tokens enter the picture.

To register a dispatcher, you must present a Shibi token specific to that dispatcher. Possession of the token is proof of capability. No token, no installation.

• Scoped Authority: Each Shibi token grants the right to install a specific dispatcher (e.g., a token for :crypto/sign cannot be used to install :sys/write-file).
• Transferable Capability: Shibi tokens can be transferred, delegated, or revoked. If a node loses its token, it loses the ability to provide that native capability.
• Auditable: Because Shibi transactions are recorded as datoms in DaoDB, the history of who installed what dispatcher is permanently auditable.

This use of Shibi tokens to gate native registration is a novel solution to the security risks of decentralized VMs. It effectively turns the "System Call" table into a cryptographic marketplace. A node cannot unilaterally expand its native capabilities; it must possess the specific "proof of right" to interact with host resources like the file system or network.

This maps the economic layer of Datom.world directly onto the security boundary of the VM. Capability and commerce become one.

Why This is a Security Sandbox

This "pre-installation" requirement is actually a massive security feature for decentralized architecture.

If a piece of malicious code is running inside Yin.VM, it cannot "guess" a function address or perform a buffer overflow on the native stack. It can only emit datoms. If the Bridge Dispatcher hasn't been explicitly configured to listen for a specific request, the request is simply ignored or logged as an error.

The Yin.VM Principle: A VM is only as powerful as the Dispatcher it is connected to.

The Dispatcher as a Stream Interpreter

In this architecture, you aren't just building a VM. You are building two distinct interpreters that form a symbiotic loop:

• Yin.VM: Interprets bytecode to produce datoms.
• Bridge Dispatcher: Interprets datoms to produce native effects.

By viewing the Bridge Dispatcher as a "Stream Interpreter," you move away from rigid API definitions and into a world of dynamic, capability-based execution.

The Dispatcher as a Logic Engine

Just as Yin.VM has an instruction pointer and a stack, the Bridge Dispatcher has a Pattern Matcher and a Registry.

When a datom enters the dispatcher, it isn't "executed" in the traditional sense. It is matched against a set of known patterns:

• If match found: Trigger the native function.
• If no match: Ignore, log, or route the datom to another node (the "Decentralized" part of the project).

The Benefits of Interpreting the FFI

Because the dispatcher is an interpreter, you gain several "superpowers" that traditional FFI lacks:

• Interception: You can inject a "Middleware" into the dispatcher that logs every native call for auditing without touching the VM's code.
• Virtualization: You can lie to the VM. If the VM asks for a production database (via a datom), the dispatcher can interpret that request and route it to a mock in-memory database instead.
• Throttling: The dispatcher can see a "stream" of requests and decide to slow them down or batch them if the native resource is under heavy load.

Structural Comparison: Bytecode vs. Datoms

The Language of the Stream

Since the Dispatcher is an interpreter, the "language" it speaks is the schema of your datoms. For Datom.world, this means the Attribute (A) in your datom acts as the "OpCode" for the dispatcher.

• If A = :sys/write-file, the dispatcher interprets this as a file system instruction.
• If A = :crypto/sign, it interprets it as a call to your Rust/C security library.

The Resulting Ecosystem

This architecture makes Yin.VM incredibly resilient. You can upgrade the "Stream Interpreter" (the Dispatcher) to add new native capabilities without ever having to recompile or change your Yin.VM bytecode. You are essentially building a plug-and-play hardware interface for your virtual logic.

Conclusion: A VM for a Decentralized Reality

By abandoning the imperative function call in favor of a fact-based stream, Yin.VM achieves a level of reactivity and auditability that traditional VMs cannot match. It aligns the low-level execution engine with the high-level philosophy of Datom.world: that data is a river of immutable truths, and computation is simply a way to navigate that flow.

The result is an architecture that is not only faster but more resilient: a necessary foundation for the future of decentralized computing.

Related Reading:

• Yin VM Documentation
• Datoms as Streams
• Stream vs Interface Architecture
• Plan 9, 9P, and DaoStream