underclock.ing

Harnessing agents in Erlang - Part 4a

2026-10-22

Welcome to part 4a of my agent harness in Erlang. I have had to split this into two parts due to an ever-growing length.

Thus far we have our system working on a multi-node setup. The database is shared across the nodes and requires a majority to write. It also has a mechanism for resuming crashed agents on a new node when the node dies. Sounds pretty good.

But there are serious problems with what we’ve built so far. The video showing our agent die and revive on a neighbouring node is interesting but it relies on a bunch of shortcuts to work. And it’s not exactly rigorous. The purpose of this chapter is to provide some rigour in the form of property tests.

Unlike the other posts, I have updated the code without writing about it. To summarise:

  1. A new llm behaviour which a provider will implement (e.g. openai, anthropic)
  2. The llm_openai and llm_fake modules which implement the behaviour.
  3. A new llm intermediate representation which provider messages are converted into. We had to delete our existing mnesia database as it stored openai format messages
  4. A property test for checking that our agent failover is working.

The point here is to stop relying on manual testing and subject our system to things we wouldn’t have thought to try.

What does the test do

I have leveraged PropEr, a property testing library in Erlang. If you are unfamiliar with property testing, it’s similar to fuzzing in that you randomly search the program space for violations of intended behaviour. When it finds a violation, the library will retry the same sequence omitting some of the commands until it can find a smaller sequence that violates the property. This makes the error more comprehensible. The catch of course is that you have to actually know the behaviours you want.

Our tests are using the statem feature in PropEr. What that lets us do is to define the set of actions that can occur in our program, the framework will randomly generate these and check that the conditions we specify hold. This is not a PropEr tutorial (this is though), so I won’t labour the details. If you just understand the following you’ll be able to follow:

  • Define the discrete events that can occur (commands)
  • Define what should be true before running them (preconditions)
  • Define what should be true after running them. (postconditions)
  • Define what should happen on said event.

What this amounts to is effectively building a reference implementation and checking that what we are testing matches that reference implementation at each generated step.

PropEr statem loop: choose a command, check its precondition, apply it to the model and call the test subject, then compare the expected result with the real one

PropEr’s statem loop, very roughly.

I won’t show all the details of the tests. The most important parts are:

command(#{up := Up, cut := Cut}) ->
    Down = [1, 2, 3] -- Up,
    frequency(
        [
            {3, {call, ?MODULE, start_agent, [elements(Up), elements(?IDS)]}},
            {5, {call, ?MODULE, message, [elements(Up), elements(?IDS)]}},
            {3, {call, ?MODULE, slow_tool, [elements(Up), elements(?IDS)]}},
            {3, {call, ?MODULE, pause, [elements([50, 200, 1000])]}}
        ] ++
            [{1, {call, ?MODULE, kill, [elements(Up)]}} || length(Up) > 1, not skipped(kill)] ++
            [{2, {call, ?MODULE, restart, [elements(Down)]}} || Down =/= [], Cut =:= none] ++
            [{1, {call, ?MODULE, isolate, [elements(Up)]}} || Cut =:= none] ++
            [{1, {call, ?MODULE, isolate_all, []}} || Cut =:= none, not skipped(isolate_all)] ++
            [{2, {call, ?MODULE, heal, []}} || Cut =/= none]
    ).

These are our commands. They operate on our cluster. We are currently testing that our agents fail over reliably: that is, in our current implementation, if their node dies a peer should revive them. The other thing to note is despite my lovingly hand-crafted diagram above we are not really using the incremental approach described there (Expected = Result?). Instead we are checking the bare minimum and leaving the important checks to the end. This is inspired by the approach Jepsen takes and, like Jepsen’s, our tests don’t run quickly.

If we run our prop_failover.erl tests we get a failure sequence of:

(6 time(s))
[{set,{var,5},{call,prop_failover,start_agent,[3,<<99>>]}},{set,{var,8},{call,prop_failover,start_agent,[1,<<99>>]}},{set,{var,11},{call,prop_failover,message,[3,<<99>>]}}]

Result: ok
State: #{up => [1,2,3],cut => none}

Steps:
  {call,prop_failover,start_agent,[3,<<"c">>]}
    -> {ok,<21765.193.0>}
  {call,prop_failover,start_agent,[1,<<"c">>]}
    -> {ok,<21763.258.0>}
  {call,prop_failover,message,[3,<<"c">>]}
    -> ok

Violations:
  {orphaned,<<"c">>,'peer-1828-1843434@archlinux',<21763.258.0>}
===>
0/1 properties passed, 1 failed
===> Failed test cases:
prop_failover:prop_failover() -> false

This is our failure but shrunk. This one is a remarkably terse sequence that violates our property. First, it’s probably easier to explain it in an unshrunken form.

  1. You start an agent{id: c} on node 3 with a slow tool
  2. You partition node 3
  3. Node 1 notices 3 is down and restarts agent{id: c}
  4. Node 3 reconnects
  5. Now there are two agents with id of c in the global registry, the registry will kill one at random
  6. If the one killed is the current owner (which should be node 1 after claiming), the agent on the wrong node will crash on first write to db
  7. Clean-up (freeing the ownership) does not run due to it being killed
  8. The agent is owned but not being run anywhere.

Our shrunken example tells us that our property breaks in merely 3 steps.

  1. Start an agent{id:c} on Node 3
  2. Start an agent{id:c} on Node 1
  3. Message “c” from node 3

There’s not even a partition in the shrunk one. At startup the nodes are connected, but global on each one hasn’t finished syncing with its peers, so each only checks its own view of the names. Both registrations succeed, when the name servers sync, they find the clash and kill one. So whenever two nodes don’t yet know about each other, through timing or a partition, we can get two agents with the same id, and the moment they find out, global may kill the wrong one. The survivor gets fenced on its first write and nothing revives it.

How could we solve it? Well the answer is obvious, we just need to tell global to kill the right process when it finds a duplicate. Erlang provides global:register_name/3 for this very purpose, you can pass a function to choose which Pid to kill. But instead of treating this like a nail to hammer we should step back for a moment.

The desired outcome here is killing the agent on the minority side. Our global resolver does it on reconnection, but we could just as easily kill the agent the moment its host realises it’s in the minority. So anytime a node is partitioned all its agents die.

Framed in this way it becomes clear that I had made a product decision that I didn’t really state. That it’s acceptable for a disconnected harness to be unable to do anything at all. Imagine that I had this agent cluster set up to run on my laptop, a cloud container and a home server. Do I really want my laptop’s harness to be unusable because Salesforce finally killed Heroku and Kiro went rogue and brought down AWS again? And the operational problem gets worse the more computers I use my harness on. To use a harness instance you need to be connected to more than half of the copies of mnesia. If I use the agent on two new computers, I’ve committed to keeping one more node running at all times.

What do we want to guarantee

It’s easy to get lost in technical details as an engineer. I myself fall victim to it all the time. It’s generally how I solve problems, a solution often just materialises in my head in the shower after chewing on the details for long enough.

Part of the motivation to build this in Erlang was to nerd out over technical details, it’s the point of this blog too. But even if this is merely an experimental system, the guiding principle behind the experimentation should still be making something that I actually want to use. And so it’s simply not acceptable that my laptop’s agent harness would cease working because the rest of the cluster is down.

This is a good time to make some decisions about what we want this system to do. Here are a few things I want:

  • A harness should be usable even when disconnected from the cluster
  • Reconnecting a partitioned instance of the harness should never lose data
  • Using a harness once shouldn’t commit us to forever running it
  • Agents that die should be revivable by peers when they die
  • Revived agents shouldn’t repeat work that isn’t safe to repeat

This all sounds good, but it’s not precise enough to test against. You might have already noticed an issue with some of these. A dead node, a partitioned node and a slow node are indistinguishable from one another. If we automatically restart agents on ‘dead’ nodes we will be duplicating work. As if wasting precious tokens wasn’t bad enough, these agents have side effects. For me at least, it’s the entire point of running them.

If we want to achieve the things on this list, we are going to have to be precise about our behaviour. Let’s go through them.

A harness should be usable even when disconnected from the cluster This means we must drop our majority setting on the tables. We must also figure out how to merge divergent histories. When two harnesses work on separate conversations it’s easy. When they’ve both touched the same one it isn’t, unless we change the data model. If you’ve ever used the tree function in Pi, you’ll know a conversation can be modelled as a DAG: every entry is immutable, has a unique key and points at its parent. Two divergent histories are then just two branches, and we can merge our databases by taking the union. Mnesia will not do this for us. We will have to do it ourselves.

Reconnecting a partitioned harness should never lose data What we really mean here is that after we merge A and B, everything that was in A or in B is in the result of their merge.

Using a harness once shouldn’t commit us to forever running it Nodes that never come back do not break the cluster. Under the majority setting of mnesia, creating a disc copy on two new nodes would mean adding another permanently running node, we don’t want this.

Agents that die should be revivable by peers Not just revivable in the sense that they can be revived. What we really mean is that the revival of an unresponsive agent should eventually complete.

Revived agents shouldn’t repeat work that isn’t safe to repeat This is the most difficult thing on the list to achieve, and it’s certainly not the case right now. It’s in direct conflict with our desire to restart failed agents. We need to redesign our tools to make this more achievable.

Many of these things are in tension with one another. If running a harness somewhere shouldn’t commit us to running it forever, then when it runs once and is partitioned we have to accept its tasks may never complete. Then there’s the tension between revival and tools. It’s the classic safety vs liveness dilemma: we want to prevent tools being run more than once, but we also want every agent to eventually complete. And we don’t want data loss, but if you use a new computer while the rest of the cluster is down and promptly brick it, its data never syncs to the hivemind. Does that count as data loss?

The reason we need to be precise is to identify what we mean by these words, where the tensions are, and then choose our guarantees and trade-offs carefully. Here are a few choices I’ve made:

Revival is opt-in. A conversation is local by default: it runs on the node that created it and nothing revives it automatically. If its node is gone we can resume it with the /resume command. Where revival is desired it can be opted into, whether the agents themselves can decide that via a tool or the user has to choose I haven’t decided, but those are the only ones we make revival promises about. This seems like the right behaviour to me. More often than not you’re just running local work which has no reason to be revived outside of the machine you’re running it on (although I do have plans in this direction too). But we still keep the ability to run these self-directed autonomous jobs that revive and fail over.

One agent per side of a partition. Guaranteeing a single agent across the whole cluster would clearly require strong consistency. Given we’ve just dropped that on our tables for good reason, we can’t have that. So when our cluster is partitioned there will be at most one agent running per side of the partition (and with three or more nodes there can be more than two sides). Their histories can merge as two branches at heal. I think we can potentially do something interesting here with the agents cooperatively choosing which should end based on progress too.

Sandbox the agent to its working directory. Bash runs in a sandbox that can only write to the conversation’s working directory and can’t use the network aside from loopback (bubblewrap, or unshare with a mount namespace and --net). A rerun can still change files in that directory, but nothing outside it. The agent still needs network access, so we’ll give it a tool called http_get, which we assume is idempotent. I am acutely aware that this isn’t actually true. A GET can trigger anything, but we have to draw the line to make this feasible. The other option is stripping query strings to limit data egress but it would break browsing.

The workspace doesn’t move. I had originally planned to use a git base commit plus a patch file of uncommitted changes as a quick and dirty way of moving work-in-progress between nodes. For now, resuming an agent moves the history only. The danger here is as articulated in part 3 - the agent has no idea where it is when resumed so we need to instruct it. I have a few ideas for the synchronisation of the workspace but it will be tackled later when the foundations are right.

Now the guarantees:

  1. Available. A node can start, message and resume any conversation whose history it holds, whether or not it’s connected to the rest of the cluster.
  2. No data loss. When nodes connect, their histories are merged without blocking either side. Once writes stop, every connected node eventually holds the same history. This history should be a superset of everything either side held.
  3. No commitment. A node that never returns never blocks anything: starting, messaging, resuming, or merging between the nodes that remain. Its unsynced data is lost forever.
  4. One agent per side. For every conversation and side of a partition there is at most one agent with a given id, unless every agent for that conversation on that side is healing. An agent is healing from the moment its side gains a node until the duplicates are resolved. Healing eventually ends with one agent, and the others stop cleanly.

#2 sounds simpler than it is. We have to sync the databases on connect without blocking, or we would break #1. So the merge has to occur while both sides keep writing. Using a DAG makes this possible. The entries never change and a union doesn’t care about order, so we should hopefully be able to sync piece by piece without locking anything.

#2 and #4 are both properties with “eventually”. Unfortunately, testing this is hard because we can’t run the test forever. So the practical approach is giving the system a generous time bound after writes have stopped in our test.

You might notice the guarantees are lacking anything about tool safety, revival or workspace synchronisation. I have split the guarantees into two groups. Every property that involves revival requires the functionality to run agents on partitioned peers, which requires ironing out our data model first. So I’ll do it in two steps. Making our cluster available, then tackling revivals after.

Our original test told us #4 was broken, it achieves one agent but it doesn’t heal them gracefully. #1, #2 and #3 are not met due to the majority setting on our tables.

Next time

In part 4b we will turn these guarantees into property tests. They’ll all be red, of course, and then we’ll turn them green. (This does not count as TDD.)