Hello,
I’ve been reading the earlier error handling discussion while working on #1206 and revisiting #309.
I noticed PLR already handles uncertain hardware state in a few different ways:
#886 checks tip presence after a failed pickup/drop to reconcile software state with the hardware.
#944 / #983 handle a CoRe 96-head case where firmware reports an error while the physical operation is still running.
HighResSampleStorage keeps an UnresolvedPlateTransfer when the physical result of a move cannot be established, and requires explicit resolution before continuing.
The Tecan Infinite backend raises errors stating the action may have started and the device state is indeterminate, and deliberately does not reinitialize in that state.
Four devices, three vendors, and roughly the same shape each time. That made me wonder whether there is a more general concept here.
For example, if communication is lost after a dispense command has been sent, PLR may not know whether the dispense:
never started,
completed,
or partially completed.
Rolling the tracker back makes sense transactionally, but it doesn’t mean the physical action didn’t happen. Retrying immediately could therefore repeat an effect that already occurred.
So my question is:
Should PLR have a generic way to represent “physical outcome unknown / reconciliation required” before retrying or continuing, or are the existing device-specific approaches the right abstraction?
If this is worth generalizing, the layer is the open question for me — driver-level (each backend exposes whether an outcome is determinable) or front-end-level (something the error handling API can branch on). Especially with the current restructuring, I’d rather ask than guess.
Separately, for anyone running long unattended protocols: has this actually bitten you — a command lost mid-flight, and no clean way to tell what the machine had already done? I’m curious what people do today, since I assume the answer is often “stop the run and inspect the deck”.
If this seems worth generalizing, I’d be happy to write up a small design proposal and take a first pass at a PoC.
yes definitely . syncing the state between hardware and software is one of the most important (and difficult) parts of PLR.
these concepts don’t exist anymore on main/v1. “interface classes” are now device specific
where possible query it from the machine, for example hamilton has a way to see if tips are mounted. OT does not.
while the concept is general, I think much of it is device-specific and really depends on what a specific device exposes…
couple of thoughts:
the python layer should do as much as possible to keep hard/software in sync
many machines are super dumb (few (accessible) sensors) and we need to keep track of state client-side
we should be more explicit with defining state machines. perhaps we can use transitions at some point
another part of the solution will be isolating driver processes from user processes. here is a draft for that for the liconic and we will want to create servers like this for all devices in PLR eventually. Those servers add an additional layer of idempotency.
Thanks @rickwierenga. Two things from the operator’s side made the distinction clearer to me, from here
The validated policy in IVD-regulated assays is to abort on hardware errors rather than recover automatically, since the error usually points to a root cause someone has to fix. Separately, when aspiration channels fail, the affected ones get flagged, their tips discarded, and fresh tips picked up, without ever working out what those channels physically did.
So two ways to re-establish state:
query the device where it answers reliably,
or discard the uncertain resource and resume from a state that’s known by construction.
The second is the one that still works on the “super dumb” machines you mentioned: it asks about the resource model, which PLR owns, rather than about firmware, which it doesn’t.
abort is just one possible action/state transition in error handling
that is not always possible or safe to do. Sometimes the problem is the PLR model is out of sync so it’s not a reliable source of information to query, and it’s also possible the machine is in an error state where discarding something is not safe. (for example, a channel just badly crashed on a star during pickup, if you call discard_tips you might be moving a bent channel across the deck)
the best is still querying the device. Using the PLR model works well in non-error scenarios, you can find countless examples of PLR keeping track of state in the codebase. But I think many of them should be expanded with a None/unkown&error state probably