PLR's safety features

General
1

Hello everyone. I know that when coding in Venus, the compiler itself provides certain safety features, like not allowing the user to move channels into the deck.

I also know that PLR has some “warning features”, such as telling the developer when a plate seems to be too high, etc. But do we have a concrete list of PLR’s safety features? Or is there an easy way to put one together? (for example looking for all the logger.warning()).

And do these features just “warn” the devs in the logs, but still allow the robot to perform whatever action?

2

I am afraid the answer to both questions is no.

Here’s a list of things that come to mind:

  • Besides the warning for resources that are exceptionally high, some commands are marked “unsafe”. E.g. the backend_kwarg use_unsafe_hotel of STARBackend.pick_up_resource (this method extends the iswap outside of the chasis).
  • Another one is STARBackend.move_channel_y has a check to make sure you don’t crash into other channels. It raises an error if that would happen.
  • With tip tracking enabled, we raise errors when we think you’re about to drop a tip in a location where we think a tip already exist
  • Similar for picking up a tip with a channel that already has one
  • There is a check for colliding resources on the Deck level

is there a specific issue you have experienced in the past that can be avoided with nice checks?

2 Likes
3

oh, another huge one: every star backend method implementing a specific firmware command checks if all values are within range

2 Likes
4

I can’t list many behaviors that I’ve experienced in the past since Venus runs these checks for me and ideally shields me from these issues, but here’s a non comprehensive list of what Venus doesn’t allow me to do with their safety checks:

  • liquid handling: all checks related to ensuring we never over aspirate liquids into the head or channel
  • head, channels and iSwap movements: all checks related to ensuring those parts don’t crash into each other or go off bound
  • movements of head and channels with tip mounting: all checks related to ensuring tips don’t crash into deck
  • tip movements: checks related to how we mount tips, how we eject them, how aware the software is of when and what tips are mounted (as you’ve said this is already present)
  • motor jams, stalls, boundaries, initializations: if we are going down to firmware commands, these kind of checks may be relevant. What happens if a labware definition is wrong and we try to move on the x axys beyond the real dimensions of the deck (let’s say at position 2000)? Along the same lines, checks around feedback and sensors: for example, with Venus, we get an error if we try pushing too hard on a labware placement. Does PLR have similar checks around motor resistance?
5

done when volume tracking is enabled (this can be kind of annoying tbh)

checked using the firmware bounds, both on our side and the star side, which may or may not be the exact bounds the deck supports. it essentially will give a firmware error Position Not Reachable when you try to got to a location it can’t access

for tip presence, this is checked by PLR with tip tracking enabled and by the firmware in any case

regarding crashing into the deck: I am not sure if we check, it would be easy to add a check, but i have never run into a situation where i accidentally told it to go into the deck. I’m not even sure if the channels go down so far (maybe with 1000uL tips)

see above

the hardware reports errors very well. it’s actually surprisingly hard to break it even if you wanted to. worst case a drive gets locked or jammed and you have to reinitialize it. This can be done using setup or by sending individual initialization commands to different components.

2 Likes
6

Great! Thank you very much

1 Like
7

I completely agree!

Many people I talked with who use VENUS but don’t know how it works in the background (and maybe have heard some hearsay about PLR :ear: ) say that they believe “one cannot break the machine in VENUS, but with PLR you can”.

Then I …

  1. open their VENUS, layout editor, click on a labware on their deck, go to probing, and move their teaching needle laterally into the cable duct at the back of the machine, breaking their channel head off their channel → VENUS-based destruction in 1 minute … just kidding, I stop right before the duct :eyes:
  2. ask them how often they’ve crashed something in VENUS

Most safety features, in general, are really the product of Hamilton’s amazing embedded software engineers → the people who write the firmware for their machines!

The firmware is actually incredibly smart on the STAR, e.g. if you are interested in what a STAR’s aspirate firmware command checks, you can find the information here:

And the firmware is catching human mistakes written into commands no matter how you call it: PLR, VENUS, PyHamilton, PyVENUS, … the firmware is what matters the most in my opinion :slight_smile:

2 Likes
8

Reading through the comments, there are 3 main concerns:

Crashing Channels (or something else) with the labware on the deck

This is mainly an issue if the labware definition is wrong, or the definition is correct but the user placed the wrong stuff on the deck. As the STAR (and the Vantage) always retract to traversal height before moving in x-/y-direction a lot of disasterous crashes are prevented. If the traversal height is correctly used (@rickwierenga / @CamilloMoschner can provide more insight on this), then using PLR is no more dangerous than using VENUS. Crashes in Z-direction, while crushing a tip, generally do not damage the channel unless you do them over and over again.

However, there must be given one big warning: The lower level the FW-Commands are, the less checks are performed on the FW-side. Everything exposed through the high level API (LiquidHandler, the tip pickup, discard and asp / disp methods) work on the Arm level and not individual channels, so they are safe.

Crashes between Channels and other components (like iSwap)

While the Arm manages the positions of the channels mounted on the arm, there are no collision avoidance checks on the firmware side for other components. This is implemented in VENUS and must currently be done manually in PLR. So in short: Crashes between channels are prevented, crashes between channels and other components are not. The same applies for multi-arm setups: No collision avoidance there!

Channel flooding, eg. by aspirating 200ul in a 10ul Tip

If implemted correctly (by PLR), channel flooding is prevented by the firmware as the volume in the tip is also tracked by the firmware. This works by giving the tip type to the firmware. There are some caveats, like 300ul tips and 50ul tips which cannot be distinguished by the firmware during pickup, so again: If the user places the correct labware / tips on the deck, a lot of safety is given by the firmware.

To check if the implementation is correct, the following quick test can be done:

  • Pickup a tip, lets say a 10ul tip
  • Try to perform a air aspiration for 100ul
  • If you get a firmware error, the PLR implementation works as expected.
  • Note: Most tips allow for more volume than the nominal given volume, so a considerably larger volume should be used when performing this test.
2 Likes
9

By “volume tracking” you mean the one of PLR or the one of the STAR / Vantage FW? Both can be annoying :sweat_smile:

I’m not 100% sure if all bounds on the firmware side are actually the hardware limitations because the STAR / Vantage comes in different configurations with different mechanical bounds. But given your deck-layout definition is correct, you should actually never move outside the deck by accident.

Regarding motor jams / stalls: Most FW-commands provide a way to limit the current / force applied. Don’t know if those limits are written by PLR, but in any case, a limitation is active and errors will be reported.

In my opinion:

  • If the deck layout is correct
  • And the user places all labware correctly
  • And crashes are reported (so those cases can be analyzed and improved / fixed)
  • Then, PLR is almost as safe to use as VENUS. VENUS is safer because it’s running since many years, so all the issues have been fixed, but there is no reason why PLR cannot be as safe as VENUS in the long run.
2 Likes
10

“correctly” is dependent on your specific deck layout.

We default to

  • 280.0 mm for iswap
  • 245.0 mm for PIP channels (this is max with 1000 uL channels)

You can adjust this by setting

or even this:

with star_backend.iswap_minimum_traversal_height(custom):
  await lh.move_plate(...)

All methods a user would call through the front end on the STARBackend are marked @need_iswap_parked: this means the iswap will park before the channels move. The methods (again, mostly used through LH but also callable directly):

  • move_channel_y
  • pick_up_tip
  • discard_tip
  • aspirate_pip
  • dispense_pip
  • get_core
  • put_core
  • core_get_plate
  • core_put_plate
  • core_move_plate_to_position
  • move_all_pipetting_channels_to_defined_position
  • position_max_free_y_for_n
  • pick_up_tips_core96
  • discard_tips_core96
  • aspirate_core_96
  • dispense_core_96
  • position_channels_in_y_direction

Feel free to add the decorator to other methods.

Note: while not important for collision detection I wanted to mention it here: the 96 head will always park when doing an operation using either the iswap or channels. This seems to be a firmware level behavior.

We don’t support this yet. (Would it be module C1??)

Implemented in PLR.

We actually give the full volume, I think. E.g. HTF = 1065 uL., HT = 1250. See pylabrobot/pylabrobot/resources/hamilton/tip_creators.py at main · PyLabRobot/pylabrobot · GitHub.

Should this be the “nominal” volume e.g. 1000 for H?

1 Like
11

It depends what you are doing with this volume. The firmware (at least on Vantage) has a Tip-Type table, so generally you only specify the tip type (i.e. 300ul filter tip, I’m not shure if this maps to the TipSize enum in PLR), the rest is already present in the FW, so the volume used by PLR is not realy relevant.