Intro to Epic: Tame the iSWAP

dev discussion
1

Hi everyone,

This is a big one:

The internal Swivel Arm Plate Handler (iSWAP) is one of Hamilton STAR(let)'s most powerful device Capabilities.

It enables…

  • movement of resources with force-sensed feedback for secure pickup, transfer and placement
  • movement to off-deck sites due to its swivel/SCARA design → ~90 mm out of the main chassis to the left, and ~20 mm out to the right
  • instantly enables horizontal integration of other devices into a workcell, i.e. without the need for additional arms

Depending on the model and firmware version it can pick up plates in either just portrait mode or portrait and landscape mode.

However, even in the OEM software the iSWAP is infamous for being a bit difficult to use.

Since we have total control over the devices we own with PyLabRobot and like to make the difficult parts of our automation life easy, I started an epic (long-term project with multiple sub-components) to make the iSWAP smarter and easier to use.

I call this epic “Tame the iSWAP”.

In this thread I would like to showcase some of the current issues with the iSWAP and how we are openly developing it into a more advanced resource transfer system:

Background: iSWAP Geometry

Fundamentally the iSWAP has 3 components:

  1. A Cartesian robot - moves the base/rotation drive/θ₁ of the iSWAP in…
    • x` (via the X-arm motor),
    • `y` (via the iSWAP channel y-motor), and
    • `z` (via the iSWAP channel z-motor).
  2. A SCARA robot
    • 2 joints - representing…
      • the “rotation drive” θ₁
      • the “wrist drive” θ₂
        and
    • 2 linkages
      • linkage_1: rotation drive ↔ wrist drive
      • linkage_2: wrist drive ↔ gripper center
  3. A Gripper with a linear motor

**More details on the SCARA parts**:

Each joint has its own specific range of motion and associated with their specific values an Orientation *with regard to* (`wrt`) the STAR coordinate system:

  • Joint 1: θ₁ has 3 “standard” yaw angles (Orientation around the Z-axis)

    • -90° / ~ -29_000 increments → LEFT (`wrt` the STAR)
    • 0° / ~ 0 increments → FRONT
    • 90° / ~ 29_000 increments → RIGHT

  • Joint 2: θ₂ has 4 “standard” yaw angles (Orientation around the Z-axis)

    • -135° / ≈ -26_577 increments → RIGHT
    • -45° / ≈ -8_860 increments → STRAIGHT
    • +45° / ≈ +9_044 increments → LEFT
    • +135° / ≈ +26_858 increments → REVERSE

260502_Moschner_iSWAP_joint_orientations_overview
260502_Moschner_iSWAP_joint_orientations_overview1500×1150 214 KB

This means there are 3 x 4 = 12 “standard” iSWAP-SCARA poses.

Moving from any given pose to any other therefore gives 12 x 11 = 132 nontrivial standard movement patterns.

Note: there is no reason why the hardware-firmware would not be able to move to any “non-standard” pose, i.e. arbitrary joint configurations (e.g. θ₁=-20°,θ₂=100°).

Current Issues

What is the problem?

The standard commands we currently have in PLR to use the iSWAP are based on the `C0` command module.

It is too (inconsistently) smart

When using our current `.iswap_get_plate()` / `C0PP` or `.move_resource()` / `C0PM` commands we only tell the commands the “grip direction” the gripper should have with regard to (`wrt`) the STAR’s entire coordinate system:

i.e. when saying `GripperDirection=Back` the gripper fingers point to the front, and the gripper is positioned behind the resource being picked up.

But … what is the angle of the two joints? :eyes:

With `GripperDirection=Back` there are 3 different joint configurations which can achieve this GripDirection:

  • θ₁=LEFT, θ₂=LEFT
  • θ₁=FRONT, θ₂=STRAIGHT
  • θ₁=RIGHT, θ₂=RIGHT

260502_Moschner_iSWAP_GripDirection_BACK_options
260502_Moschner_iSWAP_GripDirection_BACK_options1581×1035 169 KB

Which one is going to be chosen? … we don’t know - because the firmware makes this decision (with these ‘smart’ commands).

But the firmware has no deck awareness which means it might decide to orient the joints in a manner that crashes into another resource (e.g. a tip carrier next to our plate)… and we have no knowledge of when and why it would happen.

i.e. the currently exposed iSWAP movement commands are a black-box decision maker.

(Note: some movements are only possible in a specific joint configuration. e.g.: moving out of the main chassis to either side requires the linkages to straighten completely in that direction.

As a result, and confusingly, some commands do behave as we expect, clouding expected behaviour with position-dependency)

Firmware doesn’t cover all safety/risk scenarios

PLR already exposed some low-level commands to control some motors of the iSWAP assembly:

e.g. we can rotate joint_1 and joint_2 independently.

However, these commands do not perform a check whether…

  1. all channels are out of the way (i.e. are they out of x-y reach for the upcoming iSWAP movement → side-crash risk), nor
  2. out of z-height (i.e. whether the channels are raised so the iSWAP can move below → side-crash risk), nor
  3. the channels are out of z-height but have tips mounted which bring the tips into the path of the iSWAP movement → channel stop-disk beheading risk

That is dangerous.

Potential Firmware Issues

It appears that everyone I know uses an iSWAP control board from 2012 - not an issue by itself (and the iSWAP is being replaced by the Internal Plate Gripper (IPG) in current STAR releases) but iSWAP-equipped STARs will be in service for years and I found some issues at the firmware level:

As part of the “Tame the iSWAP” project, I looked carefully into the joint angles of each movement.

When parked/just initialised → the iSWAP grip center position returned by the firmware doesn’t match the actual grip center:

0_park_breaks_firmware
0_park_breaks_firmware830×502 156 KB

This shows that the parking position overwrites the firmware return value for the iSWAP grip center to the center_x-center_y position of the joint_1 / rotation drive.

-> That is off by 32.8 mm in x and 138 mm in y :anxious_face_with_sweat:

Interestingly, iSWAP movement commands in `x`, `y`, or combined joint rotation do *not* update the iSWAP grip center either!

1_simple_movements_to_not_recover_sate
1_simple_movements_to_not_recover_sate1169×676 263 KB

This looks like it could be a firmware bug, though I can’t rule out that I’m missing context.

It indicates a crucial find:

`STARBackend.request_iswap_position()` / `C0QG` does *not* appear to perform a forward kinematics calculation of where the grip center is based on queried motor encoder state.

It seems likely to me that only certain firmware commands actually update an internal tracker of the state of the iSWAP instead.

For any form of more complex/advanced iSWAP behaviour we must have a command that reliably returns the actual grip center, no matter what commands have been executed beforehand.

Why do I say that only certain commands appear to update the firmware-internal tracker system:

2_single_drive_rotation_recovers_firmware
2_single_drive_rotation_recovers_firmware835×899 288 KB

From the lowest-level commands, only single angle rotations instantly update the tracker to the correct position.

i.e. the firmware commands inconsistently update the internal tracker.

Should we inform Hamilton directly? Considerations:

  • OEMs don’t like know-it-alls :sweat_smile:
  • Lab automation OEMs do not provide an incentive to do so (no bug bounties in this industry)
  • Maybe we are wrong / don’t know the full picture, maybe they are aware and don’t consider it obstructive to most users’ needs
  • It appears this bug has been around for 14 years … it might be that nobody has detected it because the thoroughness of PyLabRobot has not existed for most of that time period, or if detected a cost/benefit analysis might have concluded not to update but instead to build a new “Internal Plate Gripper” instead

…these are all speculations; please let me know what you think is happening here

What is happening?

Via a series of PRs I’ve created a plan to engineer us out of this dilemma:

  1. Expose all position request commands, for all reference points.
  2. Replace (presumably) broken firmware `park_iswap` with PLR-based transparent primitive and update `setup_iswap` with it.
  3. Build forward kinematics model to always calculate the true position of the grip center.
  4. Add advanced safety features
  5. Build new commands from primitives which have well-defined default joint behaviour (e.g. always right-back) where possible, but log a warning if user hasn’t specified 2 intuitive parameters: joint_1 orientation AND grip direction (*both* now `wrt` STAR deck!).
  6. Build (maybe simplified) inverse kinematics model that gives granular control over grip center speed and acceleration (with caveats accepted where the firmware does not expose the required granularity) - i.e. no more “violent spinning” + speed control for anti-splashing feature during transfer.

The current list of PRs in this epic:

This means we have just reached step 3 and 4.

What’s next?

We continue to move on to the next steps.

We wanted to inform the community about this big project and the new stories this will enable, e.g.:

  • “iSWAP never crashes into the STAR anymore”
  • “liquid doesn’t splash out of plate when moved via iSWAP”

…

future stories this enables us to build:

  • “PLR warns us that iSWAP would crash into resources like carriers, other plates”,
  • “PLR’s simulator keeps complete track of the iSWAP state, including position and orientation ..”
  • “… enabling auto-computation of fastest movement patterns”
  • “… enabling automated ‘avoidance zones’ (e.g. to avoid moving over specific plates without having to declare specific paths)” :smiling_face_with_sunglasses:

Call to action

Please review the new iSWAP code, if you have improvement ideas please let us know or submit a PR.

Even though this explainer is already quite long, a lot of detail has been omitted for clarity.

Please ask questions. The more people understand this upcoming PLR feature the faster we can build it and build it even better.

Happy automation :mechanical_arm:

4 Likes
2

teaser of what is coming…

_IK_teaser

1 Like
3

Part of the iSWAP peculiarity is how alignment is handled:

adjust_iswap.mcr is the Hamilton macro (run via STARService.exe) that uses the iSWAP for conductive probing to map the deck, determine location for the calibration block, set 2 SCARA rotational offsets, and write XYZ cartesian offsets for every unique pose straight into the STAR memory.

I suspect it’s hard to turn those pose-based offsets into continuously variable ones along the full rotational path (you would need to know which of the 2 arms is out of spec), so the firmware just adds the stored offset when it hits each discrete pose (what STARBackend.request_iswap_position() / C0QG returns).

Those pose-based offsets matter if you want the iSWAP to share the same coordinate system as the pipettes for accurate pickup/dropoff at internal deck positions. They are irrelevant for external hotel locations. A fresh iSWAP still needs 1-2 mm offset per pose to drop a plate successfully. Once programmed into the STAR memory they are essentially hidden from the user. Every arm eventually gets crashed (unless PyLabRobot saves us), which adds a few more mm per pose. Once it’s badly bent, adjust_iswap.mcr can’t set offsets anymore.

We should steal an idea from the PreciseFlex400: teach positions empirically by the user. Interactive teaching via firmware (or by hijacking the existing conductive probing pins on every iSWAP) would work. A proper inverse kinematics model makes it easy: arm gets close, you fine-tune the last 10 mm. Combine that with PyLabRobot scripts for quickly teaching external hotels/nonstandard poses and it becomes as straightforward and slightly more time-consuming to teach than a PF400.

1 Like
4

My plan was to implement exactly this using interpolation between the calibrated positions :slight_smile:

i.e. my calibrated RotationDrive data:

{'home': 13000,
 'left': -29007,
 'front': 156,
 'right': 29068,
 'parking': 29500,
 'extra_1': 29068,
 'extra_2': 29068,
 'extra_3': 29068,
 'extra_4': 29068}

now we can use distance between “left: and “front” / 90 degrees to compute the actual angle forward and backward for a specific pose.

… :eyes: :joy:

Interesting, doesn’t the iSWAP need that fancy extension to perform probing?

Or can it probe even without it?

1 Like
5

The calibration kit costs $2.5k, which is very reasonable. If necessary we can use the same pins the service tool uses to engineer something more functional for taming the iSWAP with conductive probing. It is a very simple circuit.

3 Likes
6

Part 11: Expose iSWAP rotation drive request/set angle by rickwierenga · Pull Request #1022 · PyLabRobot/pylabrobot · GitHub

Part 12: Expose `STARBackend.iswap_wrist_drive_{request,rotate_to}_angle()` by rickwierenga · Pull Request #1037 · PyLabRobot/pylabrobot · GitHub

7

Part 13: Fix `STARBackend.iswap_close_gripper` defaults so it works without args by rickwierenga · Pull Request #1038 · PyLabRobot/pylabrobot · GitHub
Argless iswap_close_gripper() always raised firmware err 32 (C0 GC rejects gb=0000/gt=00). Defaults are now plate_width=86.0 / tolerance=2.0 mm, with Python-side assertions (>76.0, >=0.5) that fail fast before the firmware does.

Part 14: Add FK-based `STARBackend.iswap_request_pose()` + `iswap_request_joint_state()` by BioCam · Pull Request #1041 · PyLabRobot/pylabrobot · GitHub
FK-based grip-center accessor replacing the empirically broken firmware command C0 QG (request_iswap_position, wrong in 9/13 states, up to 255 mm off).
New iswap_request_pose() / iswap_request_joint_state() read joint encoders + per-machine EEPROM, with no firmware-state dependency - fixing a 15 year old OEM issue.
Legacy path untouched.

Part 15: https://github.com/PyLabRobot/pylabrobot/pull/1043
Bundles the 7 scattered _iswap_* setup attributes into one guarded iSWAPInformation dataclass (mirrors Head96Information/MachineConfiguration), plus a chatterbox default. fw_version now loads eagerly and get_iswap_version() raises pre-setup.

1 Like
8

Part 16: Expose joint-coupled iSWAP rotation: `STARBackend.iswap_rotate_to_angles()` by BioCam · Pull Request #1049 · PyLabRobot/pylabrobot · GitHub
iswap_rotate_to_angles(rotation_angle, wrist_angle) rotates both joints in one move.
Either arg optional (omitted axis pinned).
Float angles snap to calibrated stops if close enough to predefined position (to ensure exact increments stored in calibration are met).
Single-axis rotators (#1022, #1037) now private (and meant to only be used for troubleshooting)!

Part 17: Reroute `STARBackend.iswap_rotation_drive_request_z` to R0 RZ by BioCam · Pull Request #1051 · PyLabRobot/pylabrobot · GitHub
iswap_rotation_drive_request_z reads R0 RZ direct on the iSWAP slave, dropping the C0 QG dependency, which misreports XY in certain robot states.

2 Likes
9

Part 18: Refactor `STARBackend.move_iswap_y` for smooth motion by BioCam · Pull Request #1052 · PyLabRobot/pylabrobot · GitHub
move_iswap_y now issues one continuous absolute Y move instead of splitting into stepped relative moves.
It reads the gripper Y from forward kinematics, which is correct in all states including parked (the old path could land the gripper up to ~195 mm off when parked), exposes the speed/acceleration/make_space parameters, and inherits the underlying Y-range andchannel-collision checks.

Part 19: Refactor `STARBackend.move_iswap_x` and `move_iswap_z` for smooth motion by BioCam · Pull Request #1053 · PyLabRobot/pylabrobot · GitHub
Same smooth-motion refactor for move_iswap_x and move_iswap_z: one absolute move via forward kinematics rather than stepped relative moves.
Corrects the parked-state landing bug on X, adds caller-controllable motion parameters, and is state-independent.
Nowrequest_iswap_position has no remaining in-repo callers, i.e. is ready for removal in v1.

10

Cool work Camillo! I just noticed this thread now, after having struggled with the iSWAP for a few days based on I believe up to and including Part 12 of your epic. In that process, I also crashed it a few times unfortunately. Have you had thoughts about collision checking already? I know that PLR tends to play in the “we do everything inhouse”-corner, but I think here it might make sense to simply integrate with proven python libraries, e.g. PyBullet. One thing that I’m thinking about which would work well for our use-cases, would be not to do the collision detection at all in the realtime control backend, but instead in a simulator backend: We anyway schedule our runs first on a simulator before handing them to the real machine. Making that simulator backend pass through commands to an optional second backend would allow to opt-in for realtime collision checking by swapping the normal backend for the collision-checking simulator forwarding to the real backend.

2 Likes
11

Hi @burnpanck,
I think we have all been in the “it crashes” stage - a lot more in VENUS in my distant memory … and I am rather done with it, hence this epic :sweat_smile:

i.e. a big motivator behind this epic is the collision prevention and in general device safety aspect: with the latest PRs mentioned above this has now started.
But so far we needed to first fix the (1) lack of control, (2) lack of information, (3) firmware bugs by working around them in software/PLR, and enable a full kinematics model.

Why build in-PLR solutions:

  • We are already building the kinematics models across all PLR-integrated arms, including for Brooks and PAA arms, i.e. the kinematics model for the iSWAP is just one of many that will all follow the same nomenclature and architecture standards.
    A key component will be the full integration of the resource model :eyes:
  • Most users are using real-time development, i.e. notebooks, because it is so much faster to develop this way. This is a key feature of PLR.
    This means real-time checks are an essential requirement for safe, collision-preventative development.
  • This collision-prevention will ultimately happen inside PLR’s resource model, i.e. can be run in simulation (currently chatterbox) and execution the same way.
    The current issue is simply state tracking of what the devices actually do due to black-box firmware behaviour for many commands.
    This means we have to split these commands first into known, deterministic commands to enable real simulation.
    (another example: as part of the “Ultra-instinct Pipetting epic” we are splitting the STAR’s firmware logic behind aspirate/dispense, which decides in which order to process a sequence of positions, into the new plan_batches positioning algorithm, giving the resource model real information of what is knowingly going to happen once a command build on top of it is being executed)
    I am not sure how a different simulator would address this issue as it is based on an intrinsic lack of information of what the device will though.
  • I am not familiar with PyBullet but it looks interesting.
    Generally though, PyLabRobot has to stay very slim in terms of dependencies for wide compatibility and for maintainability reasons.

That being said, anyone can build their own simulator on top of the anti-crash prevention features that are in PLR.

The general paradigm of predictability has to be maintained:
e.g. the updated .move_iswap_y() method already raises an error if a channel is in its way but now enables via make_space = True the ability to automatically move these channels out of the way if explicitly told by the programmer.

In either scenario, we must guarantee that the device is always safe, which this epic is working towards :slight_smile:

12

more accurately, the model is “unassuming users should never crash their robot, but assuming users should be able to acknowledge risk and do whatever they want” (for example through functions and parameters prefixed unsafe_)

why not build the collision checking into the real backend directly? (the logic should always run, and be shared between execution and simulation)

1 Like
13

That’s a valid design decision, and I’d be happy with that. However, such a collision detection would always rely on the actual physical model available to PLR, and it’s very hard for these models to be 100 % accurate. Once you acknowledge that you can’t do that, for a safety feature, you’d want to err on the conservative side. But for a user who clearly sees that the movement they had in mind doesn’t crash being told by PLR that it might crash can be annoying, because it prevents them to do something that was easy before the inclusion of safety features. That’s where you need those unsafe (or ignore_collision=) escape hatches, but that is a breaking change. In code bases with a lot backwards compatibility concerns, there is often resistance to include new safety features that err on the conservative side, because it breaks valid use-cases that were working before, and obviously “we were fine without helmets and safety goggles before, weren’t we?”. If we don’t need to cater for that mentality, all the better.

1 Like
14

I actually disagree with the maintainability argument. Nowadays, dependency management is very easy to do well in python, and many big and popular libraries prove that they don’t have to implement everything on their own to be maintainable. In fact, the more external surface a particular component has, the more likely it is that maintainability decreases when moving to internal dependencies, because you end up reinventing the wheel rather than relying on community expertise.

The compatibility argument usually doesn’t hold up either, because most python libraries are extremely widely compatible. In fact, the hardware interfaces that PLR uses are often among the most incompatible dependencies you’d find. That said, PyBullet does depend on a pretty large compiled backend, and indeed, a there seems to be a wide gap of wheels available on PyPI for it. So in this case, having PyBullet as a hard dependency would likely be unacceptable. (I had a gut feeling that we wouldn’t want a hard dependency on PyBullet even before checking PyPI, and that was perhaps the other reason I suggested implementing it in a opt-in simulator backend).

15

I see your point, and I am intrigued how PyBullet or another dependency would work with the variety of lab automation arms, channels, heads, resources all together to build a real simulation.

But ultimately we must support real-time safety and creating two anti-collision solvers (one for on-device real-time execution and another for pure simulation) seems like duplication?

Also, currently we cannot even simulate x-arm movement because we haven’t yet figured out how to ask different STAR(let)’s what the size of their x-arm is (i.e. light-arm, half-arm, full-arm)… which dictates everything the STAR(let) can reach.

I don’t understand: If the information is hard to be accurate on-device, how would it be more accurate in simulation? Maybe I am misunderstanding something.

16

But ultimately we must support real-time safety and creating two anti-collision solvers (one for on-device real-time execution and another for pure simulation) seems like duplication?

I completely agree.

I don’t understand: If the information is hard to be accurate on-device, how would it be more accurate in simulation? Maybe I am misunderstanding something.

It wouldn’t. The difference is in mindset/expectation towards the feature. For simulation, we’re used to the sim-to-real gap. It’s ok for sim to be overly conservative, indicating “we can’t exclude the possibility that this operation might lead to a collision somewhere”. On the other hand, for realtime safety, erring on either side of correct is bad: If too lenient, the feature has little use, because it provides no dependable safety. Erring on the conservative side has the problem of breaking valid use-cases. Having an unsafe_ escape hatch helps a lot there, but if its disruptive in that it still requires working around “wrong library behaviour” within the user protocol.

Anyway, I’m all for having realtime safety implemented, and then of course just one implementation for both real hardware and simulation paths.

currently we cannot even simulate x-arm movement because we haven’t yet figured out how to ask different STAR(let)’s what the size of their x-arm is (i.e. light-arm, half-arm, full-arm)…

My pragmatic solution to this would be: Let the user give that information in the constructor (we can still change the meaning of None from unknown to autodetect later). Simulations anyway need that information. Here, I would actually prefer if this information (and the collision model) would live in the frontend instead of the backend, given that the collision model is part of the system model that PLR uses, not the backend specifics. I anticipate that this will become easier to do in v1 (though I haven’t looked into that one in detail). One use-case where this is important is this: To take a state snapshot from the real hardware, clone it and replace the hardware with simulators, and then simulate a trace starting from that snapshot. Today, this is a bit of a hack, because we don’t have API’s to access frontend state separately from backend state, but the idea is that front-end state should map 1:1 between real hardware and simulation, whereas backend state needs custom translation, and will always be an approximation based on the simulator’s capability (the real hardware is correct by definition).

17

Part 20: `STARBackend`: consolidate iSWAP drive parameters into `iSWAPInformation` by BioCam · Pull Request #1055 · PyLabRobot/pylabrobot · GitHub
Moves the iSWAP per-drive device facts - increment ranges, encoder resolutions, speed/acceleration ranges - out of scattered STARBackend class constants and into iSWAPInformation, alongside the per-machine setup state that Part 15 (#1043) already consolidated there. Behaviour-equivalent by construction.
Validators and unit converters now read from the dataclass instance, so a future hardware generation needs a different iSWAPInformation rather than scattered edits across the backend. The 16 superseded class constants are retained for one release as a deprecation window.

Part 21: Expose `STARBackend.measure_iswap_gripper_force()` by BioCam · Pull Request #1007 · PyLabRobot/pylabrobot · GitHub
GitHub Adds measure_iswap_gripper_force(), returning the force currently measured by the iSWAP gripper in Newtons.