TimeoutError

Does this seem like a reasonable implementation? I’d test it, but I don’t know how to reliably induce a timeout error, and I’d rather not the first run of this code be during a real protocol :') I’d be running these functions instead of the typical lh.dispense and lh.aspirate respectively.

async def safe_dispense(star, lh, resource, vols, offsets, max_attempts, text_log_file, verbose=False, prev_vol=0):
    for attempt in range(max_attempts):
        try:
            return await lh.dispense(resource, vols=vols, offsets=offsets)
        except TimeoutError:
            # use capacitative liquid level detection to check if dispense was successful
            # ASSUMPTION: the tip is in the desired dispense position
            # ASSUMPTION: len(vols) == 1
            liquid_level = await star.clld_probe_z_height_using_channel(0, move_channels_to_save_pos_after=True)
            
            if liquid_level > prev_vol + 0.1*vols[0]: # More than 10% of desired volume was dispensed
                return

    raise RuntimeError(f"Failed to dispense after {max_attempts} attempts")

async def safe_aspirate(lh, star, resource, vols, offsets, prev_vol, max_attempts, text_log_file, verbose=False):
    for attempt in range(max_attempts):
        try:
            return await lh.aspirate(resource, vols=vols, offsets=offsets)
        except TimeoutError:
            # use capacitative liquid level detection to check if aspiration was successful
            # ASSUMPTION: the tip is in the desired dispense position
            # ASSUMPTION: len(vols) == 1
            liquid_level = await star.clld_probe_z_height_using_channel(0, move_channels_to_save_pos_after=True)
            
            if liquid_level < prev_vol - 0.1*vols[0]: # More than 10% of desired volume was aspirated
                return
           
    raise RuntimeError(f"Failed to aspirate after {max_attempts} attempts")

100%

@agar, that is good we are getting closer and closer :slight_smile:

  1. Are you attempting a multi-dispense with a single channel into a list of specified wells?
    Is this what causes the command to run for a long time → the channel has to execute many different partial dispenses in different wells?

  2. Are you using a LiquidClass/are you familiar with why one is needed for this task?

wait wait @CamilloMoschner why do these things matter?

we just have to see if

  • the command is expected to take a long time (flow rate, setting time etc.)
    • can just test emperically on a machine
    • it doesn’t seem to be this since @agar set timeout to 2h for testing
  • look at the firmware log, do we get a response? do we parse it correctly?

Because…

  1. we don’t know yet whether there might be some other errors lurking in the background that aren’t caught, generating complex interactions of errors we need to unravel → i.e. identifying the unknown unknows

  2. Multi-dispense is governed by much more complex liquid physics and I want to help @agar get the result they’re expecting :slight_smile:
    (e.g. confusing piston movement volume with liquid volume, having stop back volumes that don’t match the liquid behaviour, lack of overaspirate, complexity of correction curve management across partial dispenses, …)

3 Likes

Yes I am!! Not sure if this is what makes it run for a long time. I also encounter Timeout errors when aspirating from a single reservoir into one channel.

I’ll note our machine is older and that our protocol is >2 hr (I want to eventually run protocols >10hr).

I’m not/don’t think so and don’t think one is needed for the task. I don’t think my code includes it but please lmk if it implicitly does/I’m messing smth up?

Thank you both!

2 Likes

Let’s focus on the dispense only for now, I’d say:

@agar, when you multi-dispense, are you doing so via one single command, i.e. lh.dispense([w1, w2, w3], vols=[80, 120, 100], channels_used=[0, 0, 0])?
Or are you running a loop with a single dispense/well/volume each time?

LiquidClass background:
Anytime you use an aspirate or dispense command WITHOUT specifying a LiquidClass (or in PLR-Hamilton, a hamilton_liquid_classes argument) you are NOT specifying the volume of liquid you want to aspirate/dispense in the e.g. vols=[80,120, 100] argument!
Instead you are specifying the volumes that the piston inside the channel is going to move!

The two (piston movement volume VS desired liquid volume) are related but not the same.
The job of the LiquidClass is to A) provide a correction curve that chooses the required piston movement volume for the desired liquid volume you want to achieve, and B) provide all the other aspirate/dispense parameters that are required to achieve correct liquid transfer (including flow rates, settling time, speed of tip moving out of container, …).
IF you do not specify these parameters either 1) manually, or 2) via a convenient hamilton_liquid_class, the machine takes your piston movement volume and it’s default aspirate/dispense values! I.e. you loose control.

It is possible that some bug is lurking somewhere in not specifying the LiquidClass.

I’d recommend going into pylabrobot/liquid_handling/liquid_classes/hamilton and having a look around for a water(?), 1000ul tip(?), partial dispense liquid class, and seeing whether you get the same errors in some water tests when using the hamilton_liquid_class you choose with just a dispense instead of the error handler :slight_smile:

this is not a legal PLR command. We actually generated a firmware command that would use the first three channels if you sent this. I am fixing that by raising an error in `use_channels` must be unique by rickwierenga · Pull Request #589 · PyLabRobot/pylabrobot · GitHub

2 Likes

To clarify, I want to dispense multiple times from the same tip into different wells. I’m just using a single channel tip for now. I’m running a nested for loop with a single dispense/well/volume command in the inner loop.

I see — so it won’t just default to water? Will the behavior remain the same as right now if I just specify water as the liquid class? The liquids I’m handling have similar properties to water.

hi! I tried this, got the following error when trying to stop capture:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/labauto2/Downloads/plrtest_june25/plrtest/plrtest/pylabrobot/pylabrobot/io/capture.py", line 100, in stop_capture
    capturer.stop()
  File "/home/labauto2/Downloads/plrtest_june25/plrtest/plrtest/pylabrobot/pylabrobot/io/capture.py", line 38, in stop
    json.dump(
  File "/usr/lib/python3.12/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.12/json/encoder.py", line 432, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.12/json/encoder.py", line 406, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.12/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type module is not JSON serializable

I just looked into this: it is a bit more complicated, if no hamilton_liquid_class is specified in dispense on a Hamilton STAR machine, PLR will try to get a water-based liquid class as stated in the STAR_Backend.py:

I discussed this with @rickwierenga, and I see now how this is a “necessary evil” because if someone doesn’t know how ‘air displacement technology’ pipettes (like the STAR’s, OT-2’s, …) work, it gives them a chance to get the correct behaviour for standard pipetting actions.

BUT

We never know whether an actual hamilton_liquid_class has been found (this liquid class retrieval happens in the background without us having any idea about what it ends up doing).
If it doesn’t find a liquid class that fulfils the criteria it will default to machine firmware default values, which you can also find in the STAR_Backend.py:

 @need_iswap_parked
  async def dispense_pip(
    self,
    tip_pattern: List[bool],
    dispensing_mode: List[int] = [0],
    x_positions: List[int] = [0],
    y_positions: List[int] = [0],
    minimum_height: List[int] = [3600],
    lld_search_height: List[int] = [0],
    liquid_surface_no_lld: List[int] = [3600],
    pull_out_distance_transport_air: List[int] = [50],
    immersion_depth: List[int] = [0],
    immersion_depth_direction: List[int] = [0],
    surface_following_distance: List[int] = [0],
    second_section_height: List[int] = [0],
    second_section_ratio: List[int] = [0],
    minimum_traverse_height_at_beginning_of_a_command: int = 3600,
    min_z_endpos: int = 3600,  #
    dispense_volumes: List[int] = [0],
    dispense_speed: List[int] = [500],
    cut_off_speed: List[int] = [250],
    stop_back_volume: List[int] = [0],
    transport_air_volume: List[int] = [0],
    blow_out_air_volume: List[int] = [200],
    lld_mode: List[int] = [1],
    side_touch_off_distance: int = 1,
    dispense_position_above_z_touch_off: List[int] = [5],
    gamma_lld_sensitivity: List[int] = [1],
    dp_lld_sensitivity: List[int] = [1],
    swap_speed: List[int] = [100],
    settling_time: List[int] = [5],
    mix_volume: List[int] = [0],
    mix_cycles: List[int] = [0],
    mix_position_from_liquid_surface: List[int] = [250],
    mix_speed: List[int] = [500],
    mix_surface_following_distance: List[int] = [0],
    limit_curve_index: List[int] = [0],
    tadm_algorithm: bool = False,
    recording_mode: int = 0,
  ):
    """dispense pip

    Dispensing of liquid using PIP.

    LLD restrictions!
      - "dP and Dual LLD" are used in aspiration only. During dispensation all pressure-based
        LLD is set to OFF.
      - "side touch off" turns LLD & "Z touch off" to OFF , is not available for simultaneous
        Asp/Disp. command

    Args:
      dispensing_mode: Type of dispensing mode 0 = Partial volume in jet mode
        1 = Blow out in jet mode 2 = Partial volume at surface
        3 = Blow out at surface 4 = Empty tip at fix position.
      tip_pattern: Tip pattern (channels involved). Default True.
      x_positions: x positions [0.1mm]. Must be between 0 and 25000. Default 0.
      y_positions: y positions [0.1mm]. Must be between 0 and 6500. Default 0.
      minimum_height: Minimum height (maximum immersion depth) [0.1 mm]. Must be between 0 and
        3600. Default 3600.
      lld_search_height: LLD search height [0.1 mm]. Must be between 0 and 3600. Default 0.
      liquid_surface_no_lld: Liquid surface at function without LLD [0.1mm]. Must be between 0 and
        3600. Default 3600.
      pull_out_distance_transport_air: pull out distance to take transport air in function without
        LLD [0.1mm]. Must be between 0 and 3600. Default 50.
      immersion_depth: Immersion depth [0.1mm]. Must be between 0 and 3600. Default 0.
      immersion_depth_direction: Direction of immersion depth (0 = go deeper, 1 = go up out of
        liquid). Must be between 0 and 1. Default 0.
      surface_following_distance: Surface following distance during aspiration [0.1mm]. Must be
        between 0 and 3600. Default 0.
      second_section_height: Tube 2nd section height measured from "zx" [0.1mm]. Must be between
        0 and 3600. Default 0.
      second_section_ratio: Tube 2nd section ratio (see Fig. 2 in fw guide). Must be between 0 and
        10000. Default 0.
      minimum_traverse_height_at_beginning_of_a_command: Minimum traverse height at beginning of a
        command 0.1mm] (refers to all channels independent of tip pattern parameter 'tm'). Must be
        between 0 and 3600. Default 3600.
      min_z_endpos: Minimum z-Position at end of a command [0.1 mm] (refers to all channels
        independent of tip pattern parameter 'tm'). Must be between 0 and 3600.  Default 3600.
      dispense_volumes: Dispense volume [0.1ul]. Must be between 0 and 12500. Default 0.
      dispense_speed: Dispense speed [0.1ul/s]. Must be between 4 and 5000. Default 500.
      cut_off_speed: Cut-off speed [0.1ul/s]. Must be between 4 and 5000. Default 250.
      stop_back_volume: Stop back volume [0.1ul]. Must be between 0 and 180. Default 0.
      transport_air_volume: Transport air volume [0.1ul]. Must be between 0 and 500. Default 0.
      blow_out_air_volume: Blow-out air volume [0.1ul]. Must be between 0 and 9999. Default 200.
      lld_mode: LLD mode (0 = off, 1 = gamma, 2 = dP, 3 = dual, 4 = Z touch off). Must be between 0
        and 4. Default 1.
      side_touch_off_distance: side touch off distance [0.1 mm] (0 = OFF). Must be between 0 and 45.
        Default 1.
      dispense_position_above_z_touch_off: dispense position above Z touch off [0.1 s] (0 = OFF)
        Turns LLD & Z touch off to OFF if ON!. Must be between 0 and 100. Default 5.
      gamma_lld_sensitivity: gamma LLD sensitivity (1= high, 4=low). Must be between 1 and 4.
        Default 1.
      dp_lld_sensitivity: delta p LLD sensitivity (1= high, 4=low). Must be between 1 and 4.
        Default 1.
      swap_speed: Swap speed (on leaving liquid) [0.1mm/s]. Must be between 3 and 1600.
        Default 100.
      settling_time: Settling time [0.1s]. Must be between 0 and 99. Default 5.
      mix_volume: Mix volume [0.1ul]. Must be between 0 and 12500. Default 0.
      mix_cycles: Number of mix cycles. Must be between 0 and 99. Default 0.
      mix_position_from_liquid_surface: Mix position in Z- direction from liquid surface (LLD or
        absolute terms) [0.1mm]. Must be between 0 and 900.  Default 250.
      mix_speed: Speed of mixing [0.1ul/s]. Must be between 4 and 5000. Default 500.
      mix_surface_following_distance: Surface following distance during mixing [0.1mm]. Must be
        between 0 and 3600. Default 0.
      limit_curve_index: limit curve index. Must be between 0 and 999. Default 0.
      tadm_algorithm: TADM algorithm. Default False.
      recording_mode: Recording mode 0 : no 1 : TADM errors only 2 : all TADM measurement. Must
        be between 0 and 2. Default 0.
    """

I’m not sure whether there is some default vs non-default argument setting going on in the background that generates an incorrect argument composition which somehow triggers an TimeoutError.

It would be best to see the firmware logs a complete sequence of aspirate + dispenses in which the one of the dispenses triggers this error.
Then comparing the dispenses that worked with the ones that didn’t should give us the next clue.

1 Like

wanna open a new thread or github issue for this? with a minimal reproducible example

3 Likes

So would explicitly just setting the liquid class as water solve this problem?

This would definitely solve the problem of not knowing what parameters are exactly chosen.

I don’t know whether it would solve your TimeoutError issues, because we don’t know yet what combination of parameter settings cause the TimeoutError.

Do you have a firmware log that shows the aspiration followed by all the dispense command up to and including the dispense that fails?

Done, see Capturing data during protocol execution: Object of type module is not JSON serializable · Issue #593 · PyLabRobot/pylabrobot · GitHub

1 Like

Hi Camillo, I wasn’t able to get the exact firmware log (see GH issue). But I ran Wireshark and caught some of the USB packet logs during a timeout error. It seems like the Hamilton is losing connection with the computer – there are a bunch of URB_CONTROL packets sent between my computer and the hamilton, then URB_BULK messages that culminate in an ENOENT error.

I was thinking this could be due to a faulty USB connection between my computer and the Hamilton, or a shaky power source for the Hamilton. I replaced the USB connection and made the power source more secure. Have you guys seen this pattern at all?

2 Likes

good catch! i have never seen this before.

did replacing the power/connection help at all?

Since the timeout error is stochastic, not sure yet. Does my interpretation of the Wireshark results seem reasonable?

that is entirely possible yes, and especially explains the stochastic aspect of this error

please keep us up to date

2 Likes

this seems to be the error! we are now running longer protocols without timing out

thank y’all for your help!

:smiling_face: