TimeoutError

General
1

Hi, all!

While running a protocol, I’ve been getting this error seemingly randomly during the run, causing it to stop:

TimeoutError: Timeout while waiting for response to command C0DSid0163dm2 2 2 2 2 2&tm1 1 1 1 1 0&xp09644 09644 09644 09644 09644 00000&yp2340 2221 2102 1984 1865 0000&zx1872 1872 1872 1872 1872 1872&lp2167 2167 2167 2167 2167 2167&zl1872 1872 1872 1872 1872 1872&po0100 0100 0100 0100 0100 0100&ip0000 0000 0000 0000 0000 0000&it0 0 0 0 0 0&fp0000 0000 0000 0000 0000 0000&zu0032 0032 0032 0032 0032 0032&zr06180 06180 06180 06180 06180 06180&th2450te2450dv10285 10285 10285 10285 10285 10285&ds1200 1200 1200 1200 1200 1200&ss0050 0050 0050 0050 0050 0050&rv000 000 000 000 000 000&ta300 300 300 300 300 300&ba0000 0000 0000 0000 0000 0000&lm0 0 0 0 0 0&dj00zo000 000 000 000 000 000&ll1 1 1 1 1 1&lv1 1 1 1 1 1&de0020 0020 0020 0020 0020 0020&wt10 10 10 10 10 10&mv00000 00000 00000 00000 00000 00000&mc00 00 00 00 00 00&mp000 000 000 000 000 000&ms0010 0010 0010 0010 0010 0010&mh0000 0000 0000 0000 0000 0000&gi000 000 000 000 000 000&gj0gk0.

The output is pretty vague, so I’m not sure how to understand what’s going wrong here. Any help would be appreciated!

2

On a STAR machine?

3

Yes!

4

do you see your channels move at all? that would mean the machine just needs more time to execute the command

can increase timeout by setting lh.backend.read_timeout (for all usb-based backends). for dispense it’s set to read_timeout=max(60, self.read_timeout),

if it doesn’t move, it would be in issue with the generated firmware command

5

I’ll try increasing the timeout. Part of the problem is that it isn’t consistently giving the error even when running the same command.

1 Like
6

so likely the firmware command is correct, it’s just the machine that needs more time (possibly based on the starting state when the command is first sent)

7

Great, thank you! Will try

8

Tried increasing the timeout to 4 minutes and am still getting an error randomly throughout longer runs. there isn’t a specific command that causes it, just sometime throughout a procedure running longer than 15 minutes will the TimeoutError occur. thinking this is just due to the age of the machine, but want to get your thoughts.

9

do you see the machine move on the command that times out? (perhaps need to install a camera)

10

I haven’t been there to specifically see it but I think no based on where the machine stops, e.g the error will occur on an aspirate command, but the tips will be over the wells rather than in them attempting to aspirate

11

can you check

for c in "12345678":
  version = await lh.backend.send_command(module=f"P{c}", command="RF")
  print(version)

this reports the firmware version for all channels. i am curious if one is surprisingly old

12

Output of that command:

P1RFid0007rf4.0S f 2020-07-31
P2RFid0008rf4.0S f 2020-07-31
P3RFid0009rf4.0S f 2020-07-31
P4RFid0010rf4.0S f 2020-07-31
P5RFid0011rf4.0S f 2020-07-31
P6RFid0012rf4.0S f 2020-07-31
P7RFid0013rf4.0S f 2020-07-31
P8RFid0014rf4.0S f 2020-07-31
13

all up to date. i will see if i can reproduce this error. the firmware command seems to be well-formed

14

thanks, rick! i’ve tried reproducing the error but it seems to be inconsistent. i’ll add a comment if i can get it with debugging logging on a different command

15

extending the timeout to 10 minutes seems to have done the trick

16

huh, very odd. i have never seen a single operation take that long. recording/seeing one of these happen would be interesting. does the machine just halt for minutes? is data read (check the log) which is not matched to the command?

17

working on getting cameras set up in the machine as well as an error alert system. for now the best i can do is periodically check it. it didn’t seem to stall? but again i wasn’t watching it the whole time.

1 Like
18

upon further watching, the robot will just halt for however long the time is set for, be that 60 seconds or 10 minutes. the data seems to match the command, but i’m not 100% sure what i’m looking for there so i could be wrong.

19

can you share the log?

it sounds like you’re just waiting for a timeout, and 10 minutes just makes that longer (but doesn’t fix the issue)

20

The stack trace is pretty long, but I’ll add it here:

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
Cell In[8], line 1
----> 1 await execute_second_dilutions_run(lh, tip_gen_300, tip_gen_1000)

File ~/code/labauto/src/labauto/dilutions/dilutions.py:236, in execute_second_dilutions_run(lh, tip_gen_300, tip_gen_1000)
    233 reservoir_side = dilution_info["side"]
    234 diluent_source_plate = lh.deck.get_resource(DILUENT_SOURCE_PLATE)
--> 236 await execute_serial_dilutions(lh, transfer_plate, diluent_source_plate, reservoir_side, tip_gen_1000, tip_gen_300)

File ~/code/labauto/src/labauto/dilutions/dilutions.py:163, in execute_serial_dilutions(lh, transfer_plate, diluent_source_plate, reservoir_side, tip_gen_1000, tip_gen_300)
    160 destination_well = transfer_plate[f"A{i + destination_offset}"]
    162 # Transfer the sample
--> 163 await handle_volumes(lh, transfer_split, transfer_source_well, destination_well, tip_gen_1000, tip_gen_300)
    165 # Add diluent
    166 await handle_volumes(lh, diluent_split, diluent_source_well, destination_well, tip_gen_1000, tip_gen_300)

File ~/code/labauto/src/labauto/dilutions/dilutions.py:113, in handle_volumes(lh, volumes, source, destination, tip_gen_1000, tip_gen_300)
    110 volumes_300 = [volume for volume in volumes if volume <= 300 and volume != 0]
    112 # Moves volumes with 1000 uL tips and then 300 uL tips 
--> 113 await move_volumes(lh, volumes_1000, source, destination, tip_gen_1000)
    114 await move_volumes(lh, volumes_300, source, destination, tip_gen_300)

File ~/code/labauto/src/labauto/dilutions/dilutions.py:91, in move_volumes(lh, volumes, source, destination, tip_gen)
     89 await lh.pick_up_tips(tips)
     90 await lh.aspirate(source * num_tips, volumes[:num_tips])
---> 91 await lh.dispense(destination * num_tips, volumes[:num_tips])
     92 await lh.discard_tips()
     93 volumes = volumes[num_tips:]

File ~/code/pylabrobot/pylabrobot/machines/machine.py:35, in need_setup_finished.<locals>.wrapper(*args, **kwargs)
     33 if not self.setup_finished:
     34   raise RuntimeError("The setup has not finished. See `setup`.")
---> 35 return await func(*args, **kwargs)

File ~/code/pylabrobot/pylabrobot/liquid_handling/liquid_handler.py:1125, in LiquidHandler.dispense(self, resources, vols, use_channels, flow_rates, offsets, liquid_height, blow_out_air_volume, **backend_kwargs)
   1122   self._blow_out_air_volume = None
   1124 # trigger callback
-> 1125 self._trigger_callback(
   1126   "dispense",
   1127   liquid_handler=self,
   1128   operations=dispenses,
   1129   use_channels=use_channels,
   1130   error=error,
   1131   **backend_kwargs,
   1132 )

File ~/code/pylabrobot/pylabrobot/liquid_handling/liquid_handler.py:2177, in LiquidHandler._trigger_callback(self, method_name, error, *args, **kwargs)
   2175   callback(self, *args, error=error, **kwargs)
   2176 elif error is not None:
-> 2177   raise error

File ~/code/pylabrobot/pylabrobot/liquid_handling/liquid_handler.py:1105, in LiquidHandler.dispense(self, resources, vols, use_channels, flow_rates, offsets, liquid_height, blow_out_air_volume, **backend_kwargs)
   1103 error: Optional[Exception] = None
   1104 try:
-> 1105   await self.backend.dispense(ops=dispenses, use_channels=use_channels, **backend_kwargs)
   1106 except Exception as e:
   1107   error = e

File ~/code/pylabrobot/pylabrobot/liquid_handling/backends/hamilton/STAR.py:2086, in STAR.dispense(self, ops, use_channels, lld_search_height, liquid_surface_no_lld, dispensing_mode, pull_out_distance_transport_air, second_section_height, second_section_ratio, minimum_height, immersion_depth, immersion_depth_direction, surface_following_distance, cut_off_speed, stop_back_volume, transport_air_volume, lld_mode, dispense_position_above_z_touch_off, gamma_lld_sensitivity, dp_lld_sensitivity, swap_speed, settling_time, mix_volume, mix_cycles, mix_position_from_liquid_surface, mix_speed, mix_surface_following_distance, limit_curve_index, minimum_traverse_height_at_beginning_of_a_command, min_z_endpos, side_touch_off_distance, hamilton_liquid_classes, jet, blow_out, empty)
   2083 limit_curve_index = _fill_in_defaults(limit_curve_index, [0] * n)
   2085 try:
-> 2086   ret = await self.dispense_pip(
   2087     tip_pattern=channels_involved,
   2088     x_positions=x_positions,
   2089     y_positions=y_positions,
   2090     dispensing_mode=dispensing_modes,
   2091     dispense_volumes=[round(vol * 10) for vol in volumes],
   2092     lld_search_height=[round(lsh * 10) for lsh in lld_search_height],
   2093     liquid_surface_no_lld=[round(ls * 10) for ls in liquid_surfaces_no_lld],
   2094     pull_out_distance_transport_air=[round(po * 10) for po in pull_out_distance_transport_air],
   2095     second_section_height=[round(sh * 10) for sh in second_section_height],
   2096     second_section_ratio=[round(sr * 10) for sr in second_section_ratio],
   2097     minimum_height=[round(mh * 10) for mh in minimum_height],
   2098     immersion_depth=[round(id_ * 10) for id_ in immersion_depth],  # [0, 0]
   2099     immersion_depth_direction=immersion_depth_direction,
   2100     surface_following_distance=[round(sfd * 10) for sfd in surface_following_distance],
   2101     dispense_speed=[round(fr * 10) for fr in flow_rates],
   2102     cut_off_speed=[round(cs * 10) for cs in cut_off_speed],
   2103     stop_back_volume=[round(sbv * 10) for sbv in stop_back_volume],
   2104     transport_air_volume=[round(tav * 10) for tav in transport_air_volume],
   2105     blow_out_air_volume=[round(boa * 10) for boa in blow_out_air_volumes],
   2106     lld_mode=[mode.value for mode in lld_mode],
   2107     dispense_position_above_z_touch_off=[
   2108       round(dp * 10) for dp in dispense_position_above_z_touch_off
   2109     ],
   2110     gamma_lld_sensitivity=gamma_lld_sensitivity,
   2111     dp_lld_sensitivity=dp_lld_sensitivity,
   2112     swap_speed=[round(ss * 10) for ss in swap_speed],
   2113     settling_time=[round(st * 10) for st in settling_time],
   2114     mix_volume=[round(mv * 10) for mv in mix_volume],
   2115     mix_cycles=mix_cycles,
   2116     mix_position_from_liquid_surface=[
   2117       round(mp * 10) for mp in mix_position_from_liquid_surface
   2118     ],
   2119     mix_speed=[round(ms * 10) for ms in mix_speed],
   2120     mix_surface_following_distance=[
   2121       round(msfd * 10) for msfd in mix_surface_following_distance
   2122     ],
   2123     limit_curve_index=limit_curve_index,
   2124     minimum_traverse_height_at_beginning_of_a_command=round(
   2125       (minimum_traverse_height_at_beginning_of_a_command or self._traversal_height) * 10
   2126     ),
   2127     min_z_endpos=round((min_z_endpos or self._traversal_height) * 10),
   2128     side_touch_off_distance=side_touch_off_distance,
   2129   )
   2130 except STARFirmwareError as e:
   2131   if plr_e := convert_star_firmware_error_to_plr_error(e):

File ~/code/pylabrobot/pylabrobot/liquid_handling/backends/hamilton/STAR.py:94, in need_iswap_parked.<locals>.wrapper(self, *args, **kwargs)
     89 if self.iswap_installed and not self.iswap_parked:
     90   await self.park_iswap(
     91     minimum_traverse_height_at_beginning_of_a_command=int(self._traversal_height * 10)
     92   )
---> 94 result = await method(self, *args, **kwargs)
     96 return result

File ~/code/pylabrobot/pylabrobot/liquid_handling/backends/hamilton/STAR.py:4529, in STAR.dispense_pip(self, tip_pattern, dispensing_mode, x_positions, y_positions, minimum_height, lld_search_height, liquid_surface_no_lld, pull_out_distance_transport_air, immersion_depth, immersion_depth_direction, surface_following_distance, second_section_height, second_section_ratio, minimum_traverse_height_at_beginning_of_a_command, min_z_endpos, dispense_volumes, dispense_speed, cut_off_speed, stop_back_volume, transport_air_volume, blow_out_air_volume, lld_mode, side_touch_off_distance, dispense_position_above_z_touch_off, gamma_lld_sensitivity, dp_lld_sensitivity, swap_speed, settling_time, mix_volume, mix_cycles, mix_position_from_liquid_surface, mix_speed, mix_surface_following_distance, limit_curve_index, tadm_algorithm, recording_mode)
   4524 assert any(
   4525   0 <= x <= 999 for x in limit_curve_index
   4526 ), "limit_curve_index must be between 0 and 999"
   4527 assert 0 <= recording_mode <= 2, "recording_mode must be between 0 and 2"
-> 4529 return await self.send_command(
   4530   module="C0",
   4531   command="DS",
   4532   tip_pattern=tip_pattern,
   4533   read_timeout=max(60, self.read_timeout),
   4534   dm=[f"{dm:01}" for dm in dispensing_mode],
   4535   tm=[f"{tm:01}" for tm in tip_pattern],
   4536   xp=[f"{xp:05}" for xp in x_positions],
   4537   yp=[f"{yp:04}" for yp in y_positions],
   4538   zx=[f"{zx:04}" for zx in minimum_height],
   4539   lp=[f"{lp:04}" for lp in lld_search_height],
   4540   zl=[f"{zl:04}" for zl in liquid_surface_no_lld],
   4541   po=[f"{po:04}" for po in pull_out_distance_transport_air],
   4542   ip=[f"{ip:04}" for ip in immersion_depth],
   4543   it=[f"{it:01}" for it in immersion_depth_direction],
   4544   fp=[f"{fp:04}" for fp in surface_following_distance],
   4545   zu=[f"{zu:04}" for zu in second_section_height],
   4546   zr=[f"{zr:05}" for zr in second_section_ratio],
   4547   th=f"{minimum_traverse_height_at_beginning_of_a_command:04}",
   4548   te=f"{min_z_endpos:04}",
   4549   dv=[f"{dv:05}" for dv in dispense_volumes],
   4550   ds=[f"{ds:04}" for ds in dispense_speed],
   4551   ss=[f"{ss:04}" for ss in cut_off_speed],
   4552   rv=[f"{rv:03}" for rv in stop_back_volume],
   4553   ta=[f"{ta:03}" for ta in transport_air_volume],
   4554   ba=[f"{ba:04}" for ba in blow_out_air_volume],
   4555   lm=[f"{lm:01}" for lm in lld_mode],
   4556   dj=f"{side_touch_off_distance:02}",  #
   4557   zo=[f"{zo:03}" for zo in dispense_position_above_z_touch_off],
   4558   ll=[f"{ll:01}" for ll in gamma_lld_sensitivity],
   4559   lv=[f"{lv:01}" for lv in dp_lld_sensitivity],
   4560   de=[f"{de:04}" for de in swap_speed],
   4561   wt=[f"{wt:02}" for wt in settling_time],
   4562   mv=[f"{mv:05}" for mv in mix_volume],
   4563   mc=[f"{mc:02}" for mc in mix_cycles],
   4564   mp=[f"{mp:03}" for mp in mix_position_from_liquid_surface],
   4565   ms=[f"{ms:04}" for ms in mix_speed],
   4566   mh=[f"{mh:04}" for mh in mix_surface_following_distance],
   4567   gi=[f"{gi:03}" for gi in limit_curve_index],
   4568   gj=tadm_algorithm,  #
   4569   gk=recording_mode,  #
   4570 )

File ~/code/pylabrobot/pylabrobot/liquid_handling/backends/hamilton/base.py:249, in HamiltonLiquidHandler.send_command(self, module, command, auto_id, tip_pattern, write_timeout, read_timeout, wait, fmt, **kwargs)
    224 """Send a firmware command to the Hamilton machine.
    225 
    226 Args:
   (...)
    239   A dictionary containing the parsed response, or None if no response was read within `timeout`.
    240 """
    242 cmd, id_ = self._assemble_command(
    243   module=module,
    244   command=command,
   (...)
    247   **kwargs,
    248 )
--> 249 resp = await self._write_and_read_command(
    250   id_=id_,
    251   cmd=cmd,
    252   write_timeout=write_timeout,
    253   read_timeout=read_timeout,
    254   wait=wait,
    255 )
    256 if resp is not None and fmt is not None:
    257   return self._parse_response(resp, fmt)

File ~/code/pylabrobot/pylabrobot/liquid_handling/backends/hamilton/base.py:281, in HamiltonLiquidHandler._write_and_read_command(self, id_, cmd, write_timeout, read_timeout, wait)
    279 fut = loop.create_future()
    280 self._start_reading(id_, loop, fut, cmd, read_timeout)
--> 281 result = await fut
    282 return cast(str, result)

TimeoutError: Timeout while waiting for response to command C0DSid0056dm2 2 2 2 2 2&tm1 1 1 1 1 0&xp10184 10184 10184 10184 10184 00000&yp3300 3181 3062 2944 2825 0000&zx1872 1872 1872 1872 1872 1872&lp2167 2167 2167 2167 2167 2167&zl1872 1872 1872 1872 1872 1872&po0100 0100 0100 0100 0100 0100&ip0000 0000 0000 0000 0000 0000&it0 0 0 0 0 0&fp0000 0000 0000 0000 0000 0000&zu0032 0032 0032 0032 0032 0032&zr06180 06180 06180 06180 06180 06180&th2450te2450dv10285 10285 10285 10285 10285 10285&ds1200 1200 1200 1200 1200 1200&ss0050 0050 0050 0050 0050 0050&rv000 000 000 000 000 000&ta300 300 300 300 300 300&ba0000 0000 0000 0000 0000 0000&lm0 0 0 0 0 0&dj00zo000 000 000 000 000 000&ll1 1 1 1 1 1&lv1 1 1 1 1 1&de0020 0020 0020 0020 0020 0020&wt10 10 10 10 10 10&mv00000 00000 00000 00000 00000 00000&mc00 00 00 00 00 00&mp000 000 000 000 000 000&ms0010 0010 0010 0010 0010 0010&mh0000 0000 0000 0000 0000 0000&gi000 000 000 000 000 000&gj0gk0.

This is the command log where it errored:

INFO:pylabrobot:Sent command: C0DSid0056dm2 2 2 2 2 2&tm1 1 1 1 1 0&xp10184 10184 10184 10184 10184 00000&yp3300 3181 3062 2944 2825 0000&zx1872 1872 1872 1872 1872 1872&lp2167 2167 2167 2167 2167 2167&zl1872 1872 1872 1872 1872 1872&po0100 0100 0100 0100 0100 0100&ip0000 0000 0000 0000 0000 0000&it0 0 0 0 0 0&fp0000 0000 0000 0000 0000 0000&zu0032 0032 0032 0032 0032 0032&zr06180 06180 06180 06180 06180 06180&th2450te2450dv10285 10285 10285 10285 10285 10285&ds1200 1200 1200 1200 1200 1200&ss0050 0050 0050 0050 0050 0050&rv000 000 000 000 000 000&ta300 300 300 300 300 300&ba0000 0000 0000 0000 0000 0000&lm0 0 0 0 0 0&dj00zo000 000 000 000 000 000&ll1 1 1 1 1 1&lv1 1 1 1 1 1&de0020 0020 0020 0020 0020 0020&wt10 10 10 10 10 10&mv00000 00000 00000 00000 00000 00000&mc00 00 00 00 00 00&mp000 000 000 000 000 000&ms0010 0010 0010 0010 0010 0010&mh0000 0000 0000 0000 0000 0000&gi000 000 000 000 000 000&gj0gk0
WARNING:pylabrobot:Timeout while waiting for response to command C0DSid0056dm2 2 2 2 2 2&tm1 1 1 1 1 0&xp10184 10184 10184 10184 10184 00000&yp3300 3181 3062 2944 2825 0000&zx1872 1872 1872 1872 1872 1872&lp2167 2167 2167 2167 2167 2167&zl1872 1872 1872 1872 1872 1872&po0100 0100 0100 0100 0100 0100&ip0000 0000 0000 0000 0000 0000&it0 0 0 0 0 0&fp0000 0000 0000 0000 0000 0000&zu0032 0032 0032 0032 0032 0032&zr06180 06180 06180 06180 06180 06180&th2450te2450dv10285 10285 10285 10285 10285 10285&ds1200 1200 1200 1200 1200 1200&ss0050 0050 0050 0050 0050 0050&rv000 000 000 000 000 000&ta300 300 300 300 300 300&ba0000 0000 0000 0000 0000 0000&lm0 0 0 0 0 0&dj00zo000 000 000 000 000 000&ll1 1 1 1 1 1&lv1 1 1 1 1 1&de0020 0020 0020 0020 0020 0020&wt10 10 10 10 10 10&mv00000 00000 00000 00000 00000 00000&mc00 00 00 00 00 00&mp000 000 000 000 000 000&ms0010 0010 0010 0010 0010 0010&mh0000 0000 0000 0000 0000 0000&gi000 000 000 000 000 000&gj0gk0.