Resource backend (theory-craft)

General
1

I’ve been thinking about the PLR resource model in relation to networked interfaces. Here are two problems it solves:

  1. Sharing between machines. If you have 1 incubator connected to two difference workcells, you want to be able to sync the resources in some way.
  2. Persistent connections. If you are over a network and your protocol disconnects, for whatever reason, you are screwed because now the server and client are desynced.

The logic doesn’t need to be reproduced, just the fundamental operations which sync to disk. Pretty much everything (well, not including the well tracking / item ordering) can be instantiated in a single SQL table:

CREATE TABLE resources (
    id INTEGER PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    type TEXT NOT NULL,  -- 'Resource', 'Plate', 'TipRack', 'Well', etc.

    -- Tree structure
    parent_id INTEGER REFERENCES resources(id),

    -- Spatial: Local position relative to parent
    loc_x REAL,
    loc_y REAL,
    loc_z REAL,

    -- Spatial: Local rotation (Euler angles in degrees)
    rot_x REAL DEFAULT 0,
    rot_y REAL DEFAULT 0,
    rot_z REAL DEFAULT 0,

    -- Spatial: Size
    size_x REAL NOT NULL,
    size_y REAL NOT NULL,
    size_z REAL NOT NULL,

    -- Metadata
    category TEXT,
    model TEXT,

    -- Timestamps for change tracking
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    CHECK (parent_id IS NULL OR parent_id != id)
);

CREATE INDEX idx_resources_parent ON resources(parent_id);
CREATE INDEX idx_resources_name ON resources(name);

However, I think the easiest route here is producing an API which can be the backend instead of just in-memory. You only need 5 operations, since you’re operating really off a single table:

  1. create_resource
  2. get_resource
  3. update_resource
  4. delete_resource
  5. query_resources

Though you would likely want a couple more, just for convenience:

  1. get_tree (avoid N queries)
  2. batch_create (create plate with 96 wells)
  3. batch_update (move multiple resources)
  4. batch_delete (clear deck)
  5. get_state (volume/tip state)
  6. update_state

Here is what I imagine the code looking like:

from pylabrobot.liquid_handling import LiquidHandler
from pylabrobot.liquid_handling.backends import STARBackendAPI
from pylabrobot.resources import Deck

auth_key = "AUTH_KEY"

deck = Deck.from_api("machine1", base_url = "pylabrobot.org/api/v1/state", auth = auth_key)
backend = STARBackendAPI("machine1", base_url = "pylabrobot.org/api/v1/star", auth = auth_key)
lh = LiquidHandler(backend=STARBackendAPI(), deck=deck)
await lh.setup()

await lh.pick_up_tips(lh.deck.get_resource("tip_rack")["A1"])
await lh.aspirate(lh.deck.get_resource("plate")["A1"], vols=100)
await lh.dispense(lh.deck.get_resource("plate")["A2"], vols=100)
await lh.return_tips()

This would give:

  1. Method to integrate the states of many machines together at once (same base_url, just different machine). Backend handles the pass-off of material
  2. Fully atomic transactions: pickup where you left off without ever thinking about serializing
  3. Ability for full log tracking (if there is a log table). Track movement of every plate.

However, unlike the network backend, this would require modifying the code of the deck, because we want the business logic to be able to use an API instead of just in-memory. But seems very doable.

2

very important thing to implement

choosing sql vs nosql is important. the table you presented works for the base resource, but (almost) every resource subclass adds extra attributes. For example, Container adds volume. (see Resources Introduction — PyLabRobot documentation) This is solvable of course with extra tables, just something to consider.

I have always theorized about a nosql database for this (we already serialize everything to dictionaries so this will be a couple lines of code to implement, db.store(root.serialize()) and root = Resource.deserialize(db.load()).

the user should not think about serializing, but fundamentally data will need to serialized (whether to json or another format) somewhere

not necessarily. what if you used callbacks (we need to add a resource_changed callback, in addition to resource_assigned and resource_unassigned that we already have) → you could just tag those onto any layer you want to control and could keep using the existing APIs. lazily loading/sending everything from/to the cloud every time is doable but I think it’ll be a headache.

3

Most modern SQL databases have JSON support. So you can always just push the extra data into JSON, if you need. You can even query from it JSON Functions And Operators PostgreSQL: Documentation: 18: 8.14. JSON Types

I’d also argue that the actual database backing it isn’t nearly as important as a consistent and simple interface for interacting with the resource backing system. The API-ness with a single service containing state is what really enables the different machines working together. It also allows different folks to implement different backends, which is nice (this is probably what I would do after implementing a python version). You could have a simple local SQLite backend for your particular bot (perhaps living on the raspi that you connect to with the network backend) tracking just whatever is on its own deck persistently, or you could have a central NoSQL service connected to everything in your lab. Doesn’t really matter so long as the interface is satisfied.

I imagine the tests could just be a suite of hurl tests

Why is that? The calls seem relatively sparse and don’t need to be that fast. But you might be considering something I am not

4

sure

PLR uses the model quite lazily.

almost machine every operation calls get_absolute_location (i.e. location wrt root of resource tree), which would then require many database calls.

the question is choosing:

  1. server master / client slave
  2. client master / server slave

I think 2 is better, and the server just passively mirrors things in the background rather than relying on it to pull things down every operation. it’s also significantly easier to implement.

5

Hmmm, I think I have enough to start poking around here. I’ll do some experiments and report back.

1 Like
6

Been thinking more about this. Here’s what I’ve come up with:

  1. The most important part is syncing writes
  2. Writes can be in the form of manipulating liquids, tips, or physically moving things

Furthermore, it kinda seems like the only reads that are really used often by the user are .get_resource. Things like set_liquids don’t seem to be called very often.

One thing that seems like it could work (that I think you’ve suggested @Rick) is just making sure the JSON on both sides is synced. The API could be as simple as this: it provides either the deck layout or the hash of the deck layout. Every time .get_resource or the like is called, you just check to make sure the deck is still synced.

For write operations, you simply command the server to run those. What this does behind the scenes is pull the serialized deck out of the database, deserialize it, run the changes, then serialize back to database, and return JSON / has. This isn’t strongly consistent, but is probably good enough.

It’s kinda gross, but would require very few code changes to implement. Since there are really only 2 things it can do, it could also be pretty easy to implement over a websocket to get the speed necessary.

Still, this doesn’t feel great…

7

not really, since every time you compute the location of a resource (which is pretty much any time a liquid handler does anything), you have to query the location of every resource in between it and the root.

8

also if you want to have the server responsible for resource modeling, then it will need to have an instance of LH running there to make updates as actions get executed.

and to confirm: we are designing this under the assumption that the client can’t be fully trusted

9

I was designing under the assumption that we do not trust the client to know where things are, but we do trust the client to not fuck shit up (ie, they can know where everything is and still send naughty firmware commands)

That complicates things

Hmmm yes I was mostly thinking about this in the context of separate STARBackend / Resource, but it complicates things if you want LH.

10

Important question @rickwierenga , what do you think the latency requirements are?

11

I am thinking more about the hardware to straighten out my thinking. what we want:

  • client tells server where everything is. the client requests put carrier X at position Y. In your example, this is not just a simple assignment but actually requires a physical operation to load that carrier into position.
  • the server keeps track of where everything is. the server is the location where all hardware is, so it must keep track of stuff over there
    • client might disconnect/shut down and forget

it is now obvious:

  • client needs to pull all information from server
  • client needs to push all information requests, and get confirmation it’s actually in place
    • some assignments (like loading carriers) will require work from you, other assignments (like wells to plates) are purely modeling
  • server needs LH to manage updating the resource model when it actually moves stuff
  • client benefits from LH to have errors client side, but it’s not even strictly needed because the errors could be pulled down from the server

the question is:

  • does the server push or does the client pull new diffs? I think serving pushing would be neat (simple to implement and it’s fast)

wet lab automation is incredibly slow, if every machine operation (includes maybe 10-ish resource model queries total) could stay below a second that would be good enough.

12

in pseudo system messages:

  • client: request to put carrier named X at location Y
  • <physical loading of carrier>
  • server: carrier named X now assigned to location Y
  • client: request to put plate named P at location Q
  • <physical loading of plate>
  • server: plate named P now assigned at location Q
  • [many well assignments]
  • client: move plate P to location R
  • <physical LH movement, server updates model>
  • server: Plate now assigned at location R
13

This seems very similar to how we were thinking about the LH or STARBackend implementations. Basically, unary calls are great and all, but it takes so long to actually move the robot (or in this case, have a human move some hardware) that you can’t really depend on it to happen within a ~1s time slot - so some update mechanism is required.

I think I was thinking of something subtly different: the server tells the client where everything is (or not), and the client requests that the server/humans move things around. Then the server confirms to the client that things have moved as such, and the client can continue with what it is doing.

(note: I do NOT know how this interacts with the LH actually moving things around with an iSWAP).

This isn’t too hard. Though this is sounding more and more like the realm of RPC.

14

1s applies to querying information, not assignments

yes, with “tells” I meant “commands”, not “informs” in this case