High level pylabrobot resource library (PLR-RL) discussion

[rickwierenga]

Originally started by @willem in PLR Dev Roadmap 2025 - Q1 & Q2 - #9 by willem. I am specifically responding to posts PLR Dev Roadmap 2025 - Q1 & Q2 - #11 by willem and PLR Dev Roadmap 2025 - Q1 & Q2 - #12 by CamilloMoschner

hardcoding plate information is not great as it a) requires contributors to be python programmers with access to a repository, superuser permissions, and time on their hands etc, and b) requires others to update their code in order to use new (other people’s) plates - which makes it harder on you to have a stable release for a while.
If such definitions would be in a database, or separate file, you can start thinking about crowdsourcing such info, or ask vendors if they can supply it in a certain format, independently of any ongoing coding work.

I think I can kind of see what you’re getting at: making the PLR-RL more useful to non-PLR usecases. I think it’s an interesting idea for improving the state of lab automation software. Like separating the PLR-RL from PLR. At one point I was writing a prototype for a system like that: https://github.com/rickwierenga/lwdb

this project is kind of dead (for now) because the added complexity and wanting to do everything is difficult to design from scratch without a clear use case. We (used to) often make changes to the PLR resource ontology, necessitated by specific use cases (which really show what’s needed). Standardization follows implementation. Naming conventions, physical attributes to model, resource relations, etc. When working with non PLR systems, it becomes a headache. The main reason the resource system is in Python/PLR is it’s super easy to iterate on the ontology, write code (for existing contributors), use the definitions, etc.

When we feel like we have converged on a good PLR resource model, we could think about moving the information (plate sizes etc.) to a separate system (other than the plr repo).

I agree with your point about versioning: the PLR-RL should be versioned separately from the main library. (long term goal - right now we have to get things working and i don’t see this as an immediate need.)

Manufacturer’s can already contribute as it is today: fork the repo, make a PR. Very easy to do. They can also have their own python library that can instantiate PLR classes for their resources (PLR would be a dependency of that library).

We already support JSON serialization and deserialization. So you can already communicate about PLR resources without actually running Python.

Regarding documentation / finding resources: it’s a wip. we want to have a nice UI where you can filter resources across the entire PLR-RL. Currently it’s cataloged by manufacturer > resource type. We want to have a Craigslist-level UX.

Do you have a specific use case in mind? I think a lot can be done with the current Python apis, albeit a bit hacky.

superuser permissions

for contributing to PLR? or running python code? i am a little confused.

---

but it misses the point

i am not sure if I hit “the point” above, but as far as I understand you’re approximately describing a centralized labware database slightly separated from main PLR. I agree this is necessary and we’ll probably build it. But it also comes with a lot of added complexity that would slow down development.

[willem]

Thanks Rick - yes it’s a bit thinking ahead to the longer run (you did ask a roadmap question…), and you are probably right it is not just at that point yet if we’ve not arrived at a fairly stable resource model. But even without that it is worth thinking about the ‘meta-structure’ - where does PLR like to end up, who are the users, and how will they use it; and try develop with that in mind. I just don’t know how much around that has crystallised out already, you may well have that in mind already.

And you did think about that already with your LWDB. I’d try and use an existing opensource implementation rather than build it from scratch (do have a look at django - it offers a really flexible data model, and database storage plus web interfacing for free, and like you say, it could be a separate entity/module).

It is about widening the user base and increasing usability. Yes we can define plates - and ‘we’ means people who know python. But not everyone will feel capable enough. Making life easy is the key to uptake.

As a user example: I have written a web-hosted dispensing app (I am very much not a lab person), and I know that if they switch to a a different plate type I will need to implement it.

In my app I just create a new definition in my database, say, for a 24w plate. But then I still need to go into PLR/my code and set up (‘hardcode’) that plate that will be linked to my 24w definition. Which means writing the code, testing that new version, pushing it to production (hence the superuser requirement), testing again, ask the user to check (and fix if not ok). That takes time. If I could send the user to a (my) web site to either, set up a new plate definition, or pick one that looks right, I should be able to hook that up ‘by name’ without writing new code - meaning no new release, push to production, testing etc.

As a second example, we use Greiner plates which PLR does not have many of. If I were to define these, no-one else can use them.

(I’ll answer Camillo’s questions separately)

[willem]

Camillo asked some questions - just so these are not lost, see below

@CamilloMoschner

Q1: Yes, PyLabRobot is a Python library; what is your suggestion as an alternative and why is an alternative needed?

As per previous - uptake is all about making life easy for developers. It is not a change, it is an iteration. Not everyone is at the same level, most of us do not read documentation, noone has enough time.

@CamilloMoschner

Q2: Why does one require superuser permissions to execute Python files?

Ah yes I was thinking about my use there - I am not a lab person. You probably know it is easier to find python developers outside of a lab. So I develop an app, push it to production on a server which requires permissions. I am by no means perfect but I do try to emulate a bit of a software design cycle of dev/test/versioning/prod/test/release. I don’t use github.

@CamilloMoschner

Q3: Are you suggesting a website-based database with users contributing resource information straight into the database without a pull request into the database?

Yep. Modularise it, separate it. That way you can version, augment, fix such entities separately, without affecting other parts too much.

@CamilloMoschner

Q3.a: Do you see the issue with the requirement of being a GitHub user?

My codebase is versioned with SVN. It is behind a corporate firewall. My users think github is a place where annoying people drink beer.

@CamilloMoschner

Q3.b: How would we validate contributions in that case?

Yes - this is a key issue and the biggest sticking point. Let me just bounce that back at you - who is doing that now. And if the answer is ‘I do’ then consider how much of your time that takes.

If centralising in a db, you could consider only allowing ‘trusted users’. having a ‘thumbsupflag’ where people say this worked for me etc. But you are right, QC needs consideration.
I’ll add you have the very same problem now though.

[willem]

And lastly - Camillo raised following as well, so let me show you some code, enough of my waffle in above

@CamilloMoschner

The general issue we found is simply resource naming conventions, resulting limited namespace, missing manufacturer information, and oftentimes that users misunderstand the difference between a supplier and the original manufacturer, all resources must be stored by the manufacturer’s

This will always be a pain. Automate a unique naming scheme based on attributes I’d say. It’ll be sort of user-readable. Let me just give you a very simple Django-based implementation to show what I mean when I talk about ‘plates on a database’.

from django.contrib.auth.models import User
from django.db import models

class GenericPlateType(models.Model):
    """
    Mock base data model for a plate instance
    """

    #Store who contributed the definition so you can shout at them
    #Likewise, would add a "validated_by" field
    #Django has an extensive User registration setup, you get that for free.
    contributed_by= models.ForeignKey(
        User,
        blank=False,
        null=True,
        on_delete=models.SET_NULL,  # Do not delete when user disappears
    )

    manufacturer = models.CharField(
        unique=False,
        null=False,
        max_length=100,
        help_text="Manufacturer name",
    )

    catalogId = models.CharField(
        unique=True,   #auto-checks for duplicates
        null=False,        # we won't allow null in the db
        max_length=100,
        help_text="Additional information: catalog id",
    )

    nx = models.PositiveSmallIntegerField(
        null=False,
        default=12,
        unique=False,
        validators=[MinValueValidator(1),MaxValueValidator(12)],
        verbose_name=_("Number of X wells"),
        help_text="number of wells in X direction",
    )

    ny = models.PositiveSmallIntegerField(
        null=False,
        default=8,
        unique=False,
        validators=[MinValueValidator(1),MaxValueValidator(8)],
        verbose_name=_("Number of Y wells"),
        help_text="number of wells in Y direction",
    )

    def __str__(self):
        s="%s__%s__%d__%d"%(self.manufacturer, self.catalogId, self.nx, self.ny)
        return s

    def getUniqueName(self):
        """
        The catalogId is unique so this will be a unique identifier too
        """
        return str(self)

    #All PLR plate functionality (variables, functions) goes here in 
    #the model definition (basically, everything that is in the current base class).

This is overly simplistic, and I skip some Django setup details, but I hope you get the gist.
Here’s how to use it:

#Create a new Greiner 96w plate type on the database by setting all its info 
#and storing it
my96wplatetype = GenericPlateType(nx=12,ny=8,catalogId="bla",manufacturer="Greiner")
my96wplatetype.save()  #Now it is stored on the database

#User example:
#Find a 96w Greiner plate and use the first one for whatever you do with PLR
#The mygreiner instance is now a plate object that can be used as you 
#would a usual PLR plate.
greiner96w_list = GenericPlateType.objects.filter(manufacturer="Greiner",nx=12,ny=8)
mygreiner = greiner96w_list.first()   

#print its unique identifier
print( mygreiner.getUniqueName() )

[rickwierenga]

we are in agreement that in the future to make our resource library useful for other software we need to have a separate database.

for PLR users right now, i would say it’s close to maximally useful and having it in a separate database doesn’t change anything. if anything, it will make development more painful. (versioning separately is a smart idea.)

do you envision yourself using PLR? or how would you want to use the information in the PLR RL? let’s talk specifics

you seem to be misunderstanding one specific piece:

@willem

But then I still need to go into PLR/my code and set up (‘hardcode’) that plate that will be linked to my 24w definition.

No. If your database has information in sufficient detail (most labware libraries do not: they miss material_z_thickness), you can instantiate a Plate object at runtime.

await lh.setup() # protocol starts
info = get_plate_info()
plate = Plate(
  name="name",
  size_x=info["size_x"],
  ...
)

# assign plate to deck (at runtime)
lh.deck.assign_child_resource(plate, location=...)

# use it
await lh.aspirate(plate["A1"], ...)

alternatively if you want to go by plate name / part number (from the PLR RL):

await lh.setup() # protocol starts
info = get_plate_name()
if name == "cos_...":
  plate_creator_function = Cos...
elif name == "thermo_...":
  plate_creator_function = TF...
else:
  raise UnknownPlate

plate = plate_creator_function(name=name)

# assign plate to deck (at runtime)
lh.deck.assign_child_resource(plate, location=...)

# use it
await lh.aspirate(plate["A1"], ...)

for this 2nd example, we can make some improvements eg being able to query by part number.

but ultimately it’s just python code. you can instantiate plates however you want. the PLR-RL is just a convenience but if you look closely you will see that every function returns an instance of the same general class. that means that in your script, as soon as you say plate = some_func() after that line runs it doesn’t matter which function or whatever created the Plate instance.

you can have your information in any database, and when the time comes to use it in a protocol create the PLR instance.

as far as i am aware, PLR is the only system which allows you to define your deck layout and plate types (for any device) at runtime.

again to emphasize: you can literally instantiate Plates or any other labware AT RUNTIME using information from anywhere. If you want have a nice scientist-facing UI, build it and just get the required plate information from there.

no “no new release, push to production, testing etc.” required, just dynamically create Plate instances

@willem

If I were to define these, no-one else can use them.

what does this mean?

---

@willem

Automate a unique naming scheme based on attributes I’d say. It’ll be sort of user-readable.

curious what you think about this: Almost-equivalent plate naming convention

---

it would be good to zoom out a little bit. what are we discussing here? how plr can be useful to a specific use case you have / how to make plr useful for that? or the general vision for a future labware database?

[willem]

'morning,

Yep zooming out and looking at the bigger picture is the goal here I think (a roadmap is not just for christmas!).

I agree with what you say re Plate instantiation - yes, this is possible right now. My question is more - who are the target audience and how ‘easy’ does PLR need to be to be used effectively. If I set up a Plate in my code, it is not shared until it would be -literally- included as definition in some release.

You did answer the question somewhere in the above around PLR users - PLR is envisioned as a low-level machine-interfacing library, used by people who are adept at python.

I am just conscious that there are likely more python developers outside a lab than in a lab, and the ones in a lab likely have a range of skill levels. PLR can be daunting, and ‘making life easy’ would appeal to them I think and possibly foster wider use.

Standardising Plate definitions and supplying them as a db could help on the developer side, separating that functionality out more and harmonising it (which in turn facilitates automated naming). But yes, like everything, that would take some work.

Cheers
w

[CamilloMoschner]

Hi @willem,

To answer your question who PLR is for (at this moment in time) directly from my perspective:
PLR is currently for automation developers who…

1. require standard programming practices for building physical computing workflows,
2. find the maze of incompatible OEM software obstructive to generate results with real, physical machines,
3. find that the machines they are using can do more than what the OEM software enables, but have to use their machines to their fullest potential.
   …

By necessity of these requirements these individuals are already programmers or are “go-getters” who aren’t afraid of teaching themselves Python quickly.

Python simply gives the fastest access to the programming tools required, and some more traditional C-based roboticists usually find it easy to switch to Python.
Those individuals are already familiar with standard software engineering practices used across the open-source software space on which everything from AI (e.g. PyTorch) to the web (e.g. Django) is built; e.g. git-based versioning and community development.

---

I think you are absolutely right to point out that most Python developers are found outside the lab, not in the lab.
This is one of the core issues of modern research (at least in biology and chemistry): the people generating (and often designing) data cannot scale their work due to not being able to programme automation workflows + cannot analyse their own complex data (past a certain point) once they have acquired it due to not being able to programme.

At the same time, people who programme often have an incredibly strong aversion of going into the lab and generating data themselves (the fear of the pipette is real). They often have as much knowledge about how to programme physical machines (i.e. perform physical computing) as pure wetlab scientists (which is not a critique! they simply have to focus on other aspects).

PLR will not solve this big-picture issue nor can it serve both parties at the same time.

But: the bridges between these two worlds are the automation scientists who speak domain-specific language in both worlds.
These are the people who PLR is currently for
And by empowering the bridges I personally hope to close the gap between these worlds.

---

We fully understand that this doesn’t make it “easy” (though it could be debated whether physical computing is ever easy due to the intrinisic complexity of its interdisciplinarity and whether automation sales people selling an “easy” solution are actually selling anything real).

But PLR is still developing at breath-taking speed.
It is focusing on what is working in the actual labs it is used in across the world, and is adapting accordingly. If a new idea comes along to optimise we’ll try it out.

As a result, the only users PLR can currently focus on is automation developers who can keep up with that speed

We all fully intend to make it easier down the line for people with more limited programming skillsets.
But until then we cannot spread ourselves too thin by working on too many fronts at once.

Note:
We are very happy for others to build on top of PLR, particularly if they wanted to build a “simple GUI automation interface” which is a topic that keeps cropping up anew every couple of years.
We’ve also had conversations with multiple front-end focused automation companies, and assessed whether they wanted to use PLR and collaborate, or whether they wanted to invent everything from scratch without community support.
Many chose to invent everything from scratch, i.e. the machine backends + a physical modelling system + their frontend UI… we’re seeing the first wave of these now going bankrupt.
In an odd way this might be good for PLR: it shows companies who need automation that going open-source is a form of risk mitigation - the companies pay nothing for PLR and own all code they ever write in house. They can always pick it up with their own physical computing engineers anytime.

And it is precisely this lack of commercial incentives that gives PLR the freedom to work on time-consuming tasks like generating a PLR Resource Library database in which every new addition has indeed been heavily scrutinized before being added (which still doesn’t make it bulletproof!).
Who is verifying the PLR-RL entries? - we try to always find and add “ground truth” data:
this can be technical drawings, precise dimensional information on the OEM website or if neither is available it can be 3D scanning of the items, or using a Coordinate Measurement Machine to acquire the required dimensions.
This last part is precisely the reason why PLR invented completely novel ways of making our liquid handlers identify resource dimensions themselves (see Z-probing), taking human-bias of manual measurements out of the equation (and replacing it with machine-based calibration biases ).
But, we found that many non-coding endusers are not willing to invest the time required for generating such high quality resource definitions.

As a result, yes, the PLR-RL database is growing slowly.
And it doesn’t have to grow quickly.
In fact, if growing quickly would mean sacrificing quality of resource definitions it is arguably better to have the PLR-RL database grow slowly.

---

Please do disagree with me, this is what makes open-source dev so powerful!