One Lab Memory: Why Your AI Needs an Eval Set and Context You Control

Robotic arm holding a sample vial next to a distillation analyzer

By Trucision staff

Key takeaways: AI agents in a lab are only as reliable as what they remember, what they're allowed to see, and how they're tested. If memory lives inside a vendor's tool, context is whatever the prompt happened to include, and testing means a generic benchmark, you can't defend the output to an auditor. Your lab needs three things it controls: one shared memory where every write is approved and traced, an eval set built from your own closed records, and context limited to the right systems and the right SOP revision.

Most labs are starting to use AI in small, separate ways. Staff ask ChatGPT or Microsoft Copilot about an SOP. A vendor's assistant sits inside the instrument software. Someone wrote a script to summarize deviations. Each one works well enough on its own.

The trouble starts when you ask a simple question: what does the AI actually know about our lab, and who decided that? In most labs, nobody can answer it.

The problem: every AI tool remembers something different

When each tool keeps its own memory, the same question gets different answers. One assistant still cites last year's SOP revision. Another picked up a workaround an analyst typed in once and now repeats it as if it were procedure. A third knows nothing about the calibration decision QA made last month.

None of this shows up as an error. It shows up as small inconsistencies that someone eventually has to untangle, usually during an investigation or an audit.

In a regulated lab, that's not a convenience problem. A fact an AI relies on is part of how a record was produced. If you can't say where it came from, who approved it and which version was in use, you can't defend the record.

What "lab memory" means

Lab memory is the set of facts your AI agents are allowed to rely on across tasks. Typical examples:

  • Which revision of each SOP and specification is in effect, and from when.
  • Decisions QA has made, such as how a recurring deviation type is classified.
  • Normal operating baselines for each instrument and product grade.
  • Known causes for repeat issues, with the investigation that established them.
  • Which systems hold the record of truth for samples, assets and documents.

These are facts a good senior analyst carries in their head. The difference is that an agent's memory can be written down, versioned and checked.

Why every memory write needs an owner

The safest pattern for shared lab memory is simple, and it mirrors how you already control documents:

  1. An agent proposes a new fact. For example, after a CAPA closes, the agent proposes that a specific condenser seal failure is a known cause of low recovery on one analyzer model.
  2. A qualified person approves the write. Nothing enters memory on the agent's say-so. QA, or the owner you designate, reviews and approves it.
  3. Every agent reads the same version. The deviation, CAPA, calibration and training agents all work from one approved state, not their own private copies.
  4. Each fact traces to its source. The approved fact links back to the CAPA, the LIMS results and the SOP revision that support it.

This gives you what ALCOA+ asks of any record: attributable, original, accurate, and available when an inspector asks how a conclusion was reached. It also means a fact can be retired cleanly when an SOP changes, rather than lingering in some tool's history.

Why your lab needs its own eval set

AI vendors quote benchmark scores. Those scores tell you how a model does on someone else's questions. They don't tell you how it handles your methods, your instruments, your naming conventions or the way your QA team writes a deviation.

Small changes can move results a lot, and you can see this for yourself. Take ten closed deviations and ask an assistant to classify each one. Then ask again with the categories listed in a different order, or with a few extra options added. If the answers shift, a generic benchmark wasn't going to warn you. Only testing on your own records will.

A lab eval set is a collection of your own closed, reviewed records, with known correct outcomes:

  • Closed deviations, with the final classification QA approved.
  • Completed CAPAs, with the root cause that held up.
  • Past OOS investigations and how they were resolved.
  • Questions your analysts actually ask, with the correct SOP answer.

Before anything changes, whether that's a new model version, a new prompt or a new batch of memory, you replay the eval set and compare. If the agent gets worse on your records, the change doesn't ship. Reviewer edits and rejections from live work feed back into the eval set, not silently into the model, so it gets sharper over time.

This is the same logic as qualifying an instrument. You don't accept the manufacturer's spec sheet as proof it works in your lab. You test it on your own samples.

Context controlled by you

Context is everything an agent sees when it does a task. Too little, and it guesses. Too much, and it picks up the wrong revision, an unrelated record or data it shouldn't have.

Controlled context means:

  • The right revision as of the right date. An agent drafting a deviation from March sees the SOP that was effective in March.
  • Least-privilege access. Each agent reads only the systems its job requires, and writes are limited and logged.
  • Your data stays yours. Lab records are used to run and evaluate your agents, not to train someone else's model.
  • A record of what was used. Every run keeps a list of the sources it read, so a reviewer can check the basis for a draft.

What about ChatGPT and Copilot?

Keep them. Staff like them, and they're good at general work. The point is that the lab's memory, eval set and context rules shouldn't live inside a general assistant's chat history. They should live in a system the lab controls, and the assistant should ask that system when it needs a lab answer. Standards such as MCP (Model Context Protocol) make that connection possible: the assistant calls a lab tool, and the tool answers from approved memory, with sources.

What to ask before you trust a lab AI

  • Where does the agent's memory live, and who can write to it?
  • Can we see every fact the agent relied on, with its source and approver?
  • Can we build and run an eval set from our own closed records?
  • What happens when the model, prompt or memory changes? Is it re-tested first?
  • How do we control which systems and document revisions each agent can see?
  • Is any of our data used to train models outside our environment?

If a vendor can't answer these clearly, the AI may still be useful for drafts, but it isn't ready to sit inside your quality system.

Where Trucision fits

This is the model Trucision is building toward: agents propose, people approve, and every agent reads from one versioned lab memory that traces back to the LIMS result, instrument log or SOP revision behind each fact. Changes are tested against the lab's own closed records before they go live. Trucision manages that lab operations layer end to end, and assistants such as ChatGPT and Copilot can connect to it as a tool.

FAQ

What is an eval set for lab AI?

A collection of a lab's own closed and reviewed records, such as deviations, CAPAs and OOS investigations, with known correct outcomes. It is used to test an AI agent before go-live and before any change to the model, prompt or memory.

Why shouldn't AI memory live inside a vendor tool or chat assistant?

Because the lab can't see, approve or version what the AI relies on. In a regulated lab, a fact used to produce a record needs a source, an approver and a version, just like a controlled document.

Does shared AI memory mean the AI learns from every interaction?

No. In a controlled design, agents only propose new facts. A qualified person approves each write, and reviewer feedback goes into the eval set rather than silently changing the model.

Can we keep using ChatGPT or Microsoft Copilot?

Yes. Connect them to a lab-controlled system through a standard such as MCP, so lab answers come from approved memory and your records rather than from the assistant's own chat history.

Related reading

Free download: lab eval set starter kit

A step-by-step guide to building your first eval set from 25 closed records: which records to pick, how to label the correct outcome, and a simple scoring sheet to compare any AI tool, including ChatGPT and Copilot. Send me the starter kit.

Bring one closed deviation and we'll show you what an approved, traced draft would look like. Book a 30-minute call.

Scenarios in this article are illustrative examples.

Get started

Find the right plan to modernizeyour product line

We work with OEM product and service leaders to identify the highest-value starting pointy- whether that’s a specific instrument line, a service tier launch, or a compliance gap. Most engagement scope in one session.
Name *
Company
Email *
Role
What are you working on ? *

Thanks!
Your form has been submitted.

Please complete all mandatory fields before submitting.