Medical image annotation for regulated AI: evidence, not just labels

Executive summary

Medical image annotation is where a correct label meets a compliance obligation. For regulated AI, medical data labelling has to prove three things: that protected health information was de-identified before anyone annotated it, that a trained clinician reviewed the work, and that the whole chain is logged and auditable. The label is the easy part. The evidence is what a hospital accepts.

Medical image annotation is a compliance problem, not just a labelling problem

A radiologist-grade segmentation mask can be perfectly accurate and still be useless to a hospital. Not because the label is wrong, but because nobody can show how the underlying scans were handled. In regulated medical AI, a dataset is not accepted on the quality of its labels alone. It is accepted on the evidence around them.

This is the gap most annotation content skips. Vendor pages explain how to draw a bounding box or run a segmentation workflow. That is table stakes. The harder question, the one a legal and procurement team asks before a single scan is trusted, is whether the medical data labelling process itself was controlled. Who saw the patient’s data. When it was de-identified. Who reviewed the clinical judgment. Whether any of it can be reconstructed a year later under audit. Get the labels right and fail that, and the dataset stalls in review.

If you are also selling into the EU, the pressure is dated. The EU AI Act adds high-risk obligations for in-scope systems, effective 2 December 2027. In the US, the constraint is HIPAA, and it applies the moment protected health information touches your pipeline.

What happens to PHI before anyone annotates?

The rule is simple to state and easy to get wrong: de-identification happens before annotation, not after. An annotator should never need to see a patient’s identity to draw a lesion boundary. If your process de-identifies on the way out, protected health information has already been exposed on the way in.

Under HIPAA, de-identification has two accepted routes. Safe Harbor requires removing the eighteen categories of identifiers listed at 45 CFR 164.514(b), from names and dates to medical record and account numbers. Expert Determination has a qualified expert certify that the re-identification risk is very small. Either is defensible. Guessing at a middle path is not.

DICOM makes this sharper, because protected health information hides in two places. The first is header metadata: patient name, patient ID, study dates, accession and device fields. The second is easy to forget: text burned directly into the pixels, the identifiers a technician typed onto the image itself. Scrub the header and miss the burned-in pixels, and you have shipped PHI inside the image you thought you cleaned.

Access control is the other half. Minimum-necessary access means an annotator sees the pixels the task requires and nothing that names the person. On regulated projects, that principle also has a physical shape. Annotation runs on a controlled on-premises floor, no personal devices admitted, with named entry and exit logs and no remote work. So the question “who could have removed a scan” has a physical answer, not only a policy on a page.

Who labels, and who checks the labels?

Crowd labelling is fine for stop signs and house cats. It is the wrong tool for a suspected lesion boundary. Clinical imaging usually needs an annotator with real domain training, someone who can read the modality and understand what the task is actually asking. That is the difference between a labeller and a trained medical reviewer, and it is not a detail you can wave away with volume.

Separation of duties matters just as much. The person who labels should not be the person who signs off on the label. A second, qualified reviewer is how systematic error gets caught and how you can show the review was independent rather than self-graded. It is a control, not a courtesy.

Then measure it. Ask a vendor for their inter-annotator agreement, the Cohen’s kappa on your task type, and watch the response. A team that measures quality answers with a number in one sentence. A team that only asserts quality answers with a brochure. On medical work, that single question separates the two faster than any capability deck.

Working with DICOM: segmentation and radiology workflows

A medical image annotation tool that cannot handle DICOM natively is not really built for this work. Radiology workflows expect DICOM in and structured, standards-compliant output out. That means voxel-level segmentation across a stack of slices, not just a flat box on a single frame, and it means the annotation types clinicians already use rather than a generic labelling canvas retrofitted for images.

DICOM annotation also has to respect the metadata split described above. The tool needs to preserve the technical fields a task genuinely depends on, such as windowing, slice spacing, and orientation, while stripping the identifying fields that carry PHI. Keeping windowing and deleting the patient name are two different operations, and a mature pipeline treats them that way. When the output is a DICOM SEG object or a clean overlay that maps back to the source series, a radiologist can trust that what they are reviewing lines up with the scan it came from.

What does a hospital’s legal and procurement team actually ask for?

By the time a dataset reaches procurement, the questions are predictable, and “trust us” is not an answer to any of them.

• Can you show who touched this data?
• Can you prove de-identification happened before annotation, and by which method?
• If we audit you in eighteen months, can you produce the log?

Two things answer those questions. An immutable, tamper-evident log records every action across the dataset: who de-identified, who annotated, who reviewed, when each happened, and what changed. A chain of custody proves control of every file from ingest to delivery, with no unexplained gap where a scan could have gone somewhere unrecorded. Combine that digital log with the physical floor controls, named entry and exit records on-site, and the answer to “who had access, and when” stops being a promise and becomes a record.

What goes in an evidence pack

The deliverable in regulated annotation is not just labelled data. It is labelled data plus the file that lets a legal team say yes. Call it an evidence pack. On a serious medical project it contains:

• The inter-annotator agreement or kappa score, broken out by task type.
• A control mapping that ties each control to the requirement it satisfies.
• Full provenance and chain of custody for the data.
• The de-identification method used, and an attestation that the process ran.
• The credentials of the clinical reviewers.

That pack is the whole point. It turns “we labelled your scans” into “here is the dataset, and here is what your legal and procurement team needs to accept it.” The label was step one.

When you don’t need this. If you are annotating fully public, already de-identified research imaging with no PHI and no regulatory exposure, most of this apparatus is overhead. A lighter, faster pipeline is the rational choice, and paying for evidence you will never be asked to produce is a waste. The controls earn their cost the moment a real patient’s data and a real regulator are involved. If that is not your situation, be honest about it and move faster.

The reason to treat annotation as a compliance problem is not caution for its own sake. It is that redoing a medical dataset because it cannot survive review is far more expensive than building it with the evidence the first time. For related reasoning on where to draw the line between building this in-house and using a partner, see our piece on data annotation outsourcing. Our data security and trust page spells out what a full evidence pack contains.

Next step

If you are assembling a medical imaging dataset that a hospital, a regulator, or an acquirer will eventually inspect, the label is the last thing to worry about and the evidence is the first. Scope a proof of concept on LabelFort, Predusk’s data annotation platform, and see what the evidence pack looks like on your own task type.

This article describes regulatory requirements and annotation practices for medical imaging. It is not legal advice. Confirm your de-identification method and compliance obligations for your own jurisdiction and use case.