Label The Body Planes

Annotation Workflows for Anatomical Plane Labeling

I spent about three weeks in 2022 dealing with a dataset where the scanning protocol wasn't consistent across sites. Twenty-two different CT machines, some vendors reoriented the raw DICOM data in ways that made the axial/sagittal/coronal labels ambiguous. The task was straightforward on paper: classify each volume by its primary acquisition plane and tag supporting reconstructions. What I didn't expect was how much the metadata lied.

The core problem with plane labeling isn't the classification itself. It's that radiology workflow tools, public datasets like MIDL challenges, and even some vendor implementations treat "plane" as a fixed property of the scan. It isn't. A single CT study typically has data in all three orthogonal planes because modern scanners acquire isotropic volumetric data and reconstruct along whatever axes the viewing software requests. The original acquisition plane is just one of several valid perspectives, and sometimes it's not even stored correctly in the DICOM tags. Start with the DICOM tags, but don't trust them blindly. Image Orientation (Patient) at tag (0020,0037) gives you the row and column directions in patient space. If the first vector points roughly along the patient's left-right axis and the second points anterior-posterior, you're looking at an axial acquisition. If the first vector runs head-to-toe (superior-inferior), that's likely a sagittal or coronal source. The math is basic dot products against the patient coordinate frame, but the edge cases are where this falls apart. Here's what actually tripped me up on that project: some CT vendors store reconstructed images with the same geometry but change the Slice Location tag in ways that make the apparent plane wrong if you only look at spacing. A study acquired in axial could have sagittal reconstructions where the through-plane distance is 0.5mm but the tag says something else entirely. I ended up computing the voxel dimensions from the actual pixel spacing tags Pixel Spacing (0028,0030) and Slice Thickness (0018,1150), then deriving the effective orientation from the data geometry rather than relying on the acquisition mode field. That cut my error rate from about 8% down to under 2%.

The workflow I settled on was:

• Extract Image Orientation (Patient) for the source series
• Compute the dot product of the row and column vectors against the superior-inferior axis
• Classify based on which anatomical axis each vector aligns with
• Verify against Slice Thickness to confirm through-plane resolution
• Flag any series where the computed plane disagrees with the acquisition mode

This takes about 45 seconds per study with a properly vectorized script, compared to the 8-12 minutes I was spending doing it manually when I started. The bottleneck isn't the computation. It's the edge cases where the metadata is corrupted or the scanner firmware has a bug. Beginners almost always make the same mistake: they assume the plane label is a single value per study. It's not. A typical chest CT will have axial source images, coronal MIPs, and sagittal reformats, all from the same volumetric acquisition. Labeling just one plane loses information and creates confusion downstream. The annotation standard I ended up using was to label the source acquisition plane, then tag all reconstructed series with their actual geometry, even when it differed from the source. Another trap is the patient positioning variation. Some studies are acquired with the patient's arms up, which can shift the apparent orientation in the DICOM headers if the scout view wasn't processed correctly. I encountered this with pediatric scans where the technologist positioned the child differently each time. The solution was to compute the plane from the actual image data orientation, not from the scout or localizer tags. That meant reading the pixel row and column directions from the Image Orientation (Patient) tag and projecting them onto the anatomical axes.

The counter-intuitive insight here is that coronal and sagittal planes are often more reliable than axial for certain pathologies. When you're labeling lung nodules or liver lesions, the through-plane resolution of axial acquisitions is typically 2.5-5mm, while the in-plane resolution might be 0.5-1mm. Reformatted coronal or sagittal views at isotropic 1mm give you better localization. This doesn't change the plane classification, but it does change which plane you should prioritize when creating downstream annotation tasks.

Tooling and Implementation Notes

For actual implementation, pydicom handles the tag extraction, and numpy makes the vector math trivial. The key function is computing the angle between the image row vector and the patient's right-left axis. If it's within 30 degrees, the rows run approximately left-right, which confirms an axial or coronal orientation depending on the column vector. I wrapped this in a small CLI tool that outputs JSON with the predicted planes, confidence scores, and any flagged inconsistencies. The tool takes about 30 seconds to process a full study with 200-300 series, depending on whether you're reading raw DICOM or pre-extracted metadata. Storage is minimal if you're only saving the plane labels and confidence scores, maybe 2-4KB per study. The overhead comes from the DICOM reading itself, which can be slow if you're processing thousands of files without proper caching. One thing I wish I'd known sooner: some PACS systems reorient the data on export, which can change the apparent plane if you're labeling from downloaded files rather than the original archive. This was a major issue when we shared annotations between sites using different PACS vendors. The workaround was to label from the source DICOM at the generating site, then verify the geometry matched on import. That added about 10% overhead but caught the reorientation bugs before they corrupted the dataset.

When Plane Labeling Fails Completely

There are scenarios where automated plane detection simply doesn't work. Oblique acquisitions, interventional fluoroscopy sequences, and some MRI protocols with asymmetric FOV can produce data that doesn't align cleanly with the three orthogonal planes. In those cases, the best approach is to flag the series as "non-standard" and have a radiologist or trained annotator review it manually. Don't try to force a classification where the geometry is genuinely ambiguous. That produces false confidence and corrupts downstream models. I've seen implementations that claim 99% accuracy on plane labeling. Those numbers usually come from clean, single-vendor datasets with standard protocols. On real-world multi-site data with heterogeneous equipment, expect 92-95% accuracy at best, with the errors concentrated on the edge cases I described. If you need higher reliability, combine automated detection with a lightweight review step for flagged series. That usually gets you to 98%+ effective accuracy without requiring full manual annotation of every study. The alternative when automation hits a wall is to use the acquisition protocol metadata as a fallback. Some scanners store the protocol name in DICOM tags, and certain protocol names correlate strongly with specific planes. This isn't reliable enough to replace geometry-based detection, but it can help resolve ambiguous cases where the vector math produces borderline results. I kept a small lookup table mapping common protocol names to expected planes, which resolved about 15% of the flagged cases without additional computation.

Processing time for the full pipeline, including the fallback logic, runs about 45-60 seconds per study on modern hardware. That's fast enough for retrospective dataset curation and manageable for prospective workflow integration, though it won't replace real-time PACS review where latency matters more than throughput.