How We Made Sentra's NER Model Up to 75% Faster - Without a GPU
Most data security platforms treat their scanning infrastructure as a black box. They throw compute at a problem, and the bill either makes sense or it doesn't. At Sentra, we realize that our customers’ compute is precious, so we think about scanning the way we think about everything else: the performance and economics have to work at petabyte scale. Scanning has to be fast and work flexibility in your environment, on the hardware that's actually running.
This post is about a meaningful improvement we shipped to production this week: a full replacement of our ML inference framework for Named Entity Recognition, the model that sits at the core of how Sentra identifies sensitive entities in unstructured data. The result is up to a 75% improvement in scanning performance and price/performance ratio on CPU instances, with no change in classification accuracy.
The Problem: PyTorch + IPEX Was a Dead End
Sentra's ML Server uses a model called GLiNER for NER - Named Entity Recognition - which identifies sensitive entities (PII, PHI, financial data, credentials, and more) in unstructured files like PDFs, Word documents, email archives, and the 150+ other file formats we support.
Until now, we ran inference through PyTorch, augmented by IPEX (Intel Extensions for PyTorch). IPEX gave us a meaningful 2–3x performance boost on Intel CPUs from Cascade Lake onwards. The problem: IPEX has been deprecated. The last version supports PyTorch 2.8 (August 2025), and PyTorch is now at 2.11. Staying on IPEX meant freezing our ML runtime, not an option.
The deprecation also crystallized a broader architectural question: we were tightly coupled to Torch for inference. That coupling brings dependency overhead and limits our ability to optimize for the hardware customers actually run. It was time to rethink the whole stack.
The Approach: Export to a Runtime-Agnostic Graph
The core idea is deceptively simple: instead of running GLiNER through Python and Torch at inference time, we export the model to an independent execution graph that contains both the computation logic and the model weights. This means inference can happen entirely without Python or Torch dependencies.
Two standards define this kind of portable ML representation:
ONNX (Open Neural Network Exchange) is the broadly supported IR for runtime-agnostic ML computation graphs. The ONNX CPU Execution Provider runs on any CPU and is our baseline for older hardware.
OpenVINO (Intel's open-source inference toolkit) takes ONNX a step further — specifically optimized for Intel silicon and, critically, for CPUs with BF16 (16-bit brain floating point) vectorized math support. On newer CPUs - Intel Granite Rapids, Intel Emerald Rapids, AMD Turin - the difference is significant.
The export process wasn't trivial. GLiNER has three layers with dynamic rank (a dynamic number of tensor dimensions) that neither ONNX nor OpenVINO handle cleanly out of the box. We patched those layers directly to produce a compilable graph. We also had to export in two steps - first to ONNX (to preserve high-level meaning of specific layers like LSTM), then to OpenVINO. The full export now runs in CI/CD as part of the ML models image build.
The Results: Up to 75% Improvement in Performance and Price/Performance
We benchmarked across a representative test set of ~700 unstructured files - 100 PDFs, 100 DOCX files, and equivalents across other formats - simulating the full scanning pipeline from parsing to entity extraction, matching real production conditions.
The headline: 17% to 75% improvement in both wall-clock scan time and price/performance ratio. The higher gains land on older CPU generations, where the gap between legacy Torch inference and ONNX CPU EP is widest.
On the latest CPU hardware - 5th-generation AMD Turin and Intel Granite Rapids - the optimized stack delivers scan costs under $0.16 per job. A GPU instance (G6 with L4) achieves a lower absolute scan time (9m 18s), but at $0.152/job, it no longer has a compelling price/performance advantage over the best CPU configurations. For most production workloads, modern CPUs with OpenVINO are the right answer.
Correctness: Validated Within One Percentage Point
A performance improvement that changes what you find is not an improvement - it's a different product. We validated the patched GLiNER against the reference model on 300 test cases:
Precision and recall for the patched model are within a single percentage point of the reference across all variants. At scale, scanning petabytes of unstructured data, that level of fidelity is what makes the classification trustworthy.
Already in Production: Global Logistics Company
About a week and a half before the general release, we deployed the optimized ML Server to a major logistics enterprise's environment. The results matched benchmark expectations - meaningful scan acceleration with no change in the classification findings their security team relies on. That early production validation gave us confidence to push the release broadly.
What This Means for Unstructured Data Security
The reason this matters beyond engineering is that unstructured data is where sensitive information actually lives in most organizations. Contract PDFs, employee records, medical notes, financial models, email archives, scanned documents, the files that contain the data regulators and attackers care about most are overwhelmingly unstructured.
Scanning unstructured data at petabyte scale is computationally expensive. Every efficiency gain in the ML inference layer translates directly into faster time-to-classification, lower scanning cost, and the ability to run continuous classification rather than periodic batch jobs. A 40% improvement in price/performance on a 9-petabyte environment is not a marginal gain, it's a material change in what's economically feasible. This is compounded in environments in which compute is scarce or limited for various infrastructural reasons, but scanning speed is still a priority..
This release applies to all file types Sentra supports for unstructured scanning: PDF, DOCX, XLSX, PPTX, email formats, images (via OCR), and the full range of specialized formats; DICOM, EDI, Tableau extracts, pickle/joblib, OneNote, Draw.io, and more. The NER layer that benefits from this optimization runs across all of them.
If you want to go deeper on how Sentra classifies unstructured data across cloud, SaaS, and on-premises environments, explore the platform overview at sentra.io or schedule a demo.
Discover Ron’s expertise, shaped by over 20 years of hands-on tech and leadership experience in cybersecurity, cloud, big data, and machine learning. As a serial entrepreneur and seed investor, Ron has contributed to the success of several startups, including Axonius, Firefly, Guardio, Talon Cyber Security, and Lightricks, after founding a company acquired by Oracle.
