AI in endoscopy and computer-assisted surgery
Real-time video and AI in the medical domain
Digital imaging and digital video are used more and more often in medical applications, and especially in clinical diagnostic, patient monitoring, and in the operating room (OR).
Besides its primary purpose being to carry on an instrument camera signal to the practitioner’s eye, medical video streaming eases information aggregation from different sources, collaboration between doctors, and medical training. It also opens the doors to computer assistance during the medical diagnostic and even during a surgical procedure.
A very frequent use case of video transport in the medical domain is its usage in endoscopy.
Endoscopy procedures exist since more than a Century, starting out of simple rigid tubes and of mirrors driving a candle light, then progressively evolved until nowadays endoscopes using video interfacing technologies like SDI and HDMI.
Endoscopy applications also increased and now cover use cases from simple diagnostic until surgical navigation.
In parallel, these last years recorded a booming usage of artificial intelligence (AI) in many professional and industrial domains, and the healthcare sector is a very suitable field of application for AI assistance.
AI-powered medical devices and video processors help the doctor making an accurate diagnostic, or assist the surgeon during a minimally invasive procedure.
AI-driven Computer-Assisted Decision (CAD) and Computer Assisted Surgery (CAS) systems positively impact the patient outcome.
NVIDIA Clara Holoscan
To answer the growing need of AI processing in the medical domain, NVIDIA introduced Clara Holoscan, an AI sensor processing platform that combines hardware systems for low-latency sensor and network connectivity, optimized libraries for data processing and AI, and core microservices to run streaming, imaging, and other applications, from embedded to edge to cloud.
Besides its main application domain being medical devices development, Holoscan can also be used to build streaming AI pipelines for a variety of other applications including High Performance Computing at the Edge, Industrial Inspection and more.
On the hardware side, the Clara Holoscan platform is available under two forms:
- The AGX developer kit is a prototyping device hosting a NVIDIA Jetson AGX Xavier™ Arm SoC, a RTX™ 6000 GPU and a ConnectX 100GbE NIC
- The NVIDIA IGX computing platform can be embedded directly into a medical device or attached as a sidecar for clinical edge computing. It is designed based on a Jetson AGX Orin SoC and a ConnectX 7 NIC
On top of the platform runs the Holoscan software ecosystem and SDK, an optimized framework integrating pre-trained AI models and reference architectures for endoscopy, ultrasound and colonoscopy applications.
The Holoscan software stack is based on GXF (Graphical eXecution Framework), a component-based programming environment easing the development of streaming data processing pipelines.
GXF is built for efficient data ingestion, data transfer, and AI/ML workloads, it notably integrating I/O devices like the DELTACAST cards through a series of GXF Extensions (aka input and output codelets).
In the Holoscan ecosystem, high-throughput and low-latency transfers between the DELTACAST I/O card and the GPU-based AI processing are achieved thanks to the GPUDirect RDMA technology, allowing peer-to-peer exchanges between those system components.
Besides GXF codelets, DELTACAST also provides native C++ operators addressing its video products range.
DELTACAST solutions for medical AI device developers
The complete portfolio of DELTACAST cards is compatible with the NVidia Clara Holoscan platforms – AGX and IGX – and by this way also compatible with NVIDIA Jetson AGX Xavier and Orin Arm 64-bit computers.
The DELTACAST I/O cards are all handled by a unique SDK called VideoMaster, for a neat integration and efficient porting of your application from one interface to another one. The portfolio includes I/O solutions for SDI, HDMI, DisplayPort, DVI, … and some products include a hardware keyer for on-the-fly overlay of computer graphics over the live video feed.
DELTACAST and NVIDIA collaborate since the ignition of the Holoscan program, offering a unique-on-the-market solution for live mixing of AI-generated overlay on top of SDI camera video, in 4K, and secured through an exclusive bypass feature against power outages, software crashes and system tampering.
The DELTACAST GXF codelets and C++ Holoscan operators are available in open-source on the NVIDIA Holohub repository in the Holoscan project :
https://github.com/nvidia-holoscan/holohub
