Intel® RealSense™ SDK 2.0 and Open3D

One of the benefits of working with Intel RealSense depth cameras is that the Intel RealSense SDK 2.0, works seamlessly with any of our depth cameras, allowing developers to switch between devices with ease.

The Intel RealSense SDK 2.0 is now integrated with Open3D (version 0.12+), an open-source library designed for processing 3D data. Open3D supports rapid development of software for 3D data processing, including scene reconstruction, visualization, and 3D machine learning. It can be used through both C++ and Python APIs without a separate Intel RealSense SDK installation on Linux, macOS and Windows.

Open3D features

With the recent release of version 0.13, Open3D now includes a variety of new features that will make it easier to process your 3D data for use in robotics, scanning, and edge applications.

Odometry

The raw data of a 3D scanned environment such as the point clouds created by Intel RealSense depth cameras can contain millions of individual points, each one of which has positional and color data attached to it, also known as RGB-D. The larger or more complex a scene, the more unwieldy that amount of data can become with file sizes that can reach up to 150 GB or more. In a 3D scan of a building or other non-organic objects, many of those points can be more efficiently represented by a combination of planes, lines and points. For example, the surface of a table could be millions of individual depth points, or it could be four points that represent the plane of the surface. For many Intel RealSense depth camera use cases, converting a raw point cloud like this can significantly reduce file size and increase efficiency.

The method used to turn those millions of points into more simple combinations of planes, points and lines is referred to as RGB-D Odometry. In Open3D version 0.13, a tensor based real-time RGB-D Odometry pipeline has been introduced. The ability to use this feature as part of the Intel RealSense SDK allows for more efficient storage, analysis and display of 3D scanned environments.

TSDFVoxelGrid

A Truncated Signed Distance Field (TSDF) Voxel grid represents 3D shapes by assigning each voxel a value which represents its distance from a boundary edge. This is different from a basic voxel array which can be used as a method to display and render a 3D object as an alternate to a point cloud or mesh. The value assigned to each voxel can then be used to determine which ones fall on the surface of the object, allowing for a more accurate representation without the need to sacrifice file size. In the latest version of Open3D, this function includes accelerated volumetric integration, and fast ray casting for rendering objects created by 3D scans from a device such as the Intel RealSense LiDAR Camera L515.

Real-time point cloud registration

When scanning a scene or object using an Intel RealSense depth camera, frequently you will be scanning continuous frames over time, moving the depth camera around the object or scene to capture different sides or angles and avoid occlusion. Each of these frames of data in different positions must be combined to create one final whole scene. Similarly, a robot using a depth camera for navigation must continuously scan and compare data from frame to frame so that it can understand the environment it is navigating through.

One technique for comparing two point clouds is “Iterative Closest Point” (ICP). ICP is an algorithm that compares a reference or initial point cloud to a second one and transforms the second point cloud to match it to the first. It takes each point in one data set and matches it to the closest point in the reference data set. This process happens repeatedly (or iteratively) until the best correspondence is found.

Using the Open3D tensor library, Open3D version 0.13 introduces a high-performance implementation of ICP, with support for multi-scale ICP. By iterating on different resolutions of the point cloud data in parallel, convergence of the models can be performed more quickly and efficiently with lower computation cost, something which is extremely valuable in use cases like real-time robotic navigation.

Intel RealSense and Open3D

Intel RealSense has the most comprehensive portfolio of 3D depth cameras and modules on the market today, with the widest range of technologies from LiDAR and stereo depth to standalone SLAM. With the addition of Open3D support to the Intel RealSense SDK, all of these devices can take advantage of the features that have been introduced or improved upon in Open3D version 0.12+ for high quality 3D scanning and robotics use cases.

For more information on Open3D and to get started, please visit our documentation page for Open3D.