Smart Wheelchairs with LUCI and Intel RealSense Technology

Many of us have had the experience of traveling down an unfamiliar street and discovering a pothole or steep curb a fraction too late. When you are a wheelchair rider, those potholes and curbs can be dangerous, accident-causing barriers to safe transportation. According to one study, there are over 175,000 wheelchair-related ER visits every year. More than 65% of those injuries are caused by tips and falls, and 17% of the injuries required hospital admissions. Despite these high numbers, before LUCI, nothing was being done to address this issue using today’s leading technologies.

LUCI CEO Barry Dean is driven by more than these statistics though. His daughter Katherine has been in a power wheelchair for nearly her entire life, and LUCI is designed to offer wheelchair riders like her the freedom to safely explore their world with confidence, rather than fear.

CTO Jered Dean on how LUCI started.

The rider at the heart of the solution

LUCI’s smart technology places the rider at the center of the entire driving process, enriching their lives by helping them harness the power of their chair safely. The goal of LUCI’s design team was not autonomy – this is not a self-driving car. Their goal was to give riders independence along with unprecedented levels of stability, security and connectivity.

Why LUCI chairs have names

The LUCI SmartFrame mounts onto a current power wheelchair between the wheels and seat. It combines an attachable hardware frame which includes Intel® RealSense™ depth cameras along with custom software, cloud- and sensor-fusion technologies to enable important safety features like collision avoidance, drop-off and tip protection and more.

Detailed depth data is passed to the system from three Intel RealSense depth module D430s, paired with Intel RealSense Vision Processor D4 cards in custom housings. Using the modules and ASIC cards allows the system to be more lightweight and fit better into the limited space available on power wheelchairs. The SmartFrame also features custom lenses with optical filters, similar to those described in this whitepaper.

Why depth matters

The data from the cameras is used to check the ground around the power wheelchair for unsafe steps and drop-offs. In addition, the depth data from the cameras is fused with information from other sensors – custom ultrasonic and millimeter wave radar sensors to help the power wheelchair “see” its surrounding environment. These sensors work together to help wheelchair riders avoid common dangers like curbs and stairs, but also help avoid collisions with objects in motion around them, such as people or pets. The advantage of using the stereo depth cameras specifically is their ability to accurately sense depth both indoors and outdoors in varied lighting conditions, while remaining cost effective and operating at low-power. Barry and Jered Dean, the brothers behind LUCI’s creation, tested a wide variety of sensors, but found the low-light capability enabled by the infrared projectors was critical to low-lighting operation.

Wheelchair riders talking about injuries

When it comes to autonomous driving, cars can take advantage of GPS mapping, road markings, signs, and other data that is unavailable to someone navigating the world in a power wheelchair. Unlike those autonomous systems, LUCI is striving for shared control in a dynamic world. In crowded spaces, narrow hallways or tight sidewalks, driving a power chair can be stressful and potentially dangerous. The custom sensor fusion algorithms allow LUCI to recognize movement, potential collisions and adapt user steering inputs to ensure the safety of everyone around. All of this is accomplished without the need for external signs, markers, or prior training, giving the rider the freedom to explore new areas without fear.

Safety-critical functions like ground mapping also require stereo depth, so relying on single camera computer vision without the depth camera was a non-starter for the LUCI engineering team. Depth is fundamental to the mission of keeping wheelchair riders safe, and ensuring the ground is safe to drive on.

Make existing wheelchairs smarter

One of the important product decisions the LUCI team made was to not build an entire smart wheelchair, but rather build a way to make new and existing wheelchairs smarter. This enables riders to use the chairs they already have and are comfortable with, but make those chairs smarter and safer. By attaching LUCI to their new or existing power chairs, riders gain long overdue access to a world – their world – where collisions are avoided and falling off ramps or curbs is prevented.