User Tools

Site Tools


virtualcontrolml:multisurface_control_schemes

Multi-surface Touch Control Schemes

Multi-surface controls can be applied to a relatively small group of new devices. One such device is the HP Sprout. The HP Sprout is a unique device that uses a 21“ pcap multitouch “vertical” display and an 18” pcap multitouch mat as a projected secondary interactive display. Virtual control schemes have been created for the Sprout that use component-based touch controls on the mat and complimentary global gesture controls on the main vertical display.

Ideum & HP Sprout Dual Screen Device Ideum & HP Sprout GamePlay Multitouch Controls

In addition to multi-surface touch controls Sprout can be enabled to use it's embedded RealSense camera to create an interactive volume above the mat. This volume can be configured to track hands and fingertips and enable the continuous tracking of interaction points from surface A through the volume to surface B with a consistent unique “interaction points” ID maintained in context. The intelligent use of context in multimodal input control schemes by leveraging multiple surfaces and volumes can create break-though levels of immersive, fluid interactions that seamlessly move between context spaces.

RetroDepth, Sense hands, styluses and object silhouettes as they interact on and above physical surfaces

Beyond seamless 3D motion and 2D surface interactions, using motion tracking along side touch tracking technologies can allow other input methods such as stylus, pen and 3D object tools. For example the image above shows a stereo IR camera, a retro-reflective touch mat that use novel contour analysis along with object segmentation to provide rich hand and object 3D motion tracking co-registered with dynamically defined 2D surface intersections.

HyperReal Objects

The HP Sprout can scan 3D objects using the integrated RealSense depth map camera. Once this is be done a 3D mesh of the object along with an RGB snapshot can be superimposed (1:1) on the real object using the built-in pico-projector. When this occurs it creates an interesting “hyper-real” effect that gives the real object a brighter, color accentuated look with increased contrast. The effect can be cleanly segmented from the background of the mat and developed further using object tracking to maintain the projection as the object is moved on the 2D surface.

Adding hyperreal properties to 3D objects is a great way to distinguish objects from a group or highlight object features. For example: When learning about currency, bills and coins can be placed on the HP Sprout mat and different denominations automatically scanned. Coins with specific values can then be highlighted as part of an interactive presentation. In other applications simple 3D character forms can be scanned into the system and used as complex projection surfaces by projecting animated facial expressions directly onto the head of the character. This type of enrichment can be subtle or deliberately used to make an object stand out.

Using the 3D object as a dynamic canvas uniquely leverages the context of the object. This can be used within an expanded 3D scene to directly augment observable elements of the item while maintaining the natural manipulations of the real object and it's tangible real-world properties.

Tangible Touch Control Schemes

virtualcontrolml/multisurface_control_schemes.txt · Last modified: 2016/01/27 00:26 by paul