User Tools

Site Tools




Welcome to the VirtualControlML Wiki. This wiki contains VCML samples and knowledge-base articles designed for users and developers of the Virtual Control Markup Language.

Virtual Control Markup Language (VCML) is an XML-based open standard for defining device input, gesture mapping and virtual overlay initialization, configuration, and management within immersive multiuser, multimodal and cross-modal HCI environments. It is the official markup language for authoring GestureWorks Gameplay and GestureWorks Fusion HCI applications. The goals of VCML are to create reusable and shareable UX models that are programming language-agnostic and to accelerate the development of applications with rich multiuser, multimodal natural user interfaces (NUIs). The VCML standard draws from both the CreativeML, GestureML and DeviceML standards and provides XML-based user-configurable, shareable methods to describe rich layered HCI control schemes.

Ideum: Configurable First Person Shooter MultiTouch Controls Ideum: Custom Styled, Configurable Remote Controls Ideum: Configurable Multi-surface Touch Tangible Controls


VCML provides an infrastructure for rapidly developing and sharing immersive multimodal control schemes for games, applications and operating systems.

Feature List

  • Input Device Support
    • Windows 7/8/10 multitouch screens
    • Android mobile touch devices: (cell phones, tablets)
    • Wearables: (Myo Armband, Nod Ring, Basis Watch, Gloves)
    • Motion Tracking (Kinect, RealSense, Leap Motion, Tobii)
    • Controllers: (WiiMote, PS3 Move, Gamepads)
    • Microphones: (desktop, array, wearable)
  • Multi-modal Gesture Support
    • Multitouch gestures: Drag, Swipe, Flick, Scale, Rotate, Tap, Hold+Tap
    • Motion gestures: 3D head tilt, Eye gaze, 3D hand, 3D body pose and motion
    • Sensor gestures: accelerometer tilt
    • Voice Commands: programmable two word phrases
    • inter-modal gesture sequences
    • Cross-modal gesture combinations
  • Output Emulation Support
    • key commands and keyboard events
    • cursor control and mouse events
    • system multitouch gesture events
    • macro commands
    • X-Input events
    • Direct-Input events
  • Feedback Support
    • Audio Feedback
      • confirmation beeps, tones and whistles
    • Visual Feedback
      • configurable “hudgets” (heads up display widgets) (d-pad, joystick, buttons, track-pad)
      • themes, alpha transparency
      • display object themes and skins
    • Haptic Feedback
      • wearable vibration feedback
      • Android vibration feedback
  • Specialized Control Schemes & Libraries
    • Multitouch Controls Schemes (Desktop, Ultra large displays, laptops)
    • Remote Control Schemes (Android tablet & cell phones)
    • Motion Control Schemes (body, head, eyes and hands)
    • Multimodal Control Schemes (voice, touch, motion and sensors)
    • Multi-surface Control Schemes (multiple touch surfaces)
    • Tangible Control Schemes (surface touch tangibles)
    • Virtual Reality Control Schemes (head mounted motion controls)
    • Augmented Reality & Mixed Reality Controls ()
  • Virtual Control Scheme Management
    • Multi-user controls (up to four users)
    • Multiple control modes “views” (up to eight) per virtual control scheme (pages per document)
    • Methods for switching between application VCML schemes (documents)
    • meta markup methods for documents to aid search
    • hierarchical organization methods for large VCML libraries


Virtual Control Markup Language standard was originally developed to accurately describe user defined dynamic control schemes for gaming applications. It is common in modern games to have changing scenes and dynamic gameplay requirements. As a fundamental part of each virtual control scheme there may be multiple modes of operation existing in a single control scheme description.

Virtual Control Markup Language has been designed to make it easy for developers to manage multiple, dynamic control schemes in real time. A library of virtual control schemes has been developed to provide highly tuned controls that cater to specific device form factors and ergonomics (desktops, laptops, 2-in-1s, tablets, cell phones and Head mounted displays) along with genre-specific optimizations and even scene-specific tweaks.

Each virtual controller can have up to eight unique views. Each view can contain a unique set of hudget and gesture controls. A variety of methods can be used to switch between views, from standard menus to buttons, gestures and voice commands.

VCML Example Basic Touch Control Overlay

<VirtualControlMarkupLanguage xmlns:gml="">
<VirtualControllers schemaVersion="3.0">
	<VirtualController id="0001" name="Youtube Controller"
	<View id="0000"name=""backgroundPath=""thumbnailPath=""remote="False"theme=""screenshotPath="">
		<Group id="0000" x="0" y="0.425" width="0.15" height="0.15">
			<Hudget id="0000" x="0" y="0" width="1" height="1" color="FFFFFFFF" imagePath="Default\homebutton" glyphPath="" type="HomeButton">
			     <Command isProxy="False" delay="0" sensitivity="0" triggerRate="Once" deviceRumbleTime="75" deviceSound=""/>
		<Gesture type="Motion" pose="Pinch" numPoints="1" direction="Any" motion="Hold" sensitivity="1" keypress="">
			<Command isProxy="False" delay="0" sensitivity="0" triggerRate="Once" deviceRumbleTime="0" deviceSound="">
				<Binding state="ButtonDown" value="0x4D" />
		<Gesture type="Motion" pose="Splay" numPoints="1" direction="Any" motion="Hold" sensitivity="1" keypress="">
			<Command isProxy="False" delay="0" sensitivity="0" triggerRate="Once" deviceRumbleTime="0" deviceSound="">
				<Binding state="ButtonDown" value="0x20" />

	<View id="0001"/>
	<View id="0002"/>
	<View id="0003"/>
	<View id="0004"/>
	<View id="0005"/>
	<View id="0006"/>
	<View id="0007"/>

The above example only uses one of eight available views. In the first view the control schema references two motion gestures: “pinch” and “Splay”. In this example, when the pinch gesture is detected, a “Space Bar” key code is dispatched. When a splay gesture is detected an “M” key code is dispatched. The effect of this in the targeted application window (Chrome web browser on the Youtube website) is to pause or mute the Youtube video player.

Anatomy of A View

In many ways a view can be considered a discrete control scheme or “mode”. Creating multiple views in a virtual control scheme allows users to change interaction modes or switch gesture sets during active runtime. This can be done to cater towards dynamic gameplay or simply as a way to change form-factor configurations on the fly. As a rule-of-thumb: any view should contain methods for all critical game controls. Each view may emphasize a specific control method or layout but it should still provide a means to navigate menus and other low level functions that the game or application may require. This avoids the need to switch views unnecessarily and creates a more efficient user driven UI.

The Home Button

The home button is typically visible in all views. It is primarily used for global manipulation of control schemes. When the home button is pressed it provides access to a touch keyboard, allows users to choose between views, quickly hides components in the current view or switches into a run-time view editing mode. Since VCML is designed to create and manage dynamic user-defined control schemes it has built-in affordances for run-time layout editing. Future versions of the scheme will have support for run-time defined gestures and event mapping.

Hudget and Component Controls

VCML supports over 20 types of overlay displayed input components (hudgets). Each component can be independently positioned, sized, tinted, transparency adjusted, thematically skinned and can have glyphs embedded. In addition each component can be mapped to a specific key command, mouse or X-Input events.

  • D-pad
  • Joystick
  • circular buttons
  • square buttons
  • switch buttons
  • oval buttons
  • toggle buttons
  • gesture pad
  • accelerometer

Gesture Controls

An unlimited number of gestures can be applied to any view. Since gestures are exclusive to a view, different gesture controls can be applied to different views and switching between views can change gesture options and gesture mappings. VCML supports multimodal gesture control and command descriptions. This includes multitouch gestures, 3D hand motion gestures, 3D head gestures, 3D body gestures, sensor gestures and voice commands. More information can be found at

Common Touch Control Schemes and Layouts
Multiuser Touch Control Schemes
Genre Customized Touch Controls
Highly Custom Control Schemes
Multi-modal Control Schemes
Multi-surface Control Schemes
Tangible Touch Control Schemes
Virtual Reality Control Schemes
Mixed Reality Control Schemes

Proposed Expansion of the Schema

  • Control scheme conflict resolution options
    • Gesture input conflict mediation
    • Output conflict mediation
  • VCML Document Transfer Options
    • Document differential and version control management methods
    • Simple document summaries and compressed description mode
    • Network friendly transfer document communications and update protocol
  • Control Analysis methods
    • Embedded usage statistics for each component, gesture and over all control scheme
    • Control scheme summary methods for quick parsing and task characterization
virtualcontrolml.txt · Last modified: 2017/09/13 13:19 by ideumadmin