Gesture Control in Adobe XD

25 Jun 2019

Adobe XD launched key command support recently, that allows you to trigger actions based on gamepad or keyboard input. You can use this feature to control your prototype with other inputs, by mapping values to keyboard events on the web.

I created a Chrome Extension (code on GitHub) that takes in a webcam feed and translates hand gestures to key commands.

  1. It uses TensorFlow.js and the PoseNet model to identify your hand position relative to your body.
  2. Then, the extension checks if your hands are in one of a few orientations to your shoulders and face.
  3. If the gesture matches a command, the plugin creates a keyboard event that is sent to the canvas, like so:
const ev = new KeyboardEvent("keydown", { key: "w" }); // replace "w" with the keyboard trigger value
document.querySelector("#root-1").dispatchEvent(ev);

The XD web application handles the rest!

Combine this with voice commands, that are built into XD, to prototype a multimodal experience.