Hand Tracking Developer Guide
Hand tracking lets users interact intuitively with virtual content by using natural hand gestures as an input method. The Magic Leap 2 headset camera detects hand movement in real time.
Magic Leap supports hand tracking for the Magic Leap native C-API (MLSDK), MRTK 2.8, MRTK 3, Unity, and OpenXR.
End users can enable additional hand-tracking capabilities available in the Magic Leap 2 OS. See the Hand Tracking Guide in the Magic Leap Care portal. Enabling or disabling these features does not affect hand tracking in your apps.
Unity with OpenXR
If you are developing apps in Unity with the Magic Leap 2 OpenXR Unity SDK, refer to the Unity (OpenXR) hand-tracking topics for detailed information.
The Magic Leap 2 OpenXR Unity SDK supports the OpenXR hand interaction extension, XR_EXT_hand_interaction, and the OpenXR palm pose extension XR_EXT_palm_pose.
Unity with MLSDK
If you are developing apps in Unity with MLSDK, use the Magic Leap Hand Tracking API for Unity and Gesture Classification API for Unity to access hand-tracking capabilities and assign actions to hand gestures. The Gesture Classification API helps you create more intuitive user experiences as users interact with virtual content using their hands.
Using hand-tracking keypoint data, the Gesture Classification API recognizes when the geometric properties of the user’s hands conform to certain gestures, called key poses. You assign user interaction to these gestures. The API provides predefined Supported key poses. You can also create your own custom key poses. The API exposes all the hand characteristics you’ll need, such as the relative positions and angles of the keypoints.
You can also use the Gesture Classification API to query data about the user’s hand, such as finger length and extension angle.
See this overview of the Gesture Classification API for more information.
Hand-Tracking Keypoints
Hand-tracking keypoints are locations that correspond to hand joints, finger joints, and fingertips. These keypoints are used for designing interactions and attachment of virtual content.
When you're using the MLSDK, the Magic Leap 2 device assigns 26 keypoints to each hand to create a three-dimensional representation of the hand. Twenty-one keypoints correspond to visible points on the hand. The other five keypoints are calculated based on these 21 keypoints.
Using this model of the user’s hands, the device simultaneously tracks hand position and rotation across three dimensions. This way, it can locate the position of the hand in space, classify gestures, and identify the right and left hand simultaneously.
Supported Key Poses
Key poses are static hand position classifications based on how closely the hand’s keypoints align with specified criteria. Each key pose falls under a more general grouping of hand positions called postures.
| Name | Key Pose | Description |
|---|---|---|
| OK |  | The thumb and index finger are flexed and their tips touch to form a circle. The other three fingers are partially extended. Posture: Pinch. |
| C |  | Fingers and thumb are flexed. Fingers are together and curved slightly. The hand resembles the letter C. Posture: Pinch. |
| Pinch |  | The thumb and index finger are flexed. The tips of the thumb and index finger are close to each other but not touching. The other fingers are flexed toward the palm. Posture: Pinch. |
| Finger |  | The index finger is extended. The thumb and other fingers are flexed toward the palm. Posture: Point. |
| L |  | The thumb is extended to the side of the palm and the index finger is extended to form a shape that resembles the letter L . The other fingers are flexed toward the palm. Posture: Point. |
| Thumb |  | The thumb is extended to the side of the palm. All other fingers are flexed toward the palm. Posture: Grasp. |
| Fist |  | All fingers, including the thumb, are flexed into the palm. Posture: Grasp. |
| Open |  | All fingers, including the thumb, are extended. Posture: Open. |
Hand Characteristics
Hand characteristics are the geometric properties of the hand based on its keypoints. You can use these hand characteristics to create your own hand interaction constraints and custom key poses.
| Name | Hand Characteristic | Description |
|---|---|---|
| Hand transformation |  | Hand transformation describes the directional and rotational movement of the hand center. |
| Splay angles |     | Splay angles are the angles between adjacent fingers, capturing the splay of the hand. The splay angle of each adjacent pair of fingers is calculated using the tips of the fingers and the midpoint between the MCP keypoints of the finger. |
| Finger to palm angle |  | The finger to palm is the angle between the finger and the palm. It’s calculated using the wrist-to-center vector of the hand and the MCP-to-tip vector of the finger. |
| Finger extension |    | The finger extension is the angle of the first joint of the finger. It’s calculated using the MCP-to-PIP vector and the PIP-to-DIP vector of the finger. |
| Pinch angle |  | The Pinch angle is the angle between the index finger and thumb when the hand is in a Pinch posture. It’s calculated using the tip of the thumb, the tip of the index finger, and the midpoint between the MCP keypoints of the thumb and index finger. |
| Pinch distance |  | The Pinch distance is the distance between the tip of the thumb and the tip of the index finger when the hand is in the Pinch posture. It’s calculated using the tip of the thumb and the tip of the index finger. |
Interaction Points
An interaction point is a transform which dynamically adapts translation and rotation based on the current posture of the hand. It provides a single optimal point of reference that simplifies common direct and indirect hand interactions.
| Name | Hand Characteristic | Description |
|---|---|---|
| Point |  | The Point interaction point is the tip of the index finger when the hand is in the Point posture. |
| Pinch |  | The Pinch interaction point is the midpoint between the tip of the thumb and the tip of the index finger when the hand is in the Pinch posture. |
Design Considerations
For guidelines on designing apps that use hand tracking, see the Hand tracking design guide.