XR_ML_eye_tracker
This API is still an experimental extension not included in the official OpenXR registry and is subject to change.
12.104. XR_ML_eye_tracker
- Name String
XR_ML_eye_tracker- Extension Type
Instance extension
- Registered Extension Number
485
- Revision
1
- Extension and Version Dependencies
- Last Modified Date
2024-01-17
- Contributors
Karthik Kadappan, Magic Leap
Ron Bessems, Magic Leap
12.104.1. Overview
XR_ML_eye_tracker extension enables applications to work with the Magic Leap eye tracking system.
This section provides an overview of some of the key concepts needed to understand how this extension works. Refer to the later sections for more details.
The eye tracking system data is organized into the following buckets:
Geometric Data: This provides information about geometric attributes such as openness of the eye.
Pupil Data: This provides information about pupil size. This data is gated by a dedicated permission, different from permission required for the rest of the eye tracking system.
Gaze Behavior: This provides information about eye behaviors such as saccade, pursuit, etc.
Static Data: This includes information such as width and height of the eye camera sensors which are static for a given hardware.
Pose Data: This provides information about poses.
Here are a few other related extensions that deal with eye tracking:
XR_ML_user_calibration: Applications can use this extension to ensure good eye calibration which directly affects the performance of theXR_ML_eye_trackerandXR_ML_facial_trackingXR_ML_facial_trackingXR_ML_pixel_sensors
12.104.2. Inspect system capability
An application can inspect whether the system is capable of eye tracking by extending the XrSystemProperties withXrSystemEyeTrackerPropertiesML structure when callingxrGetSystemProperties.
If a runtime returns XR_FALSE for supportsEyeTracking, the runtime must return XR_ERROR_FEATURE_UNSUPPORTED forxrCreateEyeTrackerML.
The XrSystemEyeTrackerPropertiesML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrSystemEyeTrackerPropertiesML {
XrStructureType type;
void* next;
XrBool32 supportsEyeTracking;
} XrSystemEyeTrackerPropertiesML;12.104.3. Eye Tracker Permissions
Permissions Android applications must have the com.magicleap.permission.EYE_TRACKING permission listed in their manifest to use this extension. (protection level: dangerous) If the applications want access to the pupil size the it must have the com.magicleap.permission.PUPIL_SIZE permission listed in their manifest as well (protection level: dangerous). But applications can continue to use rest of the extension without this permission. |
12.104.4. Create and destroy a tracker handle
Applications can create a eye tracker handle using thexrCreateEyeTrackerML. This will provide an XrEyeTrackerML handle.
The XrEyeTrackerML handle is defined as:
// Provided by XR_ML_eye_tracker
XR_DEFINE_HANDLE(XrEyeTrackerML)The xrCreateEyeTrackerML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrCreateEyeTrackerML(
XrSession session,
const XrEyeTrackerCreateInfoML* createInfo,
XrEyeTrackerML* eyeTracker);The XrEyeTrackerCreateInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerCreateInfoML {
XrStructureType type;
const void* next;
} XrEyeTrackerCreateInfoML;To close the eye tracker and release all the resources callxrDestroyEyeTrackerML.
// Provided by XR_ML_eye_tracker
XrResult xrDestroyEyeTrackerML(
XrEyeTrackerML eyeTracker);12.104.5. Get geometric data
The xrGetEyeTrackerGeometricDataML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrGetEyeTrackerGeometricDataML(
XrEyeTrackerML eyeTracker,
const XrEyeTrackerGeometricDataGetInfoML* info,
uint32_t geometricDataCapacityInput,
uint32_t* geometricDataCountOutput,
XrEyeTrackerGeometricDataML* geometricDatas);The XrEyeTrackerGeometricDataGetInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerGeometricDataGetInfoML {
XrStructureType type;
const void* next;
} XrEyeTrackerGeometricDataGetInfoML;The XrEyeTrackerGeometricDataML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerGeometricDataML {
XrStructureType type;
void* next;
XrEyeML eye;
XrTime time;
XrBool32 valid;
float eyeOpenness;
XrVector2f eyeInSkullPosition;
} XrEyeTrackerGeometricDataML;// Provided by XR_ML_eye_tracker
typedef enum XrEyeML {
XR_EYE_LEFT_ML = 1,
XR_EYE_RIGHT_ML = 2,
XR_EYE_MAX_ENUM_ML = 0x7FFFFFFF
} XrEyeML;| Enum | Description |
|---|---|
| Left eye. |
| Right eye. |
12.104.6. Get pupil data
Runtime must gate the pupil data behind the com.magicleap.PUPIL_SIZE permission in addition to the com.magicleap.EYE_TRACKING permission.
The xrGetEyeTrackerPupilDataML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrGetEyeTrackerPupilDataML(
XrEyeTrackerML eyeTracker,
const XrEyeTrackerPupilDataGetInfoML* info,
uint32_t pupilDataCapacityInput,
uint32_t* pupilDataCountOutput,
XrEyeTrackerPupilDataML* pupilDatas);The XrEyeTrackerPupilDataGetInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerPupilDataGetInfoML {
XrStructureType type;
const void* next;
} XrEyeTrackerPupilDataGetInfoML;The XrEyeTrackerPupilDataML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerPupilDataML {
XrStructureType type;
void* next;
XrEyeML eye;
XrTime time;
XrBool32 valid;
float pupilDiameter;
} XrEyeTrackerPupilDataML;12.104.7. Get gaze behavior
The xrGetEyeTrackerGazeBehaviorML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrGetEyeTrackerGazeBehaviorML(
XrEyeTrackerML eyeTracker,
const XrEyeTrackerGazeBehaviorGetInfoML* info,
XrEyeTrackerGazeBehaviorML* gazeData);The XrEyeTrackerGazeBehaviorGetInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerGazeBehaviorGetInfoML {
XrStructureType type;
const void* next;
} XrEyeTrackerGazeBehaviorGetInfoML;The XrEyeTrackerGazeBehaviorML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerGazeBehaviorML {
XrStructureType type;
void* next;
XrTime time;
XrBool32 valid;
XrEyeTrackerGazeBehaviorTypeML gazeBehaviorType;
XrTime onsetTime;
XrDuration duration;
} XrEyeTrackerGazeBehaviorML;// Provided by XR_ML_eye_tracker
typedef enum XrEyeTrackerGazeBehaviorTypeML {
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_UNKNOWN_ML = 0,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_EYES_CLOSED_ML = 1,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_BLINK_ML = 2,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_BLINK_LEFT_ML = 3,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_BLINK_RIGHT_ML = 4,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_FIXATION_ML = 5,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_PURSUIT_ML = 6,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_SACCADE_ML = 7,
XR_EYE_TRACKER_GAZE_BEHAVIOR_TYPE_MAX_ENUM_ML = 0x7FFFFFFF
} XrEyeTrackerGazeBehaviorTypeML;| Enum | Description |
|---|---|
| Unknown behavior. |
| Both eyes closed. |
| Blink detected. Both eyes open, close, and open.. |
| Left eye blink, right eye open. |
| Right eye blink, left eye open. |
| User is fixating, eye position is stable. |
| User is pursuing, eye velocity is low but nonzero. |
| User is making a saccade, eye velocity is high. |
Some of the gaze behaviors may have additional information and the applications can access them by chainingXrEyeTrackerGazeBehaviorMetaDataML toXrEyeTrackerGazeBehaviorML.
The XrEyeTrackerGazeBehaviorMetaDataML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerGazeBehaviorMetaDataML {
XrStructureType type;
void* next;
XrBool32 valid;
float amplitude;
float angle;
float velocity;
} XrEyeTrackerGazeBehaviorMetaDataML;12.104.8. Get static data
The xrGetEyeTrackerStaticDataML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrGetEyeTrackerStaticDataML(
XrEyeTrackerML eyeTracker,
const XrEyeTrackerStaticDataGetInfoML* info,
XrEyeTrackerStaticDataML* staticData);The XrEyeTrackerStaticDataGetInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerStaticDataGetInfoML {
XrStructureType type;
const void* next;
} XrEyeTrackerStaticDataGetInfoML;The XrEyeTrackerStaticDataML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerStaticDataML {
XrStructureType type;
void* next;
float eyeWidthMax;
float eyeHeightMax;
} XrEyeTrackerStaticDataML;12.104.9. Get pose data
The xrGetEyeTrackerPosesML function is defined as:
// Provided by XR_ML_eye_tracker
XrResult xrGetEyeTrackerPosesML(
XrEyeTrackerML eyeTracker,
const XrEyeTrackerPosesGetInfoML* info,
XrEyeTrackerPosesML* eyePoses);The XrEyeTrackerPosesGetInfoML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerPosesGetInfoML {
XrStructureType type;
const void* next;
XrSpace baseSpace;
XrTime time;
XrEyeTrackerPoseFlagsML poseFlags;
} XrEyeTrackerPosesGetInfoML;The bitmask type XrEyeTrackerPoseFlagsML is defined as:
// Provided by XR_ML_eye_tracker
typedef XrFlags64 XrEyeTrackerPoseFlagsML;As used in XrEyeTrackerPosesGetInfoML::poseFlags field,XrEyeTrackerPoseFlagsML contains a bitwise-OR of zero or more of the bits defined in XrEyeTrackerPoseFlagBitsML.
// Provided by XR_ML_eye_tracker
// Flag bits for XrEyeTrackerPoseFlagsML
static const XrEyeTrackerPoseFlagsML XR_EYE_TRACKER_POSE_GAZE_BIT_ML = 0x00000001;
static const XrEyeTrackerPoseFlagsML XR_EYE_TRACKER_POSE_LEFT_BIT_ML = 0x00000002;
static const XrEyeTrackerPoseFlagsML XR_EYE_TRACKER_POSE_RIGHT_BIT_ML = 0x00000004;
static const XrEyeTrackerPoseFlagsML XR_EYE_TRACKER_POSE_FIXATION_BIT_ML = 0x00000008;The flag bits have the following meanings:
The XrEyeTrackerPosesML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerPosesML {
XrStructureType type;
void* next;
XrEyeTrackerPoseML pose[XR_EYE_TRACKER_POSE_TYPE_COUNT_ML];
} XrEyeTrackerPosesML;The XrEyeTrackerPoseML structure is defined as:
// Provided by XR_ML_eye_tracker
typedef struct XrEyeTrackerPoseML {
XrStructureType type;
void* next;
XrBool32 valid;
XrPosef pose;
XrTime time;
XrEyeTrackerConfidenceML confidence;
} XrEyeTrackerPoseML;// Provided by XR_ML_eye_tracker
typedef enum XrEyeTrackerPoseTypeML {
XR_EYE_TRACKER_POSE_TYPE_GAZE_ML = 0,
XR_EYE_TRACKER_POSE_TYPE_LEFT_ML = 1,
XR_EYE_TRACKER_POSE_TYPE_RIGHT_ML = 2,
XR_EYE_TRACKER_POSE_TYPE_FIXATION_ML = 3,
XR_EYE_TRACKER_POSE_TYPE_COUNT_ML = 4,
XR_EYE_TRACKER_POSE_TYPE_MAX_ENUM_ML = 0x7FFFFFFF
} XrEyeTrackerPoseTypeML;| Enum | Description |
|---|---|
| Specifies the pose that originates from a point positioned between the user’s eyes. |
| Specifies the gaze of the left eye. |
| Specifies the gaze of the right eye. |
| Specifies the position of the 3D point where the left and right gaze intersect. |
// Provided by XR_ML_eye_tracker
typedef enum XrEyeTrackerConfidenceML {
XR_EYE_TRACKER_CONFIDENCE_LOW_ML = 0,
XR_EYE_TRACKER_CONFIDENCE_MEDIUM_ML = 1,
XR_EYE_TRACKER_CONFIDENCE_HIGH_ML = 2,
XR_EYE_TRACKER_CONFIDENCE_MAX_ENUM_ML = 0x7FFFFFFF
} XrEyeTrackerConfidenceML;| Enum | Description |
|---|---|
| Low quality, system has a poor view of the eyes. |
| Medium quality, system has a partial view of the eyes. |
| High quality, system has a clear view of the eyes. |
12.104.10. Example
// Create the eye tracker handle
XrEyeTrackerCreateInfoML createInfo{XR_TYPE_EYE_TRACKER_CREATE_INFO_ML};
XrEyeTrackerML eyeTracker;
CHK_XR(xrCreateEyeTrackerML(session, &createInfo, &eyeTracker));
std::vector<XrEyeTrackerPoseML> pose(4);
while(true)
{
XrTime time;
// Query Geometric information
{
XrEyeTrackerGeometricDataGetInfoML geometricDataGetInfo{XR_TYPE_EYE_TRACKER_GEOMETRIC_DATA_GET_INFO_ML};
uint32_t count;
CHK_XR(xrGetEyeTrackerGeometricDataML(eyeTracker, &geometricDataGetInfo, 0, &count, nullptr));
std::vector<XrEyeTrackerGeometricDataML> geometricDatas(count);
CHK_XR(xrGetEyeTrackerGeometricDataML(eyeTracker, &geometricDataGetInfo, count, &count, geometricDatas.data()));
// Do things
}
// Query Pupil information
{
XrEyeTrackerPupilDataGetInfoML pupilDataGetInfo{XR_TYPE_EYE_TRACKER_PUPIL_DATA_GET_INFO_ML};
uint32_t count;
CHK_XR(xrGetEyeTrackerPupilDataML(eyeTracker, &pupilDataGetInfo, 0, &count, nullptr));
std::vector<XrEyeTrackerPupilDataML> pupilDatas(count);
CHK_XR(xrGetEyeTrackerPupilDataML(eyeTracker, &pupilDataGetInfo, count, &count, pupilDatas.data()));
// Do things
}
// Query Gaze information
{
XrEyeTrackerGazeBehaviorGetInfoML gazeBehaviorGetInfo{XR_TYPE_EYE_TRACKER_GAZE_BEHAVIOR_GET_INFO_ML};
XrEyeTrackerGazeBehaviorMetaDataML gazeMetadata{XR_TYPE_EYE_TRACKER_GAZE_BEHAVIOR_META_DATA_ML};
XrEyeTrackerGazeBehaviorML gazeData{XR_TYPE_EYE_TRACKER_GAZE_BEHAVIOR_ML, &gazeMetadata};
CHK_XR(xrGetEyeTrackerGazeBehaviorML(eyeTracker, &gazeBehaviorGetInfo, &gazeData));
if(gazeData.valid == true) {
// Do things
if(gazeMetadata.valid == true) {
// Do things
}
}
}
// Query Pose information
{
XrEyeTrackerPosesGetInfoML posesGetInfo{XR_TYPE_EYE_TRACKER_POSES_GET_INFO_ML, nullptr, referenceSpace, frameState.predictedDisplayTime};
XrEyeTrackerPoseFlagsML requestedPoses = XR_EYE_TRACKER_POSE_LEFT_BIT_ML | XR_EYE_TRACKER_POSE_RIGHT_BIT_ML;
posesGetInfo.poseFlags = requestedPoses;
XrEyeTrackerPosesML posesData{XR_TYPE_EYE_TRACKER_POSES_ML, pose.data()};
CHK_XR(xrGetEyeTrackerPosesML(eyeTracker, &posesGetInfo, &posesData));
if(posesData.pose[XR_EYE_TRACKER_POSE_TYPE_GAZE_ML].valid == true) {
if(posesData.pose[XR_EYE_TRACKER_POSE_TYPE_GAZE_ML].confidence == XR_EYE_TRACKER_CONFIDENCE_HIGH_ML) {
// Do things
}
}
}
}12.104.12. New Enums
XrResult enumeration is extended with:
XR_ERROR_EYE_TRACKER_PERMISSION_DENIED_ML
XrStructureType enumeration is extended with:
XR_TYPE_EYE_TRACKER_CREATE_INFO_MLXR_TYPE_EYE_TRACKER_STATIC_DATA_GET_INFO_MLXR_TYPE_EYE_TRACKER_STATIC_DATA_MLXR_TYPE_EYE_TRACKER_GAZE_BEHAVIOR_GET_INFO_MLXR_TYPE_EYE_TRACKER_GAZE_BEHAVIOR_MLXR_TYPE_EYE_TRACKER_GEOMETRIC_DATA_GET_INFO_MLXR_TYPE_EYE_TRACKER_GEOMETRIC_DATA_MLXR_TYPE_EYE_TRACKER_PUPIL_DATA_GET_INFO_MLXR_TYPE_EYE_TRACKER_PUPIL_DATA_MLXR_TYPE_EYE_TRACKER_POSES_GET_INFO_MLXR_TYPE_EYE_TRACKER_POSES_MLXR_TYPE_EYE_TRACKER_POSE_ML