XR_ML_facial_expression

Experimental API

This API is still an experimental extension not included in the official OpenXR registry and is subject to change.

12.103. XR_ML_facial_expression

Name String

XR_ML_facial_expression

Extension Type

Instance extension

Registered Extension Number

483

Revision

1

Extension and Version Dependencies

• Requires support for OpenXR 1.0

Last Modified Date

2023-11-22

Contributors

Dushan Vasilevski, Magic Leap
Karthik Kadappan, Magic Leap
Ron Bessems, Magic Leap
Johannes Fung, Magic Leap

12.103.1. Overview

This extension provides the weights of facial blend shapes which can be used for a variety of purposes such as mood monitoring or interpolating the expression of an avatar’s face.

Permissions Android applications must have the com.magicleap.permission.FACIAL_EXPRESSION permission listed in their manifest and granted to use this extension, otherwisexrCreateFacialExpressionClientML will return aXR_ERROR_FACIAL_EXPRESSION_PERMISSION_DENIED_ML error. (protection level: dangerous)

12.103.2. Inspect system capability

An application can inspect whether the system is capable of parsing facial blend shapes by extending the XrSystemProperties withXrSystemFacialExpressionPropertiesML structure when callingxrGetSystemProperties.

// Provided by XR_ML_facial_expression
typedef struct XrSystemFacialExpressionPropertiesML {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsFacialExpression;
} XrSystemFacialExpressionPropertiesML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

• supportsFacialExpression is an XrBool32, indicating if current system is capable of parsing facial expressions enumerated inXrFacialBlendShapeML

If a runtime returns XR_FALSE for supportsFacialExpression, the runtime must return XR_ERROR_FEATURE_UNSUPPORTED fromxrCreateFacialExpressionClientML.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to using XrSystemFacialExpressionPropertiesML

• type must be XR_TYPE_SYSTEM_FACIAL_EXPRESSION_PROPERTIES_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

12.103.3. Create a facial expression client handle

The XrFacialExpressionClientML handle represents the resources for parsing facial expressions.

// Provided by XR_ML_facial_expression
XR_DEFINE_HANDLE(XrFacialExpressionClientML)

This handle is used to obtain blend shapes using thexrGetFacialExpressionBlendShapePropertiesML function.

An application can create an XrFacialExpressionClientML handle usingxrCreateFacialExpressionClientML function.

The xrCreateFacialExpressionClientML function is defined as:

// Provided by XR_ML_facial_expression
XrResult xrCreateFacialExpressionClientML(
    XrSession                                   session,
    const XrFacialExpressionClientCreateInfoML* createInfo,
    XrFacialExpressionClientML*                 facialExpressionClient);

Parameter Descriptions

• session is an XrSession in which the facial expression client will be active.

• createInfo is a pointer to theXrFacialExpressionClientCreateInfoML used to specify the facial expression client.

• facialExpressionClient is a pointer to the returnedXrFacialExpressionClientML handle.

If the system does not support parsing facial expressions, the runtime mustreturn XR_ERROR_FEATURE_UNSUPPORTED fromxrCreateFacialExpressionClientML. In this case, the runtime must return XR_FALSE forXrSystemFacialExpressionPropertiesML::supportsFacialExpressionwhen the function xrGetSystemProperties is called, so that the application can avoid creating a face tracker.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to calling xrCreateFacialExpressionClientML

• session must be a valid XrSession handle

• createInfo must be a pointer to a valid XrFacialExpressionClientCreateInfoML structure

• facialExpressionClient must be a pointer to an XrFacialExpressionClientML handle

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_VALIDATION_FAILURE

• XR_ERROR_RUNTIME_FAILURE

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_SESSION_LOST

• XR_ERROR_OUT_OF_MEMORY

• XR_ERROR_LIMIT_REACHED

• XR_ERROR_FEATURE_UNSUPPORTED

• XR_ERROR_FACIAL_EXPRESSION_PERMISSION_DENIED_ML

The XrFacialExpressionClientCreateInfoML structure is defined as follows:

// Provided by XR_ML_facial_expression
typedef struct XrFacialExpressionClientCreateInfoML {
    XrStructureType                type;
    const void*                    next;
    uint32_t                       requestedCount;
    const XrFacialBlendShapeML*    requestedFacialBlendShapes;
} XrFacialExpressionClientCreateInfoML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

• requestedCount is the size of the requestedFacialBlendShapes array

• requestedFacialBlendShapes is a non NULL pointer to a user application defined array containing the blend shapes for the runtime to target.

The XrFacialExpressionClientCreateInfoML structure describes the information to create an XrFacialExpressionClientML handle.

An application can specify the blend shapes they want to query by creating an array of type XrFacialBlendShapeML and passing it torequestedFacialBlendShapes along with the requestedCount.

The application can also pass in NULL intorequestedFacialBlendShapes to capture the entirety ofXrFacialBlendShapeML. However, for performance reasons, it would be better to be explicit about what blend shapes to query for performance reasons since some blend shapes may be queried at a greater frequency than others.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to using XrFacialExpressionClientCreateInfoML

• type must be XR_TYPE_FACIAL_EXPRESSION_CLIENT_CREATE_INFO_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

• requestedFacialBlendShapes must be a pointer to an array of requestedCount valid XrFacialBlendShapeML values

• The requestedCount parameter must be greater than 0

12.103.4. Destroy a facial expression client handle

The xrDestroyFacialExpressionClientML function releases thefacialExpressionClient and the underlying resources.

// Provided by XR_ML_facial_expression
XrResult xrDestroyFacialExpressionClientML(
    XrFacialExpressionClientML                  facialExpressionClient);

Parameter Descriptions

• facialExpressionClient is an XrFacialExpressionClientMLpreviously created by xrCreateFacialExpressionClientML.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to calling xrDestroyFacialExpressionClientML

• facialExpressionClient must be a valid XrFacialExpressionClientML handle

Thread Safety

• Access to facialExpressionClient, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_HANDLE_INVALID

12.103.5. Obtain facial expression blend shapes

The xrGetFacialExpressionBlendShapePropertiesML function returns theXrFacialExpressionBlendShapePropertiesML of each blend shape requested in XrFacialExpressionClientCreateInfoML

// Provided by XR_ML_facial_expression
XrResult xrGetFacialExpressionBlendShapePropertiesML(
    XrFacialExpressionClientML                  facialExpressionClient,
    const XrFacialExpressionBlendShapeGetInfoML* blendShapeGetInfo,
    uint32_t                                    blendShapeCount,
    XrFacialExpressionBlendShapePropertiesML*   blendShapes);

Parameter Descriptions

• facialExpressionClient is an XrFacialExpressionClientMLpreviously created by xrCreateFacialExpressionClientML.

• blendShapeGetInfo is a pointer toXrFacialExpressionBlendShapeGetInfoML describing information to obtain facial expression.

• blendShapeCount is a uint32_t specifying the size of the blendShapes array.

• blendShapes is an application allocated array ofXrFacialExpressionBlendShapePropertiesML that contains the blend shapes information. The array may be a subset of blend shapes.

Each XrFacialExpressionBlendShapePropertiesML in blendShapesshould have its requestedFacialBlendShape member variable initialized before being passed into xrGetFacialExpressionBlendShapePropertiesML.

If a blend shape in blendShapes is not enabled inxrCreateFacialExpressionClientML, the application must returnXR_ERROR_VALIDATION_FAILURE.

For unsupported blend shapes, the runtime must clear the flags member ofXrFacialExpressionBlendShapePropertiesML and the runtime must also return XR_SUCCESS.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to calling xrGetFacialExpressionBlendShapePropertiesML

• facialExpressionClient must be a valid XrFacialExpressionClientML handle

• blendShapeGetInfo must be a pointer to a valid XrFacialExpressionBlendShapeGetInfoML structure

• blendShapes must be a pointer to an XrFacialExpressionBlendShapePropertiesML structure

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_VALIDATION_FAILURE

• XR_ERROR_RUNTIME_FAILURE

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_SESSION_LOST

• XR_ERROR_TIME_INVALID

The XrFacialExpressionBlendShapeGetInfoML structure specifies properties about blend shapes desired by an application. It must be passed into xrGetFacialExpressionBlendShapePropertiesMLand is currently empty for future extensibility.

// Provided by XR_ML_facial_expression
typedef struct XrFacialExpressionBlendShapeGetInfoML {
    XrStructureType    type;
    const void*        next;
} XrFacialExpressionBlendShapeGetInfoML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to using XrFacialExpressionBlendShapeGetInfoML

• type must be XR_TYPE_FACIAL_EXPRESSION_BLEND_SHAPE_GET_INFO_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

XrFacialExpressionBlendShapePropertiesML structure holds the facial expression.

// Provided by XR_ML_facial_expression
typedef struct XrFacialExpressionBlendShapePropertiesML {
    XrStructureType                                  type;
    void*                                            next;
    XrFacialBlendShapeML                             requestedFacialBlendShape;
    float                                            weight;
    XrFacialExpressionBlendShapePropertiesFlagsML    flags;
    XrTime                                           time;
} XrFacialExpressionBlendShapePropertiesML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

• requestedFacialBlendShape is the type of XrFacialBlendShapeMLto query the runtime for.

• weight is a float describing the weight of a facial expression’s blend shape.

• flags is an XrFacialExpressionBlendShapePropertiesFlagsMLwhose bits are specified inXrFacialExpressionBlendShapePropertiesFlagBitsML

• time is the XrTime that corresponds to the moment at which the requestedFacialBlendShape is captured.

If requestedFacialBlendShape does not correspond to anyXrFacialBlendShapeML passed intoxrCreateFacialExpressionClientML then theXR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_VALID_BIT_ML andXR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_TRACKED_BIT_ML offlags must be unset. If the requestedFacialBlendShape cannot be parsed at sample timetime thenXR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_TRACKED_BIT_ML must be set to false.

The runtime must return weight which is the weight of the queried blend shape.

Valid Usage (Implicit)

• The XR_ML_facial_expression extension must be enabled prior to using XrFacialExpressionBlendShapePropertiesML

• type must be XR_TYPE_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

• requestedFacialBlendShape must be a valid XrFacialBlendShapeML value

• flags must be a valid combination of XrFacialExpressionBlendShapePropertiesFlagBitsML values

• flags must not be 0

12.103.6. Example code for obtaining facial expression information

The following example code demonstrates how to obtain weights for facial expression blend shapes.

XrInstance instance; // previously initialized
XrSystemId systemId; // previously initialized
XrSession session;   // previously initialized

// Confirm face tracking system support.
XrSystemFacialExpressionPropertiesML systemFacialExpressionClientProperties{
    XR_TYPE_SYSTEM_FACIAL_EXPRESSION_PROPERTIES_ML};
XrSystemProperties systemProperties{XR_TYPE_SYSTEM_PROPERTIES,
                                    &systemFacialExpressionClientProperties};
CHK_XR(xrGetSystemProperties(instance, systemId, &systemProperties));
if (!systemFacialExpressionClientProperties.supportsFacialExpression) {
    // The system does not support face tracking
    return;
}

// Get function pointer for xrCreateFacialExpressionClientML.
PFN_xrCreateFacialExpressionClientML pfnCreateFacialExpressionClientML;
CHK_XR(xrGetInstanceProcAddr(instance, "xrCreateFacialExpressionClientML",
                             reinterpret_cast<PFN_xrVoidFunction*>(
                             &pfnCreateFacialExpressionClientML)));

// Create a client that queries for default set of facial expressions.
XrFacialExpressionClientML facialExpressionClient = {};
  XrFacialExpressionClientCreateInfoML createInfo{XR_TYPE_FACIAL_EXPRESSION_CLIENT_CREATE_INFO_ML};
  CHK_XR(pfnCreateFacialExpressionClientML(session, &createInfo, &facialExpressionClient));

// Allocate buffers to receive facial expression data before frame
// loop starts.
const uint32_t num_blend_shapes = 2;
XrFacialExpressionBlendShapePropertiesML blendShapes[num_blend_shapes];
// User must explicitly request what facial expression blend shape to query
blendShapes[0].requestedFacialBlendShape = XR_FACIAL_BLEND_SHAPE_BROW_LOWERER_L_ML;
blendShapes[1].requestedFacialBlendShape = XR_FACIAL_BLEND_SHAPE_BROW_LOWERER_R_ML;

// Get function pointer for xrGetFacialExpressionBlendShapePropertiesML.
PFN_xrGetFacialExpressionBlendShapePropertiesML pfnGetFacialExpressionBlendShapesML;
CHK_XR(xrGetInstanceProcAddr(instance, "xrGetFacialExpressionBlendShapePropertiesML",
                             reinterpret_cast<PFN_xrVoidFunction*>(
                             &pfnGetFacialExpressionBlendShapesML)));
while (1) {
    // ...
    // For every frame in the frame loop
    // ...
    XrFrameState frameState; // previously returned from xrWaitFrame
    const XrTime time = frameState.predictedDisplayTime;

    XrFacialExpressionBlendShapeGetInfoML expressionInfo{XR_TYPE_FACIAL_EXPRESSION_BLEND_SHAPE_GET_INFO_ML};

    CHK_XR(pfnGetFacialExpressionBlendShapesML(facialExpressionClient, &expressionInfo, num_blend_shapes, blendShapes));

    for (uint32_t i = 0; i < num_blend_shapes; ++i) {
        // blendShapes[i] contains the properties of specific blend shape
    }
}

12.103.7. Conventions of blend shapes

This extension defines the following blend shapes for tracking facial expressions.

// Provided by XR_ML_facial_expression
typedef enum XrFacialBlendShapeML {
    XR_FACIAL_BLEND_SHAPE_BROW_LOWERER_L_ML = 0,
    XR_FACIAL_BLEND_SHAPE_BROW_LOWERER_R_ML = 1,
    XR_FACIAL_BLEND_SHAPE_CHEEK_RAISER_L_ML = 2,
    XR_FACIAL_BLEND_SHAPE_CHEEK_RAISER_R_ML = 3,
    XR_FACIAL_BLEND_SHAPE_CHIN_RAISER_ML = 4,
    XR_FACIAL_BLEND_SHAPE_DIMPLER_L_ML = 5,
    XR_FACIAL_BLEND_SHAPE_DIMPLER_R_ML = 6,
    XR_FACIAL_BLEND_SHAPE_EYES_CLOSED_L_ML = 7,
    XR_FACIAL_BLEND_SHAPE_EYES_CLOSED_R_ML = 8,
    XR_FACIAL_BLEND_SHAPE_INNER_BROW_RAISER_L_ML = 9,
    XR_FACIAL_BLEND_SHAPE_INNER_BROW_RAISER_R_ML = 10,
    XR_FACIAL_BLEND_SHAPE_JAW_DROP_ML = 11,
    XR_FACIAL_BLEND_SHAPE_LID_TIGHTENER_L_ML = 12,
    XR_FACIAL_BLEND_SHAPE_LID_TIGHTENER_R_ML = 13,
    XR_FACIAL_BLEND_SHAPE_LIP_CORNER_DEPRESSOR_L_ML = 14,
    XR_FACIAL_BLEND_SHAPE_LIP_CORNER_DEPRESSOR_R_ML = 15,
    XR_FACIAL_BLEND_SHAPE_LIP_CORNER_PULLER_L_ML = 16,
    XR_FACIAL_BLEND_SHAPE_LIP_CORNER_PULLER_R_ML = 17,
    XR_FACIAL_BLEND_SHAPE_LIP_FUNNELER_LB_ML = 18,
    XR_FACIAL_BLEND_SHAPE_LIP_FUNNELER_LT_ML = 19,
    XR_FACIAL_BLEND_SHAPE_LIP_FUNNELER_RB_ML = 20,
    XR_FACIAL_BLEND_SHAPE_LIP_FUNNELER_RT_ML = 21,
    XR_FACIAL_BLEND_SHAPE_LIP_PRESSOR_L_ML = 22,
    XR_FACIAL_BLEND_SHAPE_LIP_PRESSOR_R_ML = 23,
    XR_FACIAL_BLEND_SHAPE_LIP_PUCKER_L_ML = 24,
    XR_FACIAL_BLEND_SHAPE_LIP_PUCKER_R_ML = 25,
    XR_FACIAL_BLEND_SHAPE_LIP_STRETCHER_L_ML = 26,
    XR_FACIAL_BLEND_SHAPE_LIP_STRETCHER_R_ML = 27,
    XR_FACIAL_BLEND_SHAPE_LIP_SUCK_LB_ML = 28,
    XR_FACIAL_BLEND_SHAPE_LIP_SUCK_LT_ML = 29,
    XR_FACIAL_BLEND_SHAPE_LIP_SUCK_RB_ML = 30,
    XR_FACIAL_BLEND_SHAPE_LIP_SUCK_RT_ML = 31,
    XR_FACIAL_BLEND_SHAPE_LIP_TIGHTENER_L_ML = 32,
    XR_FACIAL_BLEND_SHAPE_LIP_TIGHTENER_R_ML = 33,
    XR_FACIAL_BLEND_SHAPE_LIPS_TOWARD_ML = 34,
    XR_FACIAL_BLEND_SHAPE_LOWER_LIP_DEPRESSOR_L_ML = 35,
    XR_FACIAL_BLEND_SHAPE_LOWER_LIP_DEPRESSOR_R_ML = 36,
    XR_FACIAL_BLEND_SHAPE_NOSE_WRINKLER_L_ML = 37,
    XR_FACIAL_BLEND_SHAPE_NOSE_WRINKLER_R_ML = 38,
    XR_FACIAL_BLEND_SHAPE_OUTER_BROW_RAISER_L_ML = 39,
    XR_FACIAL_BLEND_SHAPE_OUTER_BROW_RAISER_R_ML = 40,
    XR_FACIAL_BLEND_SHAPE_UPPER_LID_RAISER_L_ML = 41,
    XR_FACIAL_BLEND_SHAPE_UPPER_LID_RAISER_R_ML = 42,
    XR_FACIAL_BLEND_SHAPE_UPPER_LIP_RAISER_L_ML = 43,
    XR_FACIAL_BLEND_SHAPE_UPPER_LIP_RAISER_R_ML = 44,
    XR_FACIAL_BLEND_SHAPE_TONGUE_OUT_ML = 45,
    XR_FACIAL_BLEND_SHAPE_MAX_ENUM_ML = 0x7FFFFFFF
} XrFacialBlendShapeML;

New Handles

• XrFacialExpressionClientML

New Functions

• xrCreateFacialExpressionClientML

• xrDestroyFacialExpressionClientML

• xrGetFacialExpressionBlendShapePropertiesML

New Flag Types

The XrFacialExpressionBlendShapePropertiesML::flags member is of the following type, and contains a bitwise-OR of zero or more bits defined in XrFacialExpressionBlendShapePropertiesFlagBitsML

typedef XrFlags64 XrFacialExpressionBlendShapePropertiesFlagsML;

Valid bits for XrFacialExpressionBlendShapePropertiesFlagsML are defined by XrFacialExpressionBlendShapePropertiesFlagBitsML and are specified as

// Flag bits for XrFacialExpressionBlendShapePropertiesFlagsML
static const XrFacialExpressionBlendShapePropertiesFlagsML XR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_VALID_BIT_ML = 0x00000001;
static const XrFacialExpressionBlendShapePropertiesFlagsML XR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_TRACKED_BIT_ML = 0x00000002;

The flag bits' meaning are described as below:

Flag Descriptions

• XR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_VALID_BIT_ML — Indicates whether the blend shape is valid.

• XR_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_TRACKED_BIT_ML — Indicates whether the blend shape is being tracked by the runtime.

New Enum Constants

XrObjectType enumeration is extended with:

• XR_OBJECT_TYPE_FACIAL_EXPRESSION_CLIENT_ML

XrStructureType enumeration is extended with:

• XR_TYPE_SYSTEM_FACIAL_EXPRESSION_PROPERTIES_ML

• XR_TYPE_FACIAL_EXPRESSION_CLIENT_CREATE_INFO_ML

• XR_TYPE_FACIAL_EXPRESSION_BLEND_SHAPE_GET_INFO_ML

• XR_TYPE_FACIAL_EXPRESSION_BLEND_SHAPE_PROPERTIES_ML

New Enums

• XrFacialBlendShapeML

• XrFacialExpressionBlendShapePropertiesFlagBitsML

XrResult is extended with:

• XR_ERROR_FACIAL_EXPRESSION_PERMISSION_DENIED_ML

New Structures

• XrSystemFacialExpressionPropertiesML

• XrFacialExpressionClientCreateInfoML

• XrFacialExpressionBlendShapeGetInfoML

• XrFacialExpressionBlendShapePropertiesML

Issues

Version History

• Revision 1, 2023-11-22 (Johannes Fung)

  • Initial extension description