XR_ML_light_estimation

Experimental API

This API is still an experimental extension not included in the official OpenXR registry and is subject to change.

12.105. XR_ML_light_estimation

Name String

XR_ML_light_estimation

Extension Type

Instance extension

Registered Extension Number

482

Revision

1

Extension and Version Dependencies

• Requires support for OpenXR 1.0

Last Modified Date

2024-07-03

Contributors

Dominik Kasper, Magic Leap
Andrea Lampart, Magic Leap
Bogdan Morosanu, Magic Leap
Dario Brescianini, Magic Leap
Emanuele Vespa, Magic Leap
Karthik Kadappan, Magic Leap
Ron Bessems, Magic Leap

Overview

This extension gives access to light estimation capabilities by providing a graphical representation of environment lighting to be used in render engines. Light information is represented in terms of:

1. A 360 degree HDR environment map, formatted as a cubemap.

2. A set of spherical harmonics coefficients for a compact soft lighting representation.

3. Main light direction and color.

The three representations are provided bundled together, and it is up to the developer to choose which one to use depending on the use case.

New Flag Types

The XrLightEstimationCreateInfoML::cubemapResolution member is of the following type, and controls the resolution of a cubemap face.

// Provided by XR_ML_light_estimation
typedef enum XrLightEstimationHDRCubemapFaceResolutionML {
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_64_64_ML = 0,
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_128_128_ML = 1,
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_256_256_ML = 2,
    XR_LIGHT_ESTIMATION_HDRCUBEMAP_FACE_RESOLUTION_MAX_ENUM_ML = 0x7FFFFFFF
} XrLightEstimationHDRCubemapFaceResolutionML;

New Enum Constants

XrStructureType enumeration is extended with:

• XR_TYPE_LIGHT_ESTIMATION_CREATE_INFO_ML

• XR_TYPE_LIGHT_ESTIMATION_HDR_CUBEMAP_ML

• XR_TYPE_LIGHT_ESTIMATION_MAIN_DIRECTIONAL_LIGHT_ML

• XR_TYPE_LIGHT_ESTIMATION_SPHERICAL_HARMONICS_ML

New Structures

The XrLightEstimationML is a handle representing an single environmental light mapping session.

// Provided by XR_ML_light_estimation
XR_DEFINE_HANDLE(XrLightEstimationML)

The XrLightEstimationEstimateML is a handle to a specific light estimation estimate.

// Provided by XR_ML_light_estimation
XR_DEFINE_HANDLE(XrLightEstimationEstimateML)

The XrLightEstimationCreateInfoML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationCreateInfoML {
    XrStructureType                                type;
    const void*                                    next;
    XrSpace                                        baseSpace;
    XrTime                                         time;
    XrLightEstimationHDRCubemapFaceResolutionML    cubemapFaceResolution;
} XrLightEstimationCreateInfoML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain.

• baseSpace is the XrSpace to be used for cube mapping.

• time is the timestamp at handle creation.

• cubemapFaceResolution is the flag controlling the cubemap face resolution.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to using XrLightEstimationCreateInfoML

• type must be XR_TYPE_LIGHT_ESTIMATION_CREATE_INFO_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

• baseSpace must be a valid XrSpace handle

• cubemapFaceResolution must be a valid XrLightEstimationHDRCubemapFaceResolutionML value

Reference spaces XrLightEstimationCreateInfoML requires an XrSpace to be passed as an input parameter which will be used as the reference space for the environment mapping. This ensures that when obtaining light representations like the cubemap or the main light direction, these will be already in the correct reference frame. Notice that this extension does not support time-varying spaces: if the reference space is updated, it is up to the developer to destroy the handle and create a new one with an updated reference space.

The XrLightEstimationStateML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationStateML {
    XrStructureType    type;
    const void*        next;
    XrTime             lastUpdate;
} XrLightEstimationStateML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain.

• lastUpdate is the timestamp of the latest update to the light estimate. If no valid estimate is available, the timestamp will be set to zero.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to using XrLightEstimationStateML

• type must be XR_TYPE_LIGHT_ESTIMATION_STATE_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

The XrLightEstimationEstimateHDRCubemapML structure provides 6 cubemap images that can be used for reflections and is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateHDRCubemapML {
    XrStructureType    type;
    const void*        next;
    uint32_t           pixelCapacityInput;
    uint32_t           pixelCountOutput;
    void*              pixels;
    uint32_t           faceDim;
} XrLightEstimationEstimateHDRCubemapML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain.

• pixelCapacityInput is the size in pixels of the allocated pixels buffer.

• pixelCountOutput is the number of valid pixels written in the cubemap.This equals faceDim * faceDim * 6.

• pixels is the pixel buffer of the cubemap.

• faceDim is the size in pixels of the side of a cubemap face.

Cube map images are in RGBA format, with each channel being of 32bit floating point depth.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to using XrLightEstimationEstimateHDRCubemapML

• type must be XR_TYPE_LIGHT_ESTIMATION_ESTIMATE_HDR_CUBEMAP_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

• If pixelCapacityInput is not 0, pixels must be a pointer to an array of pixelCapacityInput bytes

The XrLightEstimationEstimateMainDirectionalLightML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateMainDirectionalLightML {
    XrStructureType    type;
    const void*        next;
    XrVector3f         direction;
    XrColor4f          color;
} XrLightEstimationEstimateMainDirectionalLightML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain.

• direction is the direction the light is shining towards.

• color encodes color and brightness of the light. The length of this vector is the absolute brightness of the light, and normalizing this vector will yield the relative RGB components. In order to get the equivalent 8bit color values, normalize the vector and then multiply the components with 256.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to using XrLightEstimationEstimateMainDirectionalLightML

• type must be XR_TYPE_LIGHT_ESTIMATION_ESTIMATE_MAIN_DIRECTIONAL_LIGHT_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

The XrLightEstimationEstimateSphericalHarmonicsML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateSphericalHarmonicsML {
    XrStructureType    type;
    const void*        next;
    float              harmonicsCoefficients[XR_LIGHT_ESTIMATION_TOTAL_HARMONICS_ML];
} XrLightEstimationEstimateSphericalHarmonicsML;

Member Descriptions

• type is the XrStructureType of this structure.

• next is NULL or a pointer to the next structure in a structure chain.

• harmonicsCoefficients are the spherical harmonics coefficients up to second order, i.e. 9 coefficients per channel. These are provided interleaved: [R0, G0, B0, R1, G1, B1, …​ R8, G8, B8].

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to using XrLightEstimationEstimateSphericalHarmonicsML

• type must be XR_TYPE_LIGHT_ESTIMATION_ESTIMATE_SPHERICAL_HARMONICS_ML

• next must be NULL or a valid pointer to the next structure in a structure chain

New Functions

The xrCreateLightEstimationML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrCreateLightEstimationML(
    XrSession                                   session,
    const XrLightEstimationCreateInfoML*        info,
    XrLightEstimationML*                        lightEstimation);

This function call creates the main session handle and at most one handle can be active per process at any one time.

Parameter Descriptions

• session is the associated XrSession.

• info is a pointer to an XrLightEstimationCreateInfoML used to configure the algorithm.

• lightEstimation is a pointer to the XrLightEstimationML handle to be populated.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrCreateLightEstimationML

• session must be a valid XrSession handle

• info must be a pointer to a valid XrLightEstimationCreateInfoML structure

• lightEstimation must be a pointer to an XrLightEstimationML handle

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_VALIDATION_FAILURE

• XR_ERROR_LIGHT_ESTIMATION_PERMISSION_DENIED_ML

• XR_ERROR_TIME_INVALID

• XR_ERROR_OUT_OF_MEMORY

• XR_ERROR_LIMIT_REACHED

• XR_ERROR_LIGHT_ESTIMATION_RGB_SENSOR_NOT_AVAILABLE_ML

Permissions Android applications must have the android.permission.CAMERA permission listed in their manifest to use this function. (protection level: dangerous)

Latency This function can take significant time to complete as it initialises a light estimation session, including opening the camera streams. It is recommended to not run this as part of a rendering loop where low latency is required.

The xrGetLightEstimationStateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationStateML(
    XrLightEstimationML                         lightEstimation,
    XrLightEstimationStateML*                   state);

This is a lightweight call that can be used to fetch the timestamp of the latest update event of the light estimate. If the light estimate is not available, the timestamp will have value of 0.

Parameter Descriptions

• lightEstimation is the associated XrLightEstimationML handle.

• state the time stamp associated with the light estimation estimate.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrGetLightEstimationStateML

• lightEstimation must be a valid XrLightEstimationML handle

• state must be a pointer to an XrLightEstimationStateML structure

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_VALIDATION_FAILURE

The xrCreateLightEstimationEstimateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrCreateLightEstimationEstimateML(
    XrLightEstimationML                         lightEstimation,
    XrLightEstimationEstimateML*                estimate);

This function provides a handle to the latest available light estimation estimate.

Parameter Descriptions

• lightEstimation is the associated XrLightEstimationML handle.

• estimate is a pointer to the XrLightEstimationEstimateMLhandle to be populated.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrCreateLightEstimationEstimateML

• lightEstimation must be a valid XrLightEstimationML handle

• estimate must be a pointer to an XrLightEstimationEstimateML handle

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_LIMIT_REACHED

• XR_ERROR_OUT_OF_MEMORY

Usage xrCreateLightEstimationEstimateML must be called after xrGetLightEstimationStateML to ensure that a valid estimate is available, otherwise it will return a runtime error.

The xrGetLightEstimationEstimateTimestampML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateTimestampML(
    XrLightEstimationEstimateML                 estimate,
    XrTime*                                     timeStamp);

It can be used to fetch the timestamp of the associated light estimation estimate.

Parameter Descriptions

• estimate is the associated XrLightEstimationEstimateML handle.

• timeStamp the time stamp associated with the light estimation estimate.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrGetLightEstimationEstimateTimestampML

• estimate must be a valid XrLightEstimationEstimateML handle

• timeStamp must be a pointer to an XrTime value

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_VALIDATION_FAILURE

The xrGetLightEstimationEstimateHDRCubemapML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateHDRCubemapML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateHDRCubemapML*      cubemap);

Parameter Descriptions

• estimate is the associated XrLightEstimationEstimateML handle.

• cubemap is a pointer to anXrLightEstimationEstimateHDRCubemapML structure to be populated.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrGetLightEstimationEstimateHDRCubemapML

• estimate must be a valid XrLightEstimationEstimateML handle

• cubemap must be a pointer to an XrLightEstimationEstimateHDRCubemapML structure

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

The xrGetLightEstimationEstimateMainDirectionalLightML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateMainDirectionalLightML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateMainDirectionalLightML* mainDirectionalLight);

Parameter Descriptions

• estimate is the associated XrLightEstimationEstimateML handle.

• mainDirectionalLight is a pointer to anXrLightEstimationEstimateMainDirectionalLightML structure to be populated.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrGetLightEstimationEstimateMainDirectionalLightML

• estimate must be a valid XrLightEstimationEstimateML handle

• mainDirectionalLight must be a pointer to an XrLightEstimationEstimateMainDirectionalLightML structure

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

• XR_ERROR_VALIDATION_FAILURE

The xrGetLightEstimationEstimateSphericalHarmonicsML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateSphericalHarmonicsML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateSphericalHarmonicsML* harmonics);

Parameter Descriptions

• estimate is the associated XrLightEstimationEstimateML handle.

• harmonics is a pointer to anXrLightEstimationEstimateSphericalHarmonicsML structure to be populated.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrGetLightEstimationEstimateSphericalHarmonicsML

• estimate must be a valid XrLightEstimationEstimateML handle

• harmonics must be a pointer to an XrLightEstimationEstimateSphericalHarmonicsML structure

Return Codes

Success

• XR_SUCCESS

• XR_SESSION_LOSS_PENDING

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_SESSION_LOST

• XR_ERROR_INSTANCE_LOST

• XR_ERROR_HANDLE_INVALID

The xrDestroyLightEstimationEstimateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrDestroyLightEstimationEstimateML(
    XrLightEstimationEstimateML                 estimate);

The function invalidates the estimate associated with the inputXrLightEstimationEstimateML.

Parameter Descriptions

• estimate is an XrLightEstimationEstimateML associated with the estimate to be destroyed.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrDestroyLightEstimationEstimateML

• estimate must be a valid XrLightEstimationEstimateML handle

Thread Safety

• Access to estimate, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_RUNTIME_FAILURE

• XR_ERROR_HANDLE_INVALID

The xrDestroyLightEstimationML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrDestroyLightEstimationML(
    XrLightEstimationML                         lightEstimation);

Parameter Descriptions

• lightEstimation is the associated XrLightEstimationML handle.

Valid Usage (Implicit)

• The XR_ML_light_estimation extension must be enabled prior to calling xrDestroyLightEstimationML

• lightEstimation must be a valid XrLightEstimationML handle

Thread Safety

• Access to lightEstimation, and any child handles, must be externally synchronized

Return Codes

Success

• XR_SUCCESS

Failure

• XR_ERROR_FUNCTION_UNSUPPORTED

• XR_ERROR_RUNTIME_FAILURE

• XR_ERROR_HANDLE_INVALID

Version History

• Revision 1, 2023-12-12 (Bogdan Morosanu, Karthik Kadappan)

  • Initial extension description