Skip to main content
Version: 21 Aug 2024

XR_ML_light_estimation

Experimental API

This API is still an experimental extension not included in the official OpenXR registry and is subject to change.

12.105. XR_ML_light_estimation

Name String

XR_ML_light_estimation

Extension Type

Instance extension

Registered Extension Number

482

Revision

1

Extension and Version Dependencies
  • Requires support for OpenXR 1.0

Last Modified Date

2024-07-03

Contributors

Dominik Kasper, Magic Leap
Andrea Lampart, Magic Leap
Bogdan Morosanu, Magic Leap
Dario Brescianini, Magic Leap
Emanuele Vespa, Magic Leap
Karthik Kadappan, Magic Leap
Ron Bessems, Magic Leap

Overview

This extension gives access to light estimation capabilities by providing a graphical representation of environment lighting to be used in render engines. Light information is represented in terms of:

  1. A 360 degree HDR environment map, formatted as a cubemap.

  2. A set of spherical harmonics coefficients for a compact soft lighting representation.

  3. Main light direction and color.

The three representations are provided bundled together, and it is up to the developer to choose which one to use depending on the use case.

New Flag Types

The XrLightEstimationCreateInfoML::cubemapResolution member is of the following type, and controls the resolution of a cubemap face.

// Provided by XR_ML_light_estimation
typedef enum XrLightEstimationHDRCubemapFaceResolutionML {
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_64_64_ML = 0,
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_128_128_ML = 1,
    XR_LIGHT_ESTIMATION_HDR_CUBEMAP_FACE_RESOLUTION_256_256_ML = 2,
    XR_LIGHT_ESTIMATION_HDRCUBEMAP_FACE_RESOLUTION_MAX_ENUM_ML = 0x7FFFFFFF
} XrLightEstimationHDRCubemapFaceResolutionML;

New Enum Constants

XrStructureType enumeration is extended with:

  • XR_TYPE_LIGHT_ESTIMATION_CREATE_INFO_ML

  • XR_TYPE_LIGHT_ESTIMATION_HDR_CUBEMAP_ML

  • XR_TYPE_LIGHT_ESTIMATION_MAIN_DIRECTIONAL_LIGHT_ML

  • XR_TYPE_LIGHT_ESTIMATION_SPHERICAL_HARMONICS_ML

New Structures

The XrLightEstimationML is a handle representing an single environmental light mapping session.

// Provided by XR_ML_light_estimation
XR_DEFINE_HANDLE(XrLightEstimationML)

The XrLightEstimationEstimateML is a handle to a specific light estimation estimate.

// Provided by XR_ML_light_estimation
XR_DEFINE_HANDLE(XrLightEstimationEstimateML)

The XrLightEstimationCreateInfoML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationCreateInfoML {
    XrStructureType                                type;
    const void*                                    next;
    XrSpace                                        baseSpace;
    XrTime                                         time;
    XrLightEstimationHDRCubemapFaceResolutionML    cubemapFaceResolution;
} XrLightEstimationCreateInfoML;
Member Descriptions
  • type is the XrStructureType of this structure.

  • next is NULL or a pointer to the next structure in a structure chain.

  • baseSpace is the XrSpace to be used for cube mapping.

  • time is the timestamp at handle creation.

  • cubemapFaceResolution is the flag controlling the cubemap face resolution.

Valid Usage (Implicit)
Reference spaces
XrLightEstimationCreateInfoML requires an XrSpace to be passed as an input parameter which will be used as the reference space for the environment mapping. This ensures that when obtaining light representations like the cubemap or the main light direction, these will be already in the correct reference frame. Notice that this extension does not support time-varying spaces: if the reference space is updated, it is up to the developer to destroy the handle and create a new one with an updated reference space.

The XrLightEstimationStateML structure is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationStateML {
    XrStructureType    type;
    const void*        next;
    XrTime             lastUpdate;
} XrLightEstimationStateML;
Member Descriptions
  • type is the XrStructureType of this structure.

  • next is NULL or a pointer to the next structure in a structure chain.

  • lastUpdate is the timestamp of the latest update to the light estimate. If no valid estimate is available, the timestamp will be set to zero.

Valid Usage (Implicit)

The XrLightEstimationEstimateHDRCubemapML structure provides 6 cubemap images that can be used for reflections and is defined as:

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateHDRCubemapML {
    XrStructureType    type;
    const void*        next;
    uint32_t           pixelCapacityInput;
    uint32_t           pixelCountOutput;
    void*              pixels;
    uint32_t           faceDim;
} XrLightEstimationEstimateHDRCubemapML;
Member Descriptions
  • type is the XrStructureType of this structure.

  • next is NULL or a pointer to the next structure in a structure chain.

  • pixelCapacityInput is the size in pixels of the allocated pixels buffer.

  • pixelCountOutput is the number of valid pixels written in the cubemap.This equals faceDim * faceDim * 6.

  • pixels is the pixel buffer of the cubemap.

  • faceDim is the size in pixels of the side of a cubemap face.

Cube map images are in RGBA format, with each channel being of 32bit floating point depth.

Valid Usage (Implicit)

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateMainDirectionalLightML {
    XrStructureType    type;
    const void*        next;
    XrVector3f         direction;
    XrColor4f          color;
} XrLightEstimationEstimateMainDirectionalLightML;
Member Descriptions
  • type is the XrStructureType of this structure.

  • next is NULL or a pointer to the next structure in a structure chain.

  • direction is the direction the light is shining towards.

  • color encodes color and brightness of the light. The length of this vector is the absolute brightness of the light, and normalizing this vector will yield the relative RGB components. In order to get the equivalent 8bit color values, normalize the vector and then multiply the components with 256.

Valid Usage (Implicit)

// Provided by XR_ML_light_estimation
typedef struct XrLightEstimationEstimateSphericalHarmonicsML {
    XrStructureType    type;
    const void*        next;
    float              harmonicsCoefficients[XR_LIGHT_ESTIMATION_TOTAL_HARMONICS_ML];
} XrLightEstimationEstimateSphericalHarmonicsML;
Member Descriptions
  • type is the XrStructureType of this structure.

  • next is NULL or a pointer to the next structure in a structure chain.

  • harmonicsCoefficients are the spherical harmonics coefficients up to second order, i.e. 9 coefficients per channel. These are provided interleaved: [R0, G0, B0, R1, G1, B1, …​ R8, G8, B8].

Valid Usage (Implicit)

New Functions

The xrCreateLightEstimationML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrCreateLightEstimationML(
    XrSession                                   session,
    const XrLightEstimationCreateInfoML*        info,
    XrLightEstimationML*                        lightEstimation);

This function call creates the main session handle and at most one handle can be active per process at any one time.

Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

  • XR_ERROR_VALIDATION_FAILURE

  • XR_ERROR_LIGHT_ESTIMATION_PERMISSION_DENIED_ML

  • XR_ERROR_TIME_INVALID

  • XR_ERROR_OUT_OF_MEMORY

  • XR_ERROR_LIMIT_REACHED

  • XR_ERROR_LIGHT_ESTIMATION_RGB_SENSOR_NOT_AVAILABLE_ML

Permissions

Android applications must have the android.permission.CAMERA permission listed in their manifest to use this function. (protection level: dangerous)

Latency

This function can take significant time to complete as it initialises a light estimation session, including opening the camera streams. It is recommended to not run this as part of a rendering loop where low latency is required.

The xrGetLightEstimationStateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationStateML(
    XrLightEstimationML                         lightEstimation,
    XrLightEstimationStateML*                   state);

This is a lightweight call that can be used to fetch the timestamp of the latest update event of the light estimate. If the light estimate is not available, the timestamp will have value of 0.

Parameter Descriptions
  • lightEstimation is the associated XrLightEstimationML handle.

  • state the time stamp associated with the light estimation estimate.

Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

  • XR_ERROR_VALIDATION_FAILURE

The xrCreateLightEstimationEstimateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrCreateLightEstimationEstimateML(
    XrLightEstimationML                         lightEstimation,
    XrLightEstimationEstimateML*                estimate);

This function provides a handle to the latest available light estimation estimate.

Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

  • XR_ERROR_LIMIT_REACHED

  • XR_ERROR_OUT_OF_MEMORY

Usage
xrCreateLightEstimationEstimateML must be called after xrGetLightEstimationStateML to ensure that a valid estimate is available, otherwise it will return a runtime error.

The xrGetLightEstimationEstimateTimestampML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateTimestampML(
    XrLightEstimationEstimateML                 estimate,
    XrTime*                                     timeStamp);

It can be used to fetch the timestamp of the associated light estimation estimate.

Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

  • XR_ERROR_VALIDATION_FAILURE

The xrGetLightEstimationEstimateHDRCubemapML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateHDRCubemapML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateHDRCubemapML*      cubemap);
Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateMainDirectionalLightML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateMainDirectionalLightML* mainDirectionalLight);
Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

  • XR_ERROR_VALIDATION_FAILURE

// Provided by XR_ML_light_estimation
XrResult xrGetLightEstimationEstimateSphericalHarmonicsML(
    XrLightEstimationEstimateML                 estimate,
    XrLightEstimationEstimateSphericalHarmonicsML* harmonics);
Parameter Descriptions
Valid Usage (Implicit)
Return Codes
Success
  • XR_SUCCESS

  • XR_SESSION_LOSS_PENDING

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_SESSION_LOST

  • XR_ERROR_INSTANCE_LOST

  • XR_ERROR_HANDLE_INVALID

The xrDestroyLightEstimationEstimateML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrDestroyLightEstimationEstimateML(
    XrLightEstimationEstimateML                 estimate);

The function invalidates the estimate associated with the inputXrLightEstimationEstimateML.

Parameter Descriptions
Valid Usage (Implicit)
Thread Safety
  • Access to estimate, and any child handles, must be externally synchronized

Return Codes
Success
  • XR_SUCCESS

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_RUNTIME_FAILURE

  • XR_ERROR_HANDLE_INVALID

The xrDestroyLightEstimationML function is defined as:

// Provided by XR_ML_light_estimation
XrResult xrDestroyLightEstimationML(
    XrLightEstimationML                         lightEstimation);
Parameter Descriptions
Valid Usage (Implicit)
Thread Safety
  • Access to lightEstimation, and any child handles, must be externally synchronized

Return Codes
Success
  • XR_SUCCESS

Failure
  • XR_ERROR_FUNCTION_UNSUPPORTED

  • XR_ERROR_RUNTIME_FAILURE

  • XR_ERROR_HANDLE_INVALID

Version History

  • Revision 1, 2023-12-12 (Bogdan Morosanu, Karthik Kadappan)

    • Initial extension description