Pixel Sensor Overview

The Magic Leap 2 OpenXR Unity SDK supports accessing the device's pixel sensors the Magic Leap 2 Pixel Sensor feature. This feature needs to be enabled in Unity's OpenXR Settings (Edit > Project Settings > XR Plug-in Management > OpenXR).

caution

The documentation for the Pixel Sensor API section is still being developed, the API is in an experimental state. If you run into any issues, please reach out to us on the developer forum.

NameSpace

using MagicLeap.OpenXR.Features.PixelSensors;

Supported Sensors & Paths

This extension supports the following cameras:

Sensor Name	XR Path	Description
Picture Center	/pixelsensor/picture/center	RGB Camera on the front of the headset
World Left	/pixelsensor/world/left	Camera located on the left corner of the headset.
World Center	/pixelsensor/world/center	Camera located on the front of the headset.
World Right	/pixelsensor/world/right	Camera located on the right corner of the headset.
Depth Center	/pixelsensor/depth/center	Depth camera located on the front of the headset.
Eye Temple Left	/pixelsensor/eye/temple/left	Left temple eye camera.
Eye Nasal Left	/pixelsensor/eye/nasal/left	Left nasal eye camera.
Eye Nasal Right	/pixelsensor/eye/nasal/right	Right nasal eye camera.
Eye Temple Right	/pixelsensor/eye/temple/right	Right temple eye camera.

Sensor Permissions

Sensors require permissions before the application can access the data from the sensor. These permissions can be enabled in your project's Permissions Settings (Edit > Project Settings > Magic Leap > Permissions) and then requested at runtime using the Android Permissions API.

Permission	Sensor Id
com.magicleap.permission.DEPTH_CAMERA (protection level: dangerous)	/pixelsensor/depth/center
com.magicleap.permission.EYE_CAMERA (protection level: dangerous)	/pixelsensor/eye/temple/left /pixelsensor/eye/nasal/left /pixelsensor/eye/nasal/right /pixelsensor/eye/temple/right
permissions android.permission.CAMERA (protection level: dangerous)	/pixelsensor/world/left, /pixelsensor/world/center /pixelsensor/world/right /pixelsensor/picture/center

Capabilities

Each sensor contains a list of capabilities. The table below outlines all of the capabilities and their data type.

Enum	Description
PixelSensorCapabilityType.UpdateRate	Data rate per second, must be specified by application. Data type is PixelSensorCapabilityDataType.UnsignedInt32.
PixelSensorCapabilityType.Resolution	Resolution to configure, must be specified by application. Data type is PixelSensorCapabilityDataType.Extent2D.
PixelSensorCapabilityType.Format	Data format, must be specified by application. Data type is MagicLeapPixelSensorFeature.PixelSensorFrameFormat.
PixelSensorCapabilityType.Depth	Range of a depth sensor. Data type is PixelSensorCapabilityDataType.Float.
PixelSensorCapabilityType.MixedReality	Camera frame and digital content will be blended into a single frame. Data type is a PixelSensorCapabilityDataType.PixelSensorRealityMode
PixelSensorCapabilityType.ManualExposureTime	Exposure time in milliseconds, if not specified runtime must use AUTO exposure. Data type is a PixelSensorCapabilityDataType.UnsignedInt32
PixelSensorCapabilityType.AnalogGain	Higher gain is useful in low light conditions but may introduce noise. Data type is PixelSensorCapabilityDataType.UnsignedInt32.
PixelSensorCapabilityType.DigitalGain	Higher gain is useful in low light conditions but may introduce noise. Data type is PixelSensorCapabilityDataType.UnsignedInt32.
PixelSensorCapabilityType.AutoExposureMode	Auto Exposure Modes. Data type is an PixelSensorCapabilityDataType.PixelSensorAutoExposureMode
PixelSensorCapabilityType.AutoExposureTargetBrightness	Set target brightness for auto exposure mode. Data type is PixelSensorCapabilityDataType.Float

Mixed Reality Modes

The World Camera's support 2 separate Auto Exposure Modes

Enum	Description
PixelSensorRealityMode.Mixed	Camera frame and digital content will be blended into a single frame.
PixelSensorRealityMode.Camera
Only camera frame will be captured.
PixelSensorRealityMode.Virtual	Only virtual content will be captured.

Auto Exposure Modes

The World Camera's support 2 separate Auto Exposure Modes

Enum	Description
PixelSensorAutoExposureMode.EnvironmentTracking	Exposure mode optimized for environment tracking.
PixelSensorAutoExposureMode.ProximityIrTracking	Exposure mode optimized for close proximity IR light source.

Frame Formats

Describes the format of data produced by the sensors:

Enum	Description
PixelSensorFrameFormat.Grayscale	Each pixel is 1 byte and represents a grayscale value. Datatype of the corresponding frame buffer is uint8_t.
PixelSensorFrameFormat.Rgba8888	Each pixel is 4 bytes and represents R,G,B, and A channels in that order. Datatype of the corresponding frame buffer is uint8_t.
PixelSensorFrameFormat.Yuv420888	Frame is represented in the YUV_420_888 planar forma. Datatype of the corresponding frame buffer is uint8_t.
PixelSensorFrameFormat.Jpeg	Frame is JPEG encoded.
PixelSensorFrameFormat.Depth32	Represents the depth. Depth is the radial distance (in meters) of the real world location with respect to the depth camera. Datatype is float.
PixelSensorFrameFormat.DepthRaw	Raw pixel data representing light captured by the sensor. For depth cameras that have a projector this raw frame will include frames captured both when the projector is on and off. Refer to PixelSensorDepthFrameIlluminationType for more details. Data type is float.

Unity Frame Formats

PixelSensorFrame structure is used to store per pixel data. The type of data stored for each pixel varies and depends on the PixelSensorFrame.FrameType. The top left corner of the frame is treated as the origin.

Enum	Description
PixelSensorFrameType.Grayscale	Refers to UnityEngine.TextureFormat.R8
PixelSensorFrameType.Rgba8888	Refers to UnityEngine.TextureFormat.Rgba8888
PixelSensorFrameType.Yuv420888	Refers to UnityEngine.TextureFormat.YUY2
PixelSensorFrameType.Depth32	Refers to UnityEngine.TextureFormat.RFloat
PixelSensorFrameType.DepthRaw	Refers to UnityEngine.TextureFormat.RFloat
PixelSensorFrameType.DepthConfidence	Refers to UnityEngine.TextureFormat.RFloat
PixelSensorFrameType.DepthFlags	Refers to UnityEngine.TextureFormat.RFloat

Metadatas

Pixel sensors may provide additional meta data for the captured frames. Application can obtain this metadata by specifying which data to capture before starting the sensor.

Enum	Description
PixelSensorExposureTime	Exposure time in milliseconds used to capture the frame
PixelSensorAnalogGain	Analog gain used to capture the frame.
PixelSensorDigitalGain	Digital gain used to capture the frame.
PixelSensorPinholeIntrinsics	Specifies the camera intrinsics and distortion coefficients for a pinhole camera model.
PixelSensorFisheyeIntrinsics	Specifies the camera matrix and distortion coefficients for a Magic Leap’s fisheye camera model.
PixelSensorDepthFrameIllumination	Illumination type used for the depth frame.
PixelSensorDepthConfidenceBuffer	Confidence values for each pixel in the camera frame. The confidence score is derived from the sensor noise and it is not normalized. The higher the value the higher the confidence. Applications can determine what confidence threshold to use based on their use case. Data type is float.
PixelSensorDepthFlagBuffer	Flag bits for each pixel in the depth camera frame. Refer to PixelSensorDepthFlagBuffer for more details. Data type is uint32_t.

In this Category

📄️ API Overview

This extension allows developers to interact with the Magic Leap 2's pixel sensors within their applications. It provides APIs for managing sensor data acquisition from various sensor types, each with unique capabilities and requirements. The following sections will delve deeper into these concepts and API usage.

📄️ Default Configuration Example

If your application needs access to one of Magic Leap's pixel sensors, and you do not plan to use a custom configuration, you can initialize the sensor with it's generic configuration.

📄️ World Camera Example

This section includes examples on how to configure the World Camera Pixel Sensor and display it's data.

📄️ Depth Camera Example

This section includes examples on how to configure the Depth Camera Pixel Sensor and display it's data.