Pixel Sensor Overview
The Magic Leap 2 OpenXR Unity SDK supports accessing the device's pixel sensors the Magic Leap 2 Pixel Sensor feature. This feature needs to be enabled in Unity's OpenXR Settings (Edit > Project Settings > XR Plug-in Management > OpenXR).
The documentation for the Pixel Sensor API section is still being developed, the API is in an experimental state. If you run into any issues, please reach out to us on the developer forum.
using MagicLeap.OpenXR.Features.PixelSensors;
Supported Sensors & Paths
This extension supports the following cameras:
| Sensor Name | XR Path | Description |
|---|---|---|
| Picture Center | /pixelsensor/picture/center | RGB Camera on the front of the headset |
| World Left | /pixelsensor/world/left | Camera located on the left corner of the headset. |
| World Center | /pixelsensor/world/center | Camera located on the front of the headset. |
| World Right | /pixelsensor/world/right | Camera located on the right corner of the headset. |
| Depth Center | /pixelsensor/depth/center | Depth camera located on the front of the headset. |
| Eye Temple Left | /pixelsensor/eye/temple/left | Left temple eye camera. |
| Eye Nasal Left | /pixelsensor/eye/nasal/left | Left nasal eye camera. |
| Eye Nasal Right | /pixelsensor/eye/nasal/right | Right nasal eye camera. |
| Eye Temple Right | /pixelsensor/eye/temple/right | Right temple eye camera. |
Sensor Permissions
Sensors require permissions before the application can access the data from the sensor. These permissions can be enabled in your project's Permissions Settings (Edit > Project Settings > Magic Leap > Permissions) and then requested at runtime using the Android Permissions API.
| Permission | Sensor Id |
|---|---|
com.magicleap.permission.DEPTH_CAMERA (protection level: dangerous) | /pixelsensor/depth/center |
com.magicleap.permission.EYE_CAMERA (protection level: dangerous) | /pixelsensor/eye/temple/left/pixelsensor/eye/nasal/left/pixelsensor/eye/nasal/right/pixelsensor/eye/temple/right |
permissions android.permission.CAMERA (protection level: dangerous) | /pixelsensor/world/left,/pixelsensor/world/center/pixelsensor/world/right/pixelsensor/picture/center |
Capabilities
Each sensor contains a list of capabilities. The table below outlines all of the capabilities and their data type.
| Enum | Description |
|---|---|
PixelSensorCapabilityType.UpdateRate | Data rate per second, must be specified by application. Data type is PixelSensorCapabilityDataType.UnsignedInt32. |
PixelSensorCapabilityType.Resolution | Resolution to configure, must be specified by application. Data type is PixelSensorCapabilityDataType.Extent2D. |
PixelSensorCapabilityType.Format | Data format, must be specified by application. Data type is MagicLeapPixelSensorFeature.PixelSensorFrameFormat. |
PixelSensorCapabilityType.Depth | Range of a depth sensor. Data type is PixelSensorCapabilityDataType.Float. |
PixelSensorCapabilityType.MixedReality | Camera frame and digital content will be blended into a single frame. Data type is a PixelSensorCapabilityDataType.PixelSensorRealityMode |
PixelSensorCapabilityType.ManualExposureTime | Exposure time in milliseconds, if not specified runtime must use AUTO exposure. Data type is a PixelSensorCapabilityDataType.UnsignedInt32 |
PixelSensorCapabilityType.AnalogGain | Higher gain is useful in low light conditions but may introduce noise. Data type is PixelSensorCapabilityDataType.UnsignedInt32. |
PixelSensorCapabilityType.DigitalGain | Higher gain is useful in low light conditions but may introduce noise. Data type is PixelSensorCapabilityDataType.UnsignedInt32. |
PixelSensorCapabilityType.AutoExposureMode | Auto Exposure Modes. Data type is an PixelSensorCapabilityDataType.PixelSensorAutoExposureMode |
PixelSensorCapabilityType.AutoExposureTargetBrightness | Set target brightness for auto exposure mode. Data type is PixelSensorCapabilityDataType.Float |
Mixed Reality Modes
The World Camera's support 2 separate Auto Exposure Modes
| Enum | Description |
|---|---|
PixelSensorRealityMode.Mixed | Camera frame and digital content will be blended into a single frame. |
PixelSensorRealityMode.Camera | |
| Only camera frame will be captured. | |
PixelSensorRealityMode.Virtual | Only virtual content will be captured. |
Auto Exposure Modes
The World Camera's support 2 separate Auto Exposure Modes
| Enum | Description |
|---|---|
PixelSensorAutoExposureMode.EnvironmentTracking | Exposure mode optimized for environment tracking. |
PixelSensorAutoExposureMode.ProximityIrTracking | Exposure mode optimized for close proximity IR light source. |
Frame Formats
Describes the format of data produced by the sensors:
| Enum | Description |
|---|---|
PixelSensorFrameFormat.Grayscale | Each pixel is 1 byte and represents a grayscale value. Datatype of the corresponding frame buffer is uint8_t. |
PixelSensorFrameFormat.Rgba8888 | Each pixel is 4 bytes and represents R,G,B, and A channels in that order. Datatype of the corresponding frame buffer is uint8_t. |
PixelSensorFrameFormat.Yuv420888 | Frame is represented in the YUV_420_888 planar forma. Datatype of the corresponding frame buffer is uint8_t. |
PixelSensorFrameFormat.Jpeg | Frame is JPEG encoded. |
PixelSensorFrameFormat.Depth32 | Represents the depth. Depth is the radial distance (in meters) of the real world location with respect to the depth camera. Datatype is float. |
PixelSensorFrameFormat.DepthRaw | Raw pixel data representing light captured by the sensor. For depth cameras that have a projector this raw frame will include frames captured both when the projector is on and off. Refer to PixelSensorDepthFrameIlluminationType for more details. Data type is float. |
Unity Frame Formats
PixelSensorFrame structure is used to store per pixel data. The type of data stored for each pixel varies and depends on the PixelSensorFrame.FrameType. The top left corner of the frame is treated as the origin.
| Enum | Description |
|---|---|
PixelSensorFrameType.Grayscale | Refers to UnityEngine.TextureFormat.R8 |
PixelSensorFrameType.Rgba8888 | Refers to UnityEngine.TextureFormat.Rgba8888 |
PixelSensorFrameType.Yuv420888 | Refers to UnityEngine.TextureFormat.YUY2 |
PixelSensorFrameType.Depth32 | Refers to UnityEngine.TextureFormat.RFloat |
PixelSensorFrameType.DepthRaw | Refers to UnityEngine.TextureFormat.RFloat |
PixelSensorFrameType.DepthConfidence | Refers to UnityEngine.TextureFormat.RFloat |
PixelSensorFrameType.DepthFlags | Refers to UnityEngine.TextureFormat.RFloat |
Metadatas
Pixel sensors may provide additional meta data for the captured frames. Application can obtain this metadata by specifying which data to capture before starting the sensor.
| Enum | Description |
|---|---|
PixelSensorExposureTime | Exposure time in milliseconds used to capture the frame |
PixelSensorAnalogGain | Analog gain used to capture the frame. |
PixelSensorDigitalGain | Digital gain used to capture the frame. |
PixelSensorPinholeIntrinsics | Specifies the camera intrinsics and distortion coefficients for a pinhole camera model. |
PixelSensorFisheyeIntrinsics | Specifies the camera matrix and distortion coefficients for a Magic Leap’s fisheye camera model. |
PixelSensorDepthFrameIllumination | Illumination type used for the depth frame. |
PixelSensorDepthConfidenceBuffer | Confidence values for each pixel in the camera frame. The confidence score is derived from the sensor noise and it is not normalized. The higher the value the higher the confidence. Applications can determine what confidence threshold to use based on their use case. Data type is float. |
PixelSensorDepthFlagBuffer | Flag bits for each pixel in the depth camera frame. Refer to PixelSensorDepthFlagBuffer for more details. Data type is uint32_t. |
In this Category
📄️ API Overview
This extension allows developers to interact with the Magic Leap 2's pixel sensors within their applications. It provides APIs for managing sensor data acquisition from various sensor types, each with unique capabilities and requirements. The following sections will delve deeper into these concepts and API usage.
📄️ Default Configuration Example
If your application needs access to one of Magic Leap's pixel sensors, and you do not plan to use a custom configuration, you can initialize the sensor with it's generic configuration.
📄️ World Camera Example
This section includes examples on how to configure the World Camera Pixel Sensor and display it's data.
📄️ Depth Camera Example
This section includes examples on how to configure the Depth Camera Pixel Sensor and display it's data.