Skip to main content
Version: 20 Mar 2024

Depth Camera

Module: Pixel Sensors

APIs to access the depth camera data. More...

Classes

Name
structMLDepthCameraStreamConfig
Structure to encapsulate the camera config for a specific stream.
structMLDepthCameraSettings
Structure to encapsulate the camera settings.
structMLDepthCameraIntrinsics
Depth camera intrinsic parameters.
structMLDepthCameraFrameBuffer
Structure to encapsulate per plane info for each camera frame.
structMLDepthCameraFrame
Structure to encapsulate output data for each camera sensor.
structMLDepthCameraData
Structure to encapsulate output data for each camera stream.
structMLDepthCameraStreamCapability
Structure to encapsulate a possible configuration for a single stream. Can be used to understand possible values for a specific #stream_configs element in MLDepthCameraSettings.
structMLDepthCameraCapability
Structure to encapsulate a possible set of streams configuration. Such set describes a possible way of setting #stream_configs in MLDepthCameraSettings and may hold multiple MLDepthCameraStreamCapability for different streams.
structMLDepthCameraCapabilityList
Structure to encapsulate a list of possible stream configurations.
structMLDepthCameraCapabilityFilter
Structure to encapsulate camera capabilities filtering.

Types

Name
typedef struct MLDepthCameraStreamConfigMLDepthCameraStreamConfig
Structure to encapsulate the camera config for a specific stream.
typedef struct MLDepthCameraSettingsMLDepthCameraSettings
Structure to encapsulate the camera settings.
typedef struct MLDepthCameraIntrinsicsMLDepthCameraIntrinsics
Depth camera intrinsic parameters.
typedef struct MLDepthCameraFrameBufferMLDepthCameraFrameBuffer
Structure to encapsulate per plane info for each camera frame.
typedef MLDepthCameraFrameBufferMLDepthCameraDepthImage
Depth image.
typedef MLDepthCameraFrameBufferMLDepthCameraConfidenceBuffer
Confidence score.
typedef MLDepthCameraFrameBufferMLDepthCameraDepthFlagsBuffer
Flags.
typedef MLDepthCameraFrameBufferMLDepthCameraAmbientRawDepthImage
Ambient raw depth image.
typedef MLDepthCameraFrameBufferMLDepthCameraRawDepthImage
Raw depth image.
typedef struct MLDepthCameraFrameMLDepthCameraFrame
Structure to encapsulate output data for each camera sensor.
typedef struct MLDepthCameraDataMLDepthCameraData
Structure to encapsulate output data for each camera stream.
typedef struct MLDepthCameraStreamCapabilityMLDepthCameraStreamCapability
Structure to encapsulate a possible configuration for a single stream. Can be used to understand possible values for a specific #stream_configs element in MLDepthCameraSettings.
typedef struct MLDepthCameraCapabilityMLDepthCameraCapability
Structure to encapsulate a possible set of streams configuration. Such set describes a possible way of setting #stream_configs in MLDepthCameraSettings and may hold multiple MLDepthCameraStreamCapability for different streams.
typedef struct MLDepthCameraCapabilityListMLDepthCameraCapabilityList
Structure to encapsulate a list of possible stream configurations.
typedef struct MLDepthCameraCapabilityFilterMLDepthCameraCapabilityFilter
Structure to encapsulate camera capabilities filtering.

Enums

Name
enumAnonymous Enum 8
{
MLDepthCameraIntrinsics_MaxDistortionCoefficients = 5
}
enumMLDepthCameraStream
{
MLDepthCameraStream_None = 0,
MLDepthCameraStream_LongRange = 1 << 0,
MLDepthCameraStream_ShortRange = 1 << 1,
MLDepthCameraStream_Ensure32Bits = 0x7FFFFFFF
}
Enumeration of depth camera streams.
enumMLDepthCameraFrameType
{
MLDepthCameraFrameType_LongRange = 0,
MLDepthCameraFrameType_ShortRange = 1,
MLDepthCameraFrameType_Count = 2,
MLDepthCameraFrameType_Ensure32Bits = 0x7FFFFFFF
}
Enumeration of camera stream used when capturing a frame.
enumMLDepthCameraFrameRate
{
MLDepthCameraFrameRate_1FPS = 0,
MLDepthCameraFrameRate_5FPS = 1,
MLDepthCameraFrameRate_25FPS = 2,
MLDepthCameraFrameRate_30FPS = 3,
MLDepthCameraFrameRate_50FPS = 4,
MLDepthCameraFrameRate_60FPS = 5,
MLDepthCameraFrameRate_Ensure32Bits = 0x7FFFFFFF
}
Enumeration of possible frame rates.
enumMLDepthCameraFlags
{
MLDepthCameraFlags_None = 0,
MLDepthCameraFlags_DepthImage = 1 << 0,
MLDepthCameraFlags_Confidence = 1 << 1,
MLDepthCameraFlags_DepthFlags = 1 << 2,
MLDepthCameraFlags_AmbientRawDepthImage = 1 << 3,
MLDepthCameraFlags_RawDepthImage = 1 << 4,
MLDepthCameraFlags_Ensure32Bits = 0x7FFFFFFF
}
Enumeration of flags to select data requested from depth camera.
enumMLDepthCameraDepthFlags
{
MLDepthCameraDepthFlags_Valid = 0 << 0,
MLDepthCameraDepthFlags_Invalid = 1 << 0,
MLDepthCameraDepthFlags_Saturated = 1 << 1,
MLDepthCameraDepthFlags_Inconsistent = 1 << 2,
MLDepthCameraDepthFlags_LowSignal = 1 << 3,
MLDepthCameraDepthFlags_FlyingPixel = 1 << 4,
MLDepthCameraDepthFlags_Masked = 1 << 5,
MLDepthCameraDepthFlags_SBI = 1 << 8,
MLDepthCameraDepthFlags_StrayLight = 1 << 9,
MLDepthCameraDepthFlags_ConnectedComponent = 1 << 10,
MLDepthCameraDepthFlags_Ensure32Bits = 0x7FFFFFFF
}
Enumeration of flags to select data requested from depth camera.

Functions

Name
voidMLDepthCameraSettingsInit(MLDepthCameraSettings * inout_handle)
Initialize the connect handle structure. Shall be called before calling MLDepthCameraConnect().
voidMLDepthCameraDataInit(MLDepthCameraData * inout_depth_camera_data)
Initialize MLDepthCameraData with version.
voidMLDepthCameraCapabilityFilterInit(MLDepthCameraCapabilityFilter * inout_handle)
Initialize the capabilities filtering structure. Shall be called before calling MLDepthCameraGetCapabilities().
MLResultMLDepthCameraConnect(const MLDepthCameraSettings settings, MLHandle out_handle)
Connect to depth camera.
MLResultMLDepthCameraUpdateSettings(MLHandle handle, const MLDepthCameraSettings * settings)
Update the depth camera settings.
MLResultMLDepthCameraGetCapabilities(MLHandle handle, const MLDepthCameraCapabilityFilter filter, MLDepthCameraCapabilityList out_caps)
Query the depth camera stream capabilities.
MLResultMLDepthCameraReleaseCapabilities(MLHandle handle, MLDepthCameraCapabilityList * out_caps)
Release resources allocated with MLDepthCameraGetCapabilities.
MLResultMLDepthCameraGetLatestDepthData(MLHandle handle, uint64_t timeout_ms, MLDepthCameraData * out_data)
Poll for Frames.
MLResultMLDepthCameraReleaseDepthData(MLHandle handle, MLDepthCameraData * depth_camera_data)
Releases specified MLDepthCameraData object.
MLResultMLDepthCameraDisconnect(MLHandle handle)
Disconnect from depth camera.

Detailed Description

APIs to access the depth camera data.

\experimental This is an experimental API which may be modified or removed without any prior notice.

Using the depth camera may result in the following APIs returning stale data:


Enums Documentation

Anonymous Enum 8

EnumeratorValueDescription
MLDepthCameraIntrinsics_MaxDistortionCoefficients5Default distortion vector size.

MLDepthCameraStream

EnumeratorValueDescription
MLDepthCameraStream_None0None.
MLDepthCameraStream_LongRange1 << 0Long range stream. Under normal operations long range stream has a maximum frequency of 5fps and a range from 1m up to 5m, in some cases this can go as far 7.5m.
MLDepthCameraStream_ShortRange1 << 1Short range stream. Under normal operations short range stream has a maximum frequency of 60fps and a range from 0.2m up to 0.9m.
MLDepthCameraStream_Ensure32Bits0x7FFFFFFFEnsure enum is represented as 32 bits.

Enumeration of depth camera streams.

API Level:

  • 29

MLDepthCameraFrameType

EnumeratorValueDescription
MLDepthCameraFrameType_LongRange0Frame captured using MLDepthCameraStream_LongRange stream.
MLDepthCameraFrameType_ShortRange1Frame captured using MLDepthCameraStream_ShortRange stream.
MLDepthCameraFrameType_Count2
MLDepthCameraFrameType_Ensure32Bits0x7FFFFFFFEnsure enum is represented as 32 bits.

Enumeration of camera stream used when capturing a frame.

API Level:

  • 29

MLDepthCameraFrameRate

EnumeratorValueDescription
MLDepthCameraFrameRate_1FPS0
MLDepthCameraFrameRate_5FPS1
MLDepthCameraFrameRate_25FPS2
MLDepthCameraFrameRate_30FPS3
MLDepthCameraFrameRate_50FPS4
MLDepthCameraFrameRate_60FPS5
MLDepthCameraFrameRate_Ensure32Bits0x7FFFFFFFEnsure enum is represented as 32 bits.

Enumeration of possible frame rates.

API Level:

  • 29

MLDepthCameraFlags

EnumeratorValueDescription
MLDepthCameraFlags_None0None.
MLDepthCameraFlags_DepthImage1 << 0Enable MLDepthCameraDepthImage. See MLDepthCameraDepthImage for more details.
MLDepthCameraFlags_Confidence1 << 1Enable MLDepthCameraConfidenceBuffer. See MLDepthCameraConfidenceBuffer for more details.
MLDepthCameraFlags_DepthFlags1 << 2Enable MLDepthCameraDepthFlagsBuffer. See MLDepthCameraDepthFlagsBuffer for more details.
MLDepthCameraFlags_AmbientRawDepthImage1 << 3Enable MLDepthCameraAmbientRawDepthImage. See MLDepthCameraAmbientRawDepthImage for more details.
MLDepthCameraFlags_RawDepthImage1 << 4Enable MLDepthCameraRawDepthImage. See MLDepthCameraRawDepthImage for more details.
MLDepthCameraFlags_Ensure32Bits0x7FFFFFFFEnsure enum is represented as 32 bits.

Enumeration of flags to select data requested from depth camera.


MLDepthCameraDepthFlags

EnumeratorValueDescription
MLDepthCameraDepthFlags_Valid0 << 0Valid pixel. Indicates that there is no additional flag data for this pixel.
MLDepthCameraDepthFlags_Invalid1 << 0Invalid. This bit is set to one to indicate that one or more flags from below have been set. Depending on the use case the application can correlate the flag data and corresponding pixel data to determine how to handle the pixel data.
MLDepthCameraDepthFlags_Saturated1 << 1Pixel saturated. The pixel intensity is either below the min or the max threshold value.
MLDepthCameraDepthFlags_Inconsistent1 << 2Inconsistent data. Inconsistent data received when capturing frames. This can happen due to fast motion.
MLDepthCameraDepthFlags_LowSignal1 << 3Low signal. Pixel has very low signal to noise ratio. One example of when this can happen is for pixels in far end of the range.
MLDepthCameraDepthFlags_FlyingPixel1 << 4Flying pixel. This typically happens when there is step jump in the distance of adjoining pixels in the scene. Example: When you open a door looking into the room the edges along the door's edges can cause flying pixels.
MLDepthCameraDepthFlags_Masked1 << 5Masked. If this bit is on it indicates that the corresponding pixel may not be within the illuminator's illumination cone.
MLDepthCameraDepthFlags_SBI1 << 8SBI. This bit will be set when there is high noise.
MLDepthCameraDepthFlags_StrayLight1 << 9Stray light. This could happen when there is another light source apart from the depth camera illuminator. This could also lead to MLDepthCameraDepthFlags_LowSignal.
MLDepthCameraDepthFlags_ConnectedComponent1 << 10Connected component. If a small group of MLDepthCameraDepthFlags_Valid is surrounded by a set of MLDepthCameraDepthFlags_Invalid then this bit will be set to 1.
MLDepthCameraDepthFlags_Ensure32Bits0x7FFFFFFFEnsure enum is represented as 32 bits.

Enumeration of flags to select data requested from depth camera.


Types Documentation

MLDepthCameraStreamConfig

typedef struct MLDepthCameraStreamConfig MLDepthCameraStreamConfig;

Structure to encapsulate the camera config for a specific stream.

More Info

API Level:

  • 29

MLDepthCameraSettings

typedef struct MLDepthCameraSettings MLDepthCameraSettings;

Structure to encapsulate the camera settings.

This structure must be initialized by calling MLDepthCameraSettingsInit before use.

These are supported configurations: Streams Exposure limits [us] Frame rate [FPS] Long Range 250 - 2000 1, 5 Short Range 50 - 375 On 50Hz PLC: 5, 25, 50On 60Hz PLC: 5, 30, 60 Use MLDepthCameraGetCapabilities to get a list of available configurations. \note The system may not be able to service all the requested streams at any given time. This parameter is treated as a hint and data will be provided for the requested streams if available. \note The API only supports a single stream. In a future release support for multiple streams will be added. @par API Level:\n 29

More Info


MLDepthCameraIntrinsics

typedef struct MLDepthCameraIntrinsics MLDepthCameraIntrinsics;

Depth camera intrinsic parameters.

More Info

API Level:

  • 29

MLDepthCameraFrameBuffer

typedef struct MLDepthCameraFrameBuffer MLDepthCameraFrameBuffer;

Structure to encapsulate per plane info for each camera frame.

More Info

API Level:

  • 29

MLDepthCameraDepthImage

typedef MLDepthCameraFrameBuffer MLDepthCameraDepthImage;

Depth image.

Depth image stores the depth data from the depth camera. Depth is represented in meters and gives the radial distance of the real world location from the depth camera coordinate frame. See MLDepthCameraIntrinsics and MLDepthCameraData.camera_pose for more details.

Cast the MLDepthCameraFrameBuffer.data to float data type.

More Info


MLDepthCameraConfidenceBuffer

typedef MLDepthCameraFrameBuffer MLDepthCameraConfidenceBuffer;

Confidence score.

Confidence buffer stores the confidence values for each pixel in the MLDepthCameraDepthImage. The confidence score is derived from the sensor noise and it is not normalized. The higher the value the higher the confidence. Applications can determine what confidence threshold to use based on their use case.

Cast the MLDepthCameraFrameBuffer.data to float data type.

More Info


MLDepthCameraDepthFlagsBuffer

typedef MLDepthCameraFrameBuffer MLDepthCameraDepthFlagsBuffer;

Flags.

Each pixel in the MLDepthCameraDepthImage will have an associated set of flags.

Cast the MLDepthCameraFrameBuffer.data to uint32_t data type.

More Info


MLDepthCameraAmbientRawDepthImage

typedef MLDepthCameraFrameBuffer MLDepthCameraAmbientRawDepthImage;

Ambient raw depth image.

This is the raw depth camera sensor data captured without the depth camera illumination and corresponds to the amount of ambient light incident on the sensor.

Cast the MLDepthCameraFrameBuffer.data to float data type.

More Info


MLDepthCameraRawDepthImage

typedef MLDepthCameraFrameBuffer MLDepthCameraRawDepthImage;

Raw depth image.

This is the raw depth camera sensor data captured with the depth camera illumination and corresponds to the amount of total light incident on the sensor.

Cast the MLDepthCameraFrameBuffer.data to float data type.

More Info


MLDepthCameraFrame

typedef struct MLDepthCameraFrame MLDepthCameraFrame;

Structure to encapsulate output data for each camera sensor.

More Info

API Level:

  • 29

MLDepthCameraData

typedef struct MLDepthCameraData MLDepthCameraData;

Structure to encapsulate output data for each camera stream.

This structure must be initialized by calling MLDepthCameraDataInit before use.

More Info

API Level:

  • 29

MLDepthCameraStreamCapability

typedef struct MLDepthCameraStreamCapability MLDepthCameraStreamCapability;

Structure to encapsulate a possible configuration for a single stream. Can be used to understand possible values for a specific #stream_configs element in MLDepthCameraSettings.

The capabilities supported by the depth camera can be queried with MLDepthCameraGetCapabilities().

More Info

API Level:

  • 29

MLDepthCameraCapability

typedef struct MLDepthCameraCapability MLDepthCameraCapability;

Structure to encapsulate a possible set of streams configuration. Such set describes a possible way of setting #stream_configs in MLDepthCameraSettings and may hold multiple MLDepthCameraStreamCapability for different streams.

Whole #stream_capabilities array describe a single configuration. Using only part of it is not recommended.

The capabilities supported by the depth camera can be queried with MLDepthCameraGetCapabilities().

More Info

API Level:

  • 29

MLDepthCameraCapabilityList

typedef struct MLDepthCameraCapabilityList MLDepthCameraCapabilityList;

Structure to encapsulate a list of possible stream configurations.

User should be able to use any of the capabilities from the list. They are grouped in usable sets of configurations - one set may contain different MLDepthCameraStreamCapability for different streams. For more information see MLDepthCameraCapability.

More Info

API Level:

  • 29

MLDepthCameraCapabilityFilter

typedef struct MLDepthCameraCapabilityFilter MLDepthCameraCapabilityFilter;

Structure to encapsulate camera capabilities filtering.

The capabilities supported by the depth camera can be queried with MLDepthCameraGetCapabilities().

More Info

API Level:

  • 29

Functions Documentation

MLDepthCameraSettingsInit

static inline void MLDepthCameraSettingsInit(
MLDepthCameraSettings * inout_handle
)

Initialize the connect handle structure. Shall be called before calling MLDepthCameraConnect().

Parameters

MLDepthCameraSettings *inout_handleMLDepthCameraSettings structure to initialize.

Required Permissions:

  • None

API Level:

  • 29

MLDepthCameraDataInit

static inline void MLDepthCameraDataInit(
MLDepthCameraData * inout_depth_camera_data
)

Initialize MLDepthCameraData with version.

Parameters

MLDepthCameraData *inout_depth_camera_dataSet up the version for inout_depth_camera_data.

API Level:

  • 29

MLDepthCameraCapabilityFilterInit

static inline void MLDepthCameraCapabilityFilterInit(
MLDepthCameraCapabilityFilter * inout_handle
)

Initialize the capabilities filtering structure. Shall be called before calling MLDepthCameraGetCapabilities().

Parameters

MLDepthCameraCapabilityFilter *inout_handleMLDepthCameraCapabilityFilter structure to initialize.

Required Permissions:

  • None

API Level:

  • 29

MLDepthCameraConnect

MLResult MLDepthCameraConnect(
const MLDepthCameraSettings * settings,
MLHandle * out_handle
)

Connect to depth camera.

Parameters

const MLDepthCameraSettings *settingsA pointer to MLDepthCameraSettings structure.
MLHandle *out_handleA pointer to camera handle to be used in later APIs.

Returns

MLResultMLResult_InvalidParamOne of the parameters is invalid.
MLResultMLResult_OkConnected to camera device(s) successfully.
MLResultMLResult_PermissionDeniedNecessary permission is missing.
MLResultMLResult_UnspecifiedFailureThe operation failed with an unspecified error.

Required Permissions:

  • com.magicleap.permission.DEPTH_CAMERA (protection level: dangerous)

API Level:

  • 29

MLDepthCameraUpdateSettings

MLResult MLDepthCameraUpdateSettings(
MLHandle handle,
const MLDepthCameraSettings * settings
)

Update the depth camera settings.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.
const MLDepthCameraSettings *settingsPointer to MLDepthCameraSettings.

Returns

MLResultMLResult_InvalidParamInvalid handle.
MLResultMLResult_OkSettings updated successfully.
MLResultMLResult_UnspecifiedFailureFailed due to internal error.

Required Permissions:

  • None

API Level:

  • 29

MLDepthCameraGetCapabilities

MLResult MLDepthCameraGetCapabilities(
MLHandle handle,
const MLDepthCameraCapabilityFilter * filter,
MLDepthCameraCapabilityList * out_caps
)

Query the depth camera stream capabilities.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.
const MLDepthCameraCapabilityFilter *filterPointer to initialized MLDepthCameraCapabilityFilter structure. Used for filtering.
MLDepthCameraCapabilityList *out_capsPointer to initialized MLDepthCameraCapabilityList structure.

Returns

MLResultMLResult_InvalidParamOne of the parameters is invalid.
MLResultMLResult_OkAPI call completed successfully.

Required Permissions:

  • None

Retrieves the capabilities supported by the depth camera based on the provided filter. Overview of the capabilities can be seen in the [configuration table].

After each call, a corresponding call to MLDepthCameraReleaseCapabilities should be made to release allocated resources.

API Level:

  • 29

MLDepthCameraReleaseCapabilities

MLResult MLDepthCameraReleaseCapabilities(
MLHandle handle,
MLDepthCameraCapabilityList * out_caps
)

Release resources allocated with MLDepthCameraGetCapabilities.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.
MLDepthCameraCapabilityList *out_capsPointer to MLDepthCameraCapabilityList filled by the call to MLDepthCameraGetCapabilities.

Returns

MLResultMLResult_InvalidParamOne of the parameters is invalid.
MLResultMLResult_OkAPI call completed successfully.

Required Permissions:

  • None

API Level:

  • 29

MLDepthCameraGetLatestDepthData

MLResult MLDepthCameraGetLatestDepthData(
MLHandle handle,
uint64_t timeout_ms,
MLDepthCameraData * out_data
)

Poll for Frames.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.
uint64_ttimeout_msTimeout in milliseconds.
MLDepthCameraData *out_dataDepth camera data. Should be an initialized MLDepthCameraData object.

Returns

MLResultMLResult_InvalidParamInvalid handle.
MLResultMLResult_OkDepth camera data fetched successfully.
MLResultMLResult_TimeoutReturned because no new frame available at this time.
MLResultMLResult_UnspecifiedFailureFailed due to internal error.

Required Permissions:

  • None

Returns MLDepthCameraData with this latest data when available. The memory is owned by the system. Application should copy the data it needs to cache it and then release the memory by calling #MLDepthCameraReleaseCameraData.

This is a blocking call. API is not thread safe.

If there are no new depth data frames within the timeout_ms duration then the API will return MLResult_Timeout.

API Level:

  • 29

MLDepthCameraReleaseDepthData

MLResult MLDepthCameraReleaseDepthData(
MLHandle handle,
MLDepthCameraData * depth_camera_data
)

Releases specified MLDepthCameraData object.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.
MLDepthCameraData *depth_camera_dataPointer to a valid MLDepthCameraData object.

Returns

MLResultMLResult_OkSuccessfully released depth camera data.
MLResultMLResult_InvalidParamdepth_camera_data parameter was not valid (NULL).
MLResultMLResult_UnspecifiedFailureFailed due to internal error.

Required Permissions:

  • None

This function should be called exactly once for each successfull call to #MLDepthCameraGetLatestCameraData.


MLDepthCameraDisconnect

MLResult MLDepthCameraDisconnect(
MLHandle handle
)

Disconnect from depth camera.

Parameters

MLHandlehandleCamera handle obtained from MLDepthCameraConnect.

Returns

MLResultMLResult_InvalidParamInvalid handle.
MLResultMLResult_OkDisconnected camera successfully.
MLResultMLResult_UnspecifiedFailureFailed to disconnect camera.

Required Permissions:

  • None

This will disconnect from the depth camera.

API Level:

  • 29