Skip to main content
Version: 21 Aug 2024

Camera2

The Magic Leap 2 supports the Android Camera2 NDK APIs. With Android Camera allows developers to capture real and virtual content inside their applications. While the Magic Leap 2 has only one physical, it is presented as 3 separate devices in the API. The Main Camera (Camera Id 0) and Mixed Reality Camera (Camera Id 3) are mutually exclusive, however they can be accessed even while the CV Camera (Camera Id 1) is being used.

Camera Devices

Magic Leap 2 allows developers to access two streams simultaneously from the same physical camera.

  • 0 - Main Camera - provides access to compressed video and still images. This device allows you to capture real-world content and is the preferred choice if you are not performing computer vision tasks on the output or if it is being used for streaming, broadcasting, or images.

  • 1 - CV Camera - best used for Computer vision scenarios, uncompressed, raw frames. If you use this device to do CV in you application, you can use the record/stream gameplay using the Capture Service, but you cannot perform Image or marker tracking using the SDK.

  • 3 - Mixed Reality Capture Camera - provides the ability to capture virtual and mixed reality content. There is a camera metadata tag that allows you to turn off mixed reality or turn it on. The tag is ML_CONTROL_MRCAMERA_CAPTURE_MODE on camera 3 only. If you try to set it on the other streams you get an error saying it does not know what the tag is.

Camera IDSupported Capture ModesStream SupportConflictsOrientation
0Camera_OnlyUp to 3 streams (shared with Camera ID 1)Conflicts with Camera ID 3Landscape
1Camera_OnlyUp to 3 streams (shared with Camera ID 0)NoneLandscape
3Mixed_Reality, Virtual_Only, Camera_Only1 streamEvicts Camera ID 0Landscape

Notes:

  • Camera IDs 0 and 1 can be accessed simultaneously.
  • Camera IDs 0 and 3 are conflicting; they cannot be accessed at the same time.
  • All cameras are rear-facing (LENS_FACING_BACK).
  • The camera has a landscape orientation, while the display has a portrait orientation.
  • Manual exposure mode may introduce a delay of 3-4 frames due to sensor configuration latencies.

You can use the same stream to capture different types of content. For example, you can capture Video, Images, and a Preview from the Main Camera stream, while also using the CV Camera at the same time.

Mixed Reality Capture

The Mixed Reality Capture Camera (camera 3) stream is the only stream that can capture Mixed Reality content. However, the Mixed Reality Capture stream only supports a single concurrent capture mode which means that you need to choose whether to capture Video OR Images but not both.

Stream Capabilities

This section details the supported Resolution, Capture Type, Formats and Frame Rates. The following table provides general information about each of the supported formats.

  • Camera type: [Main, CV]
  • Mode: [Real, MR (Mixed Reality), Virtual]
  • Capture type: [Preview, Video, Image]

These are formats supported:

FormatCameraModeCapture Type
YUVMainRealPreview/Video/Image
YUVCVRealVideo/Image
JPEGMainReal/MR/VirtualImage

Main Camera Stream

The table below shows which output formats are supported when capturing using the Main Camera Stream.

Camera Mode Resolution Capture Type Formats FrameRate
MainCam_Only640x480
1280x720
1280x960
1440x1080
1920x1080
2048x1536
PreviewYUV15/30/60
ImageYUV/JPEG15/30/60
VideoYUV/RGB/Compressed15/30/60

2880x2160
3840x2160
4096x3072

PreviewYUV15/30
ImageYUV/JPEG15/30
VideoYUV/RGB/Compressed15/30

CV Camera Stream

The table below shows which output formats are supported when capturing using the Computer Vision Camera.

Camera Mode Resolution Capture Type Formats FrameRate
CVCam_Only640x480
1280x720
1280x960
1440x1080
1920x1080
2048x1536
ImageYUV15/30/60
VideoYUV/RGB                        15/30/60

2880x2160
3840x2160
4096x3072

ImageYUV15/30
VideoYUV/RGB 15/30
info
  • 15Fps only supported in Auto Exposure Mode.
  • When multiple streams are configured frame rate is constrained by max resolution.
  • FrameRate_None can be used when only Image capture_type is used.

Mixed Reality / Virtual Only Capture

The table below shows which output formats are supported when capturing Virtual and Mixed Reality Content.

Camera Mode Resolution Capture Type Formats FrameRate
MainMR/Virtual_Only648x720 ImageJPEG30/60
VideoCompressed30/60
960x720 ImageJPEG30/60
VideoCompressed30/60
972x1080 ImageJPEG30/60
VideoCompressed30/60
1440x1080ImageJPEG30/60
VideoCompressed30/60
1944x2160ImageJPEG30
VideoCompressed30
2880x2160ImageJPEG30
VideoCompressed30
info

Quality and frame rate provided during connection should match the resolution and framerate that was used when preparing capture.

Metadata Tags

Definition of metadata tags for MagicLeap specific extensions to Camera Metadata. The metadata tags can be used with NDK apis to get and set camera metadata. The NDK APIs lack the functionality to query metadata tag from name, which makes it difficult to work with vendor specific tags without having these definitions.

This guide provides a detailed overview of the camera devices available on the Magic Leap 2 and their features, with a focus on additional vendor tags that enhance functionality and flexibility for developers. The Magic Leap 2 supports three camera IDs, each with its unique set of features and capabilities. Below is a table summarizing these camera devices and the key features accessible through the use of specific vendor tags.

Feature TagDescriptionApplicable Camera IDsData TypePossible Values / Notes
ML_CONTROL_CAMERA_FORCEAPPLY_MODEAllows overriding default camera settings precedence between Camera ID 0 and Camera ID 1.0, 1int321 = Force apply settings
ML_CONTROL_CAMERA_EFFECT_MODEConfigures special camera effects.0, 1uint80 = Off, 1 = Grayscale, 2 = Negative, 3 = Sepia, 4 = Color Selection, 5 = Sharpen, 6 = Emboss, 7 = Sketch
ML_CONTROL_CAMERA_APP_EXPOSURE_UPPER_TIME_LIMITLimits the maximum exposure time to reduce motion blur.0, 1int64Value in nanoseconds, less than 16ms/20ms
ML_CONTROL_CAMERA_MLTIME_TIMESTAMPSQueries MLTime timestamps for synchronization purposes.0, 1int64Array of timestamps [StartOfFrameTransfer, CentreOfExposure, EndOfFrameTransfer, ExternalCameraSync]
ML_CONTROL_CAMERA_NUM_INTRINSICSQueries the number of camera intrinsics.0, 1int32-
ML_CONTROL_CAMERA_INTRINSICSQueries camera intrinsics.0, 1floatArray: {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}
ML_CONTROL_MRCAMERA_TINTControls tint for virtual and camera frames composition in Mixed Reality Capture.3float0.0 to 1.0
ML_CONTROL_MRCAMERA_ALPHA_BIASControls alpha for virtual and camera frames composition in Mixed Reality Capture.3float-1.0 to 1.0
ML_CONTROL_MRCAMERA_CAPTURE_MODESets the capture mode for Mixed Reality Capture.3uint8_t0 = Mixed_Reality, 1 = Virtual_Only, 2 = Camera_Only
ML_CONTROL_MRCAMERA_MLTIME_TIMESTAMPSSimilar to ML_CONTROL_CAMERA_MLTIME_TIMESTAMPS, but for Camera ID 3. It can be used for synchronizing with data like head pose which has MLTime timestamps.3int64Array of timestamps [StartOfFrameTransfer, CentreOfExposure, EndOfFrameTransfer, ExternalCameraSync]
ML_CONTROL_MRCAMERA_NUM_INTRINSICSSimilar to ML_CONTROL_CAMERA_NUM_INTRINSICS, but for Camera ID 3. It can be used to query the number of camera intrinsics, which corresponds to the number of streams configured for the camera.3int32-
ML_CONTROL_MRCAMERA_INTRINSICSSimilar to ML_CONTROL_CAMERA_INTRINSICS, but for Camera ID 3. It can be used to query the camera intrinsics for Mixed Reality Capture.3floatArray: {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3}

Resources

For more information, see Android's Camera documentation and Camera 2 Guide.