Camera2
The Magic Leap 2 supports the Android Camera2 NDK APIs. With Android Camera allows developers to capture real and virtual content inside their applications. While the Magic Leap 2 has only one physical, it is presented as 3 separate devices in the API. The Main Camera (Camera Id 0) and Mixed Reality Camera (Camera Id 3) are mutually exclusive, however they can be accessed even while the CV Camera (Camera Id 1) is being used.
Camera Devices
Magic Leap 2 allows developers to access two streams simultaneously from the same physical camera.
0 - Main Camera - provides access to compressed video and still images. This device allows you to capture real-world content and is the preferred choice if you are not performing computer vision tasks on the output or if it is being used for streaming, broadcasting, or images.
1 - CV Camera - best used for Computer vision scenarios, uncompressed, raw frames. If you use this device to do CV in you application, you can use the record/stream gameplay using the Capture Service, but you cannot perform Image or marker tracking using the SDK.
3 - Mixed Reality Capture Camera - provides the ability to capture virtual and mixed reality content. There is a camera metadata tag that allows you to turn off mixed reality or turn it on. The tag is
ML_CONTROL_MRCAMERA_CAPTURE_MODEon camera 3 only. If you try to set it on the other streams you get an error saying it does not know what the tag is.
| Camera ID | Supported Capture Modes | Stream Support | Conflicts | Orientation |
|---|---|---|---|---|
| 0 | Camera_Only | Up to 3 streams (shared with Camera ID 1) | Conflicts with Camera ID 3 | Landscape |
| 1 | Camera_Only | Up to 3 streams (shared with Camera ID 0) | None | Landscape |
| 3 | Mixed_Reality, Virtual_Only, Camera_Only | 1 stream | Evicts Camera ID 0 | Landscape |
Notes:
- Camera IDs 0 and 1 can be accessed simultaneously.
- Camera IDs 0 and 3 are conflicting; they cannot be accessed at the same time.
- All cameras are rear-facing (LENS_FACING_BACK).
- The camera has a landscape orientation, while the display has a portrait orientation.
- Manual exposure mode may introduce a delay of 3-4 frames due to sensor configuration latencies.
You can use the same stream to capture different types of content. For example, you can capture Video, Images, and a Preview from the Main Camera stream, while also using the CV Camera at the same time.
The Mixed Reality Capture Camera (camera 3) stream is the only stream that can capture Mixed Reality content. However, the Mixed Reality Capture stream only supports a single concurrent capture mode which means that you need to choose whether to capture Video OR Images but not both.
Stream Capabilities
This section details the supported Resolution, Capture Type, Formats and Frame Rates. The following table provides general information about each of the supported formats.
- Camera type: [Main, CV]
- Mode: [Real, MR (Mixed Reality), Virtual]
- Capture type: [Preview, Video, Image]
These are formats supported:
| Format | Camera | Mode | Capture Type |
|---|---|---|---|
| YUV | Main | Real | Preview/Video/Image |
| YUV | CV | Real | Video/Image |
| JPEG | Main | Real/MR/Virtual | Image |
Main Camera Stream
The table below shows which output formats are supported when capturing using the Main Camera Stream.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| Main | Cam_Only | 640x480 1280x720 1280x960 1440x1080 1920x1080 2048x1536 | Preview | YUV | 15/30/60 |
| Image | YUV/JPEG | 15/30/60 | |||
| Video | YUV/RGB/Compressed | 15/30/60 | |||
2880x2160 3840x2160 4096x3072 | Preview | YUV | 15/30 | ||
| Image | YUV/JPEG | 15/30 | |||
| Video | YUV/RGB/Compressed | 15/30 |
CV Camera Stream
The table below shows which output formats are supported when capturing using the Computer Vision Camera.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| CV | Cam_Only | 640x480 1280x720 1280x960 1440x1080 1920x1080 2048x1536 | Image | YUV | 15/30/60 |
| Video | YUV/RGB | 15/30/60 | |||
2880x2160 3840x2160 4096x3072 | Image | YUV | 15/30 | ||
| Video | YUV/RGB | 15/30 |
- 15Fps only supported in Auto Exposure Mode.
- When multiple streams are configured frame rate is constrained by max resolution.
- FrameRate_None can be used when only Image capture_type is used.
Mixed Reality / Virtual Only Capture
The table below shows which output formats are supported when capturing Virtual and Mixed Reality Content.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| Main | MR/Virtual_Only | 648x720 | Image | JPEG | 30/60 |
| Video | Compressed | 30/60 | |||
| 960x720 | Image | JPEG | 30/60 | ||
| Video | Compressed | 30/60 | |||
| 972x1080 | Image | JPEG | 30/60 | ||
| Video | Compressed | 30/60 | |||
| 1440x1080 | Image | JPEG | 30/60 | ||
| Video | Compressed | 30/60 | |||
| 1944x2160 | Image | JPEG | 30 | ||
| Video | Compressed | 30 | |||
| 2880x2160 | Image | JPEG | 30 | ||
| Video | Compressed | 30 |
Quality and frame rate provided during connection should match the resolution and framerate that was used when preparing capture.
Metadata Tags
Definition of metadata tags for MagicLeap specific extensions to Camera Metadata. The metadata tags can be used with NDK apis to get and set camera metadata. The NDK APIs lack the functionality to query metadata tag from name, which makes it difficult to work with vendor specific tags without having these definitions.
This guide provides a detailed overview of the camera devices available on the Magic Leap 2 and their features, with a focus on additional vendor tags that enhance functionality and flexibility for developers. The Magic Leap 2 supports three camera IDs, each with its unique set of features and capabilities. Below is a table summarizing these camera devices and the key features accessible through the use of specific vendor tags.
| Feature Tag | Description | Applicable Camera IDs | Data Type | Possible Values / Notes |
|---|---|---|---|---|
ML_CONTROL_CAMERA_FORCEAPPLY_MODE | Allows overriding default camera settings precedence between Camera ID 0 and Camera ID 1. | 0, 1 | int32 | 1 = Force apply settings |
ML_CONTROL_CAMERA_EFFECT_MODE | Configures special camera effects. | 0, 1 | uint8 | 0 = Off, 1 = Grayscale, 2 = Negative, 3 = Sepia, 4 = Color Selection, 5 = Sharpen, 6 = Emboss, 7 = Sketch |
ML_CONTROL_CAMERA_APP_EXPOSURE_UPPER_TIME_LIMIT | Limits the maximum exposure time to reduce motion blur. | 0, 1 | int64 | Value in nanoseconds, less than 16ms/20ms |
ML_CONTROL_CAMERA_MLTIME_TIMESTAMPS | Queries MLTime timestamps for synchronization purposes. | 0, 1 | int64 | Array of timestamps [StartOfFrameTransfer, CentreOfExposure, EndOfFrameTransfer, ExternalCameraSync] |
ML_CONTROL_CAMERA_NUM_INTRINSICS | Queries the number of camera intrinsics. | 0, 1 | int32 | - |
ML_CONTROL_CAMERA_INTRINSICS | Queries camera intrinsics. | 0, 1 | float | Array: {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3} |
ML_CONTROL_MRCAMERA_TINT | Controls tint for virtual and camera frames composition in Mixed Reality Capture. | 3 | float | 0.0 to 1.0 |
ML_CONTROL_MRCAMERA_ALPHA_BIAS | Controls alpha for virtual and camera frames composition in Mixed Reality Capture. | 3 | float | -1.0 to 1.0 |
ML_CONTROL_MRCAMERA_CAPTURE_MODE | Sets the capture mode for Mixed Reality Capture. | 3 | uint8_t | 0 = Mixed_Reality, 1 = Virtual_Only, 2 = Camera_Only |
ML_CONTROL_MRCAMERA_MLTIME_TIMESTAMPS | Similar to ML_CONTROL_CAMERA_MLTIME_TIMESTAMPS, but for Camera ID 3. It can be used for synchronizing with data like head pose which has MLTime timestamps. | 3 | int64 | Array of timestamps [StartOfFrameTransfer, CentreOfExposure, EndOfFrameTransfer, ExternalCameraSync] |
ML_CONTROL_MRCAMERA_NUM_INTRINSICS | Similar to ML_CONTROL_CAMERA_NUM_INTRINSICS, but for Camera ID 3. It can be used to query the number of camera intrinsics, which corresponds to the number of streams configured for the camera. | 3 | int32 | - |
ML_CONTROL_MRCAMERA_INTRINSICS | Similar to ML_CONTROL_CAMERA_INTRINSICS, but for Camera ID 3. It can be used to query the camera intrinsics for Mixed Reality Capture. | 3 | float | Array: {width, height, focal_length_x, focal_length_y, principal_point_x, principal_point_y, field_of_view, k1, k2, p1, p2, k3} |
Resources
For more information, see Android's Camera documentation and Camera 2 Guide.