MLCamera
The Magic Leap 2 MLCamera API allows developers to capture real and virtual content inside their applications. While the Magic Leap 2 has only one camera for capturing content, two separate streams can be access from the camera at the same time. This allows you to create applications that can stream the user's point of view, while using the camera to perform computer vision tasks.
The Magic Leap Camera provides the same functionality as the Android Camera2 APIs and can be used when Android Camera 2 functionality is not provided by the build tools. For example, when building with Unity3D.
Camera Streams
Magic Leap 2 allows developers to access two streams from the same physical camera. The camera streams are accessed as devices in the Unity API and have the following identifiers:
Main Camera - provides access to compressed video and still images. This device allows you to capture virtual, real-world, mixed reality content and is the performed choice if you are not performing computer vision tasks on the output or if it is being used for streaming, broadcasting, or images.
CV Camera - best used for Computer vision scenarios, uncompressed, raw frames. If you use this device to do CV in you application, you can use the record/stream gameplay using the Capture Service, but you cannot perform Image or marker tracking using the SDK.
You can use the same stream to capture different types of content. For example, you can capture Video, Images, and a Preview from the Main Camera stream, while also using the CV Camera at the same time.
The Main Camera stream is the only stream that can capture Mixed Reality content. However, the Mixed Reality Capture stream only supports a single concurrent capture mode which means that you need to choose whether to capture Video OR Images but not both.
Stream Capabilities
This section details the supported Resolution, Capture Type, Formats and Frame Rates. The following table provides general information about each of the supported formats.
- Camera type: [Main, CV]
- Mode: [Real, MR (Mixed Reality), Virtual]
- Capture type: [Preview, Video, Image]
These are formats supported:
| Format | Camera | Mode | Capture Type |
|---|---|---|---|
| YUV | Main | Real | Preview/Video/Image |
| YUV | CV | Real | Video/Image |
| RGBA | Main | MR/Virtual | Video/Image |
| RGBA | CV | Real | Video |
| JPEG | Main | Real/MR/Virtual | Image |
Main Camera Stream
The table below shows which output formats are supported when capturing using the Main Camera Stream.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| Main | Cam_Only | 640x480 1280x720 1280x960 1440x1080 1920x1080 2048x1536 | Preview | YUV | 15/30/60 |
| Image | YUV/JPEG | 15/30/60 | |||
| Video | YUV/RGB/Compressed | 15/30/60 | |||
2880x2160 3840x2160 4096x3072 | Preview | YUV | 15/30 | ||
| Image | YUV/JPEG | 15/30 | |||
| Video | YUV/RGB/Compressed | 15/30 |
CV Camera Stream
The table below shows which output formats are supported when capturing using the Computer Vision Camera.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| CV | Cam_Only | 640x480 1280x720 1280x960 1440x1080 1920x1080 2048x1536 | Image | YUV | 15/30/60 |
| Video | YUV/RGB | 15/30/60 | |||
2880x2160 3840x2160 4096x3072 | Image | YUV | 15/30 | ||
| Video | YUV/RGB | 15/30 |
- 15Fps only supported in Auto Exposure Mode.
- When multiple streams are configured frame rate is constrained by max resolution.
- FrameRate_None can be used when only Image capture_type is used.
Mixed Reality / Virtual Only Capture
The table below shows which output formats are supported when capturing Virtual and Mixed Reality Content.
| Camera | Mode | Resolution | Capture Type | Formats | FrameRate |
|---|---|---|---|---|---|
| Main | MR/Virtual_Only | 648x720 | Image | RBG/JPEG | 30/60 |
| Video | RGB/Compressed | 30/60 | |||
| 960x720 | Image | RBG/JPEG | 30/60 | ||
| Video | RGB/Compressed | 30/60 | |||
| 972x1080 | Image | RBG/JPEG | 30/60 | ||
| Video | RGB/Compressed | 30/60 | |||
| 1440x1080 | Image | RBG/JPEG | 30/60 | ||
| Video | RGB/Compressed | 30/60 | |||
| 1944x2160 | Image | RBG/JPEG | 30 | ||
| Video | RGB/Compressed | 30 | |||
| 2880x2160 | Image | RBG/JPEG | 30 | ||
| Video | RGB/Compressed | 30 |
Quality and frame rate provided during connection should match the resolution and framerate that was used when preparing capture.
Upgrading from Magic Leap 1
The Magic Leap 2 offers a new Unified Camera API, which means the Mixed Reality, Computer Vision, and Standard camera streams can be accessed with the same set of APIs.
Additionally, the Magic Leap 2 allows access to two camera devices - Main Camera (Camera 0) and Computer Vision Camera (Camera 1), which allows for unique application features, such as streaming a user's point of view while tracking markers or performing custom computer vision tasks.
While the Magic Leap 2's APIs are new, the API semantics are similar to the Magic Leap 1 MLCamera API. The Magic Leap SDK provides samples that can be used for reference when creating or migrating your application.