1.4.1 OS Release Notes
Build number B3E.230928.10-R.098
This release is approved for the ML2 Medtech SKU.
New Features and Enhancements
Our most significant advances in hand tracking accuracy and latency to date
Today, we’re publishing drastic improvements to hand tracking based on your feedback. Users will notice a 6x improvement in overall accuracy and a 10% improvement in overall latency of their virtual hands. Keypoints are now more closely aligned with users’ real hands, and the depth is more consistent when rotating the hands from palmar to dorsal view. Part of this improvement comes from synthesizing views of the hand across our multiple World Cameras.
Gestures are also more responsive and reliable, with notable impacts to pinch and Home gesture detection.
For best Hand Tracking results, turn off the controller or leave it on a nearby surface. These improvements should be noticeable when using the latest MRTK, MLHandTracking Native, and Unity APIs. To update MRTK 2.8, download the latest Unity package on the Hub.
Let us know your thoughts!
Known issues
- Overlapping hands can cause aberrant key point behavior.
- Simultaneous hand and object interactions are not currently supported - key points will behave differently when tools are held.
- Strong lighting conditions can negatively impact hand tracking performance. If you notice that hand tracking is not robust in mixed lighting conditions, try moving to a different area.
Iris ID for Third Party Application Login
Developers now have the ability to enable biometric authentication as part of the login process for an app. If the user has set an unlock PIN and enrolled with Iris ID, they will be able to conveniently verify their identity after they have completed the initial login. Developer documentation for enabling this feature in an app is available on the developer portal.
Automatically Unlock with Iris ID
A new setting is available to enable automatically unlocking the device with Iris ID. After enabling this setting, when the device is in stand-by the device will be unlocked by simply wearing it, without the need to click a button.
Flashing on Homemenu and System UI
Occasionally, the homemenu and some System UI elements would observe a flickering/flashing effect on the ML2. This issue has now been fixed and should provide a more consistent experience on the ML2.
Radeon GPU Profiler failed to capture slow running applications
Previously, some applications running at a lower frame rate than 60 frames per second would have issues with taking Radeon GPU Profiler (RGP) captures. The problem was related to timeouts when gathering data from the device. The tool has been updated to have a longer timeout so this should no longer occur.
Compositor boot time crash
Previously, the device could crash when entering a doze/sleep state while an application was running on it. This exposed some edge cases causing crashes with respect to how the application interacts with our compositor while the device is in doze mode. This bug is now fixed.
Improved RenderDoc documentation
We noticed some cases where RenderDoc would have trouble launching and connecting to an application run on the device if the usb connection is slow. This in turn caused issues with profiling these applications. The solution for this is to increase the timeout timer within RenderDoc to allow it more time to connect to the device. We have added this information to our documentation for RenderDoc available to users.
Camera And Multimedia
Camera and Multimedia stack are Android Java API and NDK API compliant.
Three Camera Devices are supported: Camera Id 0, Camera Id 1, Camera Id 3.
Camera Id 0 and 1 support normal camera capture. Camera Id 3 supports mixed reality capture.
All the cameras are configured as back cameras.
Eight Audio Input Devices are available: Default Microphone Device (Voice and/or World controlled from settings - id 23), Voice (id 5), Voice+Virtual (id 11), World (id 13), Virtual (id 15), Mixed Mics (Voice + World - id 17), World + Virtual (id 19), Mixed (Voice + World + Virtual - id 21).
Improved Headpose Pixel-Stick Performance
Thanks to improvements in online calibration of the World Cameras, virtual content will better stick to the real world with a 63% improvement on average (e.g. if the error was 2 cm when walking 2 meters, it is now 0.74 cm). The calibration applied affects both extrinsic and intrinsic values, and as a result, may also benefit other algorithms that use the World Cameras. Note that these improvements are unique to each device and are dependent on how well the user maps an area before placing virtual content.
Voice Services
On-device system voice intents and app voice intents are now supported for EU French and EU Spanish, in addition to the four previously-supported languages (i.e. US English, Japanese, Modern Standard Arabic and German). VIDTK (Voice Intent Development ToolKit) now supports all on-device languages listed above. Text-To-Speech also supports the above listed on-device languages, with the additions of EU Italian and EU Portuguese.
Known limitation: in-app system voice intents are unsupported after a valid app intent file is loaded for Modern Standard Arabic, EU French, and EU Spanish.
Marker Size Estimation for OpenXR Vendor Extension
API users of the OpenXR vendor extension for marker tracking can omit the marker size in the settings passed during creation or update of a marker tracker. This will trigger automatic marker size estimation which allows the accurate tracking of the 6dof pose of static markers of variable or unknown size. The algorithm needs to gather a few samples to estimate a marker’s size correctly, so the detection may take slightly longer than when the marker’s size has been provided. It will fail for the RGB camera in a completely static scenario where the device doesn’t move at all or when the marker is moving during size estimation. The feature is available for all supported marker types that come with a 6dof pose including QR codes.
On-Device Localization
Extended Localization Feedback: The information returned by the localization service exposes additional metadata to provide more information to down-stream applications.
- Localization error codes: For failed localization attempts, the metadata provides additional insights about the failure reasons. Specifically, exposed additional failure reasons are:
- Failed localization because Headpose detected problematic conditions (low lighting conditions, low scene texture, or rapid head motion).
- The user has left the mapped area.
- The localization service could not localize the device with sufficient certainty (No Consensus), or the service failed to localize the device without a reason that can be further specified (Algorithm Failure).
- Localization Confidence: Successful localization attempts additionally provide information about the estimated quality (confidence) of the localization event. Specifically, the localization service classifies localization attempts with a confidence of either good or fair, where good indicates a high-quality localization event with a strong confidence of good accuracy, while fair indicates a localization event with a higher chance of noticeable inaccuracy.
Improved stability and robustness: Cases where AR content or spatial anchors are displayed at incorrect positions significantly reduced. Cases where AR content or spatial anchors jump to incorrect positions after localization events significantly reduced.
AR Cloud Localization and Mapping Services
Version update of the localization and mapping services to version 2.0.7.
Changes included in version 2.0.7:
- Extended Localization Feedback: The information returned by the localization service exposes additional metadata to provide more information to down-stream applications.
- Localization Confidence: Successful localization attempts additionally provide information about the estimated quality (confidence) of the localization event. Specifically, the localization service classifies localization attempts with a confidence of either good or fair, where good indicates a high-quality localization event with a strong confidence of good accuracy, while fair indicates a localization event with a higher chance of noticeable inaccuracy.
- Updated Map Maintenance: The mapping service of version 2.0.7 includes an update of map maintenance that more reliably detects and removes redundant data in maps processed by the mapping service. This results in a reduced map size, and reduced processing time of subsequent mapping events for a specific space
Changes included in version 2.0.1:
- Minimum input map size changed (reduced) to 20 keyrigs per individual uploaded map.
- Updated frame descriptors for increased localization robustness.
Changes included in version 2.0.0:
- Significantly faster loading times for maps when starting a localization session.
- Significantly faster processing of localization requests.
- Significantly faster processing of map merge events.
Calibrated Magnetometer
Developers can now access the calibrated magnetometer streams.
In order to active get the calibrated stream, the system user may need to run a magnetometer calibration app, in which the user will take off the headset and wave it through a pattern of directions.
This new stream is in addition to the uncalibrated magnetometer streams that were previously available.
Near clipping plane setting
Users can now set the near clipping plane on their Magic Leap 2 devices to a distance between 0.25m and 0.37m. For information on creating apps that account for this, see Display Zone and Clipping Planes in the Magic Leap developer portal. For end user information, see Display Zone in the Magic Leap care center.
Known Issues
Don’t forget: Some Magic Leap hardware and software features are in a pre-release state and might not be available, might contain bugs or defects, and might experience delays or failures. Read this document for guidance on the current state of Magic Leap hardware and software. This guidance helps developers make the best possible use of the Magic Leap platform.
Device Hang
When capturing a frame with either RenderDoc or Radeon GPU Profiler, it is possible (especially with higher load using applications) that the device will have a GPU Hang. This occurs mostly in Unity applications, and we are working with Unity to fix the issue. The workaround is to restart the device.
Application needs reboot after Radeon GPU Profiler
When capturing a frame with Radeon GPU Profiler, there is a chance that the application will not be usable after the capture completes. This will require the user to close and re-open the application from the home menu. We are working with AMD on the issue to fix it.
Visual glitching
- We are actively investigating an issue wherein some Unity applications exhibit visual glitching and blinking.
- For some applications, the user may see a flicker when viewing an image/video capture preview on the device. We are working on fixing this issue.
Color banding seen on recording remote render
While recording content with remote render running, the video may look darker than as seen on the device. We are actively investigating this issue.
BT Earbud connection may be lost after sleep
If Bluetooth earbuds (or headphones) are connected when the device goes to sleep, upon waking the Bluetooth connection to the earbuds may be lost, causing audio to play out of the device speakers until the earbuds are manually reconnected.
AR Cloud Localization and Mapping Services
“Loading shared spaces” message is shown and doesn’t disappear, may require device reboot to clear.
Remote Rendering
Remote Rendering will disconnect from a host computer when going to sleep, or being backgrounded to the Home Menu. When woken/refocused, Remote Rendering may rarely exhibit “Laggy” input.
This can be resolved by returning to the home menu, selecting the “X” button to terminate the Remote Viewer, and reopening the app. The Remote Viewer should immediately reconnect to the host computer without otherwise interrupting an ongoing user session, and a user may continue using it normally.
When exiting either doze or sleep modes or relaunching remote viewer; there are visual artifacts around the no laser pointer once reconnected. To workaround, close and reopen remote viewer
After flashing, Remote Rendering action tile may not be displayed immediately
Magnetometer Calibration
After a calibration failure, user needs to wait at least 2 minutes before retrying calibration due to a timeout duration mismatch between application and firmware.
Magnetometer Data
The magnetometer data will be presented with hard-iron calibration values that come from a singular test device and are not device-specific. The developers need to test if the reference hard-iron calibration values are good enough for their purposes. Otherwise, they will need to calculate the hard-iron calibration values for their device specifically.
System
Browser history may be lost upon upgrade
Browser window may exit to Home when waking from Sleep Mode. Workaround is to reenter Google Meet, and login account again
Browser window loses 6Dof cursor after captured media. Workaround is to press Bumper button to regain 6DoF cursor focus
When logged in as Admin account and attempt to install any SDK app to a specific User the app is unavailable. The workaround is to log out as Admin, log in as the desired user and install the SDK app.
Sleep Mode
Device may reboot on attempt to recover from Sleep Mode
Display may not resume from Sleep Mode, restart required
Unable to unlock the device via Iris ID when exiting sleep mode. Workaround is to place the device back on the surface, allow to re-enter doze mode, and then resume wearing the device. Device will unlock successfully using Iris ID.
WebRTC
WebRTC session disconnects after returning from Standby power state. Workaround is when using an application that includes Web RTC functionality, disable the Standby power state in Settings before launch.
Multimedia and Camera
When using manual exposure mode/autofocus trigger, applications can expect a delay of 3-4 frames between application issuing a request and receiving the result with manual exposure parameters/autofocus trigger applied for the request.
There are known issues around auto exposure compensation may not function properly under low light conditions.
MediaSession may not always receive the key events and may not be able to respond to the key events, when Home Menu or non-Java apps are in the foreground.
The related Media CTS test cases, listed below can still fail sometimes:
android.media.cts.MediaSessionTest#testCallbackOnMediaButtonEvent
android.media.cts.MediaSessionManagerTest#testRemoteUserInfo
android.media.cts.MediaSessionManagerTest#testSetOnMediaKeyListener
android.media.cts.MediaSessionManagerTest#testSetOnVolumeKeyLongPressListener
Some other Media tests might also fail occasionally, for contingent factors, examples of such tests are:
android.media.cts.AudioPlaybackCaptureTest#testCombineUid
android.media.cts.AudioPlaybackCaptureTest#testCombineUsages
android.media.cts.AudioPlaybackCaptureTest#testMatchNothing
android.media.cts.AudioPlaybackCaptureTest#testStoppedMediaProjection
android.media.cts.AudioPlaybackConfigurationTest#testCallbackMediaPlayer
android.media.cts.VisualizerTest#test2_0PollingCapture
android.media.cts.VisualizerTest#test2_1ListenerCapture
And some Camera test might also fail sometimes:
android.hardware.cts.CameraGLTest#testCameraToSurfaceTextureMetadataCamera0
android.hardware.camera2.cts.StillCaptureTest#testAeCompensationCamera0
android.hardware.cts.LegacyCameraPerformanceTest#testLegacyApiPerformanceCamera0
Audio Capture
The preview-feature 'Enhance Voice Clarity' audio setting is not present in this release, but will be supported in subsequent releases.
Assist/Wildfire Demo Apps
We recently identified a software incompatibility issue in our Assist and Wildfire Demo applications, which has resulted in an increase in the transparency of dialog boxes and other images. The workaround is to stay on OS 1.3.0-dev1 or earlier. We understand the significance of this issue and the inconvenience it has caused our customers, and we are fully committed to resolving it in the future.
Control
Controller 6DoF tracking may not work after a restart or coming out of sleep mode. To correct the issue restart the device.
During Controller firmware upgrade from 1.2.0/1.2.1, an incorrect LED pattern will be played on the Controller
Model Viewer
The Model Viewer now includes base models to get new users started with 3D experiences on the ML2.
We also introduced navigational improvements to include a gallery thumbnail view for managing model files and “drag and drop” behavior to open new models. 3D files can now be opened from the file manager on the device in addition to within the application.
For more precise scaling and rotating, the respective numerical values are shown on top of the object bounding box when making changes to a model.
Flashing Instructions
Before flashing make sure to power cycle your device without the cable plugged in. Hold down the power button for 20 seconds to do so.
When flashing, you MUST flash in a linear sequence based on the build order in the table below which can be found on our Customer Care site under Magic Leap OS Update Path
We recommend you to use ML Hub flashing capability to flash the OS. Contact your Magic Leap customer care representative for any additional support with regards to flashing your device.
Device Reset (hardware recovery)
When device (wearable and/or compute pack appear frozen or unresponsive), use the following sequence to restart:
- Remove USB-C cable from the device
- Press and Hold the power button for at least 20 seconds
- Release the power button
- Wait a few seconds, then restart the device by tapping/releasing power button again