Set Up MRTK for Magic Leap 2

This setup guide assumes you have already configured your Unity Project for ML2.

• Configure Project for Magic Leap OpenXR
• Configure Project for MLSDK

Current Status

Feature	Status
Controller	Release
Eye Tracking	Pre-Release
Hand Tracking	Release
Voice Commands	Release
Spatial Meshing	Pre-Release

MRTK 2.8 SDK and Package Compatibility

MRTK Magic Leap Package Version	ML2 Unity SDK Version	Magic Leap XR Package	OpenXR Package
1.12.3	1.12.X	Supported	Unsupported
2.00.0 Preview 1	2.4.0 or higher	Not Tested	Experimental Support

Prerequisites

• Magic Leap SDK
• MRTK Foundations v2.8 (see Import MRTK section below)
• MRTK Examples v2.8 (see Import MRTK section below)
• A configured Magic Leap 2 Unity project

note

If updating to Magic Leap Unity SDK 1.10.0 - 1.12.0, there may be a compile issue if your project contains the Unity OpenXR package. A workaround for this is to remove the OpenXR package from the project in this situation.

Getting Started

Configure Your Project

1. In Player Settings, set Active Input Handling to Both. A restart of the Unity editor may be required.
2. Import the TMP Essential Resources by selecting Window > TextMeshPro > Import TMP Essential Resources.
3. Open the Unity Preferences window and set Script Changes While Playing to Stop Playing and Recompile or Recompile and Continue Playing.
4. Install Universal RP from the Package Manager and set a UniversalRenderPipelineAsset in Graphics Settings.
5. Ensure Custom Main Manifest is selected under Publishing Settings so Magic Leap Manifest Settings can be configured.
6. [OpenXR] The Magic Leap Unity SDK v2.4.0 should install the OpenXR Plugin automatically, but confirm it is present in the project as com.unity.xr.openxr.
7. [OpenXR Hands] Hand Tracking in OpenXR requires the Unity XR Hands package com.unity.xr.hands.

How to Use OpenXR

1. In Player Settings > XR Plug-in Management, ensure Magic Leap is not checked, and select OpenXR then Magic Leap.
2. [If Switching from Magic Leap Provider] It is recommended to close the project and delete the project’s Library folder after switching providers to avoid potential asset conflicts.
3. [Project Validation Error] If a Project Validation error occurs after switching providers, but no specific error is listed in XR Plug-in Management > Project Validation, this is a known Unity issue. Switch to the Windows, Mac, Linux platform in the Build Settings, then switch back to Android to recompile the interface. A common error cause is the Target Devices in Player Settings being set to All Devices.
4. [Optional] OpenXR uses Reference Spaces to determine the scene's origin. A dropdown has been added to the camera settings with all supported reference spaces. Some require the Magic Leap 2 Reference Spaces Feature, which will be set at runtime and may cause content to visibly shift once loaded. To avoid this, or if the default Floor origin space is not desired, add an XR Origin component to the Main Camera Parent, typically MixedRealityPlayspace. Disable this if returning to the old Magic Leap Provider with meshing in the scene.
5. In XR Plug-in Management > OpenXR, specific features and interaction profiles can be added to work with MRTK 2.8 implementations without requiring scene changes.

Interaction profiles

1. Magic Leap 2 Controller Interaction Profile
2. Base Unity Eye Gaze Interaction profile [Works with Magic Leap 2 Eye Tracker Feature]
3. Base Unity Hand Interaction Profile. [Works with base Unity Hand Tracking Subsystem Feature]

Enable Magic Leap 2 OpenXR Features

1. [Required for all Applications] Magic Leap 2 support
2. Magic Leap 2 Eye Tracker
3. Magic leap 2 Meshing Subsystem - Currently works through the already established profile, but will be expanded to support additional OpenXR Settings.
4. [Optional] Magic leap 2 Reference Spaces - Adds support for Unbounded and Local Floor Reference Spaces.
5. [Optional] Magic Leap 2 Secondary View - Improves alignment when making recordings on device.
6. [Optional] Magic Leap 2 Rendering Extensions - Dimming and Blend Mode settings.

A feature labled "Experimental" means the extension is still being reviewed by Khronos to be a part of base OpenXR

Base Unity OpenXR features

1. Hand Tracking Subsystem, Hand Interaction Poses, Palm Pose - All needed for accurate Hand Tracking in OpenXR, these features are from the Unity XR Hands package.

Import MRTK

1. Download version 2.8 of MRTK Foundation and MRTK Examples from the MRTK GitHub.
2. Import the MRTK Foundation 2.8 package into your Unity project. Apply the recommended settings from the popup window that appears after doing so.
3. It will ask for Script Updating Consent, select Yes, for these and other files that might be found later.
4. Next, import the MRTK Examples 2.8 package into your project. Note: Some Oculus prefabs may log an error, these can be cleared.
5. Note that the MRTK Tools package is now required for future upgrading to a newer version of MRTK according to the releases page linked to in the first step.
6. From the top menu Select Mixed Reality > Toolkit > Utilities> Upgrade MRTK Standard Shader for Universal Render Pipeline. This will be greyed out unless a URP pipeline asset is set in the Graphics Settings

Edit MRTK Standard Shader

The standard shader in MRTK (MixedRealityStandard.shader) may throw an error or warning message in the log when building in Unity v2022.2 or higher. To resolve these, make the following changes to the MixedRealityStandard.shader shader file.

1. Change line 775 from fixed facing : VFACE to bool facing : SV_IsFrontFace

line 775

+            fixed4 frag(v2f i, bool facing : SV_IsFrontFace) : SV_Target
-            fixed4 frag(v2f i, fixed facing : VFACE) : SV_Target

2. Change line 956 from * facing; to * (facing ? 1.0 : -1.0)

line 956

+                worldNormal = normalize(worldNormal) * (facing ? 1.0 : -1.0)
-                worldNormal = normalize(worldNormal) * facing;

3. Change line 959 from * facing; to * (facing ? 1.0 : -1.0)

line 959

+                worldNormal = normalize(i.worldNormal) * (facing ? 1.0 : -1.0)
-                worldNormal = normalize(i.worldNormal) * facing;

Import MRTK Magic Leap 2

1. Download the MRTK Magic Leap 2 Unity Asset using the Magic Leap Hub Package Manager.
2. Import the asset into your project by going to Assets > Import Package > Custom Package and selecting the MRTK Magic Leap Unity package, located under USER / MagicLeap / tools / unity / mrtk / VERSION / directory.

MRTK Magic Leap 2 Examples

You can test the MRTK Magic Leap 2 implementation using the Example scenes located under MRTK-Magic Leap 2/Samples/

Make sure the MixedRealityToolkit's configuration profile is set to MagicLeap2 ...

Example Features

• Voice Intents: Refer to the SpeechCommandsDemoMagicLeap scene.
• Control: Refer to the ControlMagicLeapDemo scene.
• Hand Tracking: Refer to the HandInteractionExamplesMagicLeap scene. Note: Only partially implemented on this platform.
• Eye Tracking: Refer to the EyeTrackingDemoMagicLeap scene.
• Meshing: Refer to the MeshingDemoMagicLeap scene.
• All Interactions: Refer to the InteractionsDemoMagicLeap scene for Hand Tracking, Control, Eye Tracking, and Voice usage together.

Troubleshooting

Eye Tracking Calibration

If Eye Tracking has a noticeable offset, run through the device's Eye Tracking Calibration process.

Configure Hand Tracking Settings

The Magic Leap Hand Tracking Input Provider and Magic Leap Device Manager Input Provider can be configured to dynamically switch between controller and hand input devices.

However, if you need to override or edit this behavior you can use the following settings:

Settings for Disabling Controller

Track a Single Hand

Developres can choose to track a single hand. This can be done by setting the following changing MagicLeapHandTrackingInputProvider CurrentHandSettings property.

MagicLeapHandTrackingInputProvider.Instance.CurrentHandSettings = MagicLeapHandTrackingInputProvider.HandSettings.Left;

Disable Hand Tracking

Hand tracking can be disabled by setting the MagicLeapHandTrackingInputProvider's CurrentHandSettings to HandSettings.None or remove the Magic Leap Hand Tracking Provider from the MRTK input providers

MagicLeapHandTrackingInputProvider.Instance.CurrentHandSettings = MagicLeapDeviceManager.HandSettings.None;

Known Issues

• To use the simulator while running MRTK, set the Script Changes While Playing setting to Stop Playing and Recompile or Recompile and Continue Playing in Unity Preferences.
• Occasionally, the controller may not connect properly when launching a new application. Pausing and resuming the application usually resolves this issue.
• [OpenXR] Performance issues with MRTK shaders at startup may cause delays, resulting in a few seconds of no content. This is noticeable with multiple Data Providers or in scenes with Eye Tracking. The Interactions Example temporarily uses Head Tracking as a fallback instead of Eye Tracking while this issue is investigated.

Important Notes

• Copying a configuration files: Instead of copying a configuration file, clone the DefaultMixedReality version and make adjustments. Copying an MRTK configuration file can cause issues such as Input Data Providers not loading or visualizers not attaching properly.
• Controller Visualizer Issue: Occasionally, the Controller Visualizer may stop positioning, and the logs may indicate: Left_ControllerModel(Clone) is missing an IMixedRealityControllerVisualizer component! This issue occurs sporadically; adding the MixedRealityControllerVisualizer
• Blank or Empty Scene on Build: If your application builds and results in a blank/empty scene, you must adjust your project's quality settings. (Known issue in editor. This will be resolved in future 2022.2 editor releases) To resolve this, remove all but one of the quality presets in your projects quality settings (Edit>Player Settings>Quality)
• Tracked Pose Driver: Users may need to add the tracked pose driver to the camera themselves.