Magic Leap 2 Reprojection

The Magic Leap 2 Reprojection feature enables an application to provide additional reprojection information for a projection composition layer to help virtual content appear more stable and improve visual quality. This is an advanced feature that can be used as an alternative instead of the the existing FocusDistance, and requires some experimentation. When the Magic Leap 2 Reprojection feature is enabled, the focus distance is no longer used.

This feature enables Magic Leap 2 support for the OpenXR XR_MSFT_composition_layer_reprojection extension. See the main OpenXR Specification for complete information regarding the extension.

Experimental API

This API is experimental and is subject to change or removal without prior notice. It may be unstable and is not recommended for use in production environments.

What is Reprojection?

Reprojection uses information of objects in the scene to adjust the image rendering based on user movement. This reduces visual inconsistencies like blurriness or ghosting.

tip

See the the OpenXR | Improving Visual Stability guide to learn more about Reprojection and how the supported modes can be used to improve visual stability when rendering virtual content.

Supported Reprojection Modes

• Depth: Utilizes full depth data for high-quality reprojection. Offers the best visual stability but comes with the highest performance cost. This mode is typically used for scenes with high depth variance. Additionally, content such as UI and particle effects do not write to the depth buffer which can result in artifacts as the user moves around.

• Planar From Depth: Planar reprojection where it's parameters are automatically computed from the depth buffer. This balances performance and visual quality. This mode works best when content is mostly placed on a planar surface.

• Planar Manual: Allows the developer to manually define the reprojection plane. This mode works best when content is mostly on planar surfaces and if the plane parameters are easy to compute. It’s ideal for scenarios like positioning a browser window in an augmented reality environment where the user remains stationary.

caution

Semi-transparent materials, such as Unity UI elements may not work properly since their shaders often skip the depth buffer. To enable depth submission for UI elements, add the ZWrite flag to the top of the Pass block definition in the shader.

Modifying the Default UI Material Shader to Support Depth Submission

Step 1: Create a Custom Shader

1. In Unity, navigate to Assets > Create > Shader > Unlit Shader.
2. Name the new shader UIDepthWriteShader.
3. Open the newly created shader file.
4. Replace the shader code with the code from Unity's default Sprite Shader as a starting point. You can find Unity’s default shaders in the Unity Shader Archive.
   • Select the Unity version you're using.
   • Go to See all downloads > Other Installs > Shaders.
   • Download the shader package, locate the Default-Sprite.shader, and copy its contents into your new shader.

Step 2: Modify the Shader for Depth Submission

1. Inside the shader file, locate the Pass block (typically around lines 40-50).

2. Add the following lines to ensure the shader writes to the depth buffer and always passes the depth test:

   ZWrite On
   ZTest Always
   • ZWrite On: Enables depth writing, ensuring the shader submits depth information to the buffer.
   • ZTest Always: Ensures that the object is always drawn, even if there is other content in the scene. This is useful for UI elements that must appear regardless of scene depth.

3. Save the shader file.

Step 3: Create a Material Using the Custom Shader

1. In Unity, navigate to Assets > Create > Material.
2. Name the material UIDepthMaterial.
3. In the Inspector, set the Shader for this material to UIDepthWriteShader (the shader you created in Step 1).

Step 4: Apply the Material to UI Elements

1. Select the UI element(s) you want to modify in your scene.
2. In the Inspector panel, locate the CanvasRenderer or Image component.
3. Under the Material property, assign the UIDepthMaterial you created in Step 3.

Step 5: Verify Depth Submission

1. Enter Play Mode and observe how the UI behaves relative to 3D objects in your scene.
2. Ensure that the UI elements are now occluded by other objects when appropriate. If the UI elements are still drawn in front of other objects, review the shader settings to confirm the depth submission flags are applied correctly.

Enabling the Feature

1. Enable Magic Leap 2 Reprojection
   1. Go to Edit > Project Settings
   2. Select OpenXR under XR Plug-in Management from the left-hand menu.
   3. Under Features, scroll down and enable the Magic Leap 2 Reprojection feature.
2. Enable Depth Submission
   1. At the top of the OpenXR settings, set the Depth Submission Mode to 16 Bit or higher

Depth Submission Mode

See the Unity OpenXR package documentation for more information about the Depth Submission Modes. Project configuration | OpenXR Plugin | Depth Submission Mode

Example Script

This script allows you to test different reprojection modes quickly, including the effect of velocity on the reprojection plane:

using System;
using MagicLeap.OpenXR.Features.Reprojection;
using UnityEngine;
using UnityEngine.XR.OpenXR;

public class ReprojectionTest : MonoBehaviour
{
    public MagicLeapReprojectionFeature.ReprojectionMode reprojectionMode = MagicLeapReprojectionFeature.ReprojectionMode.Depth;
    public Transform targetObject;  // Assign in the Inspector
    private MagicLeapReprojectionFeature reprojectionFeature;
    private Vector3 previousPosition;

    private void Start()
    {
        reprojectionFeature = OpenXRSettings.Instance.GetFeature<MagicLeapReprojectionFeature>();

        if (reprojectionFeature == null || !reprojectionFeature.enabled)
        {
            Debug.LogError("MagicLeapReprojectionFeature is not enabled!");
            enabled = false;
            return;
        }
        
        // Enable reprojeciton at start
        reprojectionFeature.EnableReprojection = true;

        previousPosition = targetObject.position;
        ApplyReprojectionMode();
    }

    private void ApplyReprojectionMode()
    {
        reprojectionFeature.SetReprojectionMode(reprojectionMode);
        Debug.Log($"Reprojection mode set to: {reprojectionMode}");
    }

    private void Update()
    {
            // If you have additional info about where the user is looking, you can improve reprojeciton quality
            // by setting the PlaneInfo regardless of the selected Reprojection Mode.
            // When using PlanarManual, setting these values is required.
            Vector3 position = targetObject.position;
            Vector3 normal = GetNormal(targetObject);
            Vector3 velocity = (position - previousPosition) / Time.deltaTime;

            reprojectionFeature.SetReprojectionPlaneInfo(position, normal, velocity);
            previousPosition = position;
    }

    private Vector3 GetNormal(Transform target){
        // If the targetObject is a planar object (like a textured quad), its forward direction
        // can be used as the normal. For 3D shapes, however, the normal should point towards the user/camera.
        // The normal here is calculated to always face the main camera/user. 
        // In this example, we assume that objects will have the tag Planar if they are 2D.
        if (target.CompareTag("Planar"))
        {
            return  target.forward;  // Use forward direction for planar objects
        }
        else
        {
            return (Camera.main.transform.position - target.position).normalized;  // Normal points to the user
        }
    }
}

How to Use the Script

1. Setup:

   • Attach the script below to any GameObject in your scene.
   • Assign a target object (e.g., a moving object in the scene such as the controller) to the targetObject field in the Inspector.
   • Choose a reprojectionMode from the Inspector.

2. Run the Scene:

   • Observe the impact of different reprojection modes on visual stability.
   • In Planar Manual mode, notice how the reprojection plane adjusts based on the position, normal, and velocity of the target object.

General Recommendations for Plane Info

When using reprojection, it’s important to consider the user's gaze and whether they are likely following a moving object. In such cases, adding plane info data can enhance visual stability. If you have additional information about where the user is looking, especially when they are following a moving object (e.g., during animations or dragging), set the plane info data. This ensures better reprojection quality, independent of the selected mode.