Magic Leap 2 Reprojection

The Magic Leap 2 Reprojection feature enables an application to provide additional reprojection information for a projection composition layer to help virtual content appear more stable and improve visual quality. This is an advanced feature that can be used as an alternative instead of the the existing FocusDistance, and requires some experimentation. When the Magic Leap 2 Reprojection feature is enabled, the focus distance is no longer used.

This feature enables Magic Leap 2 support for the OpenXR XR_MSFT_composition_layer_reprojection extension. See the main OpenXR Specification for complete information regarding the extension.

Experimental API

This API is experimental and is subject to change or removal without prior notice. It may be unstable and is not recommended for use in production environments.

What is Reprojection?

Reprojection uses information of objects in the scene to adjust the image rendering based on user movement. This reduces visual inconsistencies like blurriness or ghosting.

tip

See the the OpenXR | Improving Visual Stability guide to learn more about Reprojection and how the supported modes can be used to improve visual stability when rendering virtual content.

Supported Reprojection Modes

• Depth: Utilizes full depth data for high-quality reprojection. Offers the best visual stability but comes with the highest performance cost. This mode is typically used for scenes with high depth variance. Additionally, content such as UI and particle effects do not write to the depth buffer which can result in artifacts as the user moves around.

• Planar From Depth: Planar reprojection where it's parameters are automatically computed from the depth buffer. This balances performance and visual quality. This mode works best when content is mostly placed on a planar surface.

• Planar Manual: Allows the developer to manually define the reprojection plane. This mode works best when content is mostly on planar surfaces and if the plane parameters are easy to compute. It’s ideal for scenarios like positioning a browser window in an augmented reality environment where the user remains stationary.

caution

Planar From Depth is still experimental as we continue to refine the algorithm to improve the performance and quality of this reprojection mode.

Semi-transparent materials may not work properly since their shaders often skip the depth buffer. If writing a custom shader, add the ZWrite flag to the top of the Pass block definition, to configure the shader to write to the depth buffer.

Enabling the Feature

1. Enable Magic Leap 2 Reprojection
   1. Go to Edit > Project Settings
   2. Select OpenXR under XR Plug-in Management from the left-hand menu.
   3. Under Features, scroll down and enable the Magic Leap 2 Reprojection feature.
2. Enable Depth Submission
   1. At the top of the OpenXR settings, set the Depth Submission Mode to 16 Bit or higher

This update ensures consistency in the instruction style, making the steps clear and easy to follow.

Depth Submission Mode

See the Unity OpenXR package documentation for more information about the Depth Submission Modes. Project configuration | OpenXR Plugin | Depth Submission Mode

Example Script

This script allows you to test different reprojection modes quickly, including the effect of velocity on the reprojection plane:

using System;
using MagicLeap.OpenXR.Features.Reprojection;
using UnityEngine;
using UnityEngine.XR.OpenXR;

public class ReprojectionTest : MonoBehaviour
{
    public MagicLeapReprojectionFeature.ReprojectionMode reprojectionMode = MagicLeapReprojectionFeature.ReprojectionMode.Depth;
    public Transform targetObject;  // Assign in the Inspector
    private MagicLeapReprojectionFeature reprojectionFeature;
    private Vector3 previousPosition;

    private void Start()
    {
        reprojectionFeature = OpenXRSettings.Instance.GetFeature<MagicLeapReprojectionFeature>();

        if (reprojectionFeature == null || !reprojectionFeature.enabled)
        {
            Debug.LogError("MagicLeapReprojectionFeature is not enabled!");
            enabled = false;
            return;
        }
        
        // Enable reprojeciton at start
        reprojectionFeature.EnableReprojection = true;

        previousPosition = targetObject.position;
        ApplyReprojectionMode();
    }

    private void ApplyReprojectionMode()
    {
        reprojectionFeature.SetReprojectionMode(reprojectionMode);
        Debug.Log($"Reprojection mode set to: {reprojectionMode}");
    }

    private void Update()
    {
            // If you have additional info about where the user is looking, you can improve reprojeciton quality
            // by setting the PlaneInfo regardless of the selected Reprojection Mode.
            // When using PlanarManual, setting these values is required.
            Vector3 position = targetObject.position;
            Vector3 normal = GetNormal(targetObject);
            Vector3 velocity = (position - previousPosition) / Time.deltaTime;

            reprojectionFeature.SetReprojectionPlaneInfo(position, normal, velocity);
            previousPosition = position;
    }

    private Vector3 GetNormal(Transform target){
        // If the targetObject is a planar object (like a textured quad), its forward direction
        // can be used as the normal. For 3D shapes, however, the normal should point towards the user/camera.
        // The normal here is calculated to always face the main camera/user. 
        // In this example, we assume that objects will have the tag Planar if they are 2D.
        if (target.CompareTag("Planar"))
        {
            return  target.forward;  // Use forward direction for planar objects
        }
        else
        {
            return (Camera.main.transform.position - target.position).normalized;  // Normal points to the user
        }
    }
}

How to Use the Script

1. Setup:

   • Attach the script below to any GameObject in your scene.
   • Assign a target object (e.g., a moving object in the scene such as the controller) to the targetObject field in the Inspector.
   • Choose a reprojectionMode from the Inspector.

2. Run the Scene:

   • Observe the impact of different reprojection modes on visual stability.
   • In Planar Manual mode, notice how the reprojection plane adjusts based on the position, normal, and velocity of the target object.

General Recommendations for Plane Info

When using reprojection, it’s important to consider the user's gaze and whether they are likely following a moving object. In such cases, adding plane info data can enhance visual stability. If you have additional information about where the user is looking, especially when they are following a moving object (e.g., during animations or dragging), set the plane info data. This ensures better reprojection quality, independent of the selected mode.