Skip to main content
Version: 14 Oct 2024

Marker Tracking

Marker tracking is a technique that allows you to detect and track the position and orientation of physical markers in the real world using a camera.This guide provides insights into best practices for marker tracking using various cameras and markers.

The Marker Tracking API can be used to track fiducial markers like QR codes, April tags and ArUco markers. For fiducial markers, Magic Leap provides 6DOF poses, and the encoded information. The tracker supports up to 16 markers. Identical QR codes will be treated as separate targets and reported individually. Identical ArUco markers or April tags will lead to a collision and won't be reported separately.

Supported trackable markers (with pose info):

  • ArUco markers
  • AprilTag v2
  • QR codes (model 1 & 2)

List of supported, detectable markers (no pose info):

  • EAN-13 (experimental)
  • UPC-A (experimental)

Camera Types​

There are two types of camera options you can use for marker tracking: the World Cameras and the CV Camera. Each type has its own advantages and disadvantages depending on the use case and the environment.

World Cameras​

The world cameras are a set of three cameras integral to the Magic Leap 2, primarily used for tracking the user's head pose. They offer an extensive field of view for marker detection.

  • Fixed Resolution and Frame Rate: Operate at a fixed resolution of 1018x1018 and a frame rate of 30fps, independent of the FPS hint and resolution hint settings.
  • Enhanced Low Light Performance: Capable of IR light pass-through, they are effective in dark or IR-illuminated environments.
  • Resource Efficiency: Consumes less computational power as they don’t require an additional camera stream.
  • Close Focus Capability: Optimized for close-up objects within arm’s length, although larger markers can be detected at further distances.

CV Camera​

This option utilizes the RGB camera located at the front of the headset. The CV (Computer Vision) Camera stream is used, allowing simultaneous access to the Main Camera stream even when Marker Tracking is active.

  • Adjustable Parameters: Developers can set the resolution hint and FPS hint, offering flexibility for detecting smaller or more distant objects.
  • High-Resolution Imaging: Enables the use of higher resolution images, improving the accuracy and stability of marker tracking.
  • Adaptive to Lighting Conditions: Features adjustable settings like white balance and exposure, enhancing marker visibility under varying light conditions.

Marker Settings​

The marker settings allow you to adjust the detection parameters that are used to identify and track markers in the camera frame. This section goes over each of those parameters.

Marker Tracker Profiles​

Marker tracking profiles are predefined settings that are optimized for specific use cases. Developers can select from the following:

Custom Profile SettingDescription
DefaultA Generic tracker profile. Covers standard use cases.
CustomApplication can define a custom tracker profiler. See MLMarkerTrackerCustomProfile for more details.
SpeedOptimized for speed over accuracy. Use this profile to reduce compute load and increase detection/tracker speed. Most suitable for QR Code reading.
AccuracyOptimized for accuracy. Use this profile to optimize for accurate marker poses. This can cause increased load on the computer.
Small TargetsOptimized for small targets. Use this profile to optimize for markers that are small or for larger markers that need to be detected from far.
Large FoVOptimized for field of view. Use this profile to be able to detect markers across a larger field of view. The Marker Tracker system will attempt to use the World Cameras to detect the markers.

Detection Settings​

This section provides an overview of the settings that can be fine-tuned when the Marker Tracking Profile is set to Custom. For a complete explanation of each of the parameters, refer to the API documentation.

SettingDescription
MLMarkerTrackingFpsHintDetermines the framerate for all detectors. Higher fps and resolution hint values elevate the CPU load.
MLMarkerTrackingResolutionHintSpecifies the resolution for CV camera detection, leaving world camera detection unaffected.
MLMarkerTrackerCameraHintAllows selection between the RGB CV camera stream and world camera images.
MLMarkerTrackerCornerRefineMethodThis setting influences the accuracy-speed trade-off in the detection pipeline. Options range from no refinement to various refinement methods, each with its own performance metrics.
MLMarkerTrackerFullAnalysisIntervalHintRegulates the frequency of full-frame analysis by the detector, which can impact the detection of new markers and the overall tracking performance.

Supported Marker Types​

Magic Leap 2 supports 5 different types of markers and can detect up to 16 unique marker IDs in a given frame. Here is a brief overview of the supported marker types:

QR Code (Supports 3D Pose)​

  • Two-dimensional barcodes that can store a variety of data types such as URLs, text, etc.
  • Fast readability and large storage capacity
  • The amount of data that can be stored depends on the density of the squares

ArUco (Supports 3D Pose)​

  • Primarily for camera calibration and real-time pose estimation
  • Suitable for applications in robotics and augmented reality for 3D positioning
  • Faster to detect than April Tags
  • Incorporates error detection and correction
  • Uses intensity thresholds (the difference between the whitest pixel and darkest pixel in the image or region)
    • Less resilient against changing lighting conditions

April Tags (Supports 3D Pose)​

  • Designed to be easily detected even in poor lighting conditions or when viewed at oblique angles.
  • Suitable for applications in robotics and augmented reality for 3D positioning.
  • More accurate pose estimation than some other marker systems.
  • Incorporates error detection and correction.
  • Uses topology to detect the marker which provides resilience against issues such as partial occlusion and distortion.

EAN13​

  • Requires the marker to be in close proximity to the target camera.
  • A linear barcode that encodes a 13-digit Global Trade Item Number (GTIN).
  • A 13 digit barcode standard which is a superset of the original 12-digit Universal Product Code (UPC) system.
  • Widely used worldwide for identifying products at the point of sale.

UPC A​

  • Requires the marker to be in close proximity to the target camera
  • A linear barcode that encodes a 12-digit Global Trade Item Number (GTIN)
  • The original barcode system for retail product tracking in the United States

ArUco and April Tag Dictionary Naming​

When using ArUco or April Tag marker detection, it's important to understand the difference between the different dictionary options. The following section explains their naming conventions.

  • ArUco: ArUco dictionary have the following structure: DICT_4x4_50 this can be decomposed in the following way:
    • _4x4_ - Refers to the dimension of the marker in terms of the grid cells. For instance, a 4x4 Aruco marker refers to a marker that is 4 cells wide and 4 cells high, where each cell can be either black or white and represents a bit in the binary coding of the marker.
    • _50 - Refers to the number of markers in the dictionary
  • April Tag: April Tags have a slightly different structure: DICT_APRILTAG_36h11 they can be decomposed in the following way:
    • _APRILTAG - Indicates that this dictionary is used for April Tag tracking
    • _36- Refers to the dimension of the marker in terms of the grid cells. 36 refers to a marker that is 36 cells wide and 36 cells high, where each cell can be either black or white and represents a bit in the binary coding of the marker.
    • h11 - The minimum number of bit changes needed to convert one valid marker ID into another. The larger the number, the more reliable the marker identification.

Best Practices​

When using markers in an experience, the following section goes over things to consider when using marker tracking in your application:

Marker Design​

The design of the markers affects their detection and tracking performance. There are several factors to consider when choosing or creating the markers, such as:

  • Size: There are no minimum or maximum sizes for markers, however if the markers are too big to fit the camera frame or too small or too far away, detection and tracking will be spotty or fail entirely. As a rule of thumb for stable detection and tracking, we support a “marker size / over distance” ratio of 0.04 so 4cm marker at 1m distance. That means 2cm markers would need to be as close as 0.5m and 10cm markers may be as far away as 2.5m.

  • Material: When choosing the material on which the marker is printed, be aware of the lighting conditions of your environment. For example: glossy surfaces reflect direct light and then markers may not be visible. Also while high contrast is desirable, very bright environments might lead to over-exposure of the marker. So if the marker is going to be well-lit and the brightest object in your environment, printing on a white surface is not recommended.

  • Quality: The marker tracking algorithm uses corners and edges of the printed marker to track its pose accurately. That’s why low quality prints with fuzzy edges will provide an unsatisfactory user experience. Make sure you print with a high resolution and quality, so the edges of the marker are clear and sharp. Also be aware that some printers may inadvertently shrink or grow your marker, so it’s always a good idea to verify its measurements.

  • Marker Spacing: When using a marker board, make sure that there is a white margin of a few mm between each marker. This margin helps the marker tracking algorithm to distinguish the marker from the background and to locate the corners and edges of the marker.

Motion​

Motion is another factor that can affect the detection and tracking of markers, as it can cause motion blur and distortion in the camera images. To handle motion effectively, consider the following:

  • Moving a tracked marker will result in slight lag caused by the lack of motion prediction. Static markers, on the other hand, have no lag as their pose is predicted along with the head motion.

  • Moving the headset too fast (for example, head shaking) can cause motion blur in the camera images. This can prevent the detection of markers. The RGB camera is more prone to motion blur than the world camera, so we suggest using the world camera for high-motion scenarios.

  • Use sufficient lighting to avoid motion blur. If low ambient lighting is required, use the world cameras for marker tracking and illuminate the space with non-visible 750nm wavelength IR lights.

  • Barcodes and QR codes with small details are more sensitive to motion blur. Users might need to remain stationary for optimal detection.

Optimization​

To optimize the performance and efficiency of marker tracking, you can adjust some settings and parameters depending on your needs and preferences. Here are some suggestions for optimization:

  • When using the CV Camera, you can reduce the FPS hint to diminish CPU load. This can help you save battery life and prevent overheating of the device.

  • If you are using the device world cameras, or you need to maintain a high update rate, you can minimize usage by altering FullAnalysisIntervalHint and CornerRefineMethod. These parameters control how often the marker tracking algorithm analyzes the whole image and how much it refines the edges of the detected markers. Reducing these values can improve the performance and speed of marker tracking, but it may also reduce the accuracy and stability.

  • You can boost performance by switching the Marker Type from All to the type you are using such as QR or Aruco_April. This will reduce the amount of inferences done by the algorithm.

  • Setting Corner Refinement to April Tag is computationally intensive and might conflict with other high-load operations. Avoid using this method unless precise initial marker placement is necessary.

  • To prevent excessive resource and power computation, start with the lowest settings and then, gradually increase the options until you achieve the desired results.

Learn More​