Skip to main content
Version: 14 May 2024

UI Modules

This guide describes the UI features you see in two sections. The first section describes what is common when using a device or simulator target. The second part describes what features are specific to using a simulator target.

Modules for Both Simulator and Device Targets

This section describes what UI modules are visible whether you use a simulator or a device target. The data displayed are changeable if you're using a simulator target, but read only for device target.

Head Pose Panel

The Head Pose panel represents the position, orientation, and state of the headset.

Head Pose Panel

Controller Panel

The Controller Panel represents the position, buttons, and touchpad inputs of the controllers. You can choose Snap To Hand Pose to either left or right hand, which means the controller will be held by that hand and move with that hand. You can also disable this feature and check the Follow Head Pose box instead, which will make the controller stay in position relative to the head pose.

Controller Panel

Changing Position for Controller

The panels for Controller have X, Y, and Z settings for position and orientation. You can manually enter values for these. Or, you can click on one of the letters and drag the mouse side to side to change the values.

Actions

A touchpad touch registers only when the z (force) value is greater than zero. You can also send button presses.

Touchpad Gestures

You can perform swipes and other touchpad gestures.

  1. Set the Gesture Type.
  2. Change the Touchpad Gesture State to Start, and then change it to End.
  3. Set the Gesture Direction.

Hand Tracking Panel

The Hand Tracking panel represents the position, orientation, and recognized static hand gestures.

Hand Tracking Panel

For each selected gesture, your app receives notice of the recognized gesture, the confidence value, and the world coordinates (position and orientation) of the gesture.

A gesture with a confidence value of 0 is still a detected gesture. To indicate no detected gesture, click the no-hand gesture.

No Detected Gesture Selector in Hand Tracking Panel
note

The Simulator detects gestures and key points outside the meshing frustum. This is by design, but remember that on device, gestures outside the meshing frustum are not detected.

Changing Position for Hand Tracking

The panels for Hand Tracking have X, Y, and Z settings for position and orientation. You can manually enter values for these. Or, you can click on one of the letters and drag the mouse side to side to change the values.

Eye Gaze Panel

The Eye Gaze panel represents the vergence point and the positions of the eyes.

Eye Gaze Panel

The eye positions are the positions of the center of the eyes as would be reported by the eye tracking cameras.

We do not make any effort to correlate the vergence position with the values you enter for your eye position. You can set them independently and it is up to you to enter realistic values.

Modules for Simulator Targets

Application Simulator Module Overview

You can hide or reveal panels, including Scene View or Device View, using the panels menu.

Application Simulator Menu Button for Panels

Primary Navigation Target Menu

Using this menu you can select the primary target of navigation.

Application Simulator Menu Primary Target of Navigation
Application Simulator Primary Target of Navigation Expanded Dropdown Menu

To learn more about Primary Navigation Targets see Action Bindings and Input Mapping.

A - Scene View

Scene View depicts all the entities in the scene. The red lines represent the meshing frustum used in the Simulator.

Scene View

Scene View Toolbar

Manipulation Mode

IconDescription
Select icon
Manipulates selected objects.
Move icon
Moves the selected objects.
Rotate icon
Rotates the selected objects.
Scale icon
Scales the selected objects.

Anchor Mode

IconDescription
Base pivot point icon
Set the gizmo transform pivot point to the object base.
Center pivot point icon
Set the gizmo transform pivot point to the object center.

Reference Mode

IconDescription
Local reference icon
Set gizmo transforms to be relative to local object coordinates.
World reference icon
Set gizmo transforms to be relative to world coordinates.

Scene View Camera Mode

IconDescription
Birds-eye view icon
Shows a birds-eye view of the room contents in the Mini Map.
Free Fly view icon
Free Fly Camera
Reset camera icon
Resets the camera to its starting position.

Gizmos

IconDescription
Group icon
Select which gizmos to show, such as cache grid, compass, or hands.

Editing Objects in the Scene View

If you want to move, rotate, or resize any of the objects in the room:

  1. Click the object in the Scene View to select it. A bounding box is drawn around the object.
Select Object in Scene View
  1. Set your manipulation mode in the toolbar, or press one of the hotkeys described in Action Bindings. Ensure you're manipulating the object in the correct transform space by toggling pivot or orientation.
Change Manipulation of Object in Scene View
  1. Drag the controls on the gizmo to transform the object accordingly. The Headset, Control, and other Magic Leap API placeholders can be moved and rotated, but not resized.
Move GizmoRotate GizmoScale Gizmo
Gizmo Move
Gizmo Rotate
(
Gizmo Scale
  1. Or, directly edit the Position, Orientation, or Scale of the object in the Scene Graph panel.

B - Device View

This composites the graphics from the eye buffers with a view of the scene as seen from the point of view of the headset.

C - Tool Panels

The panels provide various features.

Rendering Panel

The Rendering panel displays parameters that control how the views are rendered.

Overview of the Rendering Panel in Application Simulator

Scene Graph Panel

The Scene Graph panel allows you to load new virtual rooms or custom models, or edit the objects in the scene.

Scene Graph Toolbar

IconDescription
Load room icon
Load Virtual Room – You can load any .room file exported from the Virtual Room Generator or any session that has been converted to a .room file using the Package Session as Room option in the Session drop down.
Add model icon
Add Model – You can import individual 3D objects into your virtual room. The following formats are supported: .3ds 3ds Max; .ase 3ds Max; .blend Blender 3D; .dae Collada; .fbx Autodesk 2014/2015, not 2016+; .obj Wavefront; .ply Stanford Polygon Library; .gltf, .glb GL Transmission format
Clear room icon
Clear Virtual Room

Create a Room

You add to the virtual room layout by selecting the Add Room button in the Scene Graph panel's toolbar.

Add Room Button

If you want to completely replace the room, click the Clear Scene button, and then the Add Room button.

Clear Scene button

Virtual rooms are created in the Room Generator tool.

Moving Around in Application Simulator

You can move the hands and controller as a unit to walk around the room using the arrow buttons on your keyboard. If you hold down both mouse buttons, you can turn side to side to view other areas of the room.

Changing Position for Objects in the Room

Using the Scene View panel, you can click on an object and then use the arrows in the image to pull the object in different directions. You can click the Set Manipulation Mode icon to choose whether to move, rotate, or scale the object differently.

You can also choose an object from the the Scene Objects panel and then move it around in the Scene View.

Properties

Select an object in the Scene Graph or in the Scene View. The properties of the selected object will be shown in this panel and you can change them as needed.

Scene View Camera Panel

tip

Position values are given in meters.

Orientation values are given in degrees.

Check boxes are boolean toggles for the associated property: selected is true, clear is false.

To change the value of any numeric property:

  • Click inside the field, type the new value, and then press Tab or Enter.

    -or-

  • Pause on the label of a field. When the pointer changes to a , drag the pointer to change the value.

Marker Tracking Panel

The Marker Tracking panel enables you to add and remove markers in the virtual room, and edit attributes of the markers. For more details about this panel, see the Marker Tracking guide.

Marker Tracking Panel

Permissions Panel

You can see permissions you granted or denied for your application in this panel. If you need to change the settings, you can by clicking on the green drop-downs and choosing whether to grant or deny a permission.

Permissions Panel

The Permissions panel provides the ability to set states for several permission areas supported by the MLSDK. The permission states (Allowed, Denied, Pending) are enforced during ML API invocations, as documented in API headers.

When an API finds a permission state to be in Pending state, it signals the UI to generate a popup that lets the user allow or deny the permission.

Permissions Panel

The UI lets you load a manifest file that filters available permissions to those specified in the file. All permission requests that are not specified in the manifest are denied. Two pre-defined filters, ZI PERMISSIONS and ALL PERMISSIONS, are also provided. The former filters permissions to areas supported by the Application Simulator, whereas the latter filters to all permissions in the MLSDK.

All permission states can be saved in a named Profile and persist across Application Simulator sessions. The UI also has a ... button that brings up a menu to save, load, and delete profiles. Additionally, all permission states can be set to allowed, denied, pending, or default. When set to default, all normal permissions state are set to allowed state, whereas all dangerous permissions are set to pending state.

Permissions Panel
info

The permissions panel allows you to test workflows that require specific Dangerous permissions from the Application Simulator workflow in a more convenient way than on device, which requires fully uninstalling your app to test.

That said, the default behavior is to allow all permissions recognized by the Application Simulator, so that you can opt into the workflow as needed.

When developing ML C API applications, since the Application Simulator does not simulate the AOSP Java runtime, you should conditionally replace the AOSP Java APIs with corresponding C API calls from ml_zi_permissions.h. When developing Unity scenes, the ML Unity SDK will automatically prompt for these permissions as needed via the Magic Leap Hub or Unity Editor.

System Events Panel

The System Events panel provides the ability to trigger various System Events that the device can generate under different conditions. Common System Event sequences have been provided that attempt to mirror the behavior of the event cycles on the real device. Individual events can also be triggered manually.

System Events Panel

Simulator Limitations

note

We verify that the values you enter are valid, but not whether they are realistic. That is so you can make sure your app can handle edge cases or seemingly nonsensical API data.

Audio

Applications exercising ML Audio APIs can play audio both on host and device. Audio input and analysis support is not implemented.