Map Gaze Onto Dynamic Screen Content ​

In this guide, we will show you how to map and visualise gaze onto a screen with dynamic content, e.g. a video, web browsing or any other content of your choice, using the Reference Image Mapper enrichment and a few clicks.

What You’ll Need ​

Before continuing, ensure you are familiar with the Reference Image Mapper enrichment. Check out this explainer video for reference.

We recommend you run the enrichment, e.g. with a short recording of your desktop + monitor/screen to ensure it's working okay. Once satisfied, you can use the same reference image + scanning recording for your dynamic screen content.

You will also need to know how to record your screen's content (check out the screen recording section to learn how you can do this using the free and open-source program OBS).

Software Requirements ​

• You will need Python 3.7 or higher installed on your computer.
• Execute the following command pip install pupil-labs-dynamic-rim in the console to install the required libraries.

If you have the checklist completed, you are ready for the next steps.

Making the Recording ​

Let's assume you have everything ready to go – your participant is sat infront of the screen wearing the eye tracker, your screen content is ready to play.

So that we can capture your participant's visual interactions with the screen content, we will need to make sure that both the eye tracking and screen recordings happen at the same time.

Importantly, both sources (eye tracking and screen recording) record individually. As such, you'll need what we call an event annotation to synchronise them later.

The event annotation should be used to indicate the beginning of the screen content recording in the eye tracking recording, and be named start.video. Check here how you can create these events in the Cloud.

Once You Have Everything Recorded ​

• Create a new Reference Image Mapper enrichment, or add your new eye tracking recordings to an existing enrichment. Run the enrichment, and download the results of the enrichment from Cloud once it's computed.

• Now you'll need to get the timeseries and scene video from your new recording(s). Download them by clicking on downloads at the bottom left side of the page in the project's view.

Running the Code ​

Now you can run the code by executing the following command in your console:

pl-dynamic-rim

The application/script will prompt for a series of user inputs:

1. Select the folder path to the Reference Image Mapper download. This folder should contain gaze.csv, sections.csv files and the reference image
2. Select the subfolder corresponding to Timeseries & Scene Video export of the eye tracking recording you are using. This directory should contain event.csv, gaze.csv, world_timestamps.csv, and video files (don't select the folder with the scanning recording you made of the scene)
3. Select the video from the screen display

4. A window will show you the reference image and ask you to select the four corners of your screen.

Use the left button on your mouse to select the points. A red dot will let you know if a point has been chosen when clicked, and if you want to remove the last point, use the right button of your mouse. When the four corners are selected, you will need to press “Q”.

5. Select the folder where you want to save the final video and the CSV file with the gaze coordinates transformed to screen coordinates. (0,0) corresponds to the upper left corner.

And that's all required from you! Grab a coffee or a tea and chill. When finished, you will get a video like the one at the top of this how-to, in addition to a CSV file with your gaze mapped to the screen.

How the Code Works ​

If you only want the final visualisation and the data, there is nothing else you need to do. However, if you want to dig deeper into how the code works under the hood, continue reading by clicking on the dropdown below.

# on video/read.py
def read_screen_video()

# on video/read.py
def read_screen_video()

# on dynamic_rim.py
def merge_tables()

# on dynamic_rim.py
def merge_tables()

# on uitools/get_corners.py
def pick_point_in_image()

# on uitools/get_corners.py
def pick_point_in_image()

# on dynamic_rim.py
def get_perspective_transform()

# on dynamic_rim.py
def get_perspective_transform()

# on dynamic_rim.py
def save_videos()

# on dynamic_rim.py
def save_videos()

# on video/read.py
def get_frame()

# on video/read.py
def get_frame()

# on dynamic_rim.py
def prepare_image()

# on dynamic_rim.py
def prepare_image()

Screen Recording ​

You can use any software of your choice to record your screen, but if you do not know how to do so, here is a small guide from Gauging Gadgets.

Summary ​

1. Download and install OBS from obsproject.com
2. Launch OBS and on the bottom left, create a new scene.
3. Add a display capture source and select your monitor.
4. Configure your settings and click start recording on the bottom right.

5. To stop recording, click the stop button in OBS. You can find your recording in the output directory.

Like a Pro ​

If you want to start your screen and eye tracking recordings automatically (no visual reference required) using network event annotations, we've got you covered!