Welcome to Pupil - the open source head mounted mobile eye tracking platform.

If this is the first time hearing about the Pupil project we recommend you visit the Pupil Labs website.

This wiki is the main source of documentation for the Pupil users getting started with their Pupil headset and developers contributing to code.

Getting Started

Getting Started logo Getting Started logo

This guide will lead you through a basic workflow using Pupil hardware and software.

Once you have a Pupil Headset all you need to do is install the Pupil apps on a computer running Linux, MacOS, or Windows.

We are always working on new features, fixing bugs, and making improvements. Make sure to visit the release page frequently to download the latest version and follow the Pupil Labs blog for updates.

Capture Workflow

Go through the following steps to get familiar with the Pupil workflow. You can also check out video tutorials at the end of the guide.

1. Put on Pupil

Put on the Pupil headset and plug it in to your computer. Make sure there is space between the headset frame and your forehead. Headsets are adjustable and shipped with additional parts. For more information head over to the Pupil Hardware guide.

2. Start Pupil Capture

3. Check pupil detection

Take a look at the Eye window. If the pupil is detected you will see a red circle around the edge of your pupil and a red dot at the center of your pupil.

If the algorithm’s detection confidence is high, the red circle will be opaque. If confidence diminishes the circle will become more transparent.

Try moving your head around a bit while looking at your eye to see that the pupil is robustly detected in various orientations.

Good and bad eye video

Before calibrating, be sure to check that your eyes are well positioned for a robust eye tracking performance. For more details check out. - Pupil Headset Adjustments

Good bad eye setup Good bad eye setup

4. Calibrate

In order to know what someone is looking at, we must to establish a mapping between pupil and gaze positions. This is what we call calibration.

The calibration process establishes a mapping from pupil to gaze coordinates.

Screen Marker Calibration Method

Click c on the world screen or press c on the keyboard to start calibrate.

Follow the marker on the screen with your eyes and try to keep your head stationary

There are other calibration methods and lots more information how calibration works in the user guide.

5. Record

Start capturing data!

Pupil Capture will save the world video stream and all corresponding gaze data in a folder in your user directory named recordings.

  • Start recording: Press the r key on your keyboard or press the circular R button in the left hand side of the world window.
  • The elapsed recording time will appear next to the R button.
  • Stop recording: Press the r key on your keyboard or press the circular R button in the left hand side of the world window.

See a video demonstration of how to set recordings path, session name, and start recording – here.

Where is the recording saved?

Recording folder Recording folder

By default, each recording will live in its own unique data folder contained in the recordings folder.

You can make as many recordings as you like.

The default recordings directory will have the following hierarchy.

  • recordings
    • 2016-04-05
      • 001
      • 002
      • 003
      • ####

How recordings are saved?

Pupil capture saves the video frames in a fixed frame rate container. This means that the raw output video (world.mp4) does not show the correct duration and the correct frame rate of the recording. This information can be found in world_timestamps.npy, which tells you exactly where each frame belongs in time.

However, if you export using Pupil Player, the video will be made such that the frames will show at the exact right time. The output video will not miss any frame of the raw video, instead, output frames are spaced out exactly as they were initially captured.

Player Workflow

Use Pupil Player to visualize data recorded with Pupil Capture and export videos of visualization and datasets for further analysis.

1. Open Pupil Player

Now that you have recorded some data, you can play back the video and visualize gaze data, marker data, and more.


Player comes with a number of plugins. Plugins are classified by their use-case. Visualization plugins can be additive. This means that you can add multiple instances of a plugin to build up a visualization.

Where are Pupil Player exports saved?

Export folder Export folder

Exports are saved within a dedicated folder named exports within the original recording folder.

Each export is contained within a folder within the exports folder. The numbers of the export correlate to the trim marks (frame start and frame end) for the export.

Pupil Capture Demo Video

The video below demonstrates how to setup, calibrate, and make a recording with Pupil Capture.

Turn on closed captions CC to read annotations in the video.

Pupil Player Demo Video

The video below demonstrates how to view a dataset recorded with Pupil Capture, make and export visualizations.

Turn on closed captions CC to read annotations in the video.

Pupil Hardware

Pupil Labs is based in Berlin and ships Pupil eye tracking headsets and VR/AR eye tracking add-ons to individuals, universities, and corporate enterprises worldwide!

Go to the Pupil store for prices, versions, and specs.

Pupil Mobile Eye Tracking Headset

Pupil headset Pupil headset

You wear Pupil like a pair of glasses. Pupil connects to a computing device via a USBA or USBC cable. The headset is designed to be lightweight and adjustable in order to accommodate a wide range of users.

To the right is an illustration of a monocular Pupil Headset, depending on your configuration your headset might look different, but working principles are the same.

Pupil ships with a number of additional parts. The below sections provide an overview of their use and a guide to adjusting the Pupil headset.

Additional parts

World Camera

The high speed 2d world camera comes with two lenses. 60 degree FOV lens (shown on the left) and a wide angle 100 degree FOV lens (shown on the right).

The world camera lens are interchangeable, so you can swap between the two lenses provided for normal or wide angle FOV.

Arm Extender

If you need to adjust the eye cameras beyond the built in adjustment range, you can use the orange arm extenders that are shipped with your Pupil headset. Unplug your eye camera. Slide the existing eye camera arm off the headset. Slide the arm extender onto the triangular mount rail on the headset frame. Slide the camera onto the extended mount rail. Plug the camera back in.

The eye camera arm extender works for all existing 120 and 200hz systems.

Nose Pads

All Pupil headsets come with 2 sets of nose pads. You can swap the nose pads to customize the fit.

Pupil Headset Adjustments

A lot of design and engineering thought has gone into getting the ergonomics of the headset just right. It is designed to fit snugly, securely, and comfortably. The headset and cameras can be adjusted to accommodate a wide range of users.

To ensure a robust eye tracking performance, make sure all the cameras are in focus with a good field of view of your eyes.

Slide Eye Camera

The eye camera arm slides in and out of the headset frame. You can slide the eye camera arm along the track.

Rotate World Camera

You can rotate the world camera up and down to align with your FOV.

Rotate Eye Camera

The eye camera arm is connected to the eye camera via the ball joint. You can rotate about its ball joint.

Ball Joint Set Screw

You can adjust the set screw to control the movement of the eye camera about the ball joint. We recommend setting the set screw so that you can still move the eye camera by hand but not so loose that the eye camera moves when moving the head. You can also tighten the set screw to fix the eye camera in place.

Focus Cameras

No focus 200hz Eye Camera

No focus e200hz eye camera No focus e200hz eye camera

200hz eye cameras do not need to be focused, and can not be focused. The lens of the 200hz eye camera is arrested using glue. Twisting the lens will risk breaking the mount.

Focus 120hz Eye Camera

If you have a 120hz eye camera, make sure the eye camera is in focus. Twist the lens focus ring of the eye camera with your fingers or lens adjuster tool to bring the eye camera into focus.

Focus World Camera

Set the focus for the distance at which you will be calibrating by rotating the camera lens.

HTC Vive Add-On

HTC Vive binocular add-on HTC Vive binocular add-on

Add eye tracking powers to your HTC Vive with our 120hz binocular eye tracking add-on.

This section will guide you through all steps needed to turn your HTC Vive into an eye tracking HMD using a Pupil Labs eye tracking add-on.

Install the Vive add-on

Install the Vive PRO add-on

Install the 200Hz Vive and Vive Pro add-on

A detailed look

… at the engagement process between the eye tracking ring the the lens holding geometry. Do not follow these steps. Just have a look to get a feeling for the snap-in part to the guide above.

HTC Vive USB connection options

The HTC Vive has one free USB port hidden under the top cover that hides the cable tether connection. This gives us two options to connect the pupil eye tracking add-on:

Connect the add-on to the free htc-vive usb port.

This means the cameras share the VIVEs usb tether bandwidth with other usb components inside the Vive. This works but only if the following rules are observed:

  • Disable the HTC-Vive built-in camera in the VR settings pane to free up bandwidth for Pupil’s dual VGA120 video streams.


  • Enable the HTC-Vive built-in camera and set it to 30hz. Then set the Pupil Cameras to 320x240 resolution to share the USB bus.

Run a separate USB lane along the tether

If you want full frame rate and resolution for both the Vive’s camera and the add-on you will have to connect the Pupil add-on to a separate usb port on the host PC. We recommend this approach.

USB Connection and Camera IDs

Once you plug the usb cables into your computer:

  • the right eye camera will show up with the name: Pupil Cam 1 ID0
  • the left eye camera will show up with the name: Pupil Cam 1 ID1

Focus and Resolutions

After assembly and connection. Fire up Pupil Capture or Service and adjust the focus of the eye cameras by rotating the lens by a few degrees (not revolutions) in the lens housing.

Use 640x480 or 320x240 resolution to get 120fps and a good view of the eye. Other resolutions will crop the eye images.

Interfacing with other software or your own code

Both cameras are fully UVC compliant and will work with OpenCVs video backend, Pupil Capture, and libraries like libucv and pyuvc.

Oculus Rift DK2 Add-On

Oculus Rift DK2 add-on Oculus Rift DK2 add-on

Add eye tracking powers to your Oculus Rift DK2 with our 120hz eye tracking add-ons.

This page will guide you through all steps needed to turn your Oculus DK2 into an eye tracking HMD using the Pupil Oculus DK2 eye tracking add-on cups.

Install lens in cup

Take the lens out of an existing Oculus lens cup.

Remove the LED ring and insert the lens into the Pupil eye tracking cup.

Install the LED ring and connect the LED power supply.

Install cup in DK2

Route cables

Route the USB cables through the vent holes in the top of the Oculus DK2.

Connect cameras

Connect the eye tracking cup to the USB cable. Remove the old cup and insert the eye tracking cup in the DK2.

USB and Camera IDs

Once you plug the usb cables into your computer:

  • the right eye camera will show up with the name: Pupil Cam 1 ID0
  • the left eye camera will show up with the name: Pupil Cam 1 ID1

Both cameras are fully UVC compliant and will work with OpenCVs video backend, Pupil Capture, and libraries like libucv and pyuvc.

HoloLens Add-On

HoloLens add-on HoloLens add-on

Add eye tracking powers to your HoloLens with our 200hz eye tracking add-ons. Follow the instructions in the video to install your add-on.


DIY Pupil headset DIY Pupil headset

If you are an individual planning on using Pupil exclusively for noncommercial purposes, and are not afraid of SMD soldering and hacking – then, buy the parts, modify the cameras, and assemble a Pupil DIY headset. We have made a guide to help you and a shopping list.

Getting all the parts

The 3d-printed headset is the centerpiece of the Pupil mobile eye tracker. You can buy it from the Pupil Labs team through the Pupil shapeways store. The price for the headset is part production cost and part support to the pupil development team. This enables us to give you support and continue to work on the project.

All other parts of the Pupil DIY kit have been specifically selected with availability and affordability in mind. See the Bill of Materials to learn what else you will need to get.


You will need access to these tools:

  • Solder station, wick, flux (for SMD solder work)
  • Tweezers
  • Small philips screwdriver
  • Prying tool to help un-case the webcams

Prepare Webcams

The first step is to modify the cameras so we can use them for eye-tracking.

De-case Cameras

Take both webcams out of their casings. Follow the video guides.

  1. decase Logitech C525/C512
  2. decase Microsoft HD-6000

Solder Work on Eye Camera PCB

Microsoft HD-6000 PCB Microsoft HD-6000 PCB

This is by far the trickiest part. You will need some soldering experience, or work with someone that can help you for this step. In the video and photo the lens holder is removed, but you will do it with the lens holder attached.

  1. Cut off the microphone
  2. De-solder or break off the push button (Note: Some cameras don’t have this button.)
  3. De-solder the blue LED’s
  4. solder on the IR-LED’s. Please take note of LED polarity! video

Replace IR-blocking Filter on the Eye Camera

  1. Unscrew the lens from the mount.
  2. Carefully remove the IR filter. Be very careful! The IR filter is a thin piece of coated glass and right behind it is a lens element that must stay intact and unharmed! It is necessary to remove the IR filter, so that the image sensor will be able to “see” the IR light.
  3. Using a hole punch, cut out 1 round piece of exposed film and put it where the older filter was.
  4. Use plastic glue to fix the piece. Don’t let the glue touch the center!
  5. Put the lens back inside. You will have to manually focus the lens when you run the software for the first time by hand. Later you can use the focus control in software to fine tune.


Camera Assembly

  1. Attach the world camera onto the mount using 4 small screws, leftover from disassembly.
  2. Clip the world camera clip onto the headset
  3. Slide the eye-cam into the mount video guide
  4. Slide the arm onto the headset
  5. Route the cables
  6. Attach USB extension cable(s)


The camera mounts can be replaced by custom build parts that suit your specific camera setup or other sensors.

Windows Driver Installation

If you are using Windows, you will need to install drivers for your cameras. Please refer to the instructions here.

Pupil Hardware Development

This page contains documentation and discussion on open source camera mounts, optics, and cameras.

Camera Mounts

Headset camera mounts Headset camera mounts

We release the CAD files for the camera mounts for you to download, modify, in accordance with our license. CAD files for the frame are not open source; see explanation.

Interface Documentation

By releasing the mounts as example geometry we automatically document the interface. You can use the CAD files to take measurements and make your own mounts.


The mounts were developed as part of the whole headset and carry the revision number of the headset they were designed for.

Download Camera Mount CAD Files

All files are hosted in the pupil-hardware-diy repo here

You can clone the latest revision

git clone

Or, if you want an older version, just checkout an older version. In this example, we checkout rev006 rev006 with the git version id of 6ad49c6066d5

git clone 
git checkout 6ad49c6066d5

User Docs

This section of the documentation is targeted towards users of Pupil software and provides a deeper explanation of features and methods.

Pupil Capture

Pupil Capture is the software used with the Pupil Headset. The software reads the video streams coming in from the world camera and the eye camera. Pupil Capture uses the video streams to detect your pupil, track your gaze, detect and track markers in your environment, record video and events, and stream data in realtime.

Capture Window

Pupil Capture UI call-out Pupil Capture UI call-out

The Capture window is the main control center for Pupil Capture. It displays live video feed from pupil headset.

  1. Graphs - This area contains performance graphs. By default the graphs CPU, FPS, and pupil algorithm detection confidence will be displayed. You can control graph settings with the System Graphs plugin.
  2. Hot keys - This area contains clickable buttons for plugins.
  3. Menu - This area contains settings and contextual information for each plugin.
  4. Sidebar - This area contains clickable buttons for each plugin. System plugins are loaded in the top and user added plugins are added below the horizontal separator.

Capture Selection

By default Pupil Capture will use Local USB as the capture source. If you have a Pupil headset connected to your machine you will see the video displayed from your Pupil headset in the World and eye windows. If no headset is connected or Pupil Capture is unable to open capture devices it will fall back to the Test Image. Other options for capture source are described below.

  • Test Image - This is the fallback behavior if no capture device is found, or if you do not want to connect to any capture device.
  • Video File Source - select this option to use previously recorded videos for the capture selection.
  • Pupil Mobile - select this option when using Pupil Capture with the Pupil Mobile Android application.
  • Local USB - select this option if your Pupil Headset is connected to the machine running Pupil Capture. This is the default setting.
  • RealSense 3D - select this option if you are using an Intel RealSense 3D camera as your scene camera. Read more in the RealSense 3D section.

After switching to a different capture source, you can click the Start with default devices button. This will automatically select the correct sensor and start capturing for corresponding world and eye windows. Or, you can manually select the capture source to use from the world and eye windows.

Pupil Detection

Pupil’s algorithms automatically detect the participant’s pupil. With the 3d detection and mapping mode, Pupil uses a 3d model of the eye(s) that constantly updates based on observations of the eye. This enables the system to compensate for movements of the headset - slippage. To build up an initial model, you can just look around your field of view.

Fine-tuning Pupil Detection

As a first step it is recommended to check the eye camera resolution as some parameters are resolution dependent. For fast and robust pupil detection and tracking we recommend using the default resolution settings. For 200hz eye cameras the default resolution is set to 192x192 pixels. If you have an older 120hz eye camera, the default is 320x240 pixels.

In Pupil Capture you can view a visualization of the pupil detection algorithm in the eye windows. For fine-tuning switch to this mode: General Settings > Algorithm Mode.

Pupil Detector 2D/3D

  • Pupil Min/Max : Change to General > Algorithm Mode. The two red circles represent the min and max pupil size settings. The green circle visualizes the current apparent pupil size. Set the min and max values so the green circle (current pupil size) is within the min/max range for all eye movements.
  • Intensity Range : Defines the minimum “darkness” of a pixel to be considered as the pupil. The pixels considered for pupil detection are visualized in blue within the Algorithm Mode. Try to minimize the range so that the pupil is always fully covered while having as little leakage as possible outside of the pupil. Be aware that this is dependent on the brightness and therefore has a strong interaction with UVC Source/Sensor Settings/Absolute Exposure Time.


Pupil uses two cameras. One camera records a subject’s eye movements – we call this the eye camera. Another camera records the subject’s field of vision – we call this the world camera. In order to know what someone is looking at, we must find the parameters to a function that correlates these two streams of information.

Calibration Process



Pupil Headset comes in a variety of configurations. Calibration can be conducted with a monocular or binocular eye camera setup.

Before every calibration

Make sure that the users pupil is properly tracked. Make sure that the world camera is in focus for the distance at which you want to calibrate, and that you can see the entire area you want to calibrate within the world cameras extents (FOV).

YOur pupil is properly detected by the eye camera YOur pupil is properly detected by the eye camera

Your pupil is properly detected by the eye camera

Make sure the world camera is in focus Make sure the world camera is in focus

Make sure the world camera is in focus

Calibration Methods

Before starting calibration, ensure that eye(s) are robustly detected and that the headset is comfortable for the participant.

Screen Marker Calibration

This is the default method, and a quick method to get started. It is best suited for close range eye-tracking in a narrow field of view.

  1. Select Screen Marker Calibration
  2. Select your Monitor (if more than 1 monitor)
  3. Toggle Use fullscreen to use the entire extents of your monitor (recommended). You can adjust the scale of the pattern for a larger or smaller calibration target.
  4. Press c on your keyboard or click the blue circular C button in the left hand side of the world window to start calibration.
  5. Follow the marker on the screen with your eyes. Try to keep your head still during calibration.
  6. The calibration window will close when calibration is complete.

In the Advanced sub-menu you can set the sample duration – the number of frames to sample the eye and marker position. You can also set parameters that are used to debug and detect the circular marker on the screen.

Manual Marker Calibration

Pupil Calibration Marker v0.4 Pupil Calibration Marker v0.4

Pupil Calibration Marker v0.4

Pupil Calibration Stop Marker v0.4 Pupil Calibration Stop Marker v0.4

Pupil Calibration Stop Marker v0.4

This method is done with an operator and a subject. It is suited for midrange distances and can accommodate a wide field of view. The operator will use a printed calibration marker like the one shown in the video. Download Pupil Labs Calibration Marker v0.4 to print or display on smartphone/tablet screen.

  1. Select Manual Marker Calibration
  2. Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
  3. Stand in front of the subject (the person wearing the Pupil headset) at the distance you would like to calibrate. (1.5-2m)
  4. Ask the subject to follow the marker with their eyes and hold their head still.
  5. Show the marker to the subject and hold the marker still. You will hear a “click” sound when data sampling starts, and one second later a “tick” sound when data sampling stops.
  6. Move the marker to the next location and hold the marker still.
  7. Repeat until you have covered the subject’s field of view (generally about 9 points should suffice).
  8. Show the ‘stop marker’ or press c on your keyboard or click the blue circular C button in the left hand side of the world window to stop calibration.

You will notice that there are no standard controls, only an Advanced sub-menu to control detection parameters of the marker and to debug by showing edges of the detected marker in the world view.

Single Marker Calibration

Calibrate using a single marker displayed on screen or hand held marker. Gaze at the center of the marker and move your head in a spiral motion. You can also move your head in other patterns. This calibration method enables you to quickly sample a wide range of gaze angles and cover a large range of your FOV.

  1. Select Single Marker Calibration
  2. Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
  3. Look at the center of the marker.
  4. Slowly move your head while gazing at the center of the marker. We have found that a spiral pattern is an efficient way to cover a large area of the FOV.
  5. Press the C button on your keyboard or show the stop marker to stop calibrating.

Natural Features Calibration

This method is for special situations and far distances. Usually not required.

  1. Select Natural Features Calibration
  2. Press c on your keyboard or click the blue circular C button in the left hand side of the world window to start calibration.
  3. Ask the subject (the person wearing the Pupil headset) to look a point within their field of vision. Note – pick a salient feature in the environment.
  4. Click on that point in the world window.
  5. Data will be sampled.
  6. Repeat until you have covered the subject’s field of view (generally about 9 points should suffice)
  7. Press c on your keyboard or click the blue circular C button in the left hand side of the world window to stop calibration.

Fingertip Calibration

Calibrate using your fingertip! We have found that the easiest way to calibrate with your fingertip is as follows:

  1. Select Fingertip Calibration
  2. Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
  3. Extend your arm and hold your index finger still at the center of the field of view of the world camera.
  4. Move your head for example horizontally and then vertically while gazing at your index finger fingernail.
  5. Show five fingers or press c on your keyboard or click the blue circular C button on the left hand side of the world window to stop calibration.

This calibration method enables you to quickly sample a wide range of gaze angles and cover a large range of your FOV within 10 seconds.

A Convolutional neural network (CNN) is implemented for the fingertip detection:

First, a hand detector, based on MobileNet and SSD, searches for a hand in the image.

The position of the fingertip is then found out by a fingertip detector, adapted from YOLSE and Unet.

Notes on calibration accuracy

In 2D mode, you should easily be able to achieve tracking accuracy within the physiological limits (sub 1 deg visual degrees). Using the 3d mode you should achieve 1.5-2.5 deg of accuracy.

  • Any monocular calibration is accurate only at its depth level relative to the eye (parallax error).
  • Any calibration is only accurate inside the field of view (in the world video) you have calibrated. For example: If during your calibration you only looked at markers or natural features (depending on your calibration method) that are in the left half, you will not have good accuracy in the right half.
  • Calibration accuracy can be visualized with the Accuracy Visualizer plugin. If the Accuracy Visualizer plugin is loaded, it will display the residual between reference points and matching gaze positions that were recorded during calibration.
  • Gaze Prediction Accuracy can be estimated with an accuracy test. Start the accuracy by running a normal calibration procedure but press the T button in the world window and not the C button. After completing the test, the plugin will display the error between reference points and matching gaze positions that were recorded during the accuracy test.


Press r on your keyboard or press the blue circular R button on the left hand side of the world window to start recording. You will see red text with the elapsed time of recording next to the R button. To stop recording, press r on your keyboard or press the R button on screen.

You can set the folder or Path to recordings and the Recording session name in the Recorder sub-menu within the GUI. Note - you must specify an existing folder, otherwise the Path to recordings will revert to the default path.

What will be in the session folder?

If you open up a session folder you will see a collection of video(s) and data files. Take a look at Data format to see exactly what you get.

Open a plugin

Pupil Capture plugins Pupil Capture plugins

Open the Plugin Manager menu on the right. It lists all available plugins. Click the button next to the plugin’s name to turn on or off the plugin.

Third-party plugins

You can easily load third party plugins. Third party plugins will appear in the Pupil Capture or Pupil Player plugin list. Copy the plugin to the plugins folder within the pupil_capture_settings or pupil_player_settings folder.

Fixation Detector

The online fixation detector classifies fixations based on the dispersion-duration principle. Fixations are used by the screen and manual marker calibrations to speed up the procedure. A fixation is visualized as a yellow circle around the gaze point that is shown in the Pupil Capture world window.

You can find more information in our dedicated fixation detector section.

Network plugins

Pupil Capture has a built-in data broadcast functionality. It is based on the network library ZeroMQ and follows the PUB-SUB pattern. Data is published with an affiliated topic. Clients need to subscribe to their topic of interest to receive the respective data. To reduce network traffic, only data with at least one subscription is transferred.

Pupil Remote

Pupil Remote is the plugin that functions as the entry point to the broadcast infrastructure. It also provides a high level interface to control Pupil Capture over the network (e.g. start/stop a recording).

  • Load the Pupil Remote plugin from the General sub-menu in the GUI (it is loaded by default).
  • It will automatically open a network port at the default Address.
  • Change the address and port as desired.
  • If you want to change the address, just type in the address after the tcp://

Pupil Groups

Pupil Groups can help you to collect data from different devices and control an experiment with multiple actors (data generators and sensors) or use more than one Pupil device simultaneously:

  • Load the Pupil Groups plugin from the General sub-menu in the GUI.
  • Once the plugin is active it will show all other local network Pupil Group nodes in the GUI
  • Furthermore, actions like starting and stopping a recording on one device will be mirrored instantly on all other devices.

Pupil Time Sync

If you want to record data from multiple sensors (e.g. multiple Pupil Capture instances) with different sampling rates it is important to synchronize the clock of each sensor. You will not be able to reliably correlate the data without the synchronization.

The Pupil Time Sync protocol defines how multiple nodes can find a common clock master and synchronize their time with it.

The Pupil Time Sync plugin is able to act as clock master as well as clock follower. This means that each Pupil Capture instance can act as a clock reference for others as well as changing its own clock such that it is synchronized with another reference clock.

Pupil Time Sync nodes only synchronize time within their respective group. Be aware that each node has to implement the same protocol version to be able to talk to each other.

Frame Publisher

The Frame Publisher plugin broadcasts video frames from the world and eye cameras.

There is a pupil-helper example script that demonstrates how to receive and decode world frames.

Remote Recorder

The Pupil Mobile app can be controlled via Pupil Capture when connected. This includes changing camera and streaming settings. The Remote Recorder plugin extends this list with the possibility to start and stop recordings that are stored in the phone.

Surface Tracking

The Surface Tracker plugin allows you to define planar surfaces within your environment to track areas of interest (AOI). Surfaces are defined with Apriltag Markers.


There are many different apriltag types, currently we only support apritags of type tag36h11, which is the recommended set. There are 587 distinct markers in the set, so it should be sufficient for most use cases.

You can find images of all markers in the original Apriltag GitHub repository. The tag images are very small in the apriltags repo. Therefore you will need to scale them up before use. We have two pages of markers prepared convenience on the right, that you can download to get started. Markers can be printed on paper, stickers, or displayed on a screen.

Apriltags tag36h11 0-23 Apriltags tag36h11 0-23

Apriltags tag36h11 0-23

Apriltags tag36h11 24-47 Apriltags tag36h11 24-47

Apriltags tag36h11 24-47

Preparing your Environment

A surface can be based on one or more markers. The markers need to be placed in close proximity or within your desired AOI. If your AOI is for example a computer monitor, you could display your markers in the corners of the screen or place them somewhere on the bezel. If your AOI is a magazine page, you could place the markers in the corners of the page, or anywhere else on the page where they are not occluding the content. When placing your markers please follow the guidelines:

  • All markers of a surface need to lie within the same two dimensional plane.
  • An individual marker can be part of multiple surfaces.
  • The used markers need to be unique, i.e. you may not use multiple instances of the same marker in your environment.
  • Using more markers to define a surface yields greater robustness in the tracking of that surface.
  • Surfaces defined with more than 2 markers are detected even if some markers lie outside of the camera image or are obscured.

Defining a Surface

Surfaces can be defined with Pupil Capture in real-time, or post-hoc with Pupil Player. In both cases the necessary steps are as follows:

  • Prepare your environment as described above.
  • Turn on the Surface Tracker plugin .
  • Make sure the camera is pointing at your AOI and the markers are well detected. In the post-hoc case (using Pupil Player) seek to a frame that contains a good view of your desired AOI.
  • Add a new surface by clicking the Add surface button.
  • Give your surface a name.
  • Click the edit surface button and move the corners of your surface into the desired position. In the real-time case (using Pupil Capture) this is much easier if you freeze the video by clicking the Freeze Scene button.
  • If markers have been erroneously added or left out, click the add/remove markers button and afterwards onto the according marker to add/remove them from your surface.

Reusing Surface Definitions

Your surfaces are automatically saved in a file called surface_definitions in the pupil_capture_settings directory. If you restart Pupil Capture or the Surface Tracker plugin, your surface definitions from previous sessions will be loaded. The surface_definitions file is copied into each recording folder as well, so you will have access to your surface definitions in Pupil Player. You can copy & paste this file to move definitions from one session or recording to another.

Gaze Heatmaps for Surfaces

You can display gaze heatmaps for each surface by enabling Show Heatmap in the Surface Tracker menu. Two heatmap modes are supported: * Gaze within each surface: Visualizes the distribution of gaze points that lie within each surface. * Gaze across different surfaces: Color codes the surfaces to visualize the amount of time spend gazing on each surface in relation to other surfaces. Red color represents a lot of gaze points or time spent. Blue color represents few gaze points or little time spent.

The smoothness of the heatmap in Gaze within each surface mode can be set using the Heatmap Smoothness slider, which will effectively change the bin size of the underlying histogram. In the online case the heatmap is computed over the most recent data. The exact time window to consider can be set using the Gaze History Length field.

Further Functionality

  • You can click the Open Surface in Window button to open a view of the surface in a separate window. Gaze positions on the surface will be visualized in this window in real-time.
  • Streaming Surfaces with Pupil Capture - Detected surfaces as well as gaze positions relative to the surface are broadcast under the surface topic. Check out this video for a demonstration.
  • Surface Metrics with Pupil Player - if you have defined surfaces, you can generate surface visibility reports or gaze count per surface. See our blog post for more information.

Legacy Markers

The legacy surface system used simple square markers, which are less robust to detect. For all new projects we strongly recommend using Apriltags!

If you still need to use legacy markers, you can generate them with this script, or download this image.

The design of our markers was greatly inspired by the ArUco marker tracking library. However our markers used 5x5 grid instead of the 7x7 grid ArUco uses. This allowed us to make smaller markers that can still be detected well. The 5x5 design allowed for a total of 63 unique markers.

The pupil detection algorithm assigns a confidence value to each pupil datum. It represents the quality of the detection result. While the eye is closed the assigned confidence is very low. The Blink Detection plugin makes use of this fact by defining a blink onset as a significant confidence drop - or a blink offset as a significant confidence gain - within a short period of time. The plugin creates a blink event for each event loop iteration in the following format:

{  # blink datum
	'topic': 'blink',
    'confidence': <float>,  # blink confidence
    'timestamp': <timestamp float>,
    'base_data': [<pupil positions>, ...]
    'type': 'onset' or 'offset'}

The Filter length is the time window’s length in which the plugin tries to find such confidence drops and gains. The plugin fires the above events if the blink confidence within the current time window exceeds the onset or offset confidence threshold. Setting both thresholds to 0 will always trigger blink events, even if the confidence is very low. This means that onsets and offsets do not appear necessarily as pairs but in waves.

Audio Capture

The Audio Capture plugin provides access to a selected audio source for other plugins and writes its output to the audio.mp4 file during a recording. It also writes the Pupil Capture timestamp for each audio packet to the audio_timestamps.npy file. This way you can easily correlate single audio packets to their corresponding video frames.

Audio is recorded separately from the video in Pupil Capture. You can play back audio in sync with video in Pupil Player. Audio is automatically merged with the video when you export a video using Pupil Player.


The Annotation Capture plugin allows you to mark timestamps with a label – sometimes referred to as triggers. These labels can be created by pressing their respective hotkey or by sending a notification with the subject annotation. This is useful to mark external events (e.g. “start of condition A”) within the Pupil recording. The Annotation Player plugin is able to correlate and export these events as well as add new ones.

Remote Annotations

You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation notification.

{'subject':"annotation",'label':"Hi this is my annotation 1",'timestamp':[set a correct timestamp as float here],'duration':1.0,'source':'a test script','record':True}

Camera Intrinsics Estimation

This plugin is used to calculate camera intrinsics, which will enable one to correct camera distortion. Pupil Capture has built in, default camera intrinsics models for the high speed world camera and the high resolution world camera. You can re-calibrate your camera and/or calibrate a camera that is not supplied by Pupil Labs by running this calibration routine. We support two different distortion models, radial distortion and fisheye distortion. For cameras with a FOV of 100 degrees or greater (like e.g. the high speed world camera) the fisheye distortion model usually performs better, for cameras with a smaller FOV (e.g. the high resolution world camera) we recommend the radial distortion model.

  1. Select Camera Intrinsics Estimation
  2. Select the correct ‘Distortion Model’
  3. Click on ‘show pattern’ to display the pattern
  4. Resize the pattern to fill the screen
  5. Hold your Pupil headset and aim it at the pattern.
  6. With the world window in focus, press c on your keyboard or the circular C button in the world windows to detect and capture a pattern.
  7. Data will be sampled and displayed on the screen as a border of the calibrated pattern. (Note - make sure to move your headset at different angles and try to cover the entire FOV of the world camera for best possible calibration results)
  8. Repeat until you have captured 10 patterns.
  9. Click on show undistorted image to display the results of camera intrinsic estimation. This will display an undistorted view of your scene. If well calibrated, straight lines in the real world will appear as straight lines in the undistorted view.

Pupil Player

Pupil Player is the second tool you will use after Pupil Capture. It is a media and data visualizer at its core. You will use it to look at Pupil Capture recordings. Visualize your data and export it.

Features like surface tracking found in Pupil Capture are also available in Pupil Player.

Starting Pupil Player

Starting Pupil Player Starting Pupil Player

Drag the recording directory (the triple digit one) directly onto the app icon or launch the application and drag + drop the recording directory into Pupil Player window.

Player Window

Let’s get familiar with the Player window.

Pupil Player UI call-out Pupil Player UI call-out

The Player window is the main control center for Pupil Player. It displays the recorded video feed from pupil capture file.

  1. Graphs - This area contains performance graphs. By default the graphs CPU, FPS, and pupil algorithm detection confidence will be displayed. You can control graph settings with the System Graphs plugin.
  2. Hot keys - This area contains clickable buttons for plugins.
  3. Timeline Events - Plugins can add temporal events to this expandable panel.
  4. Timeline - Control the playback of the video with the play/pause button (or spacebar on your keyboard). Drag the playhead (vertical line) to the desired point in time.
    • Frame Stepping - You can use the arrow keys on your keyboard or the << >> buttons to advance one frame at a time.
    • Trimming - Drag either end of the timeline to set a trim beginning and ending trim marks. The trim section marks directly inform the section of video/data to export.
  5. Menu - This area contains settings and contextual information for each plugin.
  6. Sidebar - This area contains clickable buttons for each plugin. System plugins are loaded in the top and user added plugins are added below the horizontal separator.


Pupil Player is similar to a video player. You can playback recordings and can load plugins to build visualizations.

Here is an example workflow:

  • Start Pupil Player
  • Open a Plugin - From the Plugin Manager GUI menu load the Vis Circle plugin.
  • Playback - press the play button or space bar on your keyboard to view the video playback with visualization overlay, or drag the playhead in the seek bar to scrub through the dataset.
  • Set trim marks - you can drag the green rounded rectangle at the beginning and end of the seekbar to set the trim marks. This will set the start and end frame for the exporter and for other plugins.
  • Export Video & Raw Data - From the Plugin Manager view, load the Video Export Launcher plugin and the Raw Data Exporter plugin. Press e on your keyboard or the e button in the left hand side of the window to start the export.
  • Check out exported data in the exports directory within your recording directory

Plugin Overview

Pupil Player uses the same Plugin framework found in Pupil Capture to add functionality.

We implement all visualizations, marker tracking, and the exporter using this structure. Very little work (often no work) needs to be done to make a Capture Plugin work for the Pupil Player and vice versa.

There are two general types of plugins:

  • Unique - You can only launch one instance of this plugin.
  • Not unique - You can launch multiple instances of this type of plugin. For example, you can load one Vis Circle plugin to render the gaze position with a translucent green circle, and another Vis Circle plugin to render the gaze circle with a green stroke of 3 pixel thickness. You can think of these types of plugins as additive.

In the following sections we provide a summary of plugins currently available and in Pupil Player.

Visualization Plugins

We will call plugins with the Vis prefix visualization plugins. These plugins are simple plugins, are mostly additive (or not unique), and directly operate on the gaze positions to produce visualizations. Other plugins like Offline Surface Tracker also produces visualizations, but will be discussed elsewhere due to the extent of its features.

Vis Circle

Vis Circle plugin Vis Circle plugin

Visualize the gaze positions with a circle for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization.

You can set the following parameters:

  • radius - the radius of the circle around the gaze point.
  • stroke width - the thickness or width of the stoke in pixels.
  • fill - toggle on for a circle with solid fill. Toggle off for a circle with only stroke.
  • color - define the red, green, blue values for color. Alpha defines the opacity of the stroke and fill.

Here we show an example of how you could use 2 instances of the Vis Circle Plugin. The first instance renders the gaze position as a filled yellow circle. The second instance renders the same gaze position as an orange stroke circle.

Vis Cross

Vis Cross plugin Vis Cross plugin

Visualize the gaze positions with a cross for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:

  • inner offset length - the distance in pixels to offset the interior cross endpoints from the gaze position. A value of 0 will make the crosshairs intersect the gaze position.
  • outer length - The length of the cross lines in pixels from the gaze position. Note - equal values of inner offset length and outer length will result in a cross with no length, and therefore not rendered.
  • stroke width - the thickness or width of the stoke in pixels.
  • color - define the red, green, blue values for color.

Here we show an example of how you could use 2 instances of the Vis Cross Plugin. The first instance renders the gaze position as a red cross with that extends to the boundaries of the screen. The second instance renders the gaze position as a green cross, with a heavier stroke weight.

Vis Scan Path

Vis Scan Path plugin Vis Scan Path plugin

This plugin enables past gaze positions to stay visible for the duration of time specified by the user. This plugin is unique, therefore you can only load one instance of this plugin.

On its own, Scan Path does not render anything to the screen. It is designed to be used with other plugins. In some cases, it is even required to be enabled in order for other plugins to properly function. When used with Vis plugins (like Vis Circle, Vis Cross, Vis Polyline, or Vis Light Points) Scan Path will enable you to see both the current gaze positions and the past gaze positions for the specified duration of time.

Here we show an example of Scan Path set with 0.4 seconds duration used with Vis Circle. Each green circle is a gaze position within the last 0.4 seconds of the recording.

Vis Polyline

Vis Polyline plugin Vis Polyline plugin

Visualize the gaze positions with a polyline for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:

  • line thickness - the thickness or width of the polyline stroke in pixels.
  • color - define the red, green, blue values for color.

An example showing Vis Polyline used with Vis Circle and Scan Path. The polyline enables one to visualize the sequence of the gaze positions over the duration specified by Scan Path.

Vis Light Points

Vis Light Points plugin Vis Light Points plugin

Visualize the gaze positions as a point of light for each gaze position. The falloff of the light from the gaze position is specified by the user. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:

  • falloff - The distance (in pixels) at which the light begins to fall off (fade to black). A very low number will result in a very dark visualization with tiny white light points. A very large number will result in a visualization of the world view with little or no emphasis on the gaze positions.

Here is an example demonstrating Vis Light Points with a falloff of 73.

Vis Eye Video Overlay

Vis Eye Video Overlay plugin Vis Eye Video Overlay plugin

Here is an example of the Eye Video Overlay with binocular eye videos.

This plugin can be used to overlay the eye video on top of the world video. Note that the eye video is not recorded by default in Pupil Capture, so if you want to use this plugin, make sure to check record eye video in Pupil Capture. This plugin is unique, therefore you can only load one instance of this plugin.

You can set the following parameters:

  • opacity - the opacity of the overlay eye video image. 1.0 is opaque and 0.0 is transparent.
  • video scale - use the slider to increase or decrease the size of the eye videos.
  • move overlay - toggle on and then click and drag eye video to move around in the player window. Toggle off when done moving the video frames.
  • show - show or hide eye video overlays.
  • horiz. and vert. flip - flip eye videos vertically or horizontally

Pupil Data And Post-hoc Detection

Offline (post-hoc) Pupil Detection and Gaze Mapping

Offline (post-hoc) Gaze Mapping With Manual Reference Locations

Use Offline (post-hoc) Calibration For Another Recording

Offline (post-hoc) Gaze Mapping Validation

By default, Player starts with the Pupil From Recording plugin that tries to load pupil positions that were detected and stored during a Pupil Capture recording. Alternatively, one can run the pupil detection post-hoc.

Offline Pupil Detector

The Offline Pupil Detector plugin can be used with any dataset where eye videos were recorded. The plugin tries to load the eye videos, and runs the pupil detection algorithm in separate processes. This plugin is especially relevant for recordings made with Pupil Mobile, because Pupil Mobile does not perform any pupil detection or gaze estimation on the Android device. This plugin is available starting with Pupil Player v0.9.13.

The Detection Method selector sets the detection algorithm to either 2d or 3d detection (see the section on Pupil Detection for details). The Redetect button restarts the detection procedure. You can use the Offline Pupil Detector plugin to debug, improve, and gain insight into the pupil detection process.

Gaze Data And Post-hoc Calibration

By default, Player starts with the Gaze From Recording plugin that tries to load gaze positions that were detected and stored during a Pupil Capture recording. Alternatively, one can run the gaze mapping process post-hoc.

Offline Calibration

The Offline Calibration plugin enables you to calibrate, map, and validate gaze post-hoc and is available starting with Pupil Player v0.9.13. It can be used on any Pupil dataset.

The workflow is separated into three steps, each with its own submenu: Reference Locations, Calibrations, and Gaze Mappers.

Reference locations are points within the recorded world video that are known to have been fixated on by the participant/subject. They can either be automatically detected or manually annotated:

  1. Detect Circle Markers in Recording: This button starts the automatic detection of circular calibration markers within the world video. The progress is visualized in the plugin’s timeline.
  2. Manual Edit Mode: When this option is enabled, you can add new locations as well as correct and delete existing ones. There can only be one location per world frame.

As in Capture, one can have more than one calibration per recording. A calibration on its own does not result in gaze data. Rather, it contains the required parameters to map pupil to gaze data. Each has the following properties:

  • Name: Used to correctly select a calibration for each gaze mapper (see below).
  • Mapping Method: 2d uses polynomial regression, or 3d uses bundle adjustment calibration.
  • Reference Range: Time range that indicates which reference locations to use.

Calibrations are stored as plcal files in the recording’s calibration subfolder. You can copy and apply them to recordings that do not include reference locations. See the instructional video below for details.

In order to apply a calibration to pupil data, one needs a gaze mapper. Each gaze mapper has the following properties:

  • Calibration: One of the previously created or imported calibrations (see screencast)
  • Mapping Range: Time range in which pupil data will be mapped to gaze data.
  • Manual Correction: Apply a fixed offset to your gaze mapping.
  • Validation: You can validate the accuracy and precision of the mapped gaze by comparing it to reference locations in the selected Validation Range. It uses the same methodology as the Accuracy Visualizer.

Analysis Plugins

These plugins are simple unique plugins, that operate on the gaze data for analysis and visualizations.

Offline Surface Tracker

Offline Surface Tracker Offline Surface Tracker

This plugin is an offline version of the Surface Tracking plugin for Pupil Capture. You can use this plugin to detect markers in the recording, define surfaces, edit surfaces, and create and export visualizations of gaze data within the defined surfaces.

Here is an example workflow for using the Offline Surface Detector plugin to generate heatmap visualizations and export surface data reports:

  • Load Offline Surface Detector plugin - if you already have surfaces defined, the load may take a few seconds because the plugin will look through the entire video and cache the detected surfaces.
  • Add surface - if you do not have any defined surfaces, you can click on the Add surface button when the markers you want to user are visible or just click the circular A button in the left hand side of the screen.
  • Surface name and size - In the Marker Detector GUI window, define the surface name and real world size. Note - defining size is important as it will affect how heatmaps are rendered.
  • Set trim marks - optional, but if you want to export data for a specific range, then you should set the trim marks.
  • Recalculate gaze distributions - click the (Re)calculate gaze distributions button after specifying surface sizes. You should now see heatmaps in the Player window (if gaze positions were within your defined surfaces).
  • Export gaze and surface data - click e and all surface metrics reports will be exported and saved for your trim section within your export folder.

All files generated by the Offline Surface Detector will be located in the subfolder surfaces. The different reported metrics are:

  • surface_visibility.csv - Overview of how many world camera frames each surface was contained in.
  • surface_gaze_distribution.csv - Overview of how many gaze samples have been collected on each individual surface and outside of surfaces.
  • surface_events.csv - List of image-enter and image-exit events for all surfaces.

Further the following metrics are reported for every individual surface. Each surface has a name, which can be manually set as described above. This name is augmented by an automatically generated numerical identifier.

  • heatmap_<surface_name>.png - Heatmap of gaze positions on the surface aggregated over the entire export.
  • gaze_positions_on_surface_<surface_name>.csv - A list of gaze datums on the surface. The values include the gaze point in two different coordinates systems. x_norm and y_norm are coordinates between 0 and 1, where (0,0) is the bottom left corner of the surface and (1,1) is the top right corner. x_scaled and y_scaled contain the same coordinates but scaled with the size defined for the surface.
  • surf_positions_<surface_name> - List of surface positions in 3D. The position is given as the 3D pose of the surface in relation to the current position of the scene camera. img_to_surf_trans is a matrix transforming coordinates from the camera coordinate system to the surface coordinate system. surf_to_img_trans is the inverse of img_to_surf_trans.

Fixation Detector

Offline Fixation Detector Offline Fixation Detector

The offline fixation detector calculates fixations for the whole recording. The menu gives feedback about the progress of the detection, how many fixations were found, shows and detailed information about the current fixation. Press f or click the f hot key button on the left hand side of the window to seek forward to the next fixation.

Toggle Show fixations to show a visualization of fixations. The blue number is the number of the fixation (0 being the first fixation). You can export fixation reports for your current trim section by pressing e on your keyboard or the e hot key button on the left hand side of the window.

You can find more information in our dedicated fixation detector section.

Head Pose Tracking

Head Pose Tracker Tutorial

This plugin uses fiducial markers (apriltag) to build a 3d model of the environment and track the headset’s pose within it. See the surface tracking section for images of the markers to download.

See the detailed data format section for more information about the exported data.


You can export data and videos by pressing e on your keyboard or the e hot key button in the Pupil Player window.

All open plugins that have export capability will export when you press e. Exports are separated from your raw data and contained in the exports sub-directory. The exports directory lives within your recording directory.

Active video exporters will run in the background and you can see the progress bar of the export in the GUI. While exporting, you can continue working with Pupil Player and even launch new exports. Each video export creates at least one mp4 and its respective file timestamp file. See the Data Format section for details.

Export Directory

Recording folder Recording folder

Every export creates a new folder within the exports sub-directory of your recording. All data from the export is saved to this folder.

Export Handling

You can select the frame range to export by setting trim marks in the seek bar or directly in the General Settings menu.

Longer running exports, e.g. video exports, go through three phases: Queued, Running, and Completed. Export tasks can be cancelled while being queued or running. Completed tasks are kept in the list for reference.

World Video Exporter

Video Export Launcher plugin Video Export Launcher plugin

The World Video Exporter is loaded by default.

The export saves the world video as shown in Player, including all currently active visualizations (see the [Visualization Plugins](#visualization-plugins) section).

Eye Video Exporter

The Eye Video Exporter needs to be loaded explicitly through the Plugin Manager. It includes the option to render the 2d pupil detection result into the exported video.

iMotions Exporter

The iMotions Exporter creates data that can be used with

Specifically, it undistorts the world video images using the camera intrinsics. Gaze data is also undistorted and exported to the gaze.tlv file.

Raw Data Exporter

Raw Data Exporter plugin Raw Data Exporter plugin

The Raw Data Exporter export pupil and gaze data tp .csv files and is active bu default.

Developing your own Plugin

To develop your own plugin see the developer guide.

Pupil Service

Pupil Service is like Pupil Capture except it does not have a world video feed or GUI. It is intended to be used with VR and AR eye tracking setups.

Pupil Service is designed to run in the background and to be controlled via network commands only. The service process has no GUI. The tools introduced in the hmd-eyes project are made to work with Pupil Service and Pupil Capture alike.

Talking to Pupil Service

Code examples below demonstrate how to control Pupil Service over the network.

Starting and stopping Pupil Service:
import zmq, msgpack, time
ctx = zmq.Context()

#create a zmq REQ socket to talk to Pupil Service/Capture
req = ctx.socket(zmq.REQ)

#convenience functions
def send_recv_notification(n):
    # REQ REP requirese lock step communication with multipart msg (topic,msgpack_encoded dict)
    req.send_multipart(('notify.%s'%n['subject'], msgpack.dumps(n)))
    return req.recv()

def get_pupil_timestamp():
    req.send('t') #see Pupil Remote Plugin for details
    return float(req.recv())

# set start eye windows
n = {'subject':'eye_process.should_start.0','eye_id':0, 'args':{}}
n = {'subject':'eye_process.should_start.1','eye_id':1, 'args':{}}

# set calibration method to hmd calibration
n = {'subject':'start_plugin','name':'HMD_Calibration', 'args':{}}

# set calibration method to hmd calibration
n = {'subject':'service_process.should_stop'}


The code demonstrates how you can listen to all notification from Pupil Service. This requires a little helper script called

from zmq_tools import *

ctx = zmq.Context()
requester = ctx.socket(zmq.REQ)
requester.connect('tcp://localhost:50020') #change ip if using remote machine

ipc_sub_port = requester.recv()
monitor = Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',)) #change ip if using remote machine

while True:


An example client for Unity3d can be found here

Pupil Mobile

Pupil Mobile is a companion app to Pupil Capture and Pupil Service. It is currently in public beta.

Introducing Pupil Mobile

Pupil Mobile enables you to connect your Pupil eye tracking headset to your Android device via USBC. You can preview video and other sensor streams on the Android device and stream video data over a WiFi network to other computers (clients) running Pupil Capture. Seamlessly integrated with Pupil Capture and Pupil Service.

Home Screen

Pupil Mobile home call-out Pupil Mobile home call-out

Home screen is the main control center for Pupil Mobile. It displays a list of available sensors. Click any sensor for a preview.

  1. Sensors - This area contains available sensors. Pupil headset cameras along with other sensors connected to or built into the Android device like audio and IMU.
  2. Record - Click the Record button to save video and other sensor data locally on the Android device.
  3. General Settings - Main settings menu for Pupil Mobile app.

Sensor Preview

Pupil Mobile sensor call-out Pupil Mobile sensor call-out

Preview live video feed from your Pupil headset and other available sensors. Sensor preview windows will automatically close and take you back to the Home screen in order to conserve battery.

  1. Sensor settings - Settings for the sensor. For cameras you can set frame rate, exposure, white balance, and many more parameters.
  2. Sensor name and recording status - This displays the sensor name and a dot displaying the recording status of this sensor.
  3. Preview stream - A preview of sensor data.

General Settings

Pupil Mobile settings call-out Pupil Mobile settings call-out

Main settings menu for Pupil Mobile app and information about the Android Device.

  1. Device name - Text input field to name your device. This is the device name that will appear in Pupil Capture.
  2. Close/Quit app - Press this button to close the app. Pupil Mobile runs a service in the background. This enables the app to continue running even when your screen is off. Therefore, just swiping away the app view will not close the app.
  3. Save directory - Select the location where recordings should be saved. By default recordings are saved on Android’s built in storage. You can also save to an SD card, if available.

Recordings Screen

Pupil Mobile recording call-out Pupil Mobile recording call-out

View all the datasets that were recorded on your Android device.

  1. Recording folder - A directory containing all of your recordings.
  2. Delete - Permanently delete recording files from the device.

See the Transfer Recordings section below on how to transfer recordings from your phone to your computer.

Switch Views

Pupil Mobile swipe call-out Pupil Mobile swipe call-out

Pupil Mobile is designed to for efficient navigation. Swipe left or right for quick access to other views.

  1. Swipe - Swipe left or right to switch between views. Swipe right from the home screen to go to the recording view. Swipe left from the home screen to the sensor preview views.

Streaming To Subscribers

Pupil Mobile devices on the same local WiFi network are automatically detected by Pupil Capture. To subscribe to a Pupil Mobile device in Pupil Capture, go to Capture Selection and select Pupil Mobile as the capture source.

WiFi Bandwidth & Network

Make sure you have a good WiFi newtork connection and that it’s not saturated. The quality of your WiFi connection will affect the sensor streams to your subscribers.

NDSI Communication Protocol

The communication protocol is named NDSI, it is completely open. A reference client for Python exsits here.

Transfer Recordings

You will have to manually transfer recordings to your computer to open them in Player. There are two different methods to archive that:

SD card File Transfer

This method requires

  1. that you set the Save directory to SD card in the main settings
  2. a SD card reader that can be connected to your computer

Once you finished the recording, remove the sd card from your phone, insert it into the SD card reader, and connect it to your computer. The SD card should show up on your computer as if it was a USB stick.

You can find all recodings in the Pupil Mobile folder on the SD card. Copy the recording folders to a directory of your choice on your computer. Afterwards, you can remove the SD card from the computer and open the copied recordings using Pupil Player.

Direct Phone Connection

  1. Connect your phone via USB to your computer.
  2. A notification on your phone should pop up: Android System - USB charging this device.
    1. Double tap the notification to open the options.
    2. Select Use USB to transfer files.
    3. Your phone should show up on your computer as if it was an USB stick.
  3. The recordings are saved in two different locations depending on the Save directory settings chosen before the recording:
    1. Default: Internal storage/Movies/Pupil Mobile
    2. SD Card: SD card/Pupil Mobile
  4. Copy the recording folders to a directory of your choice on your computer.
  5. Disconnect the phone from your computer.
  6. Open the copied recordings in Pupil Player.

Download App

The app is free. You can download it in the Google Play Store.

Supported Hardware

  • MotoZ2 Play
  • Google Nexus 6p, Nexus 5x
  • OnePlus 3, 3T, 5, 5T
  • potentially other USB-C phones (untested)

Bugs & Features

I found a bug or need a feature!

Please check out existing issues or open a new issue at Pupil Mobile repository. This app is in Alpha state, help us make it better.


I want to use this for my experiments in the field

Feel free to do so, but do not rely on the app to work all the time! Many features and environments are still untested. If you have trouble, please open an issue.

Data Format

The data format for Pupil recordings is 100% open. In this section we will first describe how we handle high level concepts like coordinate systems, timestamps and synchronization and then describe in detail the data format present in Pupil recordings.

Coordinate Systems

We use a normalized coordinate system with the origin 0,0 at the bottom left and 1,1 at the top right.

  • Normalized Space

Origin 0,0 at the bottom left and 1,1 at the top right. This is the OpenGL convention and what we find to be an intuitive representation. This is the coordinate system we use most in Pupil. Vectors in this coordinate system are specified by a norm prefix or suffix in their variable name.

  • Image Coordinate System

In some rare cases we use the image coordinate system. This is mainly for pixel access of the image arrays. Here a unit is one pixel, origin is “top left” and “bottom right” is the maximum x,y.

  • Camera Coordinate System

Some of the raw data (such as the estimate of the 3D gaze point) is specified in the three-dimensional world camera coordinate system. The origin of this coordinate system is in the projection center located behind the midpoint of the 2D image plane. The z-axis points forward along the optical axis while the x-axis points to the right and the y-axis downwards.


All indexed data - still frames from the world camera, still frames from the eye camera(s), gaze coordinate, and pupil coordinates, etc. - have timestamps associated for synchronization purposes. The timestamp is derived from CLOCK_MONOTONIC on Linux and MacOS.

The time at which the clock starts counting is called PUPIL EPOCH. In pupil the epoch is adjustable through Pupil Remote and Pupil Timesync.

Timestamps are recorded for each sensor separately. Eye and World cameras may be capturing at very different rates (e.g. 120hz eye camera and 30hz world camera), and correlation of eye and world (and other sensors) can be done after the fact by using the timestamps. For more information on this see Synchronization below.


  • Timestamps in seconds since PUPIL EPOCH.
  • PUPIL EPOCH is usually the time since last boot.
  • In UNIX like, PUPIL EPOCH is usually not the Unix Epoch (00:00:00 UTC on 1 January 1970).

More information:

  • Unit : Seconds
  • Precision: Full float64 precision with 15 significant digits, i.e. 10 μs.
  • Accuracy:
    • If WIFI, it is ~1 ms
    • If wired or ‘localhost’, it is in the range of μs.
  • Granularity:
    • It is machine specific (depends on clock_monotonic on Linux). It is constrained by the processor cycles and software.
    • In some machines (2 GHz processor), the result comes from clock_gettime(CLOCK_MONOTONIC, &time_record) function on Linux. This function delivers a record with nanosecond, 1 GHz, granularity. Then, PUPIL software does some math and delivers a float64.
  • Maximum Sampling Rate:
    • Depends on set-up, and it is lower when more cameras are present. (120Hz maximum based on a 5.7ms latency for the cameras and a 3.0ms processing latency.


Pupil Capture software runs multiple processes. The world video feed and the eye video feeds run and record at the frame rates set by their capture devices (cameras). This allows us to be more flexible. Instead of locking everything into one frame rate, we can capture every feed at specifically set rates. But, this also means that we sometimes record world video frames with multiple gaze positions (higher eye-frame rate) or without any (no pupil detected or lower eye frame rate).

In you can find a function that takes timestamped data and correlates it with timestamps form a different source.

def correlate_data(data,timestamps):
    data:  list of data :
        each datum is a dict with at least:
            timestamp: float

    timestamps: timestamps list to correlate  data to

    this takes a data list and a timestamps list and makes a new list
    with the length of the number of timestamps.
    Each slot contains a list that will have 0, 1 or more associated data points.

    Finally we add an index field to the datum with the associated index
    timestamps = list(timestamps)
    data_by_frame = [[] for i in timestamps]

    frame_idx = 0
    data_index = 0

    data.sort(key=lambda d: d['timestamp'])

    while True:
            datum = data[data_index]
            # we can take the midpoint between two frames in time: More appropriate for SW timestamps
            ts = ( timestamps[frame_idx]+timestamps[frame_idx+1] ) / 2.
            # or the time of the next frame: More appropriate for Sart Of Exposure Timestamps (HW timestamps).
            # ts = timestamps[frame_idx+1]
        except IndexError:
            # we might loose a data point at the end but we don't care

        if datum['timestamp'] <= ts:
            datum['index'] = frame_idx
            data_index +=1

    return data_by_frame

Detailed Data Format

Every time you click record in Pupil Capture or Pupil Mobile, a new recording is started and your data is saved into a recording folder. You can use Pupil Player to playback Pupil recordings, add visualizations, and export in various formats.

Access to raw data

Note that the raw data before processing with Pupil Player is not immediately readible from other software (the raw data format is documented in the developer docs). Use the ‘Raw Data Exporter’ plugin in Pupil Player to export .csv files that contain all the data captured with Pupil Capture. Exported files will be written to a subfolder of the recording folder called exports.

The following files will be created by default in an export:

  • export_info.csv - Meta information on the export containing e.g. the export date or the dta format version.
  • pupil_positions.csv - A list of all pupil datums. See below for more infotmation.
  • gaze_positions.csv - A list of all gaze datums. See below for more infotmation.
  • pupil_gaze_positions_info.txt - Contains documentation on the contents of pupil_positions.csv and gaze_positions.csv
  • world_viz.mp4 - The exported section of world camera video.

If you are using additional plugins in Pupil Player, these might create other files. Please check the documentation of the respective plugin for the used data format.


This file contains all exported pupil datums. Each datum will have at least the following keys:

  • timestamp - timestamp of the source image frame
  • index - associated_frame: closest world video frame
  • id - 0 or 1 for left/right eye
  • confidence - is an assessment by the pupil detector on how sure we can be on this measurement. A value of 0 indicates no confidence. 1 indicates perfect confidence. In our experience useful data carries a confidence value greater than ~0.6. A confidence of exactly 0 means that we don’t know anything. So you should ignore the position data.
  • norm_pos_x - x position in the eye image frame in normalized coordinates
  • norm_pos_y - y position in the eye image frame in normalized coordinates
  • diameter - diameter of the pupil in image pixels as observed in the eye image frame (is not corrected for perspective)
  • method - string that indicates what detector was used to detect the pupil

Depending on the used gaze mapping mode, each datum may have additional keys. When using the 2D gaze mapping mode the following keys will be added:

  • 2d_ellipse_center_x - x center of the pupil in image pixels
  • 2d_ellipse_center_y - y center of the pupil in image pixels
  • 2d_ellipse_axis_a - first axis of the pupil ellipse in pixels
  • 2d_ellipse_axis_b - second axis of the pupil ellipse in pixels
  • 2d_ellipse_angle - angle of the ellipse in degrees

When using the 3D gaze mapping mode the following keys will additionally be added:

  • diameter_3d - diameter of the pupil scaled to mm based on anthropomorphic avg eye ball diameter and corrected for perspective.
  • model_confidence - confidence of the current eye model (0-1)
  • model_id - id of the current eye model. When a slippage is detected the model is replaced and the id changes.
  • sphere_center_x - x pos of the eyeball sphere is eye pinhole camera 3d space units are scaled to mm.
  • sphere_center_y - y pos of the eye ball sphere
  • sphere_center_z - z pos of the eye ball sphere
  • sphere_radius - radius of the eyeball. This is always 12mm (the anthropomorphic avg.) We need to make this assumption because of the single camera scale ambiguity.
  • circle_3d_center_x - x center of the pupil as 3d circle in eye pinhole camera 3d space units are mm.
  • circle_3d_center_y - y center of the pupil as 3d circle
  • circle_3d_center_z - z center of the pupil as 3d circle
  • circle_3d_normal_x - x normal of the pupil as 3d circle. Indicates the direction that the pupil points at in 3d space.
  • circle_3d_normal_y - y normal of the pupil as 3d circle
  • circle_3d_normal_z - z normal of the pupil as 3d circle
  • circle_3d_radius - radius of the pupil as 3d circle. Same as diameter_3d
  • theta - circle_3d_normal described in spherical coordinates
  • phi - circle_3d_normal described in spherical coordinates
  • projected_sphere_center_x - x center of the 3d sphere projected back onto the eye image frame. Units are in image pixels.
  • projected_sphere_center_y - y center of the 3d sphere projected back onto the eye image frame
  • projected_sphere_axis_a - first axis of the 3d sphere projection.
  • projected_sphere_axis_b - second axis of the 3d sphere projection.
  • projected_sphere_angle - angle of the 3d sphere projection. Units are degrees.


This file contains a list of all exported gaze datums. Each datum contains the following keys:

  • timestamp - timestamp of the source image frame
  • index - associated_frame: closest world video frame
  • confidence - computed confidence between 0 (not confident) -1 (confident)
  • norm_pos_x - x position in the world image frame in normalized coordinates
  • norm_pos_y - y position in the world image frame in normalized coordinates
  • base_data - “timestamp-id timestamp-id …” of pupil data that this gaze position is computed from

When using the 3D gaze mapping mode the following keys will additionally be available:

  • gaze_point_3d_x - x position of the 3d gaze point (the point the subject looks at) in the world camera coordinate system
  • gaze_point_3d_y - y position of the 3d gaze point
  • gaze_point_3d_z - z position of the 3d gaze point
  • eye_center0_3d_x - x center of eye-ball 0 in the world camera coordinate system (of camera 0 for binocular systems or any eye camera for monocular system)
  • eye_center0_3d_y - y center of eye-ball 0
  • eye_center0_3d_z - z center of eye-ball 0
  • gaze_normal0_x - x normal of the visual axis for eye 0 in the world camera coordinate system (of eye 0 for binocular systems or any eye for monocular system). The visual axis goes through the eye ball center and the object thats looked at.
  • gaze_normal0_y - y normal of the visual axis for eye 0
  • gaze_normal0_z - z normal of the visual axis for eye 0
  • eye_center1_3d_x - x center of eye-ball 1 in the world camera coordinate system (not available for monocular setups.)
  • eye_center1_3d_y - y center of eye-ball 1
  • eye_center1_3d_z - z center of eye-ball 1
  • gaze_normal1_x - x normal of the visual axis for eye 1 in the world camera coordinate system (not available for monocular setups.). The visual axis goes through the eye ball center and the object thats looked at.
  • gaze_normal1_y - y normal of the visual axis for eye 1
  • gaze_normal1_z - z normal of the visual axis for eye 1

World Video Stream

You can compress the videos afterwards using ffmpeg like so:

cd your_recording
ffmpeg -i world.mp4  -pix_fmt yuv420p  world_compressed.mp4; mv world_compressed.mp4 world.mp4 
ffmpeg -i eye0.mp4  -pix_fmt yuv420p  eye0_compressed.mp4; mv eye0_compressed.mp4 eye0.mp4
ffmpeg -i eye1.mp4  -pix_fmt yuv420p  eye1_compressed.mp4; mv eye1_compressed.mp4 eye1.mp4

OpenCV has a capture module that can be used to extract still frames from the video:

import cv2
capture = cv2.VideoCapture("absolute_path_to_video/world.mp4")
status, img1 = # extract the first frame
status, img2 = # second frame...

When using the setting more CPU smaller file: A mpeg4 compressed video stream of the world view will be created in an .mp4 container. The video is compressed using ffmpeg’s default settings. It gives a good balance between image quality and files size. The frame rate of this file is set to your capture frame rate.

When using the setting less CPU bigger file: A raw mjpeg stream from the world camera world view will be created in an .mp4 container. The video is compressed by the camera itself. While the file size is considerably larger than above, this will allow ultra low CPU while recording. It plays with recent version of ffmpeg and vlc player. The “frame rate” setting in the Pupil Capture sidebar (Camera Settings > Sensor Settings) controls the frame rate of the videos.

head_pose_tacker_model.csv and head_pose_tacker_poses.csv

  • head_pose_tacker_model.csv: A list of all markers used to generate the 3d model and the 3d locations of the marker vertices.
  • head_pose_tacker_poses.csv: The headset’s pose (rotation and translation) within the 3d model coordinate system for each recorded world frame.

By default, the location of the first marker occurance will be used as the origin of the 3d model’s coordinate system. In the plugin’s menu, you can change the marker that is being used as the origin.

Developer Docs

Development Overview

Overview of language, code structure, and general conventions


Pupil is written in Python 3, but no “heavy lifting” is done in Python. High performance computer vision, media compression, display libraries, and custom functions are written in external libraries or c/c++ and accessed though cython. Python plays the role of “glue” that sticks all the pieces together.

We also like writing code in Python because it’s quick and easy to move from initial idea to working proof-of-concept. If proof-of-concept code is slow, optimization and performance enhancement can happen in iterations of code.

Process Structure

When Pupil Capture starts, in default settings two processes are spawned:

Eye and World. Both processes grab image frames from a video capture stream but they have very different tasks.

Eye Process

The eye process only has one purpose - to detect the pupil and broadcast its position. The process breakdown looks like this:

  • Grabs eye camera images from eye camera video stream
  • Find the pupil position in the image
  • Broadcast/stream the detected pupil position.

World Process

This is the workhorse.

  • Grabs the world camera images from the world camera video stream
  • Receives pupil positions from the eye process
  • Performs calibration mapping from pupil positions to gaze positions
  • Loads plugins - to detect fixations, track surfaces, and more…
  • Records video and data. Most, and preferably all coordination and control happens within the World process.

Pupil Datum Format

The pupil detector, run by the Eye process are required to return a result in the form of a Python dictionary with at least the following content:

    result = {}
    result['timestamp'] = frame.timestamp
    result['norm_pos'] = (x,y) # pupil center in normalized coordinates
    result['confidence'] = # a value between 1 (very certain) and 0 (not certain, nothing found)
    result['whatever_else_you_want'] = # you can add other things to this dict

    # if no pupil was detected
    result = {}
    result['timestamp'] = frame.timestamp
    result['confidence'] = 0

This dictionary is sent on the IPC and read by gaze mapping plugins in the world process. Mapping from pupil position to gaze position happens here. The mapping plugin is initialized by a calibration plugin. The 3D pupil detector extends the 2D pupil datum with additional information. Below you can see the Python representation of a pupil and a gaze datum.

{  # pupil datum
    'topic': 'pupil',
    'method': '3d c++',
    'norm_pos': [0.5, 0.5],  # norm space, [0, 1]
    'diameter': 0.0,  # 2D image space, unit: pixel
    'timestamp': 535741.715303987,  # time, unit: seconds
    'confidence': 0.0,  # [0, 1]

    # 2D ellipse of the pupil in image coordinates
    'ellipse': {  # image space, unit: pixel
        'angle': 90.0,  # unit: degrees
        'center': [320.0, 240.0],
        'axes': [0.0, 0.0]},
    'id': 0,  # eye id, 0 or 1

    # 3D model data
    'model_birth_timestamp': -1.0,  # -1 means that the model is building up and has not finished fitting
    'model_confidence': 0.0,
    'model_id': 1

    # pupil polar coordinates on 3D eye model. The model assumes a fixed
    # eye ball size. Therefore there is no `radius` key
    'theta': 0,
    'phi': 0,

    # 3D pupil ellipse
    'circle_3d': {  # 3D space, unit: mm
        'normal': [0.0, -0.0, 0.0],
        'radius': 0.0,
        'center': [0.0, -0.0, 0.0]},
    'diameter_3d': 0.0,  # 3D space, unit: mm

    # 3D eye ball sphere
    'sphere': {  # 3D space, unit: mm
        'radius': 0.0,
        'center': [0.0, -0.0, 0.0]},
    'projected_sphere': {  # image space, unit: pixel
        'angle': 90.0,
        'center': [0, 0],
        'axes': [0, 0]}}

Gaza data is based on one (monocular) or two (binocular) pupil positions. The gaze mapper is automatically setup after calibration and maps pupil positions into world camera coordinate system. The pupil data on which the gaze datum is based on can be accessed using the base_data key.

 {  # gaze datum
    'topic': 'gaze',
    'confidence': 1.0,  # [0, 1]
    'norm_pos': [0.5238293689178297, 0.5811187961748036],  # norm space, [0, 1]
    'timestamp': 536522.568094512,  # time, unit: seconds

    # 3D space, unit: mm
    'gaze_normal_3d': [-0.03966349641933964, 0.007685562866422135, 0.9991835362811073],
    'eye_center_3d': [20.713998951917564, -22.466222119962115, 11.201474469783548],
    'gaze_point_3d': [0.8822507422478054, -18.62344068675104, 510.7932426103372],
    'base_data': [<pupil datum>]}  # list of pupil data that was used to calculate the gaze

Timing & Data Conventions

Pupil Capture is designed to work with multiple captures that free-run at different frame rates that may not be in sync. World and eye images are timestamped and any resulting artifacts (detected pupil, markers, etc) inherit the source timestamp. Any correlation of these data streams is the responsibility of the functional part that needs the data to be correlated (e.g. calibration, visualization, analyses).

For example: The pupil capture data format records the world video frames with their respective timestamps. Independent of this, the recorder also saves the detected gaze and pupil positions at their frame rate and with their timestamps. For more detail see Data Format.

Git Conventions

We make changes almost daily and sometimes features will be temporarily broken in some development branches. However, we try to keep the master branch as stable as possible and use other branches for feature development and experiments. Here’s a breakdown of conventions we try to follow.

  • tags - We make a tag following the semantic versioning protocol. Check out the releases.
  • master - this branch tries to be as stable as possible - incremental and tested features will be merged into the master. Check out the master branch.
  • branches - branches are named after features that are being developed. These branches are experimental and what could be called ‘bleeding edge’. This means features in these branches may not be fully functional, broken, or really cool… You’re certainly welcome to check them out and improve on the work!

Pull requests

If you’ve done something – even if work-in-progress – make a pull request and write a short update to the Pupil Community.

Developer Setup

Pages in the developer guide are oriented towards developers and will contain high level overview of code and organizational structure.

If you want to develop a plugin or to extend Pupil for your project, this is the place to start.

These pages will not contain detailed documentation of code. We’re working on code documentation, and when it’s done we will put code documentation online at read the docs.

If you have questions, encounter any problems, or want to share progress – chat with us on the Pupil channel on Discord. We will try our best to help you out, and answer questions quickly.

Running Pupil from Source

Pupil is a prototype and will continue to be in active development. If you plan to make changes to Pupil, want to see how it works, make a fork, install all dependencies and run Pupil source directly with Python.

Installing Dependencies

  • Linux step-by-step instructions for Ubuntu 16.04 LTS +
  • MacOS step-by-step instructions for MacOS 10.8+
  • Windows step-by-step instructions for Windows 10
  • Intel RealSense 3D instructions if you want to use an Intel RealSense R200 as scene camera.

Download and Run Pupil Source Code

Once you have all dependencies installed, you’re 99% done. Now, all you have to do fork the github repository. Or, using the terminal you can clone the Pupil repository using git:

cd /the_folder_where_Pupil_will_live/
git clone

Run Pupil Capture from Source

You’re in development land now. If you’re running from the source, there will be no icon to click. So fire up the terminal, navigate to the cloned Pupil repository, and start Pupil using Python.

cd /the_folder_where_Pupil_lives/pupil_src

Linux Dependencies

These installation instructions are tested using Ubuntu 16.04 or higher running on many machines. Do not run Pupil on a VM unless you know what you are doing. We recommend using 18.04 LTS.

Install Linux Dependencies

Let’s get started! Its time for apt! Just copy paste into the terminal and listen to your machine purr.

Ubuntu 18.04

Install dependencies with apt-get.

sudo apt install -y pkg-config git cmake build-essential nasm wget python3-setuptools libusb-1.0-0-dev  python3-dev python3-pip python3-numpy python3-scipy libglew-dev libglfw3-dev libtbb-dev

install ffmpeg >= 3.2

sudo apt install -y libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev ffmpeg x264 x265 libportaudio2 portaudio19-dev

install OpenCV >= 3.

sudo apt install -y python3-opencv libopencv-dev

Ubuntu 17.10 or lower

Pupil requires Python 3.6 or higher. Please check this resource on how to install Python 3.6 on your version of Ubuntu.

If you’re using Ubuntu <= 17.10, you will need to install OpenCV from source, and install ffmpeg-3 from a different ppa.

install ffmpeg3 from jonathonf’s ppa

sudo add-apt-repository ppa:jonathonf/ffmpeg-3
sudo apt-get update
sudo apt install -y libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev ffmpeg libav-tools x264 x265 libportaudio2 portaudio19-dev

install OpenCV from source

The requisites for opencv to build python3 library are:

  1. python3 interpreter found

  2. libpython***.so shared lib found (make sure to install python3-dev)

  3. numpy for python3 installed.

git clone
cd opencv
mkdir build
cd build
make -j2
sudo make install
sudo ldconfig


wget -O libjpeg-turbo.tar.gz
tar xvzf libjpeg-turbo.tar.gz
cd libjpeg-turbo-1.5.1
./configure --enable-static=no --prefix=/usr/local
sudo make install
sudo ldconfig

custom version of libusb Required for 17.10 and with 200hz cameras only. Otherwise IGNORE!)

  1. Build or download fixed binary from release:
  2. Replace system with this binary.

sudo cp '~/path to your fixed binary/' '/lib/x86_64-linux-gnu/'


git clone
cd libuvc
mkdir build
cd build
cmake ..
make && sudo make install

udev rules for running libuvc as normal user

echo 'SUBSYSTEM=="usb",  ENV{DEVTYPE}=="usb_device", GROUP="plugdev", MODE="0664"' | sudo tee /etc/udev/rules.d/10-libuvc.rules > /dev/null
sudo udevadm trigger

Install apriltag

git clone
cd apriltag
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4
sudo make install
sudo ldconfig

Install packages with pip

sudo pip3 install numexpr
sudo pip3 install cython
sudo pip3 install psutil
sudo pip3 install pyzmq
sudo pip3 install msgpack==0.5.6
sudo pip3 install pyopengl
sudo pip3 install pyaudio
sudo pip3 install cysignals
sudo pip3 install git+
sudo pip3 install git+
sudo pip3 install git+
sudo pip3 install git+
sudo pip3 install git+
sudo pip3 install git+
sudo pip3 install git+

3D eye model dependencies First install

sudo apt-get install -y libgoogle-glog-dev libatlas-base-dev libeigen3-dev


Ubuntu 18.04

sudo apt install -y libceres-dev

Next we need to install the Ceres library. In Ubuntu 18.04 Ceres is available as a package in the repositories. In older versions it has to be compiled from source. Choose the correct command on the right depending on your version of Ubuntu!

Ubuntu <= 17.10

# sudo apt-get install software-properties-common if add-apt-repository is not found
sudo add-apt-repository ppa:bzindovic/suitesparse-bugfix-1319687
sudo apt-get update
sudo apt-get install libsuitesparse-dev
# install ceres-solver
git clone
cd ceres-solver
mkdir build && cd build
make -j3
make test
sudo make install
sudo sh -c 'echo "/usr/local/lib" > /etc/'
sudo ldconfig

(Optional) Install PyTorch + CUDA and cuDNN.

Version 1: Without GPU acceleration: Install PyTorch via pip

pip3 install
pip3 install torchvision

Some bleeding edge features require the deep learning library PyTorch. Without GPU acceleration some of the features will probably not run in real-time.

Version 2: With GPU acceleration: Install PyTorch via pip

pip3 install torch torchvision

Please refer to the following links on how to install CUDA and cuDNN:

MacOS Dependencies

These instructions have been tested for MacOS 10.8, 10.9, 10.10, 10.11, 10.12, 10.13, and 10.14. Use the linked websites and Terminal to execute the instructions.

Install Apple Dev Tools

Trigger the install of the Command Line Tools (CLT) by typing this in your terminal and letting MacOS install the tools required:


Install Homebrew

Homebrew describes itself as “the missing package manager for OSX.” It makes development on MacOS much easier, plus it’s open source. Install with the ruby script.

ruby -e "$(curl -fsSL"

Install Homebrew Python >=3.6

brew install python3

Add Homebrew installed executables and Python scripts to your path. Add the following two lines to your ~/.bash_profile. (you can open textedit from the terminal like so: open ~/.bash_profile)

export PATH=/usr/local/bin:/usr/local/sbin:$PATH
export PYTHONPATH=/usr/local/lib/python3.6/site-packages:$PYTHONPATH

Dependencies with brew

Let’s get started! Its time to put brew to work! Just copy paste commands into your terminal and listen to your machine purr.

brew install pkg-config
brew install libjpeg-turbo
brew install libusb
brew install portaudio
# opencv will install ffmpeg, numpy, and opencv-contributions automatically
# tbb is included by default with
brew install opencv
brew install glew
brew install glfw3
# dependencies for 2d_3d c++ detector
brew install ceres-solver

Install libuvc

git clone --single-branch --branch build_fix_mac
cd libuvc
mkdir build
cd build
cmake ..
make && make install

Install apriltag

git clone
cd apriltag
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4
sudo make install

Python Packages with pip

PyOpenGL, ZMQ, …

pip3 install scipy
pip3 install PyOpenGL
pip3 install pyzmq
pip3 install numexpr
pip3 install cython
pip3 install psutil
pip3 install msgpack==0.5.6
pip3 install pyaudio
pip3 install cysignals
pip3 install torch torchvision
pip3 install git+
pip3 install git+
pip3 install git+
pip3 install git+
pip3 install git+
pip3 install git+
pip3 install git+

That’s it – you’re Done!

Windows Dependencies

System Requirements

We develop the Windows version of Pupil using 64 bit Windows 10.

Therefore we can only debug and support issues for Windows 10.

Notes Before Starting

  • Work directory - We will make a directory called work at C:\work and will use this directory for all build processes and setup scripts. Whenever we refer to the work directory, it will refer to C:\work. You can change this to whatever is convenient for you, but note that all instructions and setup files refer to C:\work, therefore you are highly encouraged to also put everything into C:\work or you will have to rewrite code! An alternative would be to create a symbolic link from C:\work to whatever work directory you have.
  • Command Prompt - We will always be using x64 Native Tools Command Prompt for VS 2017 as our command prompt. Make sure to only use this command prompt. Unlike unix systems, windows has many possible “terminals” or “cmd prompts”. We are targeting x64 systems and require the x64 command prompt. You can access this cmd prompt from the Visual Studio 2017 shortcut in your Start menu.
  • 64bit - You should be using a 64 bit system and therefore all downloads, builds, and libraries should be for x64 unless otherwise specified.
  • Windows paths and Python - path separators in windows are a forward slash \. In Python, this is a special “escape” character. When specifying Windows paths in a Python string you must use \\ instead of \ or use Python raw strings, e.g. r'\'.
  • Help - For discussion or questions on Windows head over to our #pupil Discord channel. If you run into trouble please raise an issue on github!

Install Visual Studio

Download Visual Studio 2017 Community version 15.8 from

  • Run the Visual Studio bootstrapper .exe.
  • Navigate to the Workloads tab
  • In the Workloads tab, choose Desktop Development with C++. This will install all runtimes and components we need for development. Here is a list of what you should see checked in the Desktop development with C++ in the Summary view:
    • VC++ 2017 v141 toolset (x86,x64)
    • C++ profiling tools
    • Windows 10 SDK (10.0.15063.0) for Desktop C++ x86 and x64
    • Visual C++ tools for CMAKE
    • Visual C++ ATL support
    • MFC and ATL support (x86, x64)
    • Standard Library Modules
    • VC++ 2015.3 v140 toolset for desktop (x86, x64)
  • Navigate to the Individual Components tab
  • In the Individual Components tab check Git. This will install git on your system. In the Summary Panel for Individual Components you should see:
    • Git for Windows
  • Click Install

Install 7-Zip

Install 7-zip to extract files.

Install Python

  • Download Python 3.6 x64
  • Run the Python installer.
  • Check the box Add Python to PATH. This will add Python to your System PATH Environment Variable.
  • Check the box Install for all users. This will install Python to C:\Program Files\Python36.

System Environment Variables

You will need to check to see that Python was added to your system PATH variables. You will also need to manually add other entries to the system PATH later in the setup process.

To access your System Environment Variables:

  • Right click on the Windows icon in the system tray.
  • Select System.
  • Click on Advanced system settings.
  • Click on Environment Variables....
  • You can click on Path in System Variables to view the variables that have been set.
  • You can Edit or Add new paths (this is needed later in the setup process).

Python Libs

Open your command prompt and install the following libs:

pip install numpy
pip install scipy
pip install cython
pip install opencv-python==3.*
pip install pyopengl
pip install psutil
pip install pyaudio
pip install pyzmq
pip install msgpack==0.5.6
pip install win_inet_pton
pip install git+
pip install git+
pip install git+

Now install pytorch. - Open the pytorch website for local installation: - Select options: Stable, Windows, Pip, Python 3.6, CUDA 9.0. - You will be provided with two commands. Run them in the order given to install the wheels.

Pupil Labs Python Wheels

Download the following Python wheels from Pupil Labs github repos.

pyuvc requires that you download Microsoft Visual C++ 2010 Redistributable from microsoft. The pthreadVC2 lib, which is used by libuvc, depends on msvcr100.dll.

Open your command prompt and Run as administrator in the directory where the wheels are downloaded.

  • Install all wheels with pip install X (where X is the name of the .whl file)
  • You can check that libs are installed with python import X statements in the command prompt where X is the name of the lib.

Ceres for Windows

Navigate to your work directory

  • git clone --recursive
  • Download Eigen 3.3.3
  • Unzip Eigen and rename the extracted eigen-eigen-67e894c6cd8f directory to Eigen
  • Copy the Eigen directory into ceres-windows
  • Copy C:\work\ceres-windows\ceres-solver\config\ceres\internal\config.h to C:\work\ceres-windows\ceres-solver\include\ceres\internal
  • Open ceres-2015.sln and with Visual Studio 2017 and agree to update to 2017.
  • Set configurations to Release and x64
  • Right click on libglog_static and Build
  • Right click on ceres_static and Build

Clone the Pupil Repo

  • Open a command prompt in your work dir
  • git clone

Setup pupil_external dependencies

The following steps require you to store dynamic libraries in the pupil_external folder of the cloned repository so that you do not have to add further modifications to your system PATH.

GLEW to pupil_external

  • Download GLEW Windows binaries from sourceforge
  • Unzip GLEW in your work dir
  • Copy glew32.dll to pupil_external

GLFW to pupil_external

  • Download GLFW Windows binaries from
  • Unzip GLFW to your work dir
  • Copy glfw3.dll from lib-vc2015 to pupil_external

FFMPEG to pupil_external

  • Download FFMPEG v4.0 Windows shared binaries from ffmpeg
  • Unzip ffmpeg-shared to your work dir
  • Copy the following 8 .dll files to pupil_external
    • avcodec-58.dll
    • avdevice-58.dll
    • avfilter-7.dll
    • avformat-58.dll
    • avutil-56.dll
    • postproc-55.dll
    • swresample-3.dll
    • swscale-5.dll

OpenCV to pupil_external

  • Download opencv 3.2 exe installer from sourceforge
  • Unzip OpenCV to your work dir and rename dir to opencv
  • Copy opencv\build\x64\vc14\bin\opencv_world320.dll to pupil_external

Include pupil_external in PATH variable

  • Follow the instructions under the System Environment Variables section above to add a new environment variable to PATH
  • Add the following folder: C:\work\pupil\pupil_external
  • Restart your computer so that the PATH variable is refreshed

Modify pupil_detectors

  • Open pupil\pupil_src\shared_modules\pupil_detectors\
  • Go to the if platform.system() == 'Windows' block
  • Check that paths for opencv, Eigen and ceres-windows are correctly specified. The installed opencv lib is opencv_world320.lib.
  • Edit paths if necessary
  • Save and close

In case you are using Visual Studio 2017 with v15.8 or v15.9 update, you may encounter an error regarding _ENABLE_EXTENDED_ALIGNED_STORAGE while building. Please refer to the fix here.

Modify optimization_calibration

  • Open pupil\pupil_src\shared_modules\calibration_routines\optimization_calibration\
  • Go to the if platform.system() == 'Windows' block
  • Check that paths for opencv, Eigen and ceres-windows are correctly specified. The installed opencv lib is opencv_world320.lib.
  • Edit paths if necessary
  • Save and close

Start the application

To start either of the applications – Capture, Player, or Service – you need to execute the respective run_*.bat file, i.e. run_capture.bat, run_player.bat, or run_service.bat. You can also run directly from your IDE, or with the commands python capture, python player, or python service.

Windows Driver Setup

In order to support isochronous USB transfer on Windows, you will need to install drivers for the cameras in your Pupil headset.

Download drivers and tools

  1. Download Pupil camera driver installer from the Pupil github repo - alternatively you could

Install drivers for your Pupil headset

  1. Right click PupilDrvInst.exe - and run as admin. This will install drivers.
  2. Open Windows Device Manager from System > Device Manager. Verify the drivers are correctly installed in Windows Device Manager. Your Pupil headset cameras should be listed under a new category titled: libusbK Usb Devices. Note: In some cases Pupil Cam1 may show three of the same ID as the camera name. Don’t worry - just make sure that the number of devices are the same as the number of cameras on your Pupil headset.
  3. Download the latest release of Pupil software and launch pupil_capture.exe to verify all cameras are accessible.


If you had tried to install drivers with previous driver install instructions and failed, or are not able to access cameras in Pupil Capture. Please try the following:

  1. In Device Manager (System > Device Manager)
  2. View > Show Hidden Devices
  3. Expand libUSBK Usb Devices
  4. For each device listed (even hidden devices) click Uninstall and check the box agreeing to Delete the driver software for this device and press OK
  5. Repeat for each device in libUSBK Usb Devices
  6. Unplug Pupil headset (if plugged in)
  7. Restart your computer
  8. Install drivers from step 2 in the Install drivers for your Pupil headset section

Manual Installation of DIY Camera Drivers

If any issues arise when trying to install drivers for your DIY eye or world cameras, or if you are installing them for the first time, you can try following these instructions:

  1. Unplug Pupil Headset from your computer and keep unplugged until the last step
  2. Open Device Manager
    1. Click View > Show Hidden Devices
    2. Expand the libUSBK devices category and expand the Imaging Devices category within the Device Manager (sometimes a camera may be under the Cameras category)
    3. Uninstall and delete drivers for all Pupil Cam 1 ID0, Pupil Cam 1 ID1, and Pupil Cam 1 ID2 devices within both libUSBK and Imaging Devices Category
  3. Restart Computer
  4. Start the latest version of Pupil Capture (ensure that you have admin privileges on your machine) i. General Menu > Restart with default settings
  5. Plug in Pupil Headset after Pupil Capture relaunches - Please wait, drivers should install automatically. You may need to close/cancel automatic Windows driver installation

If the above doesn’t work, uninstall all drivers by following steps 1 to 3. You can then try install libusbK drivers with Zadig as outlined in steps 1-7 in the pyuvc docs

Windows Pupil Labs Python libs from Source

This section is for Pupil core developers who want to build pyav, pyndsi, pyglui, and pyuvc from source and create Python wheels.

If you just want to run Pupil from source, go back to the Windows Dependencies section and install the prebuilt wheels.

This section assumes that you have a Windows development environment set up and all dependencies installed as specified in the Windows Dependencies section.

Please also see the Notes Before Starting section for a clarification of terms.

Clone Pupil Labs Python libs

Go to your work dir and open a cmd prompt and clone the following repos

git clone --recursive
git clone --recursive
git clone --recursive
git clone --recursive
git clone
git clone
git clone
git clone

Install wheel

In order to create wheels, you will need to install the wheel lib.

pip install wheel

Download FFMPEG Dev

You will need both .dll files as well as FFMPG libs in order to build pyav. You should have already downloaded FFMPEG shared binaries in the FFMPEG to pupil_external step above.

  • Download FFMPEG Windows Dev - ffmpeg-4.0-win64-dev - from ffmpeg
  • Unzip ffmpeg-dev to your work dir

Download libjpeg-turbo

  • Download libjpeg-turbo v1.5.1 from sourceforge
  • Open the .exe file for setup and navigate to where you want install in your work dir as libjpeg-turbo64
  • Add C:\work\libjpeg-turbo\bin to your system PATH

Build pyav

  • Copy the following ffmpeg shared libs from ffmpeg-shared\bin to pyav/av
    • avcodec
    • avdevice
    • avfilter
    • avformat
    • avutil
    • postproc
    • swresample
    • swscale
  • Open a cmd prompt
  • python clean --all build_ext --inplace --ffmpeg-dir=C:\work\ffmpeg-4.0-win64-dev -c msvc
    • replace the ffmpeg-dir with the location of your ffmpeg-dev dir
  • You can create a wheel from within this directory with pip wheel .

pyglui from source

  • Open a cmd prompt
  • python install
  • You can create a wheel from within this directory with pip wheel .

nslr-hmm from source

  • Open a cmd prompt
  • From inside the cloned nslr repository run: python install
  • From inside the cloned nslr-hmm repository run: python install

pyndsi from source

  • Open a cmd prompt
  • make sure paths to ffmpeg and libjpeg-turbo are correctly specified in
  • python install
  • You can create a wheel from within this directory with pip wheel .


  • Clone the libuvc repo
  • git clone
  • open libusb_2017.sln with MSVC 2017
  • Set to Release and x64 (libusb-1.0 (static))
  • Build solution


  • Make a dir in libuvc titled bin
  • Download CMAKE from
  • Install CMAKE. Make sure to check the box to add CMAKE to your PATH
  • Download POSIX Threads for Windows from sourceforge Note this is a 32 bit lib, but that is OK! Move PThreads to your work dir.
  • Download Microsoft Visual C++ 2010 Redistributable from microsoft. The pthreadVC2 lib depends on msvcr100.dll.
  • Open CMAKE GUI
  • Set source code directory to libuvc repo
  • Set binary directory to libuvc/bin
  • Click Configure
  • Select Visual Studio 15 2017 x64 as generator for the project
  • Select Use default native compilers
  • Fix paths in the config. Set to the following
    • LIBUSB_INCLUDE_DIR = C:\work\libusb\libusb
    • LIBUSB_LIBRARY_NAMES = C:\work\libusb\x64\Release\dll\libusb-1.0.lib
    • PTHREAD_INCLUDE_DIR = C:\work\pthreads-w32-2-9-1-release\Pre-built.2\include
    • PTHREAD_LIBRARY_NAMES = C:\work\pthreads-w32-2-9-1-release\Pre-built.2\lib\x64\pthreadVC2.lib
  • Click Configure again and resolve any path issues
  • Click Generate
  • Open libuvc/bin/libuvc.sln - this will open the project in Visual Studio 2017 Preview
  • Select ALL_BUILD and set to Release and x64 and Build Solution from the Build menu.
  • Add C:\work\libuvc\bin\Release to your system PATH


  • in pyuvc/ make sure the paths to libuvc, libjpeg-turbo, and libusb are correctly specified
  • python install
  • You can create a wheel from within this directory with pip wheel .

Intel RealSense 3D

RealSense Dependencies


All Intel RealSense cameras require librealsense to be installed. Please follow the install instructions for your operating system.


pyrealsense provides Python bindings for librealsense. Run the following command in your terminal to install it.

pip3 install  git+


Select RealSense 3D in the Capture Selection menu and activate your RealSense camera. Afterwards you should see the colored video stream of the selected camera.

Pupil Capture accesses both streams, color and depth, at all times but only previews one at a time. Enable the Preview Depth option to see the normalized depth video stream.

The Record Depth Stream option (enabled by default) will save the depth stream during a recording session to the file depth.mp4 within your recording folder.

By default, you can choose different resolutions for the color and depth streams. This is advantageous if you want to run both streams at full resolution. The Intel RealSense R200 has a maximum color resolution of 1920 x 1080 pixels and maximum depth resolution of 640 x 480 pixels. librealsense also provides the possibility to pixel-align color and depth streams. Align Streams enables this functionality. This is required if you want to infer from depth pixels to color pixels and vice versa.

The Sensor Settings menu lists all available device options. These may differ depending on your OS, installed librealsense version, and device firmware.

Color Frames

Pupil Capture accesses the YUVY color stream of the RealSense camera. All color frames are accessible through the events object using the frame key within your plugin’s recent_events method. See the plugin guide for details.

Depth Frames

Depth frame objects are accessible through the events object using the depth_frame key within your plugin’s recent_events method. The orginal 16-bit grayscale image of the camera can be accessed using the depth attribute of the frame object. The bgr attribute provides a colored image that is calculated using histogram equalization. These colored images are previewed in Pupil Capture, stored during recordings, and referred to as “normalized depth stream” in the above section. The librealsense examples use the same coloring method to visualize depth images.

Interprocess and Network Communication

This page outlines the way Pupil Capture and Pupil Service communicate via a message bus internally and how to read and write to this bus from another application on the same machine or on a remote machine.


All networking in Pupil Capture and Service is based on the ZeroMQ network library. The following socket types are most often used in our networking schemes: - REQ-REP, reliable one-to-one communication - PUB-SUB, one-to-many communication

We highly recommend to read Chapter 2 of the ZeroMQ guide to get an intuition on the philosophy behind these socket types.

The Pupil apps use the pyzmq module, a great Python ZeroMQ implementation. For auto-discovery of other Pupil app instances in the local network, we use Pyre.

The IPC Backbone

Pupil Capture/Service/Player use a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors’ messages. Therefore, it is the backbone of all communication from, to, and within Pupil apps.

IPC Backbone used by Pupil Capture and Service

The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate. - Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.) - Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.

Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.

Message Format

Currently all messages on the IPC Backbone are multipart messages containing two message frames:

  • Frame 1 contains a string we call topic. Examples are : pupil.0,, notify.recording.has_started

  • Frame 2 contains a msgpack encoded dictionary with key:value pairs. This is the actual message. We choose msgpack as the serializer due to its efficient format (45% smaller than json, 200% faster than ujson) and because encoders exist for almost every language.

Message Topics

Messages can have any topic chosen by the user. Below a list of message types used by the Pupil apps.

Pupil and Gaze Messages

Pupil data is sent from the eye0 and eye1 process with the topic pupil.0 or pupil.1. Gaze mappers receive this data and publish messages with topic gaze. See the Pupil Datum format for example messages for the topics pupil and gaze.

Notification Message

Pupil uses special messages called notifications to coordinate all activities. Notifications are dictionaries with the required field subject. Subjects are grouped by categories category.command_or_statement. Example: recording.should_stop

# message topic:
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}

The message topic construction in python:

topic = 'notify'+'.'+notification['subject']

You should use the notification topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.

In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more that 10 messages a second, it is probably not a notification but another kind of data, make a custom topic instead.

import zmq
import msgpack

topic = 'your_custom_topic'
payload = {'topic': topic}

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)

# send payload using custom topic
socket.send_string(topic, flags=zmq.SNDMORE)
socket.send(msgpack.dumps(payload, use_bin_type=True))

Log Messages

Pupil sends all log messages onto the IPC.

The topic is logging.log_level_name (debug,info,warning,error,…). The message is a dictionary that contains all attributes of the python logging.record instance.

# message topic:
# message payload, logging record attributes as dict:
{'levelname': 'WARNING', 'msg': 'Process started.', 'threadName': 'MainThread', 'name': 'eye', 'thread': 140735165432592L, 'created': 1465210820.609704, 'process': 14239, 'processName': 'eye0', 'args': [], 'module': 'eye', 'filename': '', 'levelno': 30, 'msecs': 609.7040176392, 'pathname': '/Users/mkassner/Pupil/pupil_code/pupil_src/capture/', 'lineno': 299, 'exc_text': None, 'exc_info': None, 'funcName': 'eye', 'relativeCreated': 4107.3870658875})

Message Documentation

Pupil software uses introduces a consistent naming scheme for message topics. They are used to publish and subscribe to the IPC Backbone. Pre-defined message topics are pupil, gaze, notify, delayed_notify, logging. Notifications sent with the notify_all() function of the Plugin class will be published automatically as notify.<notification subject>.

Message Reactor and Emitter Documentation

Every actor who either reacts to or emits messages is supposed to document its behavior. Therefore every actor should react to notify.meta.should_doc by emitting a message with the topic notify.meta.doc. The answer’s payload should be a serialized dictionary with the following format:

  'actor': <actor name>,
  'doc': <string containing documentation>

Plugins use notifications as their primary communication channel to the IPC Backbone. This makes plugins natural actors in the Pupil message scheme. To simplify the above mentioned documentation behavior, plugins will only have to add a docstring to their on_notify() method. It should include a list of messages to which the plugin reacts and of those which the plugin emits itself. The docstring should follow Google docstring style. The main process will automatically generate messages in the format from above using the plugin’s class name as actor and the on_notify() docstring as content for the doc key.

Notification Overview

You can use the following script to get an overview over the notification handling of the currently running actors:

import zmq, msgpack
from zmq_tools import Msg_Receiver
ctx = zmq.Context()
ip = 'localhost' #If you talk to a different machine use its IP.
port = 50020 #The port defaults to 50020 but can be set in the GUI of Pupil Capture.

# open Pupil Remote socket
requester = ctx.socket(zmq.REQ)
ipc_sub_port = requester.recv_string()

# setup message receiver
sub_url = 'tcp://%s:%s'%(ip,ipc_sub_port)
receiver = Msg_Receiver(ctx, sub_url, topics=('notify.meta.doc',))

# construct message
topic = 'notify.meta.should_doc'
payload = msgpack.dumps({'subject':'meta.should_doc'})
requester.send_string(topic, flags=zmq.SNDMORE)

# wait and print responses
while True:
    # receiver is a Msg_Receiver, that returns a topic/payload tuple on recv()
    topic, payload = receiver.recv()
    actor = payload.get('actor')
    doc = payload.get('doc')
    print('%s: %s'%(actor,doc))

Example output for v0.8:

launcher: Starts eye processes. Hosts the IPC Backbone and Logging functions.

    Reacts to notifications:
       ``launcher_process.should_stop``: Stops the launcher process
       ``eye_process.should_start``: Starts the eye process

eye0: Reads eye video and detects the pupil.

    Creates a window, gl context.
    Grabs images from a capture.
    Streams Pupil coordinates.

    Reacts to notifications:
       ``set_detection_mapping_mode``: Sets detection method
       ``eye_process.should_stop``: Stops the eye process
       ``recording.started``: Starts recording eye video
       ``recording.stopped``: Stops recording eye video

    Emits notifications:
        ``eye_process.started``: Eye process started
        ``eye_process.stopped``: Eye process stopped

    Emits data:
        ``pupil.<eye id>``: Pupil data for eye with id ``<eye id>``

capture: Reads world video and runs plugins.

    Creates a window, gl context.
    Grabs images from a capture.
    Maps pupil to gaze data
    Can run various plugins.

    Reacts to notifications:

    Emits notifications:
        ``recording.should_stop``: Emits on camera failure

    Emits data:
        ``gaze``: Gaze data from current gaze mapping plugin.``
        ``*``: any other plugin generated data in the events that it not [dt,pupil,gaze].

Pupil_Remote: send simple string messages to control application functions.

        Emits notifications:
            Any other notification received though the reqrepl port.

Screen_Marker_Calibration: Handles calibration notifications

        Reacts to notifications:
           ``calibration.should_start``: Starts the calibration procedure
           ``calibration.should_stop``: Stops the calibration procedure

        Emits notifications:
            ``calibration.started``: Calibration procedure started
            ``calibration.stopped``: Calibration procedure stopped
            ``calibration.failed``: Calibration failed
            ``calibration.successful``: Calibration succeeded

            notification (dictionary): Notification dictionary

Recorder: Handles recorder notifications

        Reacts to notifications:
            ``recording.should_start``: Starts a new recording session
            ``recording.should_stop``: Stops current recording session

        Emits notifications:
            ``recording.started``: New recording session started
            ``recording.stopped``: Current recording session stopped

            notification (dictionary): Notification dictionary

Pupil Remote

If you want to tap into the IPC backbone you will not only need the IP address but also the session unique port. You can get these by talking to ‘Pupil Remote’:

import zmq
ctx = zmq.Context()
# The requester talks to Pupil remote and receives the session unique IPC SUB PORT
requester = ctx.socket(zmq.REQ)
ip = 'localhost' #If you talk to a different machine use its IP.
port = 50020 #The port defaults to 50020 but can be set in the GUI of Pupil Capture.
sub_port = requester.recv_string()

Pupil Remote uses the fixed port 50020 and is the entry point to the IPC backbone for external applications. It also exposes a simple string-based interface for basic interaction with the Pupil apps:

Send simple string messages to control Pupil Capture functions:
    'R' start recording with auto generated session name
    'R rec_name' start recording and name new session name: rec_name
    'r' stop recording
    'C' start currently selected calibration
    'c' stop currently selected calibration
    'T 1234.56' Timesync: make timestamps count form 1234.56 from now on.
    't' get pupil capture timestamp; returns a float as string.

    # IPC Backbone communication
    'PUB_PORT' return the current pub port of the IPC Backbone
    'SUB_PORT' return the current sub port of the IPC Backbone

Reading from the Backbone

Subscribe to desired topics and receive all relevant messages (i.e. messages whose topic prefix matches the subscription). Be aware that the IPC Backbone can carry a lot of data. Do not subscribe to the whole stream unless you know that your code can drink from a firehose. (If it can not, you become the snail, see Delivery Guarantees REQ-REP.)

#...continued from above
subscriber = ctx.socket(zmq.SUB)
subscriber.set(zmq.SUBSCRIBE, 'notify.') #receive all notification messages
subscriber.set(zmq.SUBSCRIBE, 'logging.error') #receive logging error messages
#subscriber.set(zmq.SUBSCRIBE, '') #receive everything (don't do this)
# you can setup multiple subscriber sockets
# Sockets can be polled or read in different threads.

# we need a serializer
import msgpack as serializer

while True:
    topic,payload = subscriber.recv_multipart()
    message = serializer.loads(payload)
    print topic,':',message

Writing to the Backbone from outside

You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:

# continued from above
import msgpack as serializer
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)
requester.send_string(topic, flags=zmq.SNDMORE)

We say reliable transport because pupil remote will confirm every notification we send with ‘Notification received’. When we get this message we have a guarantee that the notification is on the IPC Backbone.

If we listen to the backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.

Writing to the Backbone directly

If you want to write messages other than notifications onto the IPC backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.

# continued from above
from time import time, sleep

pub_port = requester.recv_string()
publisher = ctx.socket(zmq.PUB)
publisher.connect('tcp://%s:%s'%(ip, pub_port))
sleep(1) # see Async connect in the paragraphs below
notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)
publisher.send_string(topic, flags=zmq.SNDMORE)

A full example

A full example can be found in shared_modules/

Delivery guarantees ZMQ

ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don’t want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.

  • Messages are guaranteed to be delivered whole or not at all.
  • Unlike bare TCP it is ok to connect before binding.
  • ZMQ will try to repair broken connections in the background for us.
  • It will deal with a lot of low level tcp handling so we don’t have to.

Delivery Guarantees PUB-SUB

ZMQ PUB SUB will make no guarantees for delivery. Reasons for dropped messages are:

  • Async connect: PUB sockets drop messages before a connection has been made (connections are async in the background) and topics subscribed. *1
  • The Late joiner: SUB sockets will only receive messages that have been sent after they connect. *2
  • The Snail: If SUB sockets do not consume delivered messages fast enough they start dropping them. *3
  • fast close: A PUB socket may loose packages if you close it right after sending. *1
  1. In Pupil we prevent this by using a PUSH socket as intermediary for notifications. See shared_modules/

  2. Caching all messages in the sender or proxy is not an option. This is not really considered a problem of the transport.

  3. In Pupil we pay close attention to be fast enough or to subscribe only to low volume topics. Dropping messages in this case is by design. It is better than stalling data producers or running out of memory.

Delivery Guarantees REQ-REP

When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.

Delivery Guarantees in general

We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.

However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.


Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests…

# continued from above
ts = []
for x in range(100):
    sleep(0.003) #simulate spaced requests as in real world
    t = time()
print(min(ts), sum(ts)/len(ts), max(ts))
>>>0.000266075134277 0.000597472190857 0.00339102745056

… and when talking directly to the IPC backbone and waiting for the same message to appear to the subscriber:

# continued from above
monitor = Msg_Receiver(ctx, sub_url, topics=('notify.pingback_test',))

ts = []
for x in range(100):
    sleep(0.003)  #simulate spaced requests as in real world
    t = time()
    #notify is a method of the Msg_Dispatcher class in
    notification = {'subject':'pingback_test'}
    topic = 'notify.' + notification['subject']
    payload = serializer.dumps(notification)
    publisher.send_string(topic, flags=zmq.SNDMORE)
print(min(ts), sum(ts)/len(ts) , max(ts))
>>>0.000180959701538 0.000300960540771 0.000565052032471


During a test we have run dual 120fps eye tracking with a dummy gaze mapper that turned every pupil datum into a gaze datum. This is effectively 480 messages/sec. The main process running the IPC backbone proxi showed a cpu load of 3% on a MacBook Air (late 2012).

Artificially increasing the pupil messages by a factor 100 increases the message load to 24.000 pupil messages/sec. At this rate the gaze mapper cannot keep up but the IPC backbone proxi runs at only 38% cpu load.

It appears ZMQ is indeed highly optimized for speed.

Final remarks

You can send a message anywhere in the app. Don’t send something that crashes anywhere.

Plugin Guide

Plugins Basics

Plugins encapsulate functionality in a modular fashion. Most parts of the Pupil apps are implemented as plugins. They are managed within the world process event-loop. This means that the world process can load and unload plugins during runtime. Plugins are called regularly via callback functions (see the Plugin API for details).

We recommend to use the network (see the IPC backbone) if you only need access to the data. You are only required to write a plugin if you want to interact with the Pupil apps directly, e.g. visualizations, manipulate data. In the following sections, we assume and recommend that during plugin development you run the Pupil applications from source.

Plugins in Pupil Capture

Pupil Capture’s World process can load plugins for easy integration of new features. Plugins have full access to:

  • World image frame
  • Events
    • pupil positions
    • gaze positions
    • surface events
    • note other events can be added to the event queue by other plugins
  • User input
  • Globally declared variables in the g_pool

Plugins can create their own UI elements, and even spawn their own OpenGL windows.

Pupil Player Plugins

Pupil Player uses an identical plugin structure. Little (often no work) needs to be done to use a Player Plugin in Capture and vice versa. But, it is important to keep in mind that plugins run in Pupil Capture may require more speed for real-time workflows, as opposed to plugins in Pupil Player.

Plugin API

Plugins are Python classes that inherit the Plugin class. It provides default functionality as well as series of callback functions that are called by the world process. The source contains detailed information about the use-cases of the different callback functions.

Register your plugin automatically

To add your plugin to Capture all you need to do is place the source file(s) in the plugin directory.

If you run from source:

  • Pupil Capture: [root_of_source_pupil_source_git_repo]/capture_settings/plugins/
  • Pupil Service: [root_of_source_pupil_source_git_repo]/service_settings/plugins/
  • Pupil Player: [root_of_source_pupil_source_git_repo]/player_settings/plugins/

If you want to add your plugin to a bundled version of Pupil:

  • Pupil Capture: [your_user_dir]/pupil_capture_settings/plugins/
  • Pupil Service: [your_user_dir]/pupil_service_settings/plugins/
  • Pupil Player: [your_user_dir]/pupil_player_settings/plugins/

[your_user_dir] is also called HOME (for Linux and MacOS) or USER (for Windows).

Note: if your plugin is contained in a directory, make sure to include an inside it. For example:

When a valid plugin is found in these dirs, Pupil imports your Plugin classes and adds them to the dropdown list of launchable plugins. If your plugin is a calibration plugin (i.e. it inherits from the Calibration_Plugin base class), then it will appear in the calibration drop down menu.

Example plugin development walkthrough

Inheriting from existing plugin

If you want to add or extend the functionality of an existing plugin, you should be able to apply standard inheritance principles of Python 3.

Things to keep in mind:

  • g_pool is an abbreviation for “global pool”, a system wide container full of stuff passed to all plugins.
  • if the base plugin is a system (always alive) plugin:
    • remember to close the base plugin at the __init__ method of the inheriting plugin with base_plugin.alive = False. You should find the base_plugin inside g_pool.plugins ;
    • remember to dereference the base plugin at the end of the file with del base_plugin to avoid repetition in the user plugin list;

Hacking an existing plugin

Another way to start plugin development, is to use an existing plugin as a template. For example, you could copy the plugin as a starting point.

renaming it to, for example,

Now you could give a new name to the class name:

class Open_Cv_Threshold(Plugin):

Describe what your new plugin will do for yourself in the future and for future generations:

class Open_Cv_Threshold(Plugin):
  Apply cv2.threshold filter to the world image.

Rename its reference in the persistence method:

def clone(self):
    return Open_Cv_Threshold(**self.get_init_dict())

It is good to rename its menu caption as well: = 'Threshold'

Lets determine its execution order in relation to the other plugins:

self.order = .8

You can allow or disallow multiple instances of the Custom Plugin through the uniqueness attribute:

self.uniqueness = "by_class"

See the source for a list of all available uniqueness options. Finally, lets implement what our new Plugin will do. Here we choose to apply an OpenCv threshold to the world image and give us proper feedback of the results, in real time. Good for OpenCv and related studies. It is possible by means of the recent_events method:

def recent_events(self, events):
  if 'frame' in events:
    frame = events['frame']
    img = frame.img
    height = img.shape[0]
    width = img.shape[1]

    blur = cv2.GaussianBlur(img,(5,5),0)

    edges = []
    threshold = 177
    blue, green, red = 0, 1, 2

    # apply the threshold to each channel
    for channel in (blur[:,:,blue], blur[:,:,green], blur[:,:,red]):
      retval, edg = cv2.threshold(channel, threshold, 255, cv2.THRESH_TOZERO)

    # lets merge the channels again
    edges.append(np.zeros((height, width, 1), np.uint8))
    edges_edt = cv2.max(edges[blue], edges[green])
    edges_edt = cv2.max(edges_edt, edges[red])
    merge = [edges_edt, edges_edt, edges_edt]

    # lets check the result
    frame.img = cv2.merge(merge)

recent_events is called everytime a new world frame is available but latest after a timeout of 0.05 seconds. The events dictionary will include the image frame object if it was available. It is accessible through the frame key.

You can access the image buffer through the img and the gray attributes of the frame object. They return a BGR (height x width x 3) and gray scaled (height x width) uint8-numpy array respectively. Visualization plugins (e.g. modify the img buffer such that their visualizations are visible in the Pupil Player exported video. Use OpenGL (within the Plugin.gl_display method) to draw visualizations within Pupil Player that are not visible in the exported video (e.g. surface heatmaps in Offline_Surface_Tracker. See below for more information.

The events dictionary contains other recent data, e.g. pupil_positions, gaze_positions, fixations, etc. Modifications to the events dictionary are automatically accessible by all plugins with an higher order than the modifying plugin.

Plugin Integration

pyglui UI Elements

‘pyglui’ is an OpenGL-based UI framework that provides easy to use UI components for your plugin. User plugins often have at least one menu to inform the user that they are running as well as providing the possibility to close single plugins.

from plugin import Plugin
from pyglui import ui

class Custom_Plugin(Plugin):
    # Calling add_menu() will create an icon in the icon bar that represents
    # your plugin. You can customize this icon with a symbol of your choice.
    icon_chr = '@'  # custom menu icon symbol

    # The default icon font is Roboto:
    # Alternatively, you can use icons from the Pupil Icon font:
    icon_font = 'roboto'  # or `pupil_icons` when using the Pupil Icon font

    def __init__(self, g_pool, example_param=1.0):
        # persistent attribute
        self.example_param = example_param

    def init_ui(self):
        # Create a floating menu
        self.add_menu() = '<title>'
        # Create a simple info text
        help_str = "Example info text."
        # Add a slider that represents the persistent value'example_param', self, min=0.0, step=0.05, max=1.0, label='Example Param'))

    def deinit_ui(self):

    def get_init_dict(self):
        # all keys need to exists as keyword arguments in __init__ as well
        return {'example_param': self.example_param}

Export Custom Video Visualizations

As descrbed above, plugins are able to modify the image buffers to export their visualizations. The plugins recent_events method is automatically called for each frame once by the video exporter process. Plugins might overwrite changes made by plugins with a lower order than themselves. OpenGL visualizations are not exported. See for an example visualization.

Export Custom Raw Data

Each Player plugin gets a notification with subject should_export thar includes the world frame indices range that will be exported and the directory where the recording will be exported to. Add the code to the right to your plugin and implement an export_data function. See for an example.

def on_notify(self, notification):
    if notification['subject'] is "should_export":
        self.export_data(notification['range'], notification['export_dir'])

Background Tasks

All plugins run within the world process. Doing heavy calculations within any of the periodically called Plugin methods (e.g. recent_events) can result in poor performance of the application. It is recommended to do any heavy calculations within a separate subprocess - multi-threading brings its own problems in Python. We created the Task_Proxy to simplify this procedure. It is initialized with a generator which will be executed in a subprocess. The generator’s results will automatically be piped to the main thread where the plugin can fetch them.

from plugin import Plugin
from pyglui import ui
import logging
logger = logging.getLogger(__name__)

def example_generator(mu=0., sigma=1., steps=100):
    '''samples `N(\mu, \sigma^2)`'''
    import numpy as np
    from time import sleep
    for i in range(steps):
        # yield progress, datum
        progress = (i + 1) / steps
        value = sigma * np.random.randn() + mu
        yield progress, value
        sleep(np.random.rand() * .1)

class Custom_Plugin(Plugin):
    def __init__(self, g_pool):
        self.proxy = Task_Proxy('Background', example_generator, args=(5., 3.), kwargs={'steps': 50})

    def recent_events(self, events):
        # fetch all available results
        for progress, random_number in task.fetch():
            logger.debug('[{:3.0f}%] {:0.2f}'.format(progress * 100, random_number))

        # test if task is completed
        if task.completed:
            logger.debug('Task done')

    def cleanup(self):
        if not self.proxy.completed:
            logger.debug('Cancelling task')

Recording Format

Required Files

Recording Name,2018_07_19
Start Date,19.07.2018
Start Time,14:56:21
Start Time (System),1532004981.666572
Start Time (Synced),701730.897108953
Duration Time,00:00:13
World Camera Frames,402
World Camera Resolution,1280x720
Capture Software Version,1.7.159
Data Format Version,1.8
System Info,"User: name, Platform: Linux, ..."

Each recording requires three files: 1. An info.csv file that includes two columns – key and value. (See left for example) 2. At least one video file and its corresponding timestamp file. See the Video Files section below for details.

A minimum requirement of two key, value pairs are required in the info.csv file. 1. Recording Name,<name> 2. Data Format Version,<version>

Data Files

Timestamp Files

Timestamp files must follow this strict naming convention: Given that a data file is named <name>.<ext> then its timestamps file has to be named <name>_timestamps.npy.

Timestamp files are saved in the NPY binary format. You can use numpy.load() to access the timestamps in Python.

A datum and its timestamp have the same index within their respective files, i.e. the ith timestamp in world_timestamps.npy belongs to the ith video frame in world.mp4.

Video Files

Video files are only recognized if they comply with the following constraints:

Allowed video file extentions are:

  • .mp4
  • .mkv
  • .avi
  • .h264
  • .mjpeg

Allowed video file names are:

  • world: Scene video
  • eye0: Right eye video
  • eye1: Left eye video

The video files should look like:

  • world.mp4, eye0.mjpeg, eye1.mjpeg

We also support multiple parts of video files as input. For instance:

  • world.mp4, world_001.mp4
  • eye0.mjpeg, eye0_001.mjpeg

And their corresponding timestamp files should follow the pattern:

  • world_timestamps.npy, world_001_timestamps.npy

Audio File

An audio file is only recognized in Pupil Player’s playback plugin if the file is named audio.mp4.

pldata Files

These files contain a sequence of independently msgpack-encoded messages. Each message consists of two frames: 1. frame: The payload’s topic as a string, e.g. "pupil.0" 2. frame: The payload, e.g. a pupil datum, encoded as msgpack

For clarification: The second frame is encoded twice!

Pupil Player decodes the messages into file_methods.Serialized_Dicts. Each Serialized_Dict instance holds the serialized second frame and is responsible for decoding it on demand. The class is designed such that there is a maximum number of decoded frames at the same time. This prevents Pupil Player from using excessive amounts of memory.

You can use file_methods.PLData_Writer and file_methods.load_pldata_file() to read and write pldata files.

Other Files

Files without file extention, e.g. the deprecated pupil_data file, and files with a .meta extention are msgpack-encoded dictionaries. They can be read and written using file_methods.load_object() and file_methods.save_object() and do not have a corresponding timestamps file.

Eye Movement Detector

Eye movement classification detector based on segmented linear regression.

Event identification is based on segmentation that simultaneously denoises the signal and determines event boundaries. The full gaze position time-series is segmented into an approximately optimal piecewise linear function in O(n) time. Gaze feature parameters for classification into fixations, saccades, smooth pursuits and post-saccadic oscillations are derived from human labeling in a data-driven manner.[1]

More details about this approach can be found here.

The open source implementation can be found here.


Online Eye Movement Detector

This plugin detects eye movement from the last max_sample_count frames (defaults to 1000). For every new frame, the frame buffer is re-classified and the last eye movement segment detection is published. This might result in a series of updates to an already published segment.

To see how to receive real-time eye movement classification notifications, please see this script.

Offline Eye Movement Detector

This plugin detects eye movement within a given duration window. Since the detector is able to process the entire frame buffer, the classified eye movement segments do not overlap.

Eye Movement Format

If 3d pupil data is available the eye movement will be classified based on the positional angle of the eye. These eye movement segments have their base_type field set to "pupil". If no 3d pupil data is available the plugin will assume that the gaze data is calibrated and classify the eye movement in visual angle within the coordinate system of the world camera. These eye movement segments will have their base_type field set to "gaze".


Real-time eye movement detection publishes notifications with the following structure:

  • id: Identifier uniquely identifying the segment within a detection session
  • topic: eye_movement.fixation, eye_movement.saccade, eye_movement.pso or eye_movement.smooth_pursuit
  • base_type: pupil or gaze
  • segment_class: fixation, saccade, pso or smooth_pursuit
  • timestamp: Timestamp of the eye movement segment.
  • start_frame_index: Index of the first segment frame, in the frame buffer.
  • end_frame_index: Index after the last segment frame, in the frame buffer.
  • start_frame_timestamp: Timestamp of the first frame, in the frame buffer.
  • end_frame_timestamp: Timestamp of the last frame, in the frame buffer.


Offline eye movement detection results are exported to eye_movement_by_segment.csv, with the following format:

  • id: Identifier uniquely identifying the segment within a detection session
  • base_type: pupil or gaze
  • segment_class: fixation, saccade, pso or smooth_pursuit
  • start_frame_index: Index of the first segment frame, in the frame buffer.
  • end_frame_index: Index after the last segment frame, in the frame buffer.
  • start_timestamp: Timestamp of the first frame, in the frame buffer.
  • end_timestamp: Timestamp of the last frame, in the frame buffer.
  • duration: Eye movement segment duration, in milliseconds.
  • confidence: Average pupil confidence.
  • norm_pos_x: Mean normalized position’s x coordinate.
  • norm_pos_y: Mean normalized position’s x coordinate.
  • gaze_point_3d_x: Mean 3d gaze point’s x coordinate, only available if "pupil" base_type was used
  • gaze_point_3d_y: Mean 3d gaze point’s y coordinate, only available if "pupil" base_type was used
  • gaze_point_3d_z: Mean 3d gaze point’s z coordinate, only available if "pupil" base_type was used

Fixation Detector

In Salvucci and Goldberg define different categories of fixation detectors. One of them describes dispersion-based algorithms:

I-DT identifies fixations as groups of consecutive points within a particular dispersion, or maximum separation. Because fixations typically have a duration of at least 100 ms, dispersion-based identification techniques often incorporate a minimum duration threshold of 100-200 ms to help alleviate equipment variability.[1]

Pupil’s fixation detectors implement such dispersion-based algorithms.

[1] Salvucci, D. D., & Goldberg, J. H. (2000, November). Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications (pp. 71-78). ACM.


Online Fixation Detector

This plugin detects fixations based on a dispersion threshold in terms of degrees of visual angle with a minimum duration. It publishes the fixation as soon as it complies with the constraints (dispersion and duration). This might result in a series of overlapping fixations.

  1. Maximum Dispersion (spatial, degree): Maximum distance between all gaze locations during a fixation.
  2. Minimum Duration (temporal, milliseconds): The minimum duration in which the dispersion threshold must not be exceeded.
  3. Confidence Threshold: Data with a lower confidence than this threshold is not considered during fixation detection.

Offline Fixation Detector

This plugin detects fixations based on a dispersion threshold in terms of degrees of visual angle within a given duration window. It tries to maximize the length of classified fixations within the duration window, e.g. instead of creating two consecutive fixations of length 300 ms it creates a single fixation with length 600 ms. Fixations do not overlap.

  1. Maximum Dispersion (spatial, degree): Maximum distance between all gaze locations during a fixation.
  2. Minimum Duration (temporal, milliseconds): The minimum duration in which the dispersion threshold must not be exceeded.
  3. Maximum Duration (temporal, milliseconds): The maximum duration in which the dispersion threshold must not be exceeded.

Fixation Format

If 3d pupil data is available the fixation dispersion will be calculated based on the positional angle of the eye. These fixations have their method field set to “pupil”. If no 3d pupil data is available the plugin will assume that the gaze data is calibrated and calculate the dispersion in visual angle within the coordinate system of the world camera. These fixations will have their method field set to “gaze”.


Fixations are represented as dictionaries consisting of the following keys:

  • topic: Static field set to fixation
  • norm_pos: Normalized position of the fixation’s centroid
  • base_data: Gaze data that the fixation is based on
  • duration: Exact fixation duration, in milliseconds
  • dispersion: Dispersion, in degrees
  • timestamp: Timestamp of the first related gaze datum
  • confidence: Average pupil confidence
  • method: pupil or gaze

  • gaze_point_3d: Mean 3d gaze point, only available if pupil method was used


Player detected fixations also include:

  • start_frame_index: Index of the first related frame
  • mid_frame_index: Index of the median related frame
  • end_frame_index: Index of the last related frame

USB Bandwidth And Synchronization

USB Bandwidth limits and ways to make it work regardless

The Pupil headset uses 2-3 cameras that are electrically- and firmware wise identical (except for the name in the usb descriptor). Our Pupil camera can supply frames in various resolutions and rates uncompressed (YUV) and compressed (MJPEG). When looking at uncompressed data even a single camera can saturate a high speed USB bus. This is why we always use MJPEG compression: We can squeeze the data of 3 cameras through one USB bus because the image data is compressed by a factor of ~10.

JPEG size estimation and custom video backends

However, the actual size of each image depends on the complexity of the content (JPEGs of images with more features will be bigger) and implementation details of the camera firmware. Because the cameras use isochronous usb transfers, we need to allocate bandwidth during stream initialization. Here we need to make an estimate on how much bandwidth we believe the camera will require. If we are too conservative we require more bandwidth for 3 cameras than is available and initialization will fail. If we allocate to little, we risk that image transport will fail during capture. According to the UVC specs the amount of bandwidth that is required must be read from the camera usb descriptor and usually this estimate is super conservative. This is why with the normal drivers you can never run more that one camera at decent resolutions on a single usb bus.

With our version of libuvc and pyuvc we ignore the cameras request and estimate the bandwidth ourselves like this:

//the proper way: ask the camera
config_bytes_per_packet = strmh->cur_ctrl.dwMaxPayloadTransferSize;

// our way: estimate it:
size_t bandwidth = frame_desc->wWidth * frame_desc->wHeight / 8 * bandwidth_factor; //the last one is bpp default 4 but we use if for compression, 2 is save, 1.5 is needed to run 3 high speed cameras. on one bus.
bandwidth *= 10000000 / strmh->cur_ctrl.dwFrameInterval + 1;
bandwidth /= 1000; //unit
bandwidth /= 8; // 8 high speed usb microframes per ms
bandwidth += 12; //header size
config_bytes_per_packet = bandwidth;

The scale factor bandwidth_factor is settable through the api.

We have tested these and found that we can run 3 pupil camera at 720p@60fps+2x480p@120fps on our Mac and Linux machines. If you play with the resolutions and frame rates in pupil capture you may hit a combination where the total bandwidth requirements cannot be met, thus the crash (I assume).

Use more BUS

If you want to not be limited by the bandwidth of a single usb bus, you can mod the hardware and expose every camera directly. Just make sure that you also have three free USB controllers (not plugs) on your PC.

Multi Camera Synchronization

Each camera we use is a free running capture device. Additionally each camera runs in a separate process. Instead of frame-locking the camera through special hardware we acquire timestamps for each frame. These timestamps are then used to correlate data from each camera in time and match frames based on closest proximity.

Data from each eye camera is sent via IPC to the world process. Since this involves three separate processes it can happen that data from one camera arrives earlier that another. However for each camera the frames will be ordered and timestamps are monotonically increasing. In the main process we match the available data timewise when we need. In Pupil Player we can do matching after the fact to work with perfectly sorted data from all three cameras. If you require the data to be matched over being recent I would recommend collecting data in the queue for a few more frames in before dispatching them in the events dict. (I ll actually do some tests on this subject soon.)

A note on synchronization and rolling shutters

While synchronization through hardware is preferable, its implementation would come at added hardware cost. The benefits of that become questionable at 120fps. At this rate the frame interval is about 8ms which very close to the exposure time of the eye cameras. Since our cameras use a rolling shutter the image is actually taken continuously and the time of exposure changes based on the pixel position on the sensor. You can think of the camera image stream as a scanning sensor readout with data packed into frames and timestamped with the time of the first pixel readout. If we then match frames from two or more sensors we can assume that the pixels across two camera are generally no further apart in time than the first and last pixel of one frame from a single camera.


We want Pupil to proliferate! We want you to use Pupil to empower and inspire whatever you do. Be it academic research, commercial work, teaching, art, or personally motivated projects.

We want you to be a member of the Pupil community and contribute as much as possible. The software is open and the hardware is modular and accessible. We encourage the modification of software in accordance to the open source license.


All source code written by us is open source in accordance with the GNU Lesser General Public License (LGPL v3.0) license. We encourage you to change and improve the code. We require that you will share your work with the Pupil community.


The camera mounts of the Pupil headset are open source for non-commercial use. We distribute CAD files for camera mounts and document the interface geometry in the Pupil Hardware Development so that you can use different cameras or customize for your specific needs. Again, we encourage this and are excited to see new designs, ideas, and support for other camera models.

The actual frame of the Pupil headset is not open-source and distributed via Shapeways and direct sales. We do this for a few reasons:

  • The printed geometry is not the actual CAD file. It is the result of a Finite Element Analysis (FEA) simulation that is exported as a triangle mesh. The CAD file itself is useless because the headset does not fit well without the FEA step. The FEA is useless because it is hard to properly manipulate in a CAD environment.
  • The entire design is based on the material properties of laser sintered nylon. This is what allows the headset to be so light, flexible, and strong. Unless you own an EOS SLS machine, Shapeways will always outperform the field in terms of price. In other words: The headset does not make any sense in another material and the material/manufacturing is expensive when you buy it from somewhere other than Shapeways.

We take a markup fee for every headset to finance the Pupil project. This fee supports open source development, so that you can continue to get software for free!

If you have ideas and suggestions for improvements on the actual frame of the headset we are happy to collaborate closely on improvements. Contact us.


All content of the documentation written by us is open source, according to GNU Lesser General Public License (LGPL v3.0) license.

Using Pupil in Your Research and Projects

You can use Pupil in your research, academic work, commercial work, art projects and personal work. We only ask you to credit us appropriately. See Academic Citation for samples.

Pupil is developed and maintained by Pupil Labs. If you make a contribution to open source, we will include your name in our [[Contributors]] page. For more information about the people behind the project, check out Pupil Labs.

Alternate Licensing

If you would like to use Pupil outside of the GNU Lesser General Public License (LGPL v3.0) license, contact us so we can discuss a options. Send an email to us at sales [at] pupil-labs [dot] com


Pupil Community

Pupil Community logo Pupil Community logo

The Pupil community is made up of amazing individuals around the world. It is your effort and exchanges that enable us to discover novel applications for eye tracking and help us to improve the open source repository. The Pupil community is active, growing, and thrives from your contributions.

Connect with the community, share ideas, solve problems and help make Pupil awesome!


Fork the Pupil repository and start coding!


  • Find a bug? Raise an issue in the GitHub project (pupil-labs/pupil).
  • Have you made a new feature, improvement, edit, something cool that you want to share? Send us a pull request. We check on the regular. If we merge your feature(s) you’ll be credited in the Pupil [[Contributors]] page.


For quick questions and casual discussion, chat with the community on discord.


If you want to talk directly to someone at Pupil Labs, email is the easiest way.

Academic Citation

We have been asked a few times about how to cite Pupil in academic research. Please take a look at our papers below for citation options. If you’re using Pupil as a tool in your research please cite the below UbiComp 2014 paper.

Papers that cite Pupil

We have compiled a list of publications that cite Pupil in this spreadsheet

UbiComp 2014 Paper


Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction


In this paper we present Pupil – an accessible, affordable, and extensible open source platform for pervasive eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight eye tracking headset, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a processing pipeline latency of only 0.045 seconds.

Permalink to article

Available on

BibTeX Style Citation

 author = {Kassner, Moritz and Patera, William and Bulling, Andreas},
 title = {Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
 booktitle = {Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing},
 series = {UbiComp '14 Adjunct},
 year = {2014},
 isbn = {978-1-4503-3047-3},
 location = {Seattle, Washington},
 pages = {1151--1160},
 numpages = {10},
 url = {},
 doi = {10.1145/2638728.2641695},
 acmid = {2641695},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {eye movement, gaze-based interaction, mobile eye tracking, wearable computing},

Pupil Technical Report


Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction


Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software. This limits the application areas and use of mobile eye tracking to expert users and inhibits user-driven development, customization, and extension. In this paper we present Pupil – an accessible, affordable, and extensible open source platform for mobile eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight headset with high-resolution cameras, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface (GUI) to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a latency of the processing pipeline of only 0.045 seconds.

Permalink to article

Available on

BibTeX Style Citation

  author={Kassner, Moritz and Patera, William and Bulling, Andreas},
  title={Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
  keywords={Eye Movement, Mobile Eye Tracking, Wearable Computing, Gaze-based Interaction},
  archivePrefix = "arXiv",
  eprint        = "1405.0006",
  primaryClass  = "cs-cv",
  url = {}

MIT Thesis


This thesis explores the nature of a human experience in space through a primary inquiry into vision. This inquiry begins by questioning the existing methods and instruments employed to capture and represent a human experience of space. While existing qualitative and quantitative methods and instruments – from “subjective” interviews to “objective” photographic documentation – may lead to insight in the study of a human experience in space, we argue that they are inherently limited with respect to physiological realities. As one moves about the world, one believes to see the world as continuous and fully resolved. However, this is not how human vision is currently understood to function on a physiological level. If we want to understand how humans visually construct a space, then we must examine patterns of visual attention on a physiological level. In order to inquire into patterns of visual attention in three dimensional space, we need to develop new instruments and new methods of representation. The instruments we require, directly address the physiological realities of vision, and the methods of representation seek to situate the human subject within a space of their own construction. In order to achieve this goal we have developed Pupil, a custom set of hardware and software instruments, that capture the subject’s eye movements. Using Pupil, we have conducted a series of trials from proof of concept – demonstrating the capabilities of our instruments – to critical inquiry of the relationship between a human subject and a space. We have developed software to visualize this unique spatial experience, and have posed open questions based on the initial findings of our trials. This thesis aims to contribute to spatial design disciplines, by providing a new way to capture and represent a human experience of space.

(Authors names appear in alphabetical order - equal hierarchy in authorship.)

On MIT DSpace:

BibTeX Style Citation

  title={{PUPIL: Constructing the Space of Visual Attention}},
  author={Kassner, Moritz Philipp and Patera, William Rhoades},
  school={Massachusetts Institute of Technology},

Chicago Style Citation

Moritz Kassner, William Patera, Pupil: Constructing the Space of Visual Attention, SMArchS Master Thesis, (Cambridge: Massachusetts Institute of Technology, 2012).

APA Style Citation

Kassner, M., & Patera, W. (2012). Pupil: Constructing the space of visual attention (Unpublished master’s thesis). Massachusetts Institute of Technology, Cambridge, MA. Available from

You are also welcome to link to our code repositories: Pupil Github Repository