This guide will lead you through a basic workflow using Pupil hardware and software.
Once you have a Pupil Headset all you need to do is install the Pupil apps on a computer running Linux, MacOS, or Windows.
We are always working on new features, fixing bugs, and making improvements. Make sure to visit the release page frequently to download the latest version and follow the Pupil Labs blog for updates.
Capture Workflow
Go through the following steps to get familiar with the Pupil workflow. You can also check out video tutorials at the end of the guide.
1. Put on Pupil
Put on the Pupil headset and plug it in to your computer. Make sure there is space between the headset frame and your forehead. Headsets are adjustable and shipped with additional parts. For more information head over to the Pupil Hardware guide.
2. Start Pupil Capture
3. Check pupil detection
Take a look at the Eye window. If the pupil is detected you will see a red circle around the edge of your pupil and a red dot at the center of your pupil.
If the algorithm’s detection confidence is high, the red circle will be opaque. If confidence diminishes the circle will become more transparent.
Try moving your head around a bit while looking at your eye to see that the pupil is robustly detected in various orientations.
Good and bad eye video
Before calibrating, be sure to check that your eyes are well positioned for a robust eye tracking performance. For more details check out. - Pupil Headset Adjustments
4. Calibrate
In order to know what someone is looking at, we must to establish a mapping between pupil and gaze positions. This is what we call calibration.
The calibration process establishes a mapping from pupil to gaze coordinates.
Screen Marker Calibration Method
Click c on the world screen or press c on the keyboard to start calibrate.
Follow the marker on the screen with your eyes and try to keep your head stationary
There are other calibration methods and lots more information how calibration works in the user guide.
5. Record
Start capturing data!
Pupil Capture will save the world video stream and all corresponding gaze data in a folder in your user directory named recordings.
Start recording: Press the r key on your keyboard or press the circular ‘R’ button in the left hand side of the world window.
The elapsed recording time will appear next to the ‘R’ button.
Stop recording: Press the r key on your keyboard or press the circular ‘R’ button in the left hand side of the world window.
See a video demonstration of how to set recordings path, session name, and start recording – here.
Where is the recording saved?
By default, each recording will live in its own unique data folder contained in the recordings folder.
You can make as many recordings as you like.
The default recordings directory will have the following hierarchy.
recordings
2016-04-05
001
002
003
####
How recordings are saved?
Pupil capture saves the video frames in a fixed frame rate container. This means that the raw output video (world.mp4) does not show the correct duration and the correct frame rate of the recording. This information can be found in world_timestamps.npy, which tells you exactly where each frame belongs in time.
However, if you export using Pupil Player, the video will be made such that the frames will show at the exact right time. The output video will not miss any frame of the raw video, instead, output frames are spaced out exactly as they were initially captured.
Player Workflow
Use Pupil Player to visualize data recorded with Pupil Capture and export videos of visualization and datasets for further analysis.
1. Open Pupil Player
Now that you have recorded some data, you can play back the video and visualize gaze data, marker data, and more.
Visualize
Player comes with a number of plugins. Plugins are classified by their use-case. Visualization plugins can be additive. This means that you can add multiple instances of a plugin to build up a visualization.
Where are Pupil Player exports saved?
Exports are saved within a dedicated folder named exports within the original recording folder.
Each export is contained within a folder within the exports folder. The numbers of the export correlate to the trim marks (frame start and frame end) for the export.
Pupil Capture Demo Video
The video below demonstrates how to setup, calibrate, and make a recording with Pupil Capture.
Turn on closed captions CC to read annotations in the video.
Pupil Player Demo Video
The video below demonstrates how to view a dataset recorded with Pupil Capture, make and export visualizations.
Turn on closed captions CC to read annotations in the video.
Pupil Hardware
Pupil Labs is based in Berlin and ships Pupil eye tracking headsets and VR/AR eye tracking add-ons to individuals, universities, and corporate enterprises worldwide!
Go to the Pupil store for prices, versions, and specs.
Pupil Mobile Eye Tracking Headset
You wear Pupil like a pair of glasses. Pupil connects to a computing device via a USBA or USBC cable. The headset is designed to be lightweight and adjustable in order to accommodate a wide range of users.
To the right is an illustration of a monocular Pupil Headset, depending on your configuration your headset might look different, but working principles are the same.
Pupil ships with a number of additional parts. The below sections provide an overview of their use and a guide to adjusting the Pupil headset.
Additional parts
World Camera
The high speed 2d world camera comes with two lenses. 60 degree FOV lens (shown on the left) and a wide angle 100 degree FOV lens (shown on the right).
The world camera lens are interchangeable, so you can swap between the two lenses provided for normal or wide angle FOV.
Arm Extender
If you need to adjust the eye cameras beyond the built in adjustment range, you can use the orange arm extenders that are shipped with your Pupil headset. Unplug your eye camera. Slide the existing eye camera arm off the headset. Slide the arm extender onto the triangular mount rail on the headset frame. Slide the camera onto the extended mount rail. Plug the camera back in.
The eye camera arm extender works for all existing 120 and 200hz systems.
Nose Pads
All Pupil headsets come with 2 sets of nose pads. You can swap the nose pads to customize the fit.
Pupil Headset Adjustments
A lot of design and engineering thought has gone into getting the ergonomics of the headset just right. It is designed to fit snugly, securely, and comfortably. The headset and cameras can be adjusted to accommodate a wide range of users.
To ensure a robust eye tracking performance, make sure all the cameras are in focus with a good field of view of your eyes.
Slide Eye Camera
The eye camera arm slides in and out of the headset frame. You can slide the eye camera arm along the track.
Rotate World Camera
You can rotate the world camera up and down to align with your FOV.
Rotate Eye Camera
The eye camera arm is connected to the eye camera via the ball joint. You can rotate about its ball joint.
Ball Joint Set Screw
You can adjust the set screw to control the movement of the eye camera about the ball joint. We recommend setting the set screw so that you can still move the eye camera by hand but not so loose that the eye camera moves when moving the head. You can also tighten the set screw to fix the eye camera in place.
Focus Cameras
No focus 200hz Eye Camera
200hz eye cameras do not need to be focused, and can not be focused. The lens of the 200hz eye camera is arrested using glue. Twisting the lens will risk breaking the mount.
Focus 120hz Eye Camera
If you have a 120hz eye camera, make sure the eye camera is in focus. Twist the lens focus ring of the eye camera with your fingers or lens adjuster tool to bring the eye camera into focus.
Focus World Camera
Set the focus for the distance at which you will be calibrating by rotating the camera lens.
HTC Vive Add-On
Add eye tracking powers to your HTC Vive with our 120hz binocular eye tracking add-on.
This section will guide you through all steps needed to turn your HTC Vive into an eye tracking HMD using a Pupil Labs eye tracking add-on.
Install the Vive add-on
Install the Vive PRO add-on
A detailed look
… at the engagement process between the eye tracking ring the the lens holding geometry. Do not follow these steps. Just have a look to get a feeling for the snap-in part to the guide above.
HTC Vive USB connection options
The HTC Vive has one free USB port hidden under the top cover that hides the cable tether connection. This gives us two options to connect the pupil eye tracking add-on:
Connect the add-on to the free htc-vive usb port.
This means the cameras share the VIVEs usb tether bandwidth with other usb components inside the Vive. This works but only if the following rules are observed:
Disable the HTC-Vive built-in camera in the VR settings pane to free up bandwidth for Pupil’s dual VGA120 video streams.
or
Enable the HTC-Vive built-in camera and set it to 30hz. Then set the Pupil Cameras to 320x240 resolution to share the USB bus.
Run a separate USB lane along the tether
If you want full frame rate and resolution for both the Vive’s camera and the add-on you will have to connect the Pupil add-on to a separate usb port on the host PC. We recommend this approach.
USB Connection and Camera IDs
Once you plug the usb cables into your computer:
the right eye camera will show up with the name: Pupil Cam 1 ID0
the left eye camera will show up with the name: Pupil Cam 1 ID1
Focus and Resolutions
After assembly and connection. Fire up Pupil Capture or Service and adjust the focus of the eye cameras by rotating the lens by a few degrees (not revolutions) in the lens housing.
Use 640x480 or 320x240 resolution to get 120fps and a good view of the eye. Other resolutions will crop the eye images.
Interfacing with other software or your own code
Both cameras are fully UVC compliant and will work with OpenCVs video backend, Pupil Capture, and libraries like libucv and pyuvc.
Oculus Rift DK2 Add-On
Add eye tracking powers to your Oculus Rift DK2 with our 120hz eye tracking add-ons.
This page will guide you through all steps needed to turn your Oculus DK2 into an eye tracking HMD using the Pupil Oculus DK2 eye tracking add-on cups.
Install lens in cup
Take the lens out of an existing Oculus lens cup.
Remove the LED ring and insert the lens into the Pupil eye tracking cup.
Install the LED ring and connect the LED power supply.
Install cup in DK2
Route cables
Route the USB cables through the vent holes in the top of the Oculus DK2.
Connect cameras
Connect the eye tracking cup to the USB cable. Remove the old cup and insert the eye tracking cup in the DK2.
USB and Camera IDs
Once you plug the usb cables into your computer:
the right eye camera will show up with the name: Pupil Cam 1 ID0
the left eye camera will show up with the name: Pupil Cam 1 ID1
Both cameras are fully UVC compliant and will work with OpenCVs video backend, Pupil Capture, and libraries like libucv and pyuvc.
HoloLens Add-On
Add eye tracking powers to your HoloLens with our 200hz eye tracking add-ons. Follow the instructions in the video to install your add-on.
DIY
If you are an individual planning on using Pupilexclusively for noncommercial purposes, and are not afraid of SMD soldering and hacking – then, buy the parts, modify the cameras, and assemble a Pupil DIY headset. We have made a guide to help you and a shopping list.
Getting all the parts
The 3d-printed headset is the centerpiece of the Pupil mobile eye tracker. You can buy it from the Pupil Labs team through the Pupil shapeways store. The price for the headset is part production cost and part support to the pupil development team. This enables us to give you support and continue to work on the project.
All other parts of the Pupil DIY kit have been specifically selected with availability and affordability in mind. See the Bill of Materials to learn what else you will need to get.
Tools
You will need access to these tools:
Solder station, wick, flux (for SMD solder work)
Tweezers
Small philips screwdriver
Prying tool to help un-case the webcams
Prepare Webcams
The first step is to modify the cameras so we can use them for eye-tracking.
De-case Cameras
Take both webcams out of their casings. Follow the video guides.
This is by far the trickiest part. You will need some soldering experience, or work with someone that can help you for this step. In the video and photo the lens holder is removed, but you will do it with the lens holder attached.
Cut off the microphone
De-solder or break off the push button (Note: Some cameras don’t have this button.)
De-solder the blue LED’s
solder on the IR-LED’s. Please take note of LED polarity! video
Replace IR-blocking Filter on the Eye Camera
Unscrew the lens from the mount.
Carefully remove the IR filter. Be very careful! The IR filter is a thin piece of coated glass and right behind it is a lens element that must stay intact and unharmed! It is necessary to remove the IR filter, so that the image sensor will be able to “see” the IR light.
Using a hole punch, cut out 1 round piece of exposed film and put it where the older filter was.
Use plastic glue to fix the piece. Don’t let the glue touch the center!
Put the lens back inside. You will have to manually focus the lens when you run the software for the first time by hand. Later you can use the focus control in software to fine tune.
The camera mounts can be replaced by custom build parts that suit your specific camera setup or other sensors.
Windows Driver Installation
If you are using Windows, you will need to install drivers for your cameras. Please refer to the instructions here.
Pupil Hardware Development
This page contains documentation and discussion on open source camera mounts, optics, and cameras.
Camera Mounts
We release the CAD files for the camera mounts for you to download, modify, in accordance with our license. CAD files for the frame are not open source; see explanation.
Interface Documentation
By releasing the mounts as example geometry we automatically document the interface. You can use the CAD files to take measurements and make your own mounts.
Compatibility
The mounts were developed as part of the whole headset and carry the revision number of the headset they were designed for.
Download Camera Mount CAD Files
All files are hosted in the pupil-hardware-diy repo here
This section of the documentation is targeted towards users of Pupil software and provides a deeper explanation of features and methods.
Pupil Capture
Pupil Capture is the software used with the Pupil Headset. The software reads the video streams coming in from the world camera and the eye camera. Pupil Capture uses the video streams to detect your pupil, track your gaze, detect and track markers in your environment, record video and events, and stream data in realtime.
Capture Window
The Capture window is the main control center for Pupil Capture. It displays live video feed from pupil headset.
Graphs - This area contains performance graphs. By default the graphs CPU, FPS, and pupil algorithm detection confidence will be displayed. You can control graph settings with the System Graphs plugin.
Hot keys - This area contains clickable buttons for plugins.
Menu - This area contains settings and contextual information for each plugin.
Sidebar - This area contains clickable buttons for each plugin. System plugins are loaded in the top and user added plugins are added below the horizontal separator.
Capture Selection
By default Pupil Capture will use Local USB as the capture source. If you have a Pupil headset connected to your machine you will see the video displayed from your Pupil headset in the World and eye windows. If no headset is connected or Pupil Capture is unable to open capture devices it will fall back to the Test Image. Other options for capture source are described below.
Test Image - This is the fallback behavior if no capture device is found, or if you do not want to connect to any capture device.
Video File Source - select this option to use previously recorded videos for the capture selection.
Pupil Mobile - select this option when using Pupil Capture with the Pupil Mobile Android application.
Local USB - select this option if your Pupil Headset is connected to the machine running Pupil Capture. This is the default setting.
RealSense 3D - select this option if you are using an Intel RealSense 3D camera as your scene camera. Read more in the RealSense 3D section.
Pupil Detection
Pupil’s algorithms automatically detect the participant’s pupil. With the 3d detection and mapping mode, Pupil uses a 3d model of the eye(s) that constantly updates based on observations of the eye. This enables the system to compensate for movements of the headset - slippage. To build up an initial model, you can just look around your field of view.
Calibration
Pupil uses two cameras. One camera records a subject’s eye movements – we call this the eye camera. Another camera records the subject’s field of vision – we call this the world camera. In order to know what someone is looking at, we must find the parameters to a function that correlates these two streams of information.
Calibration Process
Monocular
Binocular
Pupil Headset comes in a variety of configurations. Calibration can be conducted with a monocular or binocular eye camera setup.
Before every calibration
Make sure that the users pupil is properly tracked. Make sure that the world camera is in focus for the distance at which you want to calibrate, and that you can see the entire area you want to calibrate within the world cameras extents (FOV).
Your pupil is properly detected by the eye camera
Make sure the world camera is in focus
Calibration Methods
Before starting calibration, ensure that eye(s) are robustly detected and that the headset is comfortable for the participant.
Screen Marker Calibration
This is the default method, and a quick method to get started. It is best suited for close range eye-tracking in a narrow field of view.
Select Screen Marker Calibration
Select your Monitor (if more than 1 monitor)
Toggle Use fullscreen to use the entire extents of your monitor (recommended). You can adjust the scale of the pattern for a larger or smaller calibration target.
Press c on your keyboard or click the blue circular C button in the left hand side of the world window to start calibration.
Follow the marker on the screen with your eyes. Try to keep your head still during calibration.
The calibration window will close when calibration is complete.
In the Advanced sub-menu you can set the sample duration – the number of frames to sample the eye and marker position. You can also set parameters that are used to debug and detect the circular marker on the screen.
Manual Marker Calibration
Pupil Calibration Marker v0.4
Pupil Calibration Stop Marker v0.4
This method is done with an operator and a subject. It is suited for midrange distances and can accommodate a wide field of view. The operator will use a printed calibration marker like the one shown in the video. Download Pupil Labs Calibration Marker v0.4 to print or display on smartphone/tablet screen.
Select Manual Marker Calibration
Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
Stand in front of the subject (the person wearing the Pupil headset) at the distance you would like to calibrate. (1.5-2m)
Ask the subject to follow the marker with their eyes and hold their head still.
Show the marker to the subject and hold the marker still. You will hear a “click” sound when data sampling starts, and one second later a “tick” sound when data sampling stops.
Move the marker to the next location and hold the marker still.
Repeat until you have covered the subject’s field of view (generally about 9 points should suffice).
Show the ‘stop marker’ or press c on your keyboard or click the blue circular C button in the left hand side of the world window to stop calibration.
You will notice that there are no standard controls, only an Advanced sub-menu to control detection parameters of the marker and to debug by showing edges of the detected marker in the world view.
Single Marker Calibration
Calibrate using a single marker displayed on screen or hand held marker. Gaze at the center of the marker and move your head in a spiral motion. You can also move your head in other patterns. This calibration method enables you to quickly sample a wide range of gaze angles and cover a large range of your FOV.
Select Single Marker Calibration
Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
Look at the center of the marker.
Slowly move your head while gazing at the center of the marker. We have found that a spiral pattern is an efficient way to cover a large area of the FOV.
Press the C button on your keyboard or show the stop marker to stop calibrating.
Natural Features Calibration
This method is for special situations and far distances. Usually not required.
Select Natural Features Calibration
Press c on your keyboard or click the blue circular C button in the left hand side of the world window to start calibration.
Ask the subject (the person wearing the Pupil headset) to look a point within their field of vision. Note – pick a salient feature in the environment.
Click on that point in the world window.
Data will be sampled.
Repeat until you have covered the subject’s field of view (generally about 9 points should suffice)
Press c on your keyboard or click the blue circular C button in the left hand side of the world window to stop calibration.
Fingertip Calibration
Calibrate using your fingertip! We have found that the easiest way to calibrate with your fingertip is as follows:
Select Fingertip Calibration
Press c on your keyboard or click the blue circular C button on the left hand side of the world window to start calibration.
Extend your arm and hold your index finger still at the center of the field of view of the world camera.
Move your head for example horizontally and then vertically while gazing at your index finger fingernail.
Show five fingers or press c on your keyboard or click the blue circular C button on the left hand side of the world window to stop calibration.
This calibration method enables you to quickly sample a wide range of gaze angles and cover a large range of your FOV within 10 seconds.
A Convolutional neural network (CNN) is implemented for the fingertip detection:
First, a hand detector, based on MobileNet and SSD, searches for a hand in the image.
The position of the fingertip is then found out by a fingertip detector, adapted from YOLSE and Unet.
Notes on calibration accuracy
In 2D mode, you should easily be able to achieve tracking accuracy within the physiological limits (sub 1 deg visual degrees). Using the 3d mode you should achieve 1.5-2.5 deg of accuracy.
Any monocular calibration is accurate only at its depth level relative to the eye (parallax error).
Any calibration is only accurate inside the field of view (in the world video) you have calibrated. For example: If during your calibration you only looked at markers or natural features (depending on your calibration method) that are in the left half, you will not have good accuracy in the right half.
Calibration accuracy can be visualized with the Accuracy Visualizer plugin. If the Accuracy Visualizer plugin is loaded, it will display the residual between reference points and matching gaze positions that were recorded during calibration.
Gaze Prediction Accuracy can be estimated with an accuracy test. Start the accuracy by running a normal calibration procedure but press the T button in the world window and not the C button. After completing the test, the plugin will display the error between reference points and matching gaze positions that were recorded during the accuracy test.
Recording
Press r on your keyboard or press the blue circular R button on the left hand side of the world window to start recording. You will see red text with the elapsed time of recording next to the R button. To stop recording, press r on your keyboard or press the R button on screen.
You can set the folder or Path to recordings and the Recording session name in the Recorder sub-menu within the GUI. Note - you must specify an existing folder, otherwise the Path to recordings will revert to the default path.
What will be in the session folder?
If you open up a session folder you will see a collection of video(s) and data files. Take a look at Data format to see exactly what you get.
Open a plugin
Open the Plugin Manager menu on the right. It lists all available plugins. Click the button next to the plugin’s name to turn on or off the plugin.
Third-party plugins
You can easily load third party plugins. Third party plugins will appear in the Pupil Capture or Pupil Player plugin list. Copy the plugin to the plugins folder within the pupil_capture_settings or pupil_player_settings folder.
Fixation Detector
The online fixation detector classifies fixations based on the dispersion-duration principle. Fixations are used by the screen and manual marker calibrations to speed up the procedure. A fixation is visualized as a yellow circle around the gaze point that is shown in the Pupil Capture world window.
Pupil Capture has a built-in data broadcast functionality. It is based on the network library ZeroMQ and follows the PUB-SUB pattern. Data is published with an affiliated topic. Clients need to subscribe to their topic of interest to receive the respective data. To reduce network traffic, only data with at least one subscription is transferred.
Pupil Remote
Pupil Remote is the plugin that functions as the entry point to the broadcast infrastructure. It also provides a high level interface to control Pupil Capture over the network (e.g. start/stop a recording).
Load the Pupil Remote plugin from the General sub-menu in the GUI (it is loaded by default).
It will automatically open a network port at the default Address.
Change the address and port as desired.
If you want to change the address, just type in the address after the tcp://
Pupil Groups
Pupil Groups can help you to collect data from different devices and control an experiment with multiple actors (data generators and sensors) or use more than one Pupil device simultaneously:
Load the Pupil Groups plugin from the General sub-menu in the GUI.
Once the plugin is active it will show all other local network Pupil Group nodes in the GUI
Furthermore, actions like starting and stopping a recording on one device will be mirrored instantly on all other devices.
Pupil Time Sync
If you want to record data from multiple sensors (e.g. multiple Pupil Capture instances) with different sampling rates it is important to synchronize the clock of each sensor. You will not be able to reliably correlate the data without the synchronization.
The Pupil Time Sync protocol defines how multiple nodes can find a common clock master and synchronize their time with it.
The Pupil Time Sync plugin is able to act as clock master as well as clock follower. This means that each Pupil Capture instance can act as a clock reference for others as well as changing its own clock such that it is synchronized with another reference clock.
Pupil Time Sync nodes only synchronize time within their respective group. Be aware that each node has to implement the same protocol version to be able to talk to each other.
Frame Publisher
The Frame Publisher plugin broadcasts video frames from the world and eye cameras.
The Pupil Mobile app can be controlled via Pupil Capture when connected. This includes changing camera and streaming settings. The Remote Recorder plugin extends this list with the possibility to start and stop recordings that are stored in the phone.
Surface Tracking
The Surface Tracker plugin allows you to define surfaces within your environment and track surfaces in realtime using a 5x5 square marker. We were greatly inspired by the ArUco marker tracking library.
Markers - We use a 5x5 square marker. This is not the same marker that is used by ArUco (they use 7x7).
Using a 5x5 marker gives us 63 unique markers.
Why the 5x5 grid? The 5x5 grid allows us to make smaller markers that can still be detected. Markers can be printed on paper, stickers, or displayed on the screen.
Defining Surfaces with Markers
A surface can be defined by one or more markers. Surfaces can be defined with Pupil Capture in real-time, or offline with Pupil Player. Below we provide an outline of steps.
Define surfaces within your environment using one or more fiducial markers. Surfaces can be defined with a minimum of one marker. The maximum number of markers per surface is limited by the number of markers we can produce with a 5x5 grid.
Use Pupil Capture or Pupil Player to register surfaces, name them, and edit them.
Registered surfaces are saved automatically, so that the next time you run Pupil Capture or Pupil Player, your surfaces (if they can be seen) will appear when you start the marker tracking plugin.
Surfaces defined with more than 2 markers are detected even if some markers go outside the field of vision or are obscured.
We have created a window that shows registered surfaces within the world view and the gaze positions that occur within those surfaces in realtime.
Streaming Surfaces with Pupil Capture - Detected surfaces as well as gaze positions relative to the surface are broadcast under the surface topic. Check out this video for a demonstration.
Surface Metrics with Pupil Player - if you have defined surfaces, you can generate surface visibility reports or gaze count per surface. See our blog post for more information.
Generate markers with this script, or download the image.
Moving Surface Definitions
The surfaces are saved in a file is called surface_definitions. This file is in every pupil_capture_settings directory and in every recording directory. You can copy&paste this file to move definitions from one session or recording to another.
Surface Heatmaps
It is possbile to display gaze heatmaps for each surface by selecting Show Heatmaps mode in the Surface Tracker menu. The surface is divided into a two dimensional grid to calculate the heatmap. The grid can be refined by setting the X and Y size field of each surface. The colors are normalized such that cells with the highest amount of gaze are displayed in red and those with the least gaze are displayed in blue. The Gaze History Length option for each surface decides how many recent gaze positions will be used to calculate the heatmap.
Blink Detection
The pupil detection algorithm assigns a confidence value to each pupil datum. It represents the quality of the detection result. While the eye is closed the assigned confidence is very low. The Blink Detection plugin makes use of this fact by defining a blink onset as a significant confidence drop - or a blink offset as a significant confidence gain - within a short period of time. The plugin creates a blink event for each event loop iteration in the following format:
The Filter length is the time window’s length in which the plugin tries to find such confidence drops and gains. The plugin fires the above events if the blink confidence within the current time window exceeds the onset or offset confidence threshold. Setting both thresholds to 0 will always trigger blink events, even if the confidence is very low. This means that onsets and offsets do not appear necessarily as pairs but in waves.
Audio Capture
The Audio Capture plugin provides access to a selected audio source for other plugins and writes its output to the audio.mp4 file during a recording. It also writes the Pupil Capture timestamp for each audio packet to the audio_timestamps.npy file. This way you can easily correlate single audio packets to their corresponding video frames.
Audio is recorded separately from the video in Pupil Capture. You can play back audio in sync with video in Pupil Player. Audio is automatically merged with the video when you export a video using Pupil Player.
Annotations
The Annotation Capture plugin allows you to mark timestamps with a label – sometimes referred to as triggers. These labels can be created by pressing their respective hotkey or by sending a notification with the subject annotation. This is useful to mark external events (e.g. “start of condition A”) within the Pupil recording. The Annotation Player plugin is able to correlate and export these events as well as add new ones.
Remote Annotations
You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation notification.
{'subject':"annotation",'label':"Hi this is my annotation 1",'timestamp':[setacorrecttimestampasfloathere],'duration':1.0,'source':'a test script','record':True}
Camera Intrinsics Estimation
This plugin is used to calculate camera intrinsics, which will enable one to correct camera distortion. Pupil Capture has built in, default camera intrinsics models for the high speed world camera and the high resolution world camera. You can re-calibrate your camera and/or calibrate a camera that is not supplied by Pupil Labs by running this calibration routine. We support two different distortion models, radial distortion and fisheye distortion. For cameras with a FOV of 100 degrees or greater (like e.g. the high speed world camera) the fisheye distortion model usually performs better, for cameras with a smaller FOV (e.g. the high resolution world camera) we recommend the radial distortion model.
Select Camera Intrinsics Estimation
Select the correct ‘Distortion Model’
Click on ‘show pattern’ to display the pattern
Resize the pattern to fill the screen
Hold your Pupil headset and aim it at the pattern.
With the world window in focus, press c on your keyboard or the circular C button in the world windows to detect and capture a pattern.
Data will be sampled and displayed on the screen as a border of the calibrated pattern. (Note - make sure to move your headset at different angles and try to cover the entire FOV of the world camera for best possible calibration results)
Repeat until you have captured 10 patterns.
Click on show undistorted image to display the results of camera intrinsic estimation. This will display an undistorted view of your scene. If well calibrated, straight lines in the real world will appear as straight lines in the undistorted view.
Pupil Player
Pupil Player is the second tool you will use after Pupil Capture. It is a media and data visualizer at its core. You will use it to look at Pupil Capture recordings. Visualize your data and export it.
Features like surface tracking found in Pupil Capture are also available in Pupil Player.
Starting Pupil Player
Drag the recording directory (the triple digit one) directly onto the app icon or launch the application and drag + drop the recording directory into Pupil Player window.
Player Window
Let’s get familiar with the Player window.
The Player window is the main control center for Pupil Player. It displays the recorded video feed from pupil capture file.
Graphs - This area contains performance graphs. By default the graphs CPU, FPS, and pupil algorithm detection confidence will be displayed. You can control graph settings with the System Graphs plugin.
Hot keys - This area contains clickable buttons for plugins.
Timeline Events - Plugins can add temporal events to this expandable panel.
Timeline - Control the playback of the video with the play/pause button (or spacebar on your keyboard). Drag the playhead (vertical line) to the desired point in time.
Frame Stepping - You can use the arrow keys on your keyboard or the <<>> buttons to advance one frame at a time.
Trimming - Drag either end of the timeline to set a trim beginning and ending trim marks. The trim section marks directly inform the section of video/data to export.
Menu - This area contains settings and contextual information for each plugin.
Sidebar - This area contains clickable buttons for each plugin. System plugins are loaded in the top and user added plugins are added below the horizontal separator.
Workflow
Pupil Player is similar to a video player. You can playback recordings and can load plugins to build visualizations.
Here is an example workflow:
Start Pupil Player
Open a Plugin - From the Plugin Manager GUI menu load the Vis Circle plugin.
Playback - press the play button or space bar on your keyboard to view the video playback with visualization overlay, or drag the playhead in the seek bar to scrub through the dataset.
Set trim marks - you can drag the green rounded rectangle at the beginning and end of the seekbar to set the trim marks. This will set the start and end frame for the exporter and for other plugins.
Export Video & Raw Data - From the Plugin Manager view, load the Video Export Launcher plugin and the Raw Data Exporter plugin. Press e on your keyboard or the e button in the left hand side of the window to start the export.
Check out exported data in the exports directory within your recording directory
Plugin Overview
Pupil Player uses the same Plugin framework found in Pupil Capture to add functionality.
We implement all visualizations, marker tracking, and the exporter using this structure. Very little work (often no work) needs to be done to make a Capture Plugin work for the Pupil Player and vice versa.
There are two general types of plugins:
Unique - You can only launch one instance of this plugin.
Not unique - You can launch multiple instances of this type of plugin. For example, you can load one Vis Circle plugin to render the gaze position with a translucent green circle, and another Vis Circle plugin to render the gaze circle with a green stroke of 3 pixel thickness. You can think of these types of plugins as additive.
In the following sections we provide a summary of plugins currently available and in Pupil Player.
Visualization Plugins
We will call plugins with the Vis prefix visualization plugins. These plugins are simple plugins, are mostly additive (or not unique), and directly operate on the gaze positions to produce visualizations. Other plugins like Offline Surface Tracker also produces visualizations, but will be discussed elsewhere due to the extent of its features.
Vis Circle
Visualize the gaze positions with a circle for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization.
You can set the following parameters:
radius - the radius of the circle around the gaze point.
stroke width - the thickness or width of the stoke in pixels.
fill - toggle on for a circle with solid fill. Toggle off for a circle with only stroke.
color - define the red, green, blue values for color. Alpha defines the opacity of the stroke and fill.
Here we show an example of how you could use 2 instances of the Vis Circle Plugin. The first instance renders the gaze position as a filled yellow circle. The second instance renders the same gaze position as an orange stroke circle.
Vis Cross
Visualize the gaze positions with a cross for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:
inner offset length - the distance in pixels to offset the interior cross endpoints from the gaze position. A value of 0 will make the crosshairs intersect the gaze position.
outer length - The length of the cross lines in pixels from the gaze position. Note - equal values of inner offset length and outer length will result in a cross with no length, and therefore not rendered.
stroke width - the thickness or width of the stoke in pixels.
color - define the red, green, blue values for color.
Here we show an example of how you could use 2 instances of the Vis Cross Plugin. The first instance renders the gaze position as a red cross with that extends to the boundaries of the screen. The second instance renders the gaze position as a green cross, with a heavier stroke weight.
Vis Scan Path
This plugin enables past gaze positions to stay visible for the duration of time specified by the user. This plugin is unique, therefore you can only load one instance of this plugin.
On its own, Scan Path does not render anything to the screen. It is designed to be used with other plugins. In some cases, it is even required to be enabled in order for other plugins to properly function. When used with Vis plugins (like Vis Circle, Vis Cross, Vis Polyline, or Vis Light Points) Scan Path will enable you to see both the current gaze positions and the past gaze positions for the specified duration of time.
Here we show an example of Scan Path set with 0.4 seconds duration used with Vis Circle. Each green circle is a gaze position within the last 0.4 seconds of the recording.
Vis Polyline
Visualize the gaze positions with a polyline for each gaze position. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:
line thickness - the thickness or width of the polyline stroke in pixels.
color - define the red, green, blue values for color.
An example showing Vis Polyline used with Vis Circle and Scan Path. The polyline enables one to visualize the sequence of the gaze positions over the duration specified by Scan Path.
Vis Light Points
Visualize the gaze positions as a point of light for each gaze position. The falloff of the light from the gaze position is specified by the user. This plugin is not unique, therefore you can add multiple instances of the plugin to build your visualization. You can set the following parameters:
falloff - The distance (in pixels) at which the light begins to fall off (fade to black). A very low number will result in a very dark visualization with tiny white light points. A very large number will result in a visualization of the world view with little or no emphasis on the gaze positions.
Here is an example demonstrating Vis Light Points with a falloff of 73.
Vis Eye Video Overlay
Here is an example of the Eye Video Overlay with binocular eye videos.
This plugin can be used to overlay the eye video on top of the world video. Note that the eye video is not recorded by default in Pupil Capture, so if you want to use this plugin, make sure to check record eye video in Pupil Capture. This plugin is unique, therefore you can only load one instance of this plugin.
You can set the following parameters:
opacity - the opacity of the overlay eye video image. 1.0 is opaque and 0.0 is transparent.
video scale - use the slider to increase or decrease the size of the eye videos.
move overlay - toggle on and then click and drag eye video to move around in the player window. Toggle off when done moving the video frames.
show - show or hide eye video overlays.
horiz. and vert. flip - flip eye videos vertically or horizontally
Pupil Data And Post-hoc Detection
By default, Player starts with the Pupil From Recording plugin that tries to load pupil positions that were detected and stored during a Pupil Capture recording. Alternatively, one can run the pupil detection post-hoc.
Offline Pupil Detector
The Offline Pupil Detector plugin can be used with any dataset where eye videos were recorded. The plugin tries to load the eye videos, and runs the pupil detection algorithm in separate processes. This plugin is especially relevant for recordings made with Pupil Mobile, because Pupil Mobile does not perform any pupil detection or gaze estimation on the Android device. This plugin is available starting with Pupil Player v0.9.13.
The Detection Method selector sets the detection algorithm to either 2d or 3d detection (see the section on Pupil Detection for details). The Redetect button restarts the detection procedure. You can use the Offline Pupil Detector plugin to debug, improve, and gain insight into the pupil detection process.
Gaze Data And Post-hoc Calibration
By default, Player starts with the Gaze From Recording plugin that tries to load gaze positions that were detected and stored during a Pupil Capture recording. Alternatively, one can run the gaze mapping process post-hoc.
Offline Calibration
The Offline Calibration plugin enables you to calibrate, map, and validate gaze post-hoc and is available starting with Pupil Player v0.9.13. It can be used on any Pupil dataset.
The workflow is separated into three steps, each with its own submenu: Reference Locations, Calibrations, and Gaze Mappers.
Reference locations are points within the recorded world video that are known to have been fixated on by the participant/subject. They can either be automatically detected or manually annotated:
Detect Circle Markers in Recording: This button starts the automatic detection of circular calibration markers within the world video. The progress is visualized in the plugin’s timeline.
Manual Edit Mode: When this option is enabled, you can add new locations as well as correct and delete existing ones. There can only be one location per world frame.
As in Capture, one can have more than one calibration per recording. A calibration on its own does not result in gaze data. Rather, it contains the required parameters to map pupil to gaze data. Each has the following properties:
Name: Used to correctly select a calibration for each gaze mapper (see below).
Mapping Method: 2d uses polynomial regression, or 3d uses bundle adjustment calibration.
Reference Range: Time range that indicates which reference locations to use.
Calibrations are stored as plcal files in the recording’s calibration subfolder. You can copy and apply them to recordings that do not include reference locations. See the instructional video below for details.
In order to apply a calibration to pupil data, one needs a gaze mapper. Each gaze mapper has the following properties:
Calibration: One of the previously created or imported calibrations (see screencast)
Mapping Range: Time range in which pupil data will be mapped to gaze data.
Manual Correction: Apply a fixed offset to your gaze mapping.
Validation: You can validate the accuracy and precision of the mapped gaze by comparing it to reference locations in the selected Validation Range. It uses the same methodology as the Accuracy Visualizer.
Offline (post-hoc) Pupil Detection and Gaze Mapping
Offline (post-hoc) Gaze Mapping With Manual Reference Locations
Use Offline (post-hoc) Calibration For Another Recording
Offline (post-hoc) Gaze Mapping Validation
Analysis Plugins
These plugins are simple unique plugins, that operate on the gaze data for analysis and visualizations.
Offline Surface Tracker
This plugin is an offline version of the Surface Tracking plugin for Pupil Capture. You can use this plugin to detect markers in the recording, define surfaces, edit surfaces, and create and export visualizations of gaze data within the defined surfaces.
Here is an example workflow for using the Offline Surface Detector plugin to generate heatmap visualizations and export surface data reports:
Load Offline Surface Detector plugin - if you already have surfaces defined, the load may take a few seconds because the plugin will look through the entire video and cache the detected surfaces.
Add surface - if you do not have any defined surfaces, you can click on the Add surface button when the markers you want to user are visible or just click the circular A button in the left hand side of the screen.
Surface name and size - In the Marker Detector GUI window, define the surface name and real world size. Note - defining size is important as it will affect how heatmaps are rendered.
Set trim marks - optional, but if you want to export data for a specific range, then you should set the trim marks.
Recalculate gaze distributions - click the (Re)calculate gaze distributions button after specifying surface sizes. You should now see heatmaps in the Player window (if gaze positions were within your defined surfaces).
Export gaze and surface data - click e and all surface metrics reports will be exported and saved for your trim section within your export folder.
All files generated by the Offline Surface Detector will be located in the subfolder surfaces. The different reported metrics are:
surface_visibility.csv - Overview of how many world camera frames each surface was contained in.
surface_gaze_distribution.csv - Overview of how many gaze samples have been collected on each individual surface and outside of surfaces.
surface_events.csv - List of image-enter and image-exit events for all surfaces.
Further the following metrics are reported for every individual surface. Each surface has a name, which can be manually set as described above. This name is augmented by an automatically generated numerical identifier.
heatmap_<surface_name>_<surface_id>.png - Heatmap of gaze positions on the surface aggregated over the entire export.
fixations_on_surface_<surface_name>_<surface_id>.csv - A list of fixations that have occurred on the surface.
gaze_positions_on_surface_<surface_name>_<surface_id>.csv - A list of gaze datums on the surface. The values include the gaze point in two different coordinates systems. x_norm and y_norm are coordinates between 0 and 1, where (0,0) is the bottom left corner of the surface and (1,1) is the top right corner. x_scaled and y_scaled contain the same coordinates but scaled with the size defined for the surface.
srf_positons_<surface_name>_<surface_id> - List of surface positions in 3D. The position is given as the 3D pose of the surface in relation to the current position of the scene camera. m_to_screen is a matrix transforming coordinates from the camera coordinate system to the surface coordinate system. m_from_screen is the inverse of m_to_screen.
Fixation Detector
The offline fixation detector calculates fixations for the whole recording. The menu gives feedback about the progress of the detection, how many fixations were found, shows and detailed information about the current fixation. Press f or click the f hot key button on the left hand side of the window to seek forward to the next fixation.
Toggle Show fixations to show a visualization of fixations. The blue number is the number of the fixation (0 being the first fixation). You can export fixation reports for your current trim section by pressing e on your keyboard or the e hot key button on the left hand side of the window.
You can export data and videos by pressing e on your keyboard or the e hot key button in the Pupil Player window.
All open plugins that have export capability will export when you press e. Exports are separated from your raw data and contained in the exports sub-directory. The exports directory lives within your recording directory.
Active video exporters will run in the background and you can see the progress bar of the export in the GUI. While exporting, you can continue working with Pupil Player and even launch new exports. Each video export creates at least one mp4 and its respective file timestamp file. See the Data Format section for details.
Export Directory
Every export creates a new folder within the exports sub-directory of your recording. All data from the export is saved to this folder.
Export Handling
You can select the frame range to export by setting trim marks in the seek bar or directly in the General Settings menu.
Longer running exports, e.g. video exports, go through three phases: Queued, Running, and Completed. Export tasks can be cancelled while being queued or running. Completed tasks are kept in the list for reference.
World Video Exporter
The World Video Exporter is loaded by default.
The export saves the world video as shown in Player, including all currently active visualizations (see the [Visualization Plugins](#visualization-plugins) section).
Eye Video Exporter
The Eye Video Exporter needs to be loaded explicitly through the Plugin Manager. It includes the option to render the 2d pupil detection result into the exported video.
Pupil Service is like Pupil Capture except it does not have a world video feed or GUI. It is intended to be used with VR and AR eye tracking setups.
Pupil Service is designed to run in the background and to be controlled via network commands only. The service process has no GUI. The tools introduced in the hmd-eyes project are made to work with Pupil Service and Pupil Capture alike.
Talking to Pupil Service
Code examples below demonstrate how to control Pupil Service over the network.
Starting and stopping Pupil Service:
importzmq,msgpack,timectx=zmq.Context()#create a zmq REQ socket to talk to Pupil Service/Capturereq=ctx.socket(zmq.REQ)req.connect('tcp://localhost:50020')#convenience functionsdefsend_recv_notification(n):# REQ REP requirese lock step communication with multipart msg (topic,msgpack_encoded dict)req.send_multipart(('notify.%s'%n['subject'],msgpack.dumps(n)))returnreq.recv()defget_pupil_timestamp():req.send('t')#see Pupil Remote Plugin for detailsreturnfloat(req.recv())# set start eye windowsn={'subject':'eye_process.should_start.0','eye_id':0,'args':{}}print(send_recv_notification(n))n={'subject':'eye_process.should_start.1','eye_id':1,'args':{}}print(send_recv_notification(n))time.sleep(2)# set calibration method to hmd calibrationn={'subject':'start_plugin','name':'HMD_Calibration','args':{}}print(send_recv_notification(n))time.sleep(2)# set calibration method to hmd calibrationn={'subject':'service_process.should_stop'}print(send_recv_notification(n))
Notifications
The code demonstrates how you can listen to all notification from Pupil Service. This requires a little helper script called zmq_tools.py.
fromzmq_toolsimport*ctx=zmq.Context()requester=ctx.socket(zmq.REQ)requester.connect('tcp://localhost:50020')#change ip if using remote machinerequester.send('SUB_PORT')ipc_sub_port=requester.recv()monitor=Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',))#change ip if using remote machinewhileTrue:print(monitor.recv())
Pupil Mobile is a companion app to Pupil Capture and Pupil Service. It is currently in public beta.
Introducing Pupil Mobile
Pupil Mobile enables you to connect your Pupil eye tracking headset to your Android device via USBC. You can preview video and other sensor streams on the Android device and stream video data over a WiFi network to other computers (clients) running Pupil Capture. Seamlessly integrated with Pupil Capture and Pupil Service.
Home Screen
Home screen is the main control center for Pupil Mobile. It displays a list of available sensors. Click any sensor for a preview.
Sensors - This area contains available sensors. Pupil headset cameras along with other sensors connected to or built into the Android device like audio and IMU.
Record - Click the Record button to save video and other sensor data locally on the Android device.
General Settings - Main settings menu for Pupil Mobile app.
Sensor Preview
Preview live video feed from your Pupil headset and other available sensors. Sensor preview windows will automatically close and take you back to the Home screen in order to conserve battery.
Sensor settings - Settings for the sensor. For cameras you can set frame rate, exposure, white balance, and many more parameters.
Sensor name and recording status - This displays the sensor name and a dot displaying the recording status of this sensor.
Preview stream - A preview of sensor data.
General Settings
Main settings menu for Pupil Mobile app and information about the Android Device.
Device name - Text input field to name your device. This is the device name that will appear in Pupil Capture.
Close/Quit app - Press this button to close the app. Pupil Mobile runs a service in the background. This enables the app to continue running even when your screen is off. Therefore, just swiping away the app view will not close the app.
Save directory - Select the location where recordings should be saved. By default recordings are saved on Android’s built in storage. You can also save to an SD card, if available.
Recordings Screen
View all the datasets that were recorded on your Android device.
Recording folder - A directory containing all of your recordings.
Delete - Permanently delete recording files from the device.
See the Transfer Recordings section below on how to transfer recordings from your phone to your computer.
Switch Views
Pupil Mobile is designed to for efficient navigation. Swipe left or right for quick access to other views.
Swipe - Swipe left or right to switch between views. Swipe right from the home screen to go to the recording view. Swipe left from the home screen to the sensor preview views.
Streaming To Subscribers
Pupil Mobile devices on the same local WiFi network are automatically detected by Pupil Capture. To subscribe to a Pupil Mobile device in Pupil Capture, go to Capture Selection and select Pupil Mobile as the capture source.
WiFi Bandwidth & Network
Make sure you have a good WiFi newtork connection and that it’s not saturated. The quality of your WiFi connection will affect the sensor streams to your subscribers.
NDSI Communication Protocol
The communication protocol is named NDSI, it is completely open. A reference client for Python exsits here.
Transfer Recordings
You will have to manually transfer recordings to your computer to open them in Player. There are two different methods to archive that:
SD card File Transfer
This method requires
that you set the Save directory to SD card in the main settings
a SD card reader that can be connected to your computer
Once you finished the recording, remove the sd card from your phone, insert it into the SD card reader, and connect it to your computer. The SD card should show up on your computer as if it was a USB stick.
You can find all recodings in the Pupil Mobile folder on the SD card. Copy the recording folders to a directory of your choice on your computer. Afterwards, you can remove the SD card from the computer and open the copied recordings using Pupil Player.
Direct Phone Connection
Connect your phone via USB to your computer.
A notification on your phone should pop up: Android System - USB charging this device.
Double tap the notification to open the options.
Select Use USB to transfer files.
Your phone should show up on your computer as if it was an USB stick.
The recordings are saved in two different locations depending on the Save directory settings chosen before the recording:
Default: Internal storage/Movies/Pupil Mobile
SD Card: SD card/Pupil Mobile
Copy the recording folders to a directory of your choice on your computer.
Please check out existing issues or open a new issue at Pupil Mobile repository. This app is in Alpha state, help us make it better.
Experiments
I want to use this for my experiments in the field
Feel free to do so, but do not rely on the app to work all the time! Many features and environments are still untested. If you have trouble, please open an issue.
Data Format
The data format for Pupil recordings is 100% open. In this section we will first describe how we handle high level concepts like coordinate systems, timestamps and synchronization and then describe in detail the data format present in Pupil recordings.
Coordinate Systems
We use a normalized coordinate system with the origin 0,0 at the bottom left and 1,1 at the top right.
Normalized Space
Origin 0,0 at the bottom left and 1,1 at the top right. This is the OpenGL convention and what we find to be an intuitive representation. This is the coordinate system we use most in Pupil. Vectors in this coordinate system are specified by a norm prefix or suffix in their variable name.
Image Coordinate System
In some rare cases we use the image coordinate system. This is mainly for pixel access of the image arrays. Here a unit is one pixel, origin is “top left” and “bottom right” is the maximum x,y.
Timestamps
All indexed data - still frames from the world camera, still frames from the eye camera(s), gaze coordinate, and pupil coordinates, etc. - have timestamps associated for synchronization purposes. The timestamp is derived from CLOCK_MONOTONIC on Linux and MacOS.
The time at which the clock starts counting is called PUPIL EPOCH. In pupil the epoch is adjustable through Pupil Remote and Pupil Timesync.
Timestamps are recorded for each sensor separately. Eye and World cameras may be capturing at very different rates (e.g. 120hz eye camera and 30hz world camera), and correlation of eye and world (and other sensors) can be done after the fact by using the timestamps. For more information on this see Synchronization below.
Observations:
Timestamps in seconds since PUPIL EPOCH.
PUPIL EPOCH is usually the time since last boot.
In UNIX like, PUPIL EPOCH is usually not the Unix Epoch (00:00:00 UTC on 1 January 1970).
More information:
Unit : Seconds
Precision: Full float64 precision with 15 significant digits, i.e. 10 μs.
Accuracy:
If WIFI, it is ~1 ms
If wired or ‘localhost’, it is in the range of μs.
Granularity:
It is machine specific (depends on clock_monotonic on Linux). It is constrained by the processor cycles and software.
In some machines (2 GHz processor), the result comes from clock_gettime(CLOCK_MONOTONIC, &time_record) function on Linux. This function delivers a record with nanosecond, 1 GHz, granularity. Then, PUPIL software does some math and delivers a float64.
Maximum Sampling Rate:
Depends on set-up, and it is lower when more cameras are present. (120Hz maximum based on a 5.7ms latency for the cameras and a 3.0ms processing latency.
Synchronization
Pupil Capture software runs multiple processes. The world video feed and the eye video feeds run and record at the frame rates set by their capture devices (cameras). This allows us to be more flexible. Instead of locking everything into one frame rate, we can capture every feed at specifically set rates. But, this also means that we sometimes record world video frames with multiple gaze positions (higher eye-frame rate) or without any (no pupil detected or lower eye frame rate).
In player_methods.py you can find a function that takes timestamped data and correlates it with timestamps form a different source.
defcorrelate_data(data,timestamps):'''
data: list of data :
each datum is a dict with at least:
timestamp: float
timestamps: timestamps list to correlate data to
this takes a data list and a timestamps list and makes a new list
with the length of the number of timestamps.
Each slot contains a list that will have 0, 1 or more associated data points.
Finally we add an index field to the datum with the associated index
'''timestamps=list(timestamps)data_by_frame=[[]foriintimestamps]frame_idx=0data_index=0data.sort(key=lambdad:d['timestamp'])whileTrue:try:datum=data[data_index]# we can take the midpoint between two frames in time: More appropriate for SW timestampsts=(timestamps[frame_idx]+timestamps[frame_idx+1])/2.# or the time of the next frame: More appropriate for Sart Of Exposure Timestamps (HW timestamps).# ts = timestamps[frame_idx+1]exceptIndexError:# we might loose a data point at the end but we don't carebreakifdatum['timestamp']<=ts:datum['index']=frame_idxdata_by_frame[frame_idx].append(datum)data_index+=1else:frame_idx+=1returndata_by_frame
Detailed Data Format
Every time you click record in Pupil Capture or Pupil Mobile, a new recording is started and your data is saved into a recording folder. You can use Pupil Player to playback Pupil recordings, add visualizations, and export in various formats.
Access to raw data
Note that the raw data before processing with Pupil Player is not immediately readible from other software (the raw data format is documented in the developer docs). Use the ‘Raw Data Exporter’ plugin in Pupil Player to export .csv files that contain all the data captured with Pupil Capture. Exported files will be written to a subfolder of the recording folder called exports.
The following files will be created by default in an export:
export_info.csv - Meta information on the export containing e.g. the export date or the dta format version.
pupil_positions.csv - A list of all pupil datums. See below for more infotmation.
gaze_positions.csv - A list of all gaze datums. See below for more infotmation.
pupil_gaze_positions_info.txt - Contains documentation on the contents of pupil_positions.csv and gaze_positions.csv
world_viz.mp4 - The exported section of world camera video.
If you are using additional plugins in Pupil Player, these might create other files. Please check the documentation of the respective plugin for the used data format.
pupil_positions.csv
This file contains all exported pupil datums. Each datum will have at least the following keys:
timestamp - timestamp of the source image frame
index - associated_frame: closest world video frame
id - 0 or 1 for left/right eye
confidence - is an assessment by the pupil detector on how sure we can be on this measurement. A value of 0 indicates no confidence. 1 indicates perfect confidence. In our experience useful data carries a confidence value greater than ~0.6. A confidence of exactly 0 means that we don’t know anything. So you should ignore the position data.
norm_pos_x - x position in the eye image frame in normalized coordinates
norm_pos_y - y position in the eye image frame in normalized coordinates
diameter - diameter of the pupil in image pixels as observed in the eye image frame (is not corrected for perspective)
method - string that indicates what detector was used to detect the pupil
Depending on the used gaze mapping mode, each datum may have additional keys. When using the 2D gaze mapping mode the following keys will be added:
2d_ellipse_center_x - x center of the pupil in image pixels
2d_ellipse_center_y - y center of the pupil in image pixels
2d_ellipse_axis_a - first axis of the pupil ellipse in pixels
2d_ellipse_axis_b - second axis of the pupil ellipse in pixels
2d_ellipse_angle - angle of the ellipse in degrees
When using the 3D gaze mapping mode the following keys will additionally be added:
diameter_3d - diameter of the pupil scaled to mm based on anthropomorphic avg eye ball diameter and corrected for perspective.
model_confidence - confidence of the current eye model (0-1)
model_id - id of the current eye model. When a slippage is detected the model is replaced and the id changes.
sphere_center_x - x pos of the eyeball sphere is eye pinhole camera 3d space units are scaled to mm.
sphere_center_y - y pos of the eye ball sphere
sphere_center_z - z pos of the eye ball sphere
sphere_radius - radius of the eyeball. This is always 12mm (the anthropomorphic avg.) We need to make this assumption because of the single camera scale ambiguity.
circle_3d_center_x - x center of the pupil as 3d circle in eye pinhole camera 3d space units are mm.
circle_3d_center_y - y center of the pupil as 3d circle
circle_3d_center_z - z center of the pupil as 3d circle
circle_3d_normal_x - x normal of the pupil as 3d circle. Indicates the direction that the pupil points at in 3d space.
circle_3d_normal_y - y normal of the pupil as 3d circle
circle_3d_normal_z - z normal of the pupil as 3d circle
circle_3d_radius - radius of the pupil as 3d circle. Same as diameter_3d
theta - circle_3d_normal described in spherical coordinates
phi - circle_3d_normal described in spherical coordinates
projected_sphere_center_x - x center of the 3d sphere projected back onto the eye image frame. Units are in image pixels.
projected_sphere_center_y - y center of the 3d sphere projected back onto the eye image frame
projected_sphere_axis_a - first axis of the 3d sphere projection.
projected_sphere_axis_b - second axis of the 3d sphere projection.
projected_sphere_angle - angle of the 3d sphere projection. Units are degrees.
gaze_positions.csv
This file contains a list of all exported gaze datums. Each datum contains the following keys:
timestamp - timestamp of the source image frame
index - associated_frame: closest world video frame
confidence - computed confidence between 0 (not confident) -1 (confident)
norm_pos_x - x position in the world image frame in normalized coordinates
norm_pos_y - y position in the world image frame in normalized coordinates
base_data - “timestamp-id timestamp-id …” of pupil data that this gaze position is computed from
When using the 3D gaze mapping mode the following keys will additionally be available:
gaze_point_3d_x - x position of the 3d gaze point (the point the subject looks at) in the world camera coordinate system
gaze_point_3d_y - y position of the 3d gaze point
gaze_point_3d_z - z position of the 3d gaze point
eye_center0_3d_x - x center of eye-ball 0 in the world camera coordinate system (of camera 0 for binocular systems or any eye camera for monocular system)
eye_center0_3d_y - y center of eye-ball 0
eye_center0_3d_z - z center of eye-ball 0
gaze_normal0_x - x normal of the visual axis for eye 0 in the world camera coordinate system (of eye 0 for binocular systems or any eye for monocular system). The visual axis goes through the eye ball center and the object thats looked at.
gaze_normal0_y - y normal of the visual axis for eye 0
gaze_normal0_z - z normal of the visual axis for eye 0
eye_center1_3d_x - x center of eye-ball 1 in the world camera coordinate system (not available for monocular setups.)
eye_center1_3d_y - y center of eye-ball 1
eye_center1_3d_z - z center of eye-ball 1
gaze_normal1_x - x normal of the visual axis for eye 1 in the world camera coordinate system (not available for monocular setups.). The visual axis goes through the eye ball center and the object thats looked at.
gaze_normal1_y - y normal of the visual axis for eye 1
gaze_normal1_z - z normal of the visual axis for eye 1
World Video Stream
When using the setting more CPU smaller file: A mpeg4 compressed video stream of the world view in a .mp4 container. The video is compressed using ffmpeg’s default settings. It gives a good balance between image quality and files size. The frame rate of this file is set to your capture frame rate.
When using the setting less CPU bigger file: A raw mjpeg stream from the world camera world view in a .mp4 container. The video is compressed by the camera itself. While the file size is considerably larger than above, this will allow ultra low CPU while recording. It plays with recent version of ffmpeg and vlc player. The “frame rate” setting in the Pupil Capture sidebar (Camera Settings > Sensor Settings) controls the frame rate of the videos.
You can compress the videos afterwards using ffmpeg like so:
OpenCV has a capture module that can be used to extract still frames from the video:
importcv2capture=cv2.VideoCapture("absolute_path_to_video/world.mp4")status,img1=capture.read()# extract the first framestatus,img2=capture.read()# second frame...
Developer Docs
Development Overview
Overview of language, code structure, and general conventions
Language
Pupil is written in Python 3, but no “heavy lifting” is done in Python. High performance computer vision, media compression, display libraries, and custom functions are written in external libraries or c/c++ and accessed though cython. Python plays the role of “glue” that sticks all the pieces together.
We also like writing code in Python because it’s quick and easy to move from initial idea to working proof-of-concept. If proof-of-concept code is slow, optimization and performance enhancement can happen in iterations of code.
Process Structure
When Pupil Capture starts, in default settings two processes are spawned:
Eye and World. Both processes grab image frames from a video capture stream but they have very different tasks.
Eye Process
The eye process only has one purpose - to detect the pupil and broadcast its position. The process breakdown looks like this:
Grabs eye camera images from eye camera video stream
Find the pupil position in the image
Broadcast/stream the detected pupil position.
World Process
This is the workhorse.
Grabs the world camera images from the world camera video stream
Receives pupil positions from the eye process
Performs calibration mapping from pupil positions to gaze positions
Loads plugins - to detect fixations, track surfaces, and more…
Records video and data. Most, and preferably all coordination and control happens within the World process.
Pupil Datum Format
The pupil detector, run by the Eye process are required to return a result in the form of a Python dictionary with at least the following content:
result={}result['timestamp']=frame.timestampresult['norm_pos']=(x,y)# pupil center in normalized coordinatesresult['confidence']=# a value between 1 (very certain) and 0 (not certain, nothing found)result['whatever_else_you_want']=# you can add other things to this dict# if no pupil was detectedresult={}result['timestamp']=frame.timestampresult['confidence']=0
This dictionary is sent on the IPC and read by gaze mapping plugins in the world process. Mapping from pupil position to gaze position happens here. The mapping plugin is initialized by a calibration plugin. The 3D pupil detector extends the 2D pupil datum with additional information. Below you can see the Python representation of a pupil and a gaze datum.
{# pupil datum'topic':'pupil','method':'3d c++','norm_pos':[0.5,0.5],# norm space, [0, 1]'diameter':0.0,# 2D image space, unit: pixel'timestamp':535741.715303987,# time, unit: seconds'confidence':0.0,# [0, 1]# 2D ellipse of the pupil in image coordinates'ellipse':{# image space, unit: pixel'angle':90.0,# unit: degrees'center':[320.0,240.0],'axes':[0.0,0.0]},'id':0,# eye id, 0 or 1# 3D model data'model_birth_timestamp':-1.0,# -1 means that the model is building up and has not finished fitting'model_confidence':0.0,'model_id':1# pupil polar coordinates on 3D eye model. The model assumes a fixed# eye ball size. Therefore there is no `radius` key'theta':0,'phi':0,# 3D pupil ellipse'circle_3d':{# 3D space, unit: mm'normal':[0.0,-0.0,0.0],'radius':0.0,'center':[0.0,-0.0,0.0]},'diameter_3d':0.0,# 3D space, unit: mm# 3D eye ball sphere'sphere':{# 3D space, unit: mm'radius':0.0,'center':[0.0,-0.0,0.0]},'projected_sphere':{# image space, unit: pixel'angle':90.0,'center':[0,0],'axes':[0,0]}}
Gaza data is based on one (monocular) or two (binocular) pupil positions. The gaze mapper is automatically setup after calibration and maps pupil positions into world camera coordinate system. The pupil data on which the gaze datum is based on can be accessed using the base_data key.
{# gaze datum'topic':'gaze','confidence':1.0,# [0, 1]'norm_pos':[0.5238293689178297,0.5811187961748036],# norm space, [0, 1]'timestamp':536522.568094512,# time, unit: seconds# 3D space, unit: mm'gaze_normal_3d':[-0.03966349641933964,0.007685562866422135,0.9991835362811073],'eye_center_3d':[20.713998951917564,-22.466222119962115,11.201474469783548],'gaze_point_3d':[0.8822507422478054,-18.62344068675104,510.7932426103372],'base_data':[<pupildatum>]}# list of pupil data that was used to calculate the gaze
Timing & Data Conventions
Pupil Capture is designed to work with multiple captures that free-run at different frame rates that may not be in sync. World and eye images are timestamped and any resulting artifacts (detected pupil, markers, etc) inherit the source timestamp. Any correlation of these data streams is the responsibility of the functional part that needs the data to be correlated (e.g. calibration, visualization, analyses).
For example: The pupil capture data format records the world video frames with their respective timestamps. Independent of this, the recorder also saves the detected gaze and pupil positions at their frame rate and with their timestamps. For more detail see Data Format.
Git Conventions
We make changes almost daily and sometimes features will be temporarily broken in some development branches. However, we try to keep the master branch as stable as possible and use other branches for feature development and experiments. Here’s a breakdown of conventions we try to follow.
master - this branch tries to be as stable as possible - incremental and tested features will be merged into the master. Check out the master branch.
branches - branches are named after features that are being developed. These branches are experimental and what could be called ‘bleeding edge’. This means features in these branches may not be fully functional, broken, or really cool… You’re certainly welcome to check them out and improve on the work!
Pull requests
If you’ve done something – even if work-in-progress – make a pull request and write a short update to the Pupil Community.
Developer Setup
Pages in the developer guide are oriented towards developers and will contain high level overview of code and organizational structure.
If you want to develop a plugin or to extend Pupil for your project, this is the place to start.
These pages will not contain detailed documentation of code. We’re working on code documentation, and when it’s done we will put code documentation online at read the docs.
If you have questions, encounter any problems, or want to share progress – chat with us on the Pupil channel on Discord. We will try our best to help you out, and answer questions quickly.
Running Pupil from Source
Pupil is a prototype and will continue to be in active development. If you plan to make changes to Pupil, want to see how it works, make a fork, install all dependencies and run Pupil source directly with Python.
Installing Dependencies
Linux step-by-step instructions for Ubuntu 16.04 LTS +
Intel RealSense 3D instructions if you want to use an Intel RealSense R200 as scene camera.
Download and Run Pupil Source Code
Once you have all dependencies installed, you’re 99% done. Now, all you have to do fork the github repository. Or, using the terminal you can clone the Pupil repository using git:
cd /the_folder_where_Pupil_will_live/
git clone https://github.com/pupil-labs/pupil.git
Run Pupil Capture from Source
You’re in development land now. If you’re running from the source, there will be no icon to click. So fire up the terminal, navigate to the cloned Pupil repository, and start Pupil using Python.
cd /the_folder_where_Pupil_lives/pupil_src
python3 main.py
Linux Dependencies
These installation instructions are tested using Ubuntu 16.04 or higher running on many machines. Do not run Pupil on a VM unless you know what you are doing.
Install Linux Dependencies
Let’s get started! Its time for apt! Just copy paste into the terminal and listen to your machine purr.
Next we need to install the Ceres library. In Ubuntu 18.04 Ceres is available as a package in the repositories. In older versions it has to be compiled from source. Choose the correct command on the right depending on your version of Ubuntu!
Ubuntu <= 17.10
# sudo apt-get install software-properties-common if add-apt-repository is not found
sudo add-apt-repository ppa:bzindovic/suitesparse-bugfix-1319687
sudo apt-get update
sudo apt-get install libsuitesparse-dev
# install ceres-solver
git clone https://ceres-solver.googlesource.com/ceres-solver
cd ceres-solver
mkdir build &&cd build
cmake .. -DBUILD_SHARED_LIBS=ON
make -j3
make test
sudo make install
sudo sh -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/ceres.conf'
sudo ldconfig
(Optional) Install PyTorch + CUDA and cuDNN.
Version 1: Without GPU acceleration: Install PyTorch via pip
Some bleeding edge features require the deep learning library PyTorch. Without GPU acceleration some of the features will probably not run in real-time.
Version 2: With GPU acceleration: Install PyTorch via pip
pip3 install torch torchvision
Please refer to the following links on how to install CUDA and cuDNN:
These instructions have been tested for MacOS 10.8, 10.9, 10.10, 10.11, and 10.12. Use the linked websites and Terminal to execute the instructions.
Install Apple Dev Tools
Trigger the install of the Command Line Tools (CLT) by typing this in your terminal and letting MacOS install the tools required:
git
Install Homebrew
Homebrew describes itself as “the missing package manager for OSX.” It makes development on MacOS much easier, plus it’s open source. Install with the ruby script.
Add Homebrew installed executables and Python scripts to your path. Add the following two lines to your ~/.bash_profile. (you can open textedit from the terminal like so: open ~/.bash_profile)
We develop the Windows version of Pupil using 64 bitWindows 10.
Therefore we can only debug and support issues for Windows 10.
Notes Before Starting
Work directory - We will make a directory called work at C:\work and will use this directory for all build processes and setup scripts. Whenever we refer to the work directory, it will refer to C:\work. You can change this to whatever is convenient for you, but note that all instructions and setup files refer to C:\work
Command Prompt - We will always be using x64 Native Tools Command Prompt for VS 2017 as our command prompt. Make sure to only use this command prompt. Unlike unix systems, windows has many possible “terminals” or “cmd prompts”. We are targeting x64 systems and require the x64 command prompt. You can access this cmd prompt from the Visual Studio 2017 shortcut in your Start menu.
64bit - You should be using a 64 bit system and therefore all downloads, builds, and libraries should be for x64 unless otherwise specified.
Windows paths and Python - path separators in windows are a forward slash \. In Python, this is a special “escape” character. When specifying Windows paths in a Python string you must use \\ instead of \ or use Python raw strings, e.g. r'\'.
Download Visual Studio 2017 Community version 15.8 from visualstudio.com
Run the Visual Studio bootstrapper .exe.
Navigate to the Workloads tab
In the Workloads tab, choose Desktop Development with C++. This will install all runtimes and components we need for development. Here is a list of what you should see checked in the Desktop development with C++ in the Summary view:
VC++ 2017 v141 toolset (x86,x64)
C++ profiling tools
Windows 10 SDK (10.0.15063.0) for Desktop C++ x86 and x64
Visual C++ tools for CMAKE
Visual C++ ATL support
MFC and ATL support (x86, x64)
Standard Library Modules
VC++ 2015.3 v140 toolset for desktop (x86, x64)
Navigate to the Individual Components tab
In the Individual Components tab check Git. This will install git on your system. In the Summary Panel for Individual Components you should see:
Check the box Add Python to PATH. This will add Python to your System PATH Environment Variable.
Check the box Install for all users. This will install Python to C:\Program Files\Python36.
System Environment Variables
You will need to check to see that Python was added to your system PATH variables. You will also need to manually add other entries to the system PATH later in the setup process.
To access your System Environment Variables:
Right click on the Windows icon in the system tray.
Select System.
Click on Advanced system settings.
Click on Environment Variables....
You can click on Path in System Variables to view the variables that have been set.
You can Edit or Add new paths (this is needed later in the setup process).
Python Wheels
Most Python extensions can be installed via pip. We recommend to download and install the pre-built wheel (*.whl) packages maintained by Christoph Gohlke. (@Gohlke Thanks for creating and sharing these packages!)
Download the most recent version of the following wheels Python3.6 x64 systems.
pyuvc requires that you download Microsoft Visual C++ 2010 Redistributable from microsoft. The pthreadVC2 lib, which is used by libuvc, depends on msvcr100.dll.
Extract boost to work directory and name the boost dir boost
Open C:\work\boost\boost\python\detail\config.hpp with Visual Studio 2017
Change L108 from define BOOST_LIB_NAME boost_python to define BOOST_LIB_NAME boost_python3
Save the file and close Visual Studio
The prebuilt boost.python depends on python27.dll. The files from package boost.python are built with Visual Studio 2015. One solution to this issue is to build boost from source.
Open your command prompt
cd to C:\work\boost
Run boostrap.bat. This will generate b2.exe.
Change user config before compiling boost.
Copy C:\work\boost\tools\build\example\user-config.jam to boost\tools\build\src\user-config.jam.
Uncomment and edit following lines in the user-config.jam file according your msvc and python version:
using msvc : 14.1 ; in section MSVC configuration
using python : 3.6 : C:\\Python36 : C:\\Python36\\include : C:\\Python36\\libs ; in section Python configuration
Build boost.python
Open your command prompt and navigate to your work dir
cd to boost
b2 --with-python link=shared address-model=64
The generated DLL and Lib files are in C:\work\boost\stage.
Add Boost libs to your system path
Add C:\work\boost\stage\lib to your system PATH in your System Environment Variables
Clone the Pupil Repo
Open a command prompt in your work dir
git clone https://github.com/pupil-labs/pupil.git
Setup pupil_external dependencies
The following steps require you to store dynamic libraries in the pupil_external folder of the cloned repository so that you do not have to add further modifications to your system PATH.
Download FFMPEG v4.0 Windows shared binaries from ffmpeg
Unzip ffmpeg-shared to your work dir
Copy the following 8 .dll files to pupil_external
avcodec-58.dll
avdevice-58.dll
avfilter-7.dll
avformat-58.dll
avutil-56.dll
postproc-55.dll
swresample-3.dll
swscale-5.dll
OpenCV to pupil_external
Download opencv 3.2 exe installer from sourceforge
Unzip OpenCV to your work dir and rename dir to opencv
Copy opencv\build\x64\vc14\bin\opencv_world320.dll to pupil_external
Include pupil_external in PATH variable
Follow the instructions under the System Environment Variables section above to add a new environment variable to PATH
Add the following folder: C:\work\pupil\pupil_external
Restart your computer so that the PATH variable is refreshed
Modify pupil_detectors setup.py
Open pupil\pupil_src\capture\pupil_detectors\setup.py
Go to the if platform.system() == 'Windows' block
Check that paths for opencv, Eigen, ceres-windows and boost are correctly specified. The installed opencv lib is opencv_world320.lib.
Edit paths if necessary
Edit C:\\work\\boost\\stage\\lib if necessary
Save and close setup.py
Modify optimization_calibration setup.py
Open pupil\pupil_src\shared_modules\calibration_routines\optimization_calibration\setup.py
Go to the if platform.system() == 'Windows' block
Check that paths for opencv, Eigen, ceres-windows and boost are correctly specified. The installed opencv lib is opencv_world320.lib.
Edit paths if necessary
Edit C:\\work\\boost\\stage\\lib if necessary
Save and close setup.py
Start the application
To start either of the applications – Capture, Player, or Service – you need to execute the respective run_*.bat file, i.e. run_capture.bat, run_player.bat, or run_service.bat. You can also run main.py directly from your IDE, or with the commands python main.py capture, python main.py player, or python main.py service.
Windows Driver Setup
In order to support isochronous USB transfer on Windows, you will need to install drivers for the cameras in your Pupil headset.
Right click PupilDrvInst.exe - and run as admin. This will install drivers.
Open Windows Device Manager from System > Device Manager. Verify the drivers are correctly installed in Windows Device Manager. Your Pupil headset cameras should be listed under a new category titled: libusbK Usb Devices. Note: In some cases Pupil Cam1 may show three of the same ID as the camera name. Don’t worry - just make sure that the number of devices are the same as the number of cameras on your Pupil headset.
Download the latest release of Pupil software and launch pupil_capture.exe to verify all cameras are accessible.
Troubleshooting
If you had tried to install drivers with previous driver install instructions and failed, or are not able to access cameras in Pupil Capture. Please try the following:
In Device Manager (System > Device Manager)
View > Show Hidden Devices
Expand libUSBK Usb Devices
For each device listed (even hidden devices) click Uninstall and check the box agreeing to Delete the driver software for this device and press OK
Repeat for each device in libUSBK Usb Devices
Unplug Pupil headset (if plugged in)
Restart your computer
Install drivers from step 2 in the Install drivers for your Pupil headset section
Manual Installation of DIY Camera Drivers
If any issues arise when trying to install drivers for your DIY eye or world cameras, or if you are installing them for the first time, you can try following these instructions:
Unplug Pupil Headset from your computer and keep unplugged until the last step
Open Device Manager
Click View > Show Hidden Devices
Expand the libUSBK devices category and expand the Imaging Devices category within the Device Manager (sometimes a camera may be under the Cameras category)
Uninstall and delete drivers for all Pupil Cam 1 ID0, Pupil Cam 1 ID1, and Pupil Cam 1 ID2 devices within both libUSBK and Imaging Devices Category
Restart Computer
Start the latest version of Pupil Capture (ensure that you have admin privileges on your machine) i. General Menu > Restart with default settings
Plug in Pupil Headset after Pupil Capture relaunches - Please wait, drivers should install automatically. You may need to close/cancel automatic Windows driver installation
If the above doesn’t work, uninstall all drivers by following steps 1 to 3. You can then try install libusbK drivers with Zadig as outlined in steps 1-7 in the pyuvc docs
Windows Pupil Labs Python libs from Source
This section is for Pupil core developers who want to build pyav, pyndsi, pyglui, and pyuvc from source and create Python wheels.
If you just want to run Pupil from source, go back to the Windows Dependencies section and install the prebuilt wheels.
This section assumes that you have a Windows development environment set up and all dependencies installed as specified in the Windows Dependencies section.
In order to create wheels, you will need to install the wheel lib.
pip install wheel
Download FFMPEG Dev
You will need both .dll files as well as FFMPG libs in order to build pyav. You should have already downloaded FFMPEG shared binaries in the FFMPEG to pupil_external step above.
Download FFMPEG Windows Dev - ffmpeg-4.0-win64-dev - from ffmpeg
Select RealSense 3D in the Capture Selection menu and activate your RealSense camera. Afterwards you should see the colored video stream of the selected camera.
Pupil Capture accesses both streams, color and depth, at all times but only previews one at a time. Enable the Preview Depth option to see the normalized depth video stream.
The Record Depth Stream option (enabled by default) will save the depth stream during a recording session to the file depth.mp4 within your recording folder.
By default, you can choose different resolutions for the color and depth streams. This is advantageous if you want to run both streams at full resolution. The Intel RealSense R200 has a maximum color resolution of 1920 x 1080 pixels and maximum depth resolution of 640 x 480 pixels. librealsense also provides the possibility to pixel-align color and depth streams. Align Streams enables this functionality. This is required if you want to infer from depth pixels to color pixels and vice versa.
The Sensor Settings menu lists all available device options. These may differ depending on your OS, installed librealsense version, and device firmware.
Color Frames
Pupil Capture accesses the YUVY color stream of the RealSense camera. All color frames are accessible through the events object using the frame key within your plugin’s recent_events method. See the plugin guide for details.
Depth Frames
Depth frame objects are accessible through the events object using the depth_frame key within your plugin’s recent_events method. The orginal 16-bit grayscale image of the camera can be accessed using the depth attribute of the frame object. The bgr attribute provides a colored image that is calculated using histogram equalization. These colored images are previewed in Pupil Capture, stored during recordings, and referred to as “normalized depth stream” in the above section. The librealsense examples use the same coloring method to visualize depth images.
Interprocess and Network Communication
This page outlines the way Pupil Capture and Pupil Service communicate via a message bus internally and how to read and write to this bus from another application on the same machine or on a remote machine.
Networking
All networking in Pupil Capture and Service is based on the ZeroMQ network library. The following socket types are most often used in our networking schemes: - REQ-REP, reliable one-to-one communication - PUB-SUB, one-to-many communication
We highly recommend to read Chapter 2 of the ZeroMQ guide to get an intuition on the philosophy behind these socket types.
The Pupil apps use the pyzmq module, a great Python ZeroMQ implementation. For auto-discovery of other Pupil app instances in the local network, we use Pyre.
The IPC Backbone
Pupil Capture/Service/Player use a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors’ messages. Therefore, it is the backbone of all communication from, to, and within Pupil apps.
IPC Backbone used by Pupil Capture and Service
The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate. - Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.) - Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.
Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.
Message Format
Currently all messages on the IPC Backbone are multipart messages containing two message frames:
Frame 1 contains a string we call topic. Examples are : pupil.0, logging.info, notify.recording.has_started
Frame 2 contains a msgpack encoded dictionary with key:value pairs. This is the actual message. We choose msgpack as the serializer due to its efficient format (45% smaller than json, 200% faster than ujson) and because encoders exist for almost every language.
Message Topics
Messages can have any topic chosen by the user. Below a list of message types used by the Pupil apps.
Pupil and Gaze Messages
Pupil data is sent from the eye0 and eye1 process with the topic pupil.0 or pupil.1. Gaze mappers receive this data and publish messages with topic gaze. See the Pupil Datum format for example messages for the topics pupil and gaze.
Notification Message
Pupil uses special messages called notifications to coordinate all activities. Notifications are dictionaries with the required field subject. Subjects are grouped by categories category.command_or_statement. Example: recording.should_stop
# message topic:'notify.recording.should_start'# message payload, a notification dict{'subject':'recording.should_start','session_name':'my session'}
The message topic construction in python:
topic='notify'+'.'+notification['subject']
You should use the notification topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.
In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more that 10 messages a second, it is probably not a notification but another kind of data, make a custom topic instead.
importzmqimportmsgpacktopic='your_custom_topic'payload={'topic':topic}# create and connect PUB socket to IPCpub_socket=zmq.Socket(zmq.Context(),zmq.PUB)pub_socket.connect(ipc_pub_url)# send payload using custom topicsocket.send_string(topic,flags=zmq.SNDMORE)socket.send(msgpack.dumps(payload,use_bin_type=True))
Log Messages
Pupil sends all log messages onto the IPC.
The topic is logging.log_level_name (debug,info,warning,error,…). The message is a dictionary that contains all attributes of the python logging.record instance.
# message topic:'logging.warning'# message payload, logging record attributes as dict:{'levelname':'WARNING','msg':'Process started.','threadName':'MainThread','name':'eye','thread':140735165432592L,'created':1465210820.609704,'process':14239,'processName':'eye0','args':[],'module':'eye','filename':'eye.py','levelno':30,'msecs':609.7040176392,'pathname':'/Users/mkassner/Pupil/pupil_code/pupil_src/capture/eye.py','lineno':299,'exc_text':None,'exc_info':None,'funcName':'eye','relativeCreated':4107.3870658875})
Message Documentation
Pupil software uses introduces a consistent naming scheme for message topics. They are used to publish and subscribe to the IPC Backbone. Pre-defined message topics are pupil, gaze, notify, delayed_notify, logging. Notifications sent with the notify_all() function of the Plugin class will be published automatically as notify.<notification subject>.
Message Reactor and Emitter Documentation
Every actor who either reacts to or emits messages is supposed to document its behavior. Therefore every actor should react to notify.meta.should_doc by emitting a message with the topic notify.meta.doc. The answer’s payload should be a serialized dictionary with the following format:
Plugins use notifications as their primary communication channel to the IPC Backbone. This makes plugins natural actors in the Pupil message scheme. To simplify the above mentioned documentation behavior, plugins will only have to add a docstring to their on_notify() method. It should include a list of messages to which the plugin reacts and of those which the plugin emits itself. The docstring should follow Google docstring style. The main process will automatically generate messages in the format from above using the plugin’s class name as actor and the on_notify() docstring as content for the doc key.
Notification Overview
You can use the following script to get an overview over the notification handling of the currently running actors:
importzmq,msgpackfromzmq_toolsimportMsg_Receiverctx=zmq.Context()ip='localhost'#If you talk to a different machine use its IP.port=50020#The port defaults to 50020 but can be set in the GUI of Pupil Capture.# open Pupil Remote socketrequester=ctx.socket(zmq.REQ)requester.connect('tcp://%s:%s'%(ip,port))requester.send_string('SUB_PORT')ipc_sub_port=requester.recv_string()# setup message receiversub_url='tcp://%s:%s'%(ip,ipc_sub_port)receiver=Msg_Receiver(ctx,sub_url,topics=('notify.meta.doc',))# construct messagetopic='notify.meta.should_doc'payload=msgpack.dumps({'subject':'meta.should_doc'})requester.send_string(topic,flags=zmq.SNDMORE)requester.send(payload)requester.recv_string()# wait and print responseswhileTrue:# receiver is a Msg_Receiver, that returns a topic/payload tuple on recv()topic,payload=receiver.recv()actor=payload.get('actor')doc=payload.get('doc')print('%s: %s'%(actor,doc))
Example output for v0.8:
launcher: Starts eye processes. Hosts the IPC Backbone and Logging functions.
Reacts to notifications:
``launcher_process.should_stop``: Stops the launcher process
``eye_process.should_start``: Starts the eye process
eye0: Reads eye video and detects the pupil.
Creates a window, gl context.
Grabs images from a capture.
Streams Pupil coordinates.
Reacts to notifications:
``set_detection_mapping_mode``: Sets detection method
``eye_process.should_stop``: Stops the eye process
``recording.started``: Starts recording eye video
``recording.stopped``: Stops recording eye video
Emits notifications:
``eye_process.started``: Eye process started
``eye_process.stopped``: Eye process stopped
Emits data:
``pupil.<eye id>``: Pupil data for eye with id ``<eye id>``
capture: Reads world video and runs plugins.
Creates a window, gl context.
Grabs images from a capture.
Maps pupil to gaze data
Can run various plugins.
Reacts to notifications:
``set_detection_mapping_mode``
``eye_process.started``
``start_plugin``
Emits notifications:
``eye_process.should_start``
``eye_process.should_stop``
``set_detection_mapping_mode``
``world_process.started``
``world_process.stopped``
``recording.should_stop``: Emits on camera failure
``launcher_process.should_stop``
Emits data:
``gaze``: Gaze data from current gaze mapping plugin.``
``*``: any other plugin generated data in the events that it not [dt,pupil,gaze].
Pupil_Remote: send simple string messages to control application functions.
Emits notifications:
``recording.should_start``
``recording.should_stop``
``calibration.should_start``
``calibration.should_stop``
Any other notification received though the reqrepl port.
Screen_Marker_Calibration: Handles calibration notifications
Reacts to notifications:
``calibration.should_start``: Starts the calibration procedure
``calibration.should_stop``: Stops the calibration procedure
Emits notifications:
``calibration.started``: Calibration procedure started
``calibration.stopped``: Calibration procedure stopped
``calibration.failed``: Calibration failed
``calibration.successful``: Calibration succeeded
Args:
notification (dictionary): Notification dictionary
Recorder: Handles recorder notifications
Reacts to notifications:
``recording.should_start``: Starts a new recording session
``recording.should_stop``: Stops current recording session
Emits notifications:
``recording.started``: New recording session started
``recording.stopped``: Current recording session stopped
Args:
notification (dictionary): Notification dictionary
Pupil Remote
If you want to tap into the IPC backbone you will not only need the IP address but also the session unique port. You can get these by talking to ‘Pupil Remote’:
importzmqctx=zmq.Context()# The requester talks to Pupil remote and receives the session unique IPC SUB PORTrequester=ctx.socket(zmq.REQ)ip='localhost'#If you talk to a different machine use its IP.port=50020#The port defaults to 50020 but can be set in the GUI of Pupil Capture.requester.connect('tcp://%s:%s'%(ip,port))requester.send_string('SUB_PORT')sub_port=requester.recv_string()
Pupil Remote uses the fixed port 50020 and is the entry point to the IPC backbone for external applications. It also exposes a simple string-based interface for basic interaction with the Pupil apps:
Send simple string messages to control Pupil Capture functions:
'R' start recording with auto generated session name
'R rec_name' start recording and name new session name: rec_name
'r' stop recording
'C' start currently selected calibration
'c' stop currently selected calibration
'T 1234.56' Timesync: make timestamps count form 1234.56 from now on.
't' get pupil capture timestamp; returns a float as string.
# IPC Backbone communication
'PUB_PORT' return the current pub port of the IPC Backbone
'SUB_PORT' return the current sub port of the IPC Backbone
Reading from the Backbone
Subscribe to desired topics and receive all relevant messages (i.e. messages whose topic prefix matches the subscription). Be aware that the IPC Backbone can carry a lot of data. Do not subscribe to the whole stream unless you know that your code can drink from a firehose. (If it can not, you become the snail, see Delivery Guarantees REQ-REP.)
#...continued from abovesubscriber=ctx.socket(zmq.SUB)subscriber.connect('tcp://%s:%s'%(ip,sub_port))subscriber.set(zmq.SUBSCRIBE,'notify.')#receive all notification messagessubscriber.set(zmq.SUBSCRIBE,'logging.error')#receive logging error messages#subscriber.set(zmq.SUBSCRIBE, '') #receive everything (don't do this)# you can setup multiple subscriber sockets# Sockets can be polled or read in different threads.# we need a serializerimportmsgpackasserializerwhileTrue:topic,payload=subscriber.recv_multipart()message=serializer.loads(payload)printtopic,':',message
Writing to the Backbone from outside
You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:
# continued from aboveimportmsgpackasserializernotification={'subject':'recording.should_start','session_name':'my session'}topic='notify.'+notification['subject']payload=serializer.dumps(notification)requester.send_string(topic,flags=zmq.SNDMORE)requester.send(payload)print(requester.recv_string())
We say reliable transport because pupil remote will confirm every notification we send with ‘Notification received’. When we get this message we have a guarantee that the notification is on the IPC Backbone.
If we listen to the backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.
Writing to the Backbone directly
If you want to write messages other than notifications onto the IPC backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.
# continued from abovefromtimeimporttime,sleeprequester.send_string('PUB_PORT')pub_port=requester.recv_string()publisher=ctx.socket(zmq.PUB)publisher.connect('tcp://%s:%s'%(ip,pub_port))sleep(1)# see Async connect in the paragraphs belownotification={'subject':'calibration.should_start'}topic='notify.'+notification['subject']payload=serializer.dumps(notification)publisher.send_string(topic,flags=zmq.SNDMORE)publisher.send(payload)
A full example
A full example can be found in shared_modules/zmq-tools.py.
Delivery guarantees ZMQ
ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don’t want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.
Messages are guaranteed to be delivered whole or not at all.
Unlike bare TCP it is ok to connect before binding.
ZMQ will try to repair broken connections in the background for us.
It will deal with a lot of low level tcp handling so we don’t have to.
Delivery Guarantees PUB-SUB
ZMQ PUB SUB will make no guarantees for delivery. Reasons for dropped messages are:
Async connect: PUB sockets drop messages before a connection has been made (connections are async in the background) and topics subscribed. *1
The Late joiner: SUB sockets will only receive messages that have been sent after they connect. *2
The Snail: If SUB sockets do not consume delivered messages fast enough they start dropping them. *3
fast close: A PUB socket may loose packages if you close it right after sending. *1
In Pupil we prevent this by using a PUSH socket as intermediary for notifications. See shared_modules/zmq_tools.py.
Caching all messages in the sender or proxy is not an option. This is not really considered a problem of the transport.
In Pupil we pay close attention to be fast enough or to subscribe only to low volume topics. Dropping messages in this case is by design. It is better than stalling data producers or running out of memory.
Delivery Guarantees REQ-REP
When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.
Delivery Guarantees in general
We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.
However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.
Latency
Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests…
# continued from abovets=[]forxinrange(100):sleep(0.003)#simulate spaced requests as in real worldt=time()requester.send_string('t')requester.recv_string()ts.append(time()-t)print(min(ts),sum(ts)/len(ts),max(ts))>>>0.0002660751342770.0005974721908570.00339102745056
… and when talking directly to the IPC backbone and waiting for the same message to appear to the subscriber:
# continued from abovemonitor=Msg_Receiver(ctx,sub_url,topics=('notify.pingback_test',))sleep(1.)ts=[]forxinrange(100):sleep(0.003)#simulate spaced requests as in real worldt=time()#notify is a method of the Msg_Dispatcher class in zmq_tools.pynotification={'subject':'pingback_test'}topic='notify.'+notification['subject']payload=serializer.dumps(notification)publisher.send_string(topic,flags=zmq.SNDMORE)publisher.send(payload)monitor.recv()ts.append(time()-t)print(min(ts),sum(ts)/len(ts),max(ts))>>>0.0001809597015380.0003009605407710.000565052032471
Throughput
During a test we have run dual 120fps eye tracking with a dummy gaze mapper that turned every pupil datum into a gaze datum. This is effectively 480 messages/sec. The main process running the IPC backbone proxi showed a cpu load of 3% on a MacBook Air (late 2012).
Artificially increasing the pupil messages by a factor 100 increases the message load to 24.000 pupil messages/sec. At this rate the gaze mapper cannot keep up but the IPC backbone proxi runs at only 38% cpu load.
It appears ZMQ is indeed highly optimized for speed.
Final remarks
You can send a message anywhere in the app. Don’t send something that crashes anywhere.
Plugin Guide
Plugins Basics
Plugins encapsulate functionality in a modular fashion. Most parts of the Pupil apps are implemented as plugins. They are managed within the world process event-loop. This means that the world process can load and unload plugins during runtime. Plugins are called regularly via callback functions (see the Plugin API for details).
We recommend to use the network (see the IPC backbone) if you only need access to the data. You are only required to write a plugin if you want to interact with the Pupil apps directly, e.g. visualizations, manipulate data. In the following sections, we assume and recommend that during plugin development you run the Pupil applications from source.
Plugins in Pupil Capture
Pupil Capture’s World process can load plugins for easy integration of new features. Plugins have full access to:
World image frame
Events
pupil positions
gaze positions
surface events
note other events can be added to the event queue by other plugins
User input
Globally declared variables in the g_pool
Plugins can create their own UI elements, and even spawn their own OpenGL windows.
Pupil Player Plugins
Pupil Player uses an identical plugin structure. Little (often no work) needs to be done to use a Player Plugin in Capture and vice versa. But, it is important to keep in mind that plugins run in Pupil Capture may require more speed for real-time workflows, as opposed to plugins in Pupil Player.
Plugin API
Plugins are Python classes that inherit the Plugin class. It provides default functionality as well as series of callback functions that are called by the world process. The source contains detailed information about the use-cases of the different callback functions.
Register your plugin automatically
To add your plugin to Capture all you need to do is place the source file(s) in the plugin directory.
[your_user_dir] is also called HOME (for Linux and MacOS) or USER (for Windows).
Note: if your plugin is contained in a directory, make sure to include an __init__.py inside it. For example:
When a valid plugin is found in these dirs, Pupil imports your Plugin classes and adds them to the dropdown list of launchable plugins. If your plugin is a calibration plugin (i.e. it inherits from the Calibration_Plugin base class), then it will appear in the calibration drop down menu.
Example plugin development walkthrough
Inheriting from existing plugin
If you want to add or extend the functionality of an existing plugin, you should be able to apply standard inheritance principles of Python 3.
Things to keep in mind:
g_pool is an abbreviation for “global pool”, a system wide container full of stuff passed to all plugins.
if the base plugin is a system (always alive) plugin:
remember to close the base plugin at the __init__ method of the inheriting plugin with base_plugin.alive = False. You should find the base_plugin inside g_pool.plugins ;
remember to dereference the base plugin at the end of the file with del base_plugin to avoid repetition in the user plugin list;
Hacking an existing plugin
Another way to start plugin development, is to use an existing plugin as a template. For example, you could copy the vis_circle.py plugin as a starting point.
renaming it to, for example, open_cv_threshold.py.
Now you could give a new name to the class name:
classOpen_Cv_Threshold(Plugin):
Describe what your new plugin will do for yourself in the future and for future generations:
classOpen_Cv_Threshold(Plugin):"""
Apply cv2.threshold filter to the world image.
"""
Lets determine its execution order in relation to the other plugins:
self.order=.8
You can allow or disallow multiple instances of the Custom Plugin through the uniqueness attribute:
self.uniqueness="by_class"
See the source for a list of all available uniqueness options. Finally, lets implement what our new Plugin will do. Here we choose to apply an OpenCv threshold to the world image and give us proper feedback of the results, in real time. Good for OpenCv and related studies. It is possible by means of the recent_events method:
defrecent_events(self,events):if'frame'inevents:frame=events['frame']img=frame.imgheight=img.shape[0]width=img.shape[1]blur=cv2.GaussianBlur(img,(5,5),0)edges=[]threshold=177blue,green,red=0,1,2# apply the threshold to each channelforchannelin(blur[:,:,blue],blur[:,:,green],blur[:,:,red]):retval,edg=cv2.threshold(channel,threshold,255,cv2.THRESH_TOZERO)edges.append(edg)# lets merge the channels againedges.append(np.zeros((height,width,1),np.uint8))edges_edt=cv2.max(edges[blue],edges[green])edges_edt=cv2.max(edges_edt,edges[red])merge=[edges_edt,edges_edt,edges_edt]# lets check the resultframe.img=cv2.merge(merge)
recent_events is called everytime a new world frame is available but latest after a timeout of 0.05 seconds. The events dictionary will include the image frame object if it was available. It is accessible through the frame key.
You can access the image buffer through the img and the gray attributes of the frame object. They return a BGR (height x width x 3) and gray scaled (height x width) uint8-numpy array respectively. Visualization plugins (e.g. vis_circle.py modify the img buffer such that their visualizations are visible in the Pupil Player exported video. Use OpenGL (within the Plugin.gl_display method) to draw visualizations within Pupil Player that are not visible in the exported video (e.g. surface heatmaps in Offline_Surface_Tracker. See below for more information.
The events dictionary contains other recent data, e.g. pupil_positions, gaze_positions, fixations, etc. Modifications to the events dictionary are automatically accessible by all plugins with an higher order than the modifying plugin.
Plugin Integration
pyglui UI Elements
‘pyglui’ is an OpenGL-based UI framework that provides easy to use UI components for your plugin. User plugins often have at least one menu to inform the user that they are running as well as providing the possibility to close single plugins.
frompluginimportPluginfrompygluiimportuiclassCustom_Plugin(Plugin):# Calling add_menu() will create an icon in the icon bar that represents# your plugin. You can customize this icon with a symbol of your choice.icon_chr='@'# custom menu icon symbol# The default icon font is Roboto: https://fonts.google.com/specimen/Roboto# Alternatively, you can use icons from the Pupil Icon font:# https://github.com/pupil-labs/pupil-icon-fonticon_font='roboto'# or `pupil_icons` when using the Pupil Icon fontdef__init__(self,g_pool,example_param=1.0):super().__init__(g_pool)# persistent attributeself.example_param=example_paramdefinit_ui(self):# Create a floating menuself.add_menu()self.menu.label='<title>'# Create a simple info texthelp_str="Example info text."self.menu.append(ui.Info_Text(help_str))# Add a slider that represents the persistent valueself.menu.append(ui.Slider('example_param',self,min=0.0,step=0.05,max=1.0,label='Example Param'))defdeinit_ui(self):self.remove_menu()defget_init_dict(self):# all keys need to exists as keyword arguments in __init__ as wellreturn{'example_param':self.example_param}
Export Custom Video Visualizations
As descrbed above, plugins are able to modify the image buffers to export their visualizations. The plugins recent_events method is automatically called for each frame once by the video exporter process. Plugins might overwrite changes made by plugins with a lower order than themselves. OpenGL visualizations are not exported. See vis_circle.py for an example visualization.
Export Custom Raw Data
Each Player plugin gets a notification with subject should_export thar includes the world frame indices range that will be exported and the directory where the recording will be exported to. Add the code to the right to your plugin and implement an export_data function. See fixation_detector.py for an example.
All plugins run within the world process. Doing heavy calculations within any of the periodically called Plugin methods (e.g. recent_events) can result in poor performance of the application. It is recommended to do any heavy calculations within a separate subprocess - multi-threading brings its own problems in Python. We created the Task_Proxy to simplify this procedure. It is initialized with a generator which will be executed in a subprocess. The generator’s results will automatically be piped to the main thread where the plugin can fetch them.
frompluginimportPluginfrompygluiimportuiimportlogginglogger=logging.getLogger(__name__)defexample_generator(mu=0.,sigma=1.,steps=100):'''samples `N(\mu, \sigma^2)`'''importnumpyasnpfromtimeimportsleepforiinrange(steps):# yield progress, datumprogress=(i+1)/stepsvalue=sigma*np.random.randn()+muyieldprogress,valuesleep(np.random.rand()*.1)classCustom_Plugin(Plugin):def__init__(self,g_pool):super().__init__(g_pool)self.proxy=Task_Proxy('Background',example_generator,args=(5.,3.),kwargs={'steps':50})defrecent_events(self,events):# fetch all available resultsforprogress,random_numberintask.fetch():logger.debug('[{:3.0f}%] {:0.2f}'.format(progress*100,random_number))# test if task is completediftask.completed:logger.debug('Task done')defcleanup(self):ifnotself.proxy.completed:logger.debug('Cancelling task')self.proxy.cancel()
Recording Format
Required Files
key,value
Recording Name,2018_07_19
Start Date,19.07.2018
Start Time,14:56:21
Start Time (System),1532004981.666572
Start Time (Synced),701730.897108953
Duration Time,00:00:13
World Camera Frames,402
World Camera Resolution,1280x720
Capture Software Version,1.7.159
Data Format Version,1.8
System Info,"User: name, Platform: Linux, ..."
Each recording requires three files: 1. An info.csv file that includes two columns – key and value. (See left for example) 2. At least one video file and its corresponding timestamp file. See the Video Files section below for details.
A minimum requirement of two key, value pairs are required in the info.csv file. 1. Recording Name,<name> 2. Data Format Version,<version>
Data Files
Timestamp Files
Timestamp files must follow this strict naming convention: Given that a data file is named <name>.<ext> then its timestamps file has to be named <name>_timestamps.npy.
A datum and its timestamp have the same index within their respective files, i.e. the ith timestamp in world_timestamps.npy belongs to the ith video frame in world.mp4.
Video Files
Video files are only recognized if they comply with the following constraints:
Allowed video file extentions are:
.mp4
.mkv
.avi
.h264
.mjpeg
Allowed video file names are:
world: Scene video
eye0: Right eye video
eye1: Left eye video
The video files should look like:
world.mp4, eye0.mjpeg, eye1.mjpeg
We also support multiple parts of video files as input. For instance:
world.mp4, world_001.mp4
eye0.mjpeg, eye0_001.mjpeg
And their corresponding timestamp files should follow the pattern:
world_timestamps.npy, world_001_timestamps.npy
Audio File
An audio file is only recognized in Pupil Player’s playback plugin if the file is named audio.mp4.
pldata Files
These files contain a sequence of independently msgpack-encoded messages. Each message consists of two frames: 1. frame: The payload’s topic as a string, e.g. "pupil.0" 2. frame: The payload, e.g. a pupil datum, encoded as msgpack
For clarification: The second frame is encoded twice!
Pupil Player decodes the messages into file_methods.Serialized_Dicts. Each Serialized_Dict instance holds the serialized second frame and is responsible for decoding it on demand. The class is designed such that there is a maximum number of decoded frames at the same time. This prevents Pupil Player from using excessive amounts of memory.
Files without file extention, e.g. the deprecated pupil_data file, and files with a .meta extention are msgpack-encoded dictionaries. They can be read and written using file_methods.load_object() and file_methods.save_object() and do not have a corresponding timestamps file.
Fixation Detector
In Salvucci and Goldberg define different categories of fixation detectors. One of them describes dispersion-based algorithms:
I-DT identifies fixations as groups of consecutive points within a particular dispersion, or maximum separation. Because fixations typically have a duration of at least 100 ms, dispersion-based identification techniques often incorporate a minimum duration threshold of 100-200 ms to help alleviate equipment variability.[1]
Pupil’s fixation detectors implement such dispersion-based algorithms.
[1] Salvucci, D. D., & Goldberg, J. H. (2000, November). Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 symposium on Eye tracking research & applications (pp. 71-78). ACM.
Usage
Online Fixation Detector
This plugin detects fixations based on a dispersion threshold in terms of degrees of visual angle with a minimum duration. It publishes the fixation as soon as it complies with the constraints (dispersion and duration). This might result in a series of overlapping fixations.
Maximum Dispersion (spatial, degree): Maximum distance between all gaze locations during a fixation.
Minimum Duration (temporal, milliseconds): The minimum duration in which the dispersion threshold must not be exceeded.
Confidence Threshold: Data with a lower confidence than this threshold is not considered during fixation detection.
Offline Fixation Detector
This plugin detects fixations based on a dispersion threshold in terms of degrees of visual angle within a given duration window. It tries to maximize the length of classified fixations within the duration window, e.g. instead of creating two consecutive fixations of length 300 ms it creates a single fixation with length 600 ms. Fixations do not overlap.
Maximum Dispersion (spatial, degree): Maximum distance between all gaze locations during a fixation.
Minimum Duration (temporal, milliseconds): The minimum duration in which the dispersion threshold must not be exceeded.
Maximum Duration (temporal, milliseconds): The maximum duration in which the dispersion threshold must not be exceeded.
Fixation Format
If 3d pupil data is available the fixation dispersion will be calculated based on the positional angle of the eye. These fixations have their method field set to “pupil”. If no 3d pupil data is available the plugin will assume that the gaze data is calibrated and calculate the dispersion in visual angle within the coordinate system of the world camera. These fixations will have their method field set to “gaze”.
Capture
Fixations are represented as dictionaries consisting of the following keys:
topic: Static field set to fixation
norm_pos: Normalized position of the fixation’s centroid
base_data: Gaze data that the fixation is based on
duration: Exact fixation duration, in milliseconds
dispersion: Dispersion, in degrees
timestamp: Timestamp of the first related gaze datum
confidence: Average pupil confidence
method: pupil or gaze
gaze_point_3d: Mean 3d gaze point, only available if pupil method was used
Player
Player detected fixations also include:
start_frame_index: Index of the first related frame
mid_frame_index: Index of the median related frame
end_frame_index: Index of the last related frame
USB Bandwidth And Synchronization
USB Bandwidth limits and ways to make it work regardless
The Pupil headset uses 2-3 cameras that are electrically- and firmware wise identical (except for the name in the usb descriptor). Our Pupil camera can supply frames in various resolutions and rates uncompressed (YUV) and compressed (MJPEG). When looking at uncompressed data even a single camera can saturate a high speed USB bus. This is why we always use MJPEG compression: We can squeeze the data of 3 cameras through one USB bus because the image data is compressed by a factor of ~10.
JPEG size estimation and custom video backends
However, the actual size of each image depends on the complexity of the content (JPEGs of images with more features will be bigger) and implementation details of the camera firmware. Because the cameras use isochronous usb transfers, we need to allocate bandwidth during stream initialization. Here we need to make an estimate on how much bandwidth we believe the camera will require. If we are too conservative we require more bandwidth for 3 cameras than is available and initialization will fail. If we allocate to little, we risk that image transport will fail during capture. According to the UVC specs the amount of bandwidth that is required must be read from the camera usb descriptor and usually this estimate is super conservative. This is why with the normal drivers you can never run more that one camera at decent resolutions on a single usb bus.
With our version of libuvc and pyuvc we ignore the cameras request and estimate the bandwidth ourselves like this:
//the proper way: ask the camera
config_bytes_per_packet=strmh->cur_ctrl.dwMaxPayloadTransferSize;// our way: estimate it:
size_tbandwidth=frame_desc->wWidth*frame_desc->wHeight/8*bandwidth_factor;//the last one is bpp default 4 but we use if for compression, 2 is save, 1.5 is needed to run 3 high speed cameras. on one bus.
bandwidth*=10000000/strmh->cur_ctrl.dwFrameInterval+1;bandwidth/=1000;//unit
bandwidth/=8;// 8 high speed usb microframes per ms
bandwidth+=12;//header size
config_bytes_per_packet=bandwidth;
The scale factor bandwidth_factor is settable through the api.
We have tested these and found that we can run 3 pupil camera at 720p@60fps+2x480p@120fps on our Mac and Linux machines. If you play with the resolutions and frame rates in pupil capture you may hit a combination where the total bandwidth requirements cannot be met, thus the crash (I assume).
Use more BUS
If you want to not be limited by the bandwidth of a single usb bus, you can mod the hardware and expose every camera directly. Just make sure that you also have three free USB controllers (not plugs) on your PC.
Multi Camera Synchronization
Each camera we use is a free running capture device. Additionally each camera runs in a separate process. Instead of frame-locking the camera through special hardware we acquire timestamps for each frame. These timestamps are then used to correlate data from each camera in time and match frames based on closest proximity.
Data from each eye camera is sent via IPC to the world process. Since this involves three separate processes it can happen that data from one camera arrives earlier that another. However for each camera the frames will be ordered and timestamps are monotonically increasing. In the main process we match the available data timewise when we need. In Pupil Player we can do matching after the fact to work with perfectly sorted data from all three cameras. If you require the data to be matched over being recent I would recommend collecting data in the queue for a few more frames in world.py before dispatching them in the events dict. (I ll actually do some tests on this subject soon.)
A note on synchronization and rolling shutters
While synchronization through hardware is preferable, its implementation would come at added hardware cost. The benefits of that become questionable at 120fps. At this rate the frame interval is about 8ms which very close to the exposure time of the eye cameras. Since our cameras use a rolling shutter the image is actually taken continuously and the time of exposure changes based on the pixel position on the sensor. You can think of the camera image stream as a scanning sensor readout with data packed into frames and timestamped with the time of the first pixel readout. If we then match frames from two or more sensors we can assume that the pixels across two camera are generally no further apart in time than the first and last pixel of one frame from a single camera.
License
We want Pupil to proliferate! We want you to use Pupil to empower and inspire whatever you do. Be it academic research, commercial work, teaching, art, or personally motivated projects.
We want you to be a member of the Pupil community and contribute as much as possible. The software is open and the hardware is modular and accessible. We encourage the modification of software in accordance to the open source license.
Software
All source code written by us is open source in accordance with the GNU Lesser General Public License (LGPL v3.0) license. We encourage you to change and improve the code. We require that you will share your work with the Pupil community.
Hardware
The camera mounts of the Pupil headset are open source for non-commercial use. We distribute CAD files for camera mounts and document the interface geometry in the Pupil Hardware Development so that you can use different cameras or customize for your specific needs. Again, we encourage this and are excited to see new designs, ideas, and support for other camera models.
The actual frame of the Pupil headset is not open-source and distributed via Shapeways and direct sales. We do this for a few reasons:
The printed geometry is not the actual CAD file. It is the result of a Finite Element Analysis (FEA) simulation that is exported as a triangle mesh. The CAD file itself is useless because the headset does not fit well without the FEA step. The FEA is useless because it is hard to properly manipulate in a CAD environment.
The entire design is based on the material properties of laser sintered nylon. This is what allows the headset to be so light, flexible, and strong. Unless you own an EOS SLS machine, Shapeways will always outperform the field in terms of price. In other words: The headset does not make any sense in another material and the material/manufacturing is expensive when you buy it from somewhere other than Shapeways.
We take a markup fee for every headset to finance the Pupil project. This fee supports open source development, so that you can continue to get software for free!
If you have ideas and suggestions for improvements on the actual frame of the headset we are happy to collaborate closely on improvements. Contact us.
You can use Pupil in your research, academic work, commercial work, art projects and personal work. We only ask you to credit us appropriately. See Academic Citation for samples.
Pupil is developed and maintained by Pupil Labs. If you make a contribution to open source, we will include your name in our [[Contributors]] page. For more information about the people behind the project, check out Pupil Labs.
The Pupil community is made up of amazing individuals around the world. It is your effort and exchanges that enable us to discover novel applications for eye tracking and help us to improve the open source repository. The Pupil community is active, growing, and thrives from your contributions.
Connect with the community, share ideas, solve problems and help make Pupil awesome!
Find a bug? Raise an issue in the GitHub project (pupil-labs/pupil).
Have you made a new feature, improvement, edit, something cool that you want to share? Send us a pull request. We check on the regular. If we merge your feature(s) you’ll be credited in the Pupil [[Contributors]] page.
Discord
For quick questions and casual discussion, chat with the community on discord.
Email
If you want to talk directly to someone at Pupil Labs, email is the easiest way.
We have been asked a few times about how to cite Pupil in academic research. Please take a look at our papers below for citation options. If you’re using Pupil as a tool in your research please cite the below UbiComp 2014 paper.
Papers that cite Pupil
We have compiled a list of publications that cite Pupil in this spreadsheet
UbiComp 2014 Paper
Title
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
Abstract
In this paper we present Pupil – an accessible, affordable, and extensible open source platform for pervasive eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight eye tracking headset, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a processing pipeline latency of only 0.045 seconds.
@inproceedings{Kassner:2014:POS:2638728.2641695,
author = {Kassner, Moritz and Patera, William and Bulling, Andreas},
title = {Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
booktitle = {Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing},
series = {UbiComp '14 Adjunct},
year = {2014},
isbn = {978-1-4503-3047-3},
location = {Seattle, Washington},
pages = {1151--1160},
numpages = {10},
url = {http://doi.acm.org/10.1145/2638728.2641695},
doi = {10.1145/2638728.2641695},
acmid = {2641695},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {eye movement, gaze-based interaction, mobile eye tracking, wearable computing},
}
Pupil Technical Report
Title
Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
Abstract
Commercial head-mounted eye trackers provide useful features to customers in industry and research but are expensive and rely on closed source hardware and software. This limits the application areas and use of mobile eye tracking to expert users and inhibits user-driven development, customization, and extension. In this paper we present Pupil – an accessible, affordable, and extensible open source platform for mobile eye tracking and gaze-based interaction. Pupil comprises 1) a light-weight headset with high-resolution cameras, 2) an open source software framework for mobile eye tracking, as well as 3) a graphical user interface (GUI) to playback and visualize video and gaze data. Pupil features high-resolution scene and eye cameras for monocular and binocular gaze estimation. The software and GUI are platform-independent and include state-of-the-art algorithms for real-time pupil detection and tracking, calibration, and accurate gaze estimation. Results of a performance evaluation show that Pupil can provide an average gaze estimation accuracy of 0.6 degree of visual angle (0.08 degree precision) with a latency of the processing pipeline of only 0.045 seconds.
@article{KassnerPateraBulling:2014,
author={Kassner, Moritz and Patera, William and Bulling, Andreas},
title={Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction},
keywords={Eye Movement, Mobile Eye Tracking, Wearable Computing, Gaze-based Interaction},
year={2014},
month={April},
archivePrefix = "arXiv",
eprint = "1405.0006",
primaryClass = "cs-cv",
url = {http://arxiv.org/abs/1405.0006}
}
MIT Thesis
Abstract
This thesis explores the nature of a human experience in space through a primary inquiry into vision. This inquiry begins by questioning the existing methods and instruments employed to capture and represent a human experience of space. While existing qualitative and quantitative methods and instruments – from “subjective” interviews to “objective” photographic documentation – may lead to insight in the study of a human experience in space, we argue that they are inherently limited with respect to physiological realities. As one moves about the world, one believes to see the world as continuous and fully resolved. However, this is not how human vision is currently understood to function on a physiological level. If we want to understand how humans visually construct a space, then we must examine patterns of visual attention on a physiological level. In order to inquire into patterns of visual attention in three dimensional space, we need to develop new instruments and new methods of representation. The instruments we require, directly address the physiological realities of vision, and the methods of representation seek to situate the human subject within a space of their own construction. In order to achieve this goal we have developed Pupil, a custom set of hardware and software instruments, that capture the subject’s eye movements. Using Pupil, we have conducted a series of trials from proof of concept – demonstrating the capabilities of our instruments – to critical inquiry of the relationship between a human subject and a space. We have developed software to visualize this unique spatial experience, and have posed open questions based on the initial findings of our trials. This thesis aims to contribute to spatial design disciplines, by providing a new way to capture and represent a human experience of space.
(Authors names appear in alphabetical order - equal hierarchy in authorship.)
@mastersthesis{Kassner:Patera:2012,
title={{PUPIL: Constructing the Space of Visual Attention}},
author={Kassner, Moritz Philipp and Patera, William Rhoades},
year={2012},
school={Massachusetts Institute of Technology},
url={http://hdl.handle.net/1721.1/72626}
}
Chicago Style Citation
Moritz Kassner, William Patera, Pupil: Constructing the Space of Visual Attention, SMArchS Master Thesis, (Cambridge: Massachusetts Institute of Technology, 2012).
APA Style Citation
Kassner, M., & Patera, W. (2012). Pupil: Constructing the space of visual attention (Unpublished master’s thesis). Massachusetts Institute of Technology, Cambridge, MA. Available from http://hdl.handle.net/1721.1/72626
