Overview

Welcome to Pupil Core developer documentation. If you haven't already, we highly recommend reading the Getting Started section and the User Guide, before continuing with the developer documentation.

Have a question? Get in touch with developers and other community members on the #pupil-software-dev channel on Discord.


Where to start?

There are a number of ways you can interact with Pupil Core as a developer.

  • Use the realtime API: The Pupil Core Network API allows you to remote control Pupil Core software and send/receive data realtime. It provides access to nearly all data generated by Pupil Core software: gaze, fixation, video data, and much more! The API can be used with any programming language that supports zeromq and msgpack.
  • Develop a plugin: Pupil Core software is desgined with extendability in mind. It provides a simple yet powerful Plugin API that is used by nearly all existing Pupil Core components. Develop a plugin when you need to access to data that is not provided via the Network API.
  • Modify source code: Can't do what you need to do with the network based API or plugin? Then get ready to dive into the inner workings of Pupil, set up dependencies, and run from source!

In most cases you can simply download Pupil Core app bundles and extend the functionality via API or Plugin.

Terminology

There are a lot of new terms that are specific to eye and to Pupil Core. We have compiled a small list in the terminology section.

Timing & Data Conventions

Pupil Capture is designed to work with multiple cameras that free-run at different frame rates that may not be in sync. World and eye images are timestamped and any resulting artifacts (detected pupil, markers, etc) inherit the source timestamp. Any correlation of these data streams is the responsibility of the functional part that needs the data to be correlated (e.g. calibration, visualization, analyses).

Example: Pupil Capture data format records the world video frames with their respective timestamps. Independent of this, the recorder saves the detected gaze and pupil positions at their own frame rate and with their timestamps. For details about the stored data, see the recording format section.

Pupil Core software uses simple key-value structures to represent single data points. Key-value structures can easily be serialized and nearly all programming languages have an implementation for them.

Each data point should have at least two keys topic and timestamp. Each data point should be uniquely identifiable by its topic and timestamp.

  1. topic: Identifies the type of the object. We recommend that you specify subtypes, separated by a .
  2. timestamp: Pupil time at which the datum was generated.

Pupil Datum Format

The pupil detector generates pupil data from eye images. In addition to the pupil topic and the timestamp (inherited from the eye image), the pupil detector adds fields most importantly:

  • norm_pos: Pupil location in normalised eye coordinates, and
  • confidence: Value indicating quality of the measurement

By default, the Pupil Core software uses the 3d detector for pupil detection. Since it is an extension of the 2d detector, its data contains keys that were inherited from the 2d detection, as well as 3d detector specific keys. Below you can see the Python representation of a 3d pupil datum:

{
    # pupil datum
    'topic': 'pupil.0',
    'method': '3d c++',
    'norm_pos': [0.5, 0.5],  # norm space, [0, 1]
    'diameter': 0.0,  # 2D image space, unit: pixel
    'timestamp': 535741.715303987,  # time, unit: seconds
    'confidence': 0.0,  # [0, 1]
    
    # 2D ellipse of the pupil in image coordinates
    'ellipse': {  # image space, unit: pixel
        'angle': 90.0,  # unit: degrees
        'center': [320.0, 240.0],
        'axes': [0.0, 0.0],
    },
    'id': 0,  # eye id, 0 or 1
    
    ## 3D model data
    # -1 means that the model is building up and has not finished fitting
    'model_birth_timestamp': -1.0,
    'model_confidence': 0.0,
    'model_id': 1,
    
    # pupil polar coordinates on 3D eye model. The model assumes a fixed
    # eye ball size. Therefore there is no `radius` key
    'theta': 0,
    'phi': 0,
    
    # 3D pupil ellipse
    'circle_3d': {  # 3D space, unit: mm
        'normal': [0.0, -0.0, 0.0],
        'radius': 0.0,
        'center': [0.0, -0.0, 0.0],
    },
    'diameter_3d': 0.0,  # 3D space, unit: mm
    
    # 3D eye ball sphere
    'sphere': {  # 3D space, unit: mm
        'radius': 0.0,
        'center': [0.0, -0.0, 0.0],
    },
    'projected_sphere': {  # image space, unit: pixel
        'angle': 90.0,
        'center': [0, 0],
        'axes': [0, 0],
    },
}

Gaze Datum Format

Gaza data is based on one (monocular) or two (binocular) pupil positions. The gaze mapper is automatically setup after calibration and maps pupil positions into the world camera coordinate system. The pupil data on which the gaze datum is based on can be accessed using the base_data key.

{
    # monocular gaze datum
    'topic': 'gaze.3d.1.',
    'confidence': 1.0,  # [0, 1]
    'norm_pos': [x, y],  # norm space, [0, 1]
    'timestamp': ts,  # time, unit: seconds

    # 3D space, unit: mm
    'gaze_normal_3d': [x, y, z],
    'eye_center_3d': [x, y, z],
    'gaze_point_3d': [x, y, z],
    'base_data': [<pupil datum>]  # list of pupil data used to calculate gaze
} 
{
    # binocular gaze datum
    'topic': 'gaze.3d.01.',
    'confidence': 1.0,  # [0, 1]
    'norm_pos': [x, y],  # norm space, [0, 1]
    'timestamp': ts,  # time, unit: seconds

    # 3D space, unit: mm
    'gaze_normals_3d': {
        0: [x, y, z],
        1: [x, y, z],
    },
    'eye_centers_3d': {
        0: [x, y, z],
        1: [x, y, z],
    },
    'gaze_point_3d': [x, y, z],
    'base_data': [<pupil datum>]  # list of pupil data used to calculate gaze
}

Running From Source

Follow the setup instructions for your OS on the Pupil Core Github repo

When running from source, the user settings are not placed in the user's home directory but in the root directory of the cloned repository.