Plugin API ​

In Pupil Core, plugins are distributed as Python files that are loaded and executed at runtime. To be recognized as such, they need to be installed in the correct place and implement the Plugin API.

The usage of plugins has multiple advantages. For the user, they make it easy to turn features on and off as required. For the developer, it increases maintainability through separation.

Plugins can also be loaded at runtime, extending Pupil's functionality by sharing a simple Python file. See our pupil-community repository for a list of third-party plugins. See below on how to add them.

Adding A Plugin ​

Each Pupil Core software creates its own user directory. It is directly placed in your user's home directory and follows this naming convention: pupil_<name>_settings, e.g. pupil_capture_settings.

Each user directory has a plugins subdirectory into which the plugin files need to be placed. The Pupil Core software will attempt to load the files during the next launch.

If the plugin was installed correctly, it should appear in the Plugin Manager of the corresponding Pupil Core software. Check the log file (~/pupil_<name>_settings/<name>.log) for errors if this is not the case.

Development ​

For the plugin development process, we recommend to run from source.

Language ​

Pupil is written in Python 3.6, but no "heavy lifting" is done in Python. High performance computer vision, media compression, display libraries, and custom functions are written in external libraries or c/c++ and accessed though cython. Python plays the role of "glue" that sticks all the pieces together.

We also like writing code in Python because it's quick and easy to move from initial idea to working proof-of-concept. If proof-of-concept code is slow, optimization and performance enhancement can happen in iterations of code.

Process Structure ​

When Pupil Capture starts, in default settings two processes are spawned:

Eye and World. Both processes grab image frames from a video capture stream but they have very different tasks.

Pupil Player spawns with one main process that visualizes the recording but will spawn more for background calculations, e.g. post-hoc pupil detection or video exports.

Eye Process ​

The eye process only has one purpose - to detect the pupil and broadcast its position. The process breakdown looks like this:

• Grabs eye camera images from eye camera video stream
• Find the pupil position in the image
• Broadcast/stream the detected pupil position.

See the terminology section for the difference between pupil and gaze data.

World Process ​

This is the workhorse in Pupil Capture.

• Grabs the world camera images from the world camera video stream
• Receives pupil positions from the eye process
• Performs calibration mapping from pupil positions to gaze positions
• Loads plugins - to detect fixations, track surfaces, and more...
• Records video and data. Most, and preferably all coordination and control happens within the World process.

Player Process ​

Main process within Pupil Player.

• Visualizes recorded data
• Performs post-hoc analyses
• Exports video and csv data

API Reference ​

Plugins are the recommended way to extend Pupil Core software functionality. They are typically managed within the application's main process which communicates with the plugins via callbacks (see below). These are defined in the root Plugin class. In order to create a new plugin, one has to inherit from this class and overwrite them as desired.

from plugin import Plugin

class MyCustomPlugin(Plugin):
    pass

from plugin import Plugin

class MyCustomPlugin(Plugin):
    pass

Plugins are automatically listed in the Plugin Manager unless they inherit from a set of special classes (System_Plugin_Base, Base_Manager, Base_Source, CalibrationChoreographyPlugin, GazerBase).

Plugin class attributes ​

These class attributes define general plugin behavior. Overwriting them is optional but recommended. alive is an exception and should only be set to False if you want the plugin to close autonomously. Otherwise, this attribute is managed by the enclosing application.

Plugin uniqueness ​

• not unique - plugins can be instantiated multiple times, e.g. gaze visualization plugins
• unique by class - only one plugin instance can exist at a time, e.g. blink detector
• unique by base class - if two plugins share the same base class they are only allowed to be active one at a time. Calibration choreographies are examples of unique by base class plugins. They each implement a separate by-base-class-unique plugin but since they all share the same base class (CalibrationChoreographyPlugin) only one can be active at a time.

If a new instance of a unique plugin is started the old instance will be cleaned up and replaced.

Plugin callback methods ​

Callback methods are triggered by the enclosing application. These can be divided into three categories: Startup/cleanup, processing, and UI interactions.

Startup/cleanup callbacks ​

Callbacks of this kind are only called once in the life cycle of a plugin.

A typical use case of the session settings is to persistently store plugin parameters, e.g. the minimum duration parameter of the fixation detector.

The code below shows how to store a custom value (my_custom_setting) in the session settings. my_custom_setting=5 defines the default value in case no session settings were found when the application was started or the plugin has just been enabled. If session settings were loaded successfully the class will be instantiated with the dictionary previously returned by get_init_dict().

from plugin import Plugin

class MyCustomPlugin(Plugin):
    def __init__(self, g_pool, my_custom_setting=5):
        self._my_custom_setting = my_custom_setting
    
    def get_init_dict(self):
        return {"my_custom_setting": self._my_custom_setting}
        # return {}  # on next launch, recover plugin with default settings
        # raise NotImplementedError  # on next launch, do not recover plugin

from plugin import Plugin

class MyCustomPlugin(Plugin):
    def __init__(self, g_pool, my_custom_setting=5):
        self._my_custom_setting = my_custom_setting
    
    def get_init_dict(self):
        return {"my_custom_setting": self._my_custom_setting}
        # return {}  # on next launch, recover plugin with default settings
        # raise NotImplementedError  # on next launch, do not recover plugin

Processing callbacks ​

As described in the process structure section above, Pupil Core software launches several processes. Each is driven by an infinite loop that processes data in each iteration, the so called "application event cycle". Data is fetched, generated, or processed by calling the plugin processing callbacks below in increasing Plugin.order.

Custom data example:

from plugin import Plugin

CUSTOM_TOPIC = "custom_topic"


class CustomDataExample(Plugin):
    def recent_events(self, events):
        custom_datum = {
            "topic": CUSTOM_TOPIC,
            "timestamp": self.g_pool.get_timestamp(),  # Timestamp in pupil time
            "custom field": 42,
            # Further fields can be added here.
            # Their values should be serializable with msgpack.
        }
        events[CUSTOM_TOPIC] = [custom_datum]

from plugin import Plugin

CUSTOM_TOPIC = "custom_topic"


class CustomDataExample(Plugin):
    def recent_events(self, events):
        custom_datum = {
            "topic": CUSTOM_TOPIC,
            "timestamp": self.g_pool.get_timestamp(),  # Timestamp in pupil time
            "custom field": 42,
            # Further fields can be added here.
            # Their values should be serializable with msgpack.
        }
        events[CUSTOM_TOPIC] = [custom_datum]

Plugin utility methods ​

In addition to the callbacks, the plugin implements a series of useful functions to interact with the application.

Pupil Detection Plugins ​

Starting with version v2.6, Pupil Core supports custom pupil detection plugins that run in the eye process. These plugins can be used to implement custom algorithms for extracting pupillometry data from the eye images and feeding it to the rest of the gaze mapping pipeline.

Pupil detection plugins are supported in Pupil Capture, Pupil Player, and Pupil Service. Similar to regular user plugins, custom pupil detection plugins should be placed inside pupil_capture_settings/plugins for Pupil Capture, pupil_player_settings/plugins for Pupil Player, and pupil_service_settings/plugin for Pupil Service, all of which are located in the user directory of your system. The plugins are automatically loaded when any of the applications are started.

Conceptually, pupil detection plugins are wrappers around pupil detectors to make them work within Pupil Core. Pupil detectors are objects that are responsible for extracting the raw pupillometry data from the eye images. Pupil detector plugins use pupil detectors for extracting the raw data, package that data into pupil datums compatible for the gaze mapping pipeline, and integrates the detector into the rest of the lifecycle of a plugin.

Custom pupil detectors must subclass the DetectorBase class, while custom pupil detection plugins must subclass the PupilDetectorPlugin class.

from pupil_detector_plugins import PupilDetectorPlugin

class MyCustomPupilDetectorPlugin(PupilDetectorPlugin):

    def __init__(self, g_pool):
        super().__init__(g_pool)
        # In some cases, it might be desirable to disable other pupil detectors running in the eye process
        # e.g. to increase performance on systems with limited computing resources.
        # In such cases, these pupil detection plugins can be disabled with the helper method:
        self._stop_other_pupil_detectors()

    def _stop_other_pupil_detectors(self):
        # Getting the Plugin_List instance from the g_pool object
        plugin_list = self.g_pool.plugins
        # Iterating over every plugin in the Plugin_List instance
        for plugin in plugin_list:
            # Deactivating every PupilDetectorPlugin instances except self
            if isinstance(plugin, PupilDetectorPlugin) and plugin is not self:
                plugin.alive = False
        # Forcing Plugin_List instance to remove deactivated plugins
        plugin_list.clean()

    @property
    def pupil_detector(self):
        # This read-only property must be implemented by the custom subclass.
        #
        # Returns an instance of pupil detector;
        # See: https://github.com/pupil-labs/pupil-detectors
        pass

    def detect(self, frame, **kwargs):
        # This method must be implemented by the custom subclass.
        #
        # Returns a pupil datum dictionary containing the pupil location and related metadata.
        # See Pupil Datum Format for a list of required keys:
        # https://docs.pupil-labs.com/core/developer/#pupil-datum-format
        pass

from pupil_detector_plugins import PupilDetectorPlugin

class MyCustomPupilDetectorPlugin(PupilDetectorPlugin):

    def __init__(self, g_pool):
        super().__init__(g_pool)
        # In some cases, it might be desirable to disable other pupil detectors running in the eye process
        # e.g. to increase performance on systems with limited computing resources.
        # In such cases, these pupil detection plugins can be disabled with the helper method:
        self._stop_other_pupil_detectors()

    def _stop_other_pupil_detectors(self):
        # Getting the Plugin_List instance from the g_pool object
        plugin_list = self.g_pool.plugins
        # Iterating over every plugin in the Plugin_List instance
        for plugin in plugin_list:
            # Deactivating every PupilDetectorPlugin instances except self
            if isinstance(plugin, PupilDetectorPlugin) and plugin is not self:
                plugin.alive = False
        # Forcing Plugin_List instance to remove deactivated plugins
        plugin_list.clean()

    @property
    def pupil_detector(self):
        # This read-only property must be implemented by the custom subclass.
        #
        # Returns an instance of pupil detector;
        # See: https://github.com/pupil-labs/pupil-detectors
        pass

    def detect(self, frame, **kwargs):
        # This method must be implemented by the custom subclass.
        #
        # Returns a pupil datum dictionary containing the pupil location and related metadata.
        # See Pupil Datum Format for a list of required keys:
        # https://docs.pupil-labs.com/core/developer/#pupil-datum-format
        pass

See examples of custom pupil detection plugins here.