# Network API

There are two different ways that you can interact with the Pupil Core Network API:

  1. Pupil Remote: Simple text-based API to remote control Pupil Core software.
  2. IPC Backbone: msgpack based API with access to realtime data.

In the sections below, we outline how the basic communication works and how you can access the different stages of the Network API.

If you want to run the Python examples below you will have to install the following dependencies:

pip install zmq msgpack==0.5.6

# Pupil Remote

Pupil Remote provides a simple, text-based API to remote control the Pupil Core software, as well as to access the second Network API stage (IPC Backbone). It uses ZeroMQ's REQ-REP pattern for reliable one-to-one communication.

This is how you connect to Pupil Remote using pyzmq:

import zmq
ctx = zmq.Context()
pupil_remote = zmq.Socket(ctx, zmq.REQ)

Pupil Remote accepts requests via a REP socket, by default on port 50020. Alternatively, you can provide a custom port via the --port application argument.

After opening the REQ socket, you can send simple text messages to control Pupil Capture and Pupil Service functions:

# start recording

For every message that you send to Pupil Remote, you need to receive the response. If you do not call recv(), Pupil Capture might become unresponsive!

This is an overview over all available Pupil Remote commands:

'R'  # start recording with auto generated session name
'R rec_name'  # start recording named "rec_name"
'r'  # stop recording
'C'  # start currently selected calibration
'c'  # stop currently selected calibration
'T 1234.56'  # resets current Pupil time to given timestamp
't'  # get current Pupil time; returns a float as string.
'v'  # get the Pupil Core software version string
# IPC Backbone communication
'PUB_PORT'  # return the current pub port of the IPC Backbone
'SUB_PORT'  # return the current sub port of the IPC Backbone

Delayed Execution

Pupil Remote commands can be subject to transmission delay (e.g. from network latency). This is especially important to keep in mind for the T, t, and R commands.

We do not recommend using R for time synchronization. In addition to transmission delay, not all recording processes are guaranteed to start simultaneously when Pupil Capture receives the command. Please read our Best Practices for more appropriate methods of time synchronization.

Pupil Service does not support the creation of recordings, i.e. the R and r commands do not work with Pupil Service.

See this Python script for a full example interaction with Pupil Remote.

# Pupil Groups

The Pupil Groups plugin uses the ZRE protocol to implement real-time local network discovery and many-to-many communication. Common workflows like starting and stopping a recording are already implemented by Pupil Capture to use and respond to the Pupil Groups interface, if available.

If you want to integrate Pupil Groups in your own app or device, have a look at the ZRE protocol specification. From Python we use the pyre library to communicate between groups, but any ZRE implementation can be used. The Pupil Groups plugin joins a user-defined group, by default pupil-groups. Make sure that all devices are in the same group. You can broadcast notifications to all members in the group, by adding the key-value-pair "remote_notify": "all". All messages, whose topic stars with remote_notify are also broadcasted. The following notifications are broadcasted by default:

  • recording.should_start
  • recording.should_stop

# IPC Backbone

The IPC Backbone grants you realtime access to nearly all data generated by Pupil Capture and Pupil Service. It uses ZeroMQ's PUB-SUB pattern for one-to-many communication.

If you want to tap into the IPC Backbone you will need both the IP address and the session's unique port. You can request them from Pupil Remote:

import zmq
ctx = zmq.Context()
# The REQ talks to Pupil remote and receives the session unique IPC SUB PORT
pupil_remote = ctx.socket(zmq.REQ)
ip = 'localhost'  # If you talk to a different machine use its IP.
port = 50020  # The port defaults to 50020. Set in Pupil Capture GUI.
# Request 'SUB_PORT' for reading data
sub_port = pupil_remote.recv_string()
# Request 'PUB_PORT' for writing data
pub_port = pupil_remote.recv_string()

# Reading from the IPC Backbone

To start reading from the IPC Backbone, you need to subscribe to the topic of your desired data. Once the subscription was successful, you will start receiving data.

#...continued from above
# Assumes `sub_port` to be set to the current subscription port
subscriber = ctx.socket(zmq.SUB)
subscriber.subscribe('gaze.')  # receive all gaze messages
# we need a serializer
import msgpack
while True:
    topic, payload = subscriber.recv_multipart()
    message = msgpack.loads(payload)
    print(f"{topic}: {message}")

See the data conventions and message format sections for details on the data format.

# IPC Backbone Message Format

All messages on the IPC Backbone are multipart messages containing (at least) two message frames:

  • Frame 1 contains the topic string, e.g. pupil.0, logging.info, notify.recording.has_started

  • Frame 2 contains a msgpack encoded key-value mapping. This is the actual message. We choose msgpack as the serializer due to its efficient format (45% smaller than json, 200% faster than ujson) and because encoders exist for almost every language.

# Message Topics

Messages can have any topic chosen by the user. See topics below for a list of message types used by Pupil apps.

# Pupil and Gaze Messages

Pupil data is sent from the eye0 and eye1 process with the topic format pupil.<EYE_ID>.<PUPIL_DETECTOR_IDENTIFIER>, where EYE_ID is 0 or 1 for eye0 and eye1 respectively, and PUPIL_DETECTOR_IDENTIFIER is a string that uniquely identifies the pupil detector plugin that produced the message. In the case of the built-in pupil detectors, the identifier corresponds to 2d and 3d respectively. Therefore, the built-in detectors publish the following four topics: pupil.0.2d, pupil.1.2d, pupil.0.3d, pupil.1.3d.

Gaze mappers receive this data and publish messages with topic gaze. See the Timing & Data Conventions section for example messages for the pupil and gaze topics.

# Notification Message

Pupil uses special messages called notifications to coordinate all activities. Notifications are key-value mappings with the required field subject. Subjects are grouped by categories category.command_or_statement. Example: recording.should_stop.

# message topic:
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}

The message topic construction:

topic = f"notify.{notification['subject']}"

You should use the notify topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.

In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more than 10 messages a second, it is probably not a notification but another kind of data. Use a custom topic instead.

import zmq
import msgpack
topic = 'your_custom_topic'
payload = {'topic': topic}
# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
# send payload using custom topic
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

The script above requires you to implement a custom Plugin to process the incoming messages. Alternatively, you can use remote annotations.

# Fixation Messages

The Online Fixation Detector in Pupil Capture publishes the following notification:

    'topic': 'fixation',
    'start_timestamp': 534.5628765  # Timestamp of the first related gaze datum
    'duration': 219.61850000002414  # Exact fixation duration, in milliseconds
    'norm_pos_x': 0.4086194786082147  # Normalized x position of the fixation’s centroid
    'norm_pos_y': 0.5458605572970497  # Normalized y position of the fixation’s centroid
    'dispersion': 1.1102585185279166  # Dispersion, in degrees
    'confidence': 0.9907119103981579  # Average pupil confidence
    'method': '2d gaze'  # Which calibration method was used
    'base_data': "545.53418 545.5361829999999 545.53817 ..."  # Timestamps of all data of the fixation

When method is set to 3d gaze, it will also contain the gaze point position:

    # ...
    'gaze_point_3d_x': -258.140474196683  # x position of mean 3d gaze point
    'gaze_point_3d_y': 5.005121102396152  # y position of mean 3d gaze point
    'gaze_point_3d_z': 463.9099936463522  # z position of mean 3d gaze point
    # ...

The Offline Fixation Detector in Pupil Player additionally includes the following keys:

    # ...
    'start_frame_index': 679  # Index of the first related frame
    'end_frame_index': 685  # Index of the last related frame
    # ...

The online Blink Detector in Pupil Capture publishes the following notification:

{   # blink datum
    'topic': 'blink',
    'confidence': <float>,  # blink confidence
    'timestamp': <timestamp float>,
    'base_data': [<pupil positions>, ...]
    'type': 'onset' or 'offset'

# Remote Annotations

You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation.

    'topic': "annotation",
    'label': "Hi this is my annotation 1",
    'timestamp': <pupil time>,
    'duration': 1.0,

You can add custom fields to your annotation which will be included in the csv export.

This script demonstrates how to send remote annotations. Use this script as a starting point for your integrations.

# Log Messages

The topic is logging.log_level_name (debug, info, warning, error,...). The message is a key-value mapping that contains all attributes of the python logging.record instance.

# message topic:
# message payload, logging record attributes as dict:
    'levelname': 'WARNING',
    'msg': 'Process started.',
    'name': 'eye',

# Pupil Detector Plugin Notifications

The following are notifications handled by pupil detector plugins, that can be sent to the IPC Backbone:

# Notification for enabling/disabling any or all pupil detector plugins
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional

In response to pupil_detector.broadcast_properties, zero or more pupil detector plugins will respond with the following messages:

# This message will be sent on the topic `pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>`
    "subject": "pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>",
    "values": <detector_properties: dict>,

For an example script that showcases pupil detector plugins' network API, please consult this helper script.

# Writing to the IPC Backbone

You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:

# continued from above
import msgpack
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = msgpack.dumps(notification)
pupil_remote.send_string(topic, flags=zmq.SNDMORE)

We say reliable transport because Pupil Remote will confirm every notification we send with 'Notification received'. When we get this message we have a guarantee that the notification was received by the Pupil Core software.

If we listen to the Backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.

# Writing to the Backbone directly

If you want to write messages other than notifications onto the IPC Backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.

# continued from above...
# Assumes pupil_remote to be a connected REQ socket
import msgpack
from time import time, sleep
pub_port = pupil_remote.recv_string()
publisher = ctx.socket(zmq.PUB)
# Wait for the connection to be established...
# See Async connect in "Delivery Guarantees PUB-SUB" below
notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)
publisher.send_string(topic, flags=zmq.SNDMORE)

# Communicating with Pupil Service

This code shows how to use notifications to start the eye windows, set a calibration method and close Pupil Service:

import zmq, msgpack, time
# create a zmq REQ socket to talk to Pupil Service
ctx = zmq.Context()
pupil_remote = ctx.socket(zmq.REQ)
# convenience function
def send_recv_notification(n):
    pupil_remote.send_string(f"notify.{n['subject']}", flags=zmq.SNDMORE)
    return pupil_remote.recv_string()
# set start eye windows
n = {'subject':'eye_process.should_start.0','eye_id': 0, 'args':{}}
n = {'subject':'eye_process.should_start.1','eye_id':1, 'args':{}}
# set calibration method to hmd calibration
n = {'subject':'start_plugin','name':'HMD_Calibration', 'args':{}}
# close Pupil Service
n = {'subject':'service_process.should_stop'}

The code demonstrates how you can listen to all notifications from Pupil Service. It requires a little helper script called zmq_tools.py.

from zmq_tools import *
ctx = zmq.Context()
requester = ctx.socket(zmq.REQ)
requester.connect('tcp://localhost:50020') #change ip if using remote machine
# request 'SUB_PORT' for reading data
ipc_sub_port = requester.recv_string()
monitor = Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',)) #change ip if using remote machine
while True:

# IPC Backbone Implementation

This section provides detailed inside information about the IPC Backbone implementation. Please see specifically the subsection about delivery guarantees.

Pupil Core software uses a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors' messages. Therefore, it is the Backbone of all communication to/from and within the Pupil Core software.

Note - The main process does not do any CPU heavy work. It only runs the proxy, launches other processes and does a few other light tasks.

# IPC Backbone used by Pupil Capture and Service

The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate.

  • Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.)
  • Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.

Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.

# Delivery guarantees ZMQ

ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don't want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.

  • Messages are guaranteed to be delivered whole or not at all.
  • Unlike bare TCP it is ok to connect before binding.
  • ZMQ will try to repair broken connections in the background for us.
  • It will deal with a lot of low level tcp handling so we don't have to.

# Delivery Guarantees PUB-SUB

ZMQ PUB SUB will make no guarantees for delivery. Reasons for not receiving messages are:

  • Async Connect/The Late joiner: PUB sockets drop messages before a connection has been established and topics subscribed. ZMQ connects asynchronously in the background.
  • The Snail: If SUB sockets do not consume delivered messages fast enough they start dropping them.
  • Fast close: A PUB socket may loose packages if you close it right after sending.

For more information see ZMQ Guide Chapter 5 - Advanced Pub-Sub Patterns.

In order to avoid accidentally dropping notifications in Pupil, we use a PUSH instead of an PUB socket. It acts as an intermediary for notifications and guarantees that any notification sent to the IPC Backbone, is processed and published by it.

# Delivery Guarantees REQ-REP

When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.

# Delivery Guarantees in general

We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.

However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.

# Latency

Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests...

# continued from above
ts = []
for x in range(100):
    sleep(0.003) #simulate spaced requests as in real world
    t = time()
print(min(ts), sum(ts)/len(ts), max(ts))
>>> 0.000266075134277 0.000597472190857 0.00339102745056

... and when talking directly to the IPC Backbone and waiting for the same message to appear to the subscriber:

# continued from above
monitor = Msg_Receiver(ctx, sub_url, topics=('notify.pingback_test',))
ts = []
for x in range(100):
    sleep(0.003)  #simulate spaced requests as in real world
    t = time()
    #notify is a method of the Msg_Dispatcher class in zmq_tools.py
    notification = {'subject':'pingback_test'}
    topic = 'notify.' + notification['subject']
    payload = serializer.dumps(notification)
    publisher.send_string(topic, flags=zmq.SNDMORE)
print(min(ts), sum(ts)/len(ts) , max(ts))
>>>0.000180959701538 0.000300960540771 0.000565052032471

# Throughput

During a test we have run dual 120fps eye tracking with a dummy gaze mapper that turned every pupil datum into a gaze datum. This is effectively 480 messages/sec. The main process running the IPC Backbone proxy showed a cpu load of 3% on a MacBook Air (late 2012).

Artificially increasing the pupil messages by a factor 100 increases the message load to 24.000 pupil messages/sec. At this rate the gaze mapper cannot keep up but the IPC Backbone proxy runs at only 38% cpu load.

It appears ZMQ is indeed highly optimized for speed.