Network API

There are two different ways that you can interact with the Pupil Core Network API:

Pupil Remote: Simple text-based API to remote control Pupil Core software.
IPC Backbone: msgpack based API with access to realtime data.

In the sections below, we outline how the basic communication works and how you can access the different stages of the Network API.

If you want to run the Python examples below you will have to install the following dependencies:

pip install zmq msgpack==0.5.6

pip install zmq msgpack==0.5.6

Pupil Remote

Pupil Remote provides a simple, text-based API to remote control the Pupil Core software, as well as to access the second Network API stage (IPC Backbone). It uses ZeroMQ's REQ-REP pattern for reliable one-to-one communication.

This is how you connect to Pupil Remote using pyzmq:

import zmq

ctx = zmq.Context()
pupil_remote = zmq.Socket(ctx, zmq.REQ)
pupil_remote.connect('tcp://127.0.0.1:50020')

import zmq

ctx = zmq.Context()
pupil_remote = zmq.Socket(ctx, zmq.REQ)
pupil_remote.connect('tcp://127.0.0.1:50020')

TIP

Pupil Remote accepts requests via a REP socket, by default on port 50020. Alternatively, you can provide a custom port via the --port application argument.

After opening the REQ socket, you can send simple text messages to control Pupil Capture and Pupil Service functions:

# start recording
pupil_remote.send_string('R')
print(pupil_remote.recv_string())

sleep(5)
pupil_remote.send_string('r')
print(pupil_remote.recv_string())

# start recording
pupil_remote.send_string('R')
print(pupil_remote.recv_string())

sleep(5)
pupil_remote.send_string('r')
print(pupil_remote.recv_string())

WARNING

For every message that you send to Pupil Remote, you need to receive the response. If you do not call recv(), Pupil Capture might become unresponsive!

This is an overview over all available Pupil Remote commands:

'R'  # start recording with auto generated session name
'R rec_name'  # start recording named "rec_name"
'r'  # stop recording
'C'  # start currently selected calibration
'c'  # stop currently selected calibration
'T 1234.56'  # resets current Pupil time to given timestamp
't'  # get current Pupil time; returns a float as string.
'v'  # get the Pupil Core software version string

# IPC Backbone communication
'PUB_PORT'  # return the current pub port of the IPC Backbone
'SUB_PORT'  # return the current sub port of the IPC Backbone

'R'  # start recording with auto generated session name
'R rec_name'  # start recording named "rec_name"
'r'  # stop recording
'C'  # start currently selected calibration
'c'  # stop currently selected calibration
'T 1234.56'  # resets current Pupil time to given timestamp
't'  # get current Pupil time; returns a float as string.
'v'  # get the Pupil Core software version string

# IPC Backbone communication
'PUB_PORT'  # return the current pub port of the IPC Backbone
'SUB_PORT'  # return the current sub port of the IPC Backbone

Delayed Execution

Pupil Remote commands can be subject to transmission delay (e.g. from network latency). This is especially important to keep in mind for the T, t, and R commands.

We do not recommend using R for time synchronization. In addition to transmission delay, not all recording processes are guaranteed to start simultaneously when Pupil Capture receives the command. Please read our Best Practices for more appropriate methods of time synchronization.

TIP

Pupil Service does not support the creation of recordings, i.e. the R and r commands do not work with Pupil Service.

See this Python script for a full example interaction with Pupil Remote.

Pupil Groups

The Pupil Groups plugin uses the ZRE protocol to implement real-time local network discovery and many-to-many communication. Common workflows like starting and stopping a recording are already implemented by Pupil Capture to use and respond to the Pupil Groups interface, if available.

If you want to integrate Pupil Groups in your own app or device, have a look at the ZRE protocol specification. From Python we use the pyre library to communicate between groups, but any ZRE implementation can be used. The Pupil Groups plugin joins a user-defined group, by default pupil-groups. Make sure that all devices are in the same group. You can broadcast notifications to all members in the group, by adding the key-value-pair "remote_notify": "all". All messages, whose topic stars with remote_notify are also broadcasted. The following notifications are broadcasted by default:

recording.should_start
recording.should_stop

IPC Backbone

The IPC Backbone grants you realtime access to nearly all data generated by Pupil Capture and Pupil Service. It uses ZeroMQ's PUB-SUB pattern for one-to-many communication.

If you want to tap into the IPC Backbone you will need both the IP address and the session's unique port. You can request them from Pupil Remote:

python

import zmq
ctx = zmq.Context()
# The REQ talks to Pupil remote and receives the session unique IPC SUB PORT
pupil_remote = ctx.socket(zmq.REQ)

ip = 'localhost'  # If you talk to a different machine use its IP.
port = 50020  # The port defaults to 50020. Set in Pupil Capture GUI.

pupil_remote.connect(f'tcp://{ip}:{port}')

# Request 'SUB_PORT' for reading data
pupil_remote.send_string('SUB_PORT')
sub_port = pupil_remote.recv_string()

# Request 'PUB_PORT' for writing data
pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

import zmq
ctx = zmq.Context()
# The REQ talks to Pupil remote and receives the session unique IPC SUB PORT
pupil_remote = ctx.socket(zmq.REQ)

ip = 'localhost'  # If you talk to a different machine use its IP.
port = 50020  # The port defaults to 50020. Set in Pupil Capture GUI.

pupil_remote.connect(f'tcp://{ip}:{port}')

# Request 'SUB_PORT' for reading data
pupil_remote.send_string('SUB_PORT')
sub_port = pupil_remote.recv_string()

# Request 'PUB_PORT' for writing data
pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

Reading from the IPC Backbone

To start reading from the IPC Backbone, you need to subscribe to the topic of your desired data. Once the subscription was successful, you will start receiving data.

#...continued from above
# Assumes `sub_port` to be set to the current subscription port
subscriber = ctx.socket(zmq.SUB)
subscriber.connect(f'tcp://{ip}:{sub_port}')
subscriber.subscribe('gaze.')  # receive all gaze messages

# we need a serializer
import msgpack

while True:
    topic, payload = subscriber.recv_multipart()
    message = msgpack.loads(payload)
    print(f"{topic}: {message}")

#...continued from above
# Assumes `sub_port` to be set to the current subscription port
subscriber = ctx.socket(zmq.SUB)
subscriber.connect(f'tcp://{ip}:{sub_port}')
subscriber.subscribe('gaze.')  # receive all gaze messages

# we need a serializer
import msgpack

while True:
    topic, payload = subscriber.recv_multipart()
    message = msgpack.loads(payload)
    print(f"{topic}: {message}")

See the data conventions and message format sections for details on the data format.

IPC Backbone Message Format

All messages on the IPC Backbone are multipart messages containing (at least) two message frames:

Frame 1 contains the topic string, e.g. pupil.0, logging.info, notify.recording.has_started
Frame 2 contains a msgpack encoded key-value mapping. This is the actual message. We choose msgpack as the serializer due to its efficient format (45% smaller than json, 200% faster than ujson) and because encoders exist for almost every language.

Message Topics

Messages can have any topic chosen by the user. See topics below for a list of message types used by Pupil apps.

Pupil and Gaze Messages

Pupil data is sent from the eye0 and eye1 process with the topic format pupil.<EYE_ID>.<PUPIL_DETECTOR_IDENTIFIER>, where EYE_ID is 0 or 1 for eye0 and eye1 respectively, and PUPIL_DETECTOR_IDENTIFIER is a string that uniquely identifies the pupil detector plugin that produced the message. In the case of the built-in pupil detectors, the identifier corresponds to 2d and 3d respectively. Therefore, the built-in detectors publish the following four topics: pupil.0.2d, pupil.1.2d, pupil.0.3d, pupil.1.3d.

Gaze mappers receive this data and publish messages with topic gaze. See the Timing & Data Conventions section for example messages for the pupil and gaze topics.

Notification Message

Pupil uses special messages called notifications to coordinate all activities. Notifications are key-value mappings with the required field subject. Subjects are grouped by categories category.command_or_statement. Example: recording.should_stop.

python

# message topic:
'notify.recording.should_start'
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}

# message topic:
'notify.recording.should_start'
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}

The message topic construction:

python

topic = f"notify.{notification['subject']}"

topic = f"notify.{notification['subject']}"

You should use the notify topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.

In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more than 10 messages a second, it is probably not a notification but another kind of data. Use a custom topic instead.

python

import zmq
import msgpack

topic = 'your_custom_topic'
payload = {'topic': topic}

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)

# send payload using custom topic
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

import zmq
import msgpack

topic = 'your_custom_topic'
payload = {'topic': topic}

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)

# send payload using custom topic
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

The script above requires you to implement a custom Plugin to process the incoming messages. Alternatively, you can use remote annotations.

Fixation Messages

The Online Fixation Detector in Pupil Capture publishes the following notification:

python

{
    'topic': 'fixation',
    'start_timestamp': 534.5628765  # Timestamp of the first related gaze datum
    'duration': 219.61850000002414  # Exact fixation duration, in milliseconds
    'norm_pos_x': 0.4086194786082147  # Normalized x position of the fixation’s centroid
    'norm_pos_y': 0.5458605572970497  # Normalized y position of the fixation’s centroid
    'dispersion': 1.1102585185279166  # Dispersion, in degrees
    'confidence': 0.9907119103981579  # Average pupil confidence
    'method': '2d gaze'  # Which calibration method was used
    'base_data': "545.53418 545.5361829999999 545.53817 ..."  # Timestamps of all data of the fixation
}

{
    'topic': 'fixation',
    'start_timestamp': 534.5628765  # Timestamp of the first related gaze datum
    'duration': 219.61850000002414  # Exact fixation duration, in milliseconds
    'norm_pos_x': 0.4086194786082147  # Normalized x position of the fixation’s centroid
    'norm_pos_y': 0.5458605572970497  # Normalized y position of the fixation’s centroid
    'dispersion': 1.1102585185279166  # Dispersion, in degrees
    'confidence': 0.9907119103981579  # Average pupil confidence
    'method': '2d gaze'  # Which calibration method was used
    'base_data': "545.53418 545.5361829999999 545.53817 ..."  # Timestamps of all data of the fixation
}

When method is set to 3d gaze, it will also contain the gaze point position:

python

    # ...
    'gaze_point_3d_x': -258.140474196683  # x position of mean 3d gaze point
    'gaze_point_3d_y': 5.005121102396152  # y position of mean 3d gaze point
    'gaze_point_3d_z': 463.9099936463522  # z position of mean 3d gaze point
    # ...

    # ...
    'gaze_point_3d_x': -258.140474196683  # x position of mean 3d gaze point
    'gaze_point_3d_y': 5.005121102396152  # y position of mean 3d gaze point
    'gaze_point_3d_z': 463.9099936463522  # z position of mean 3d gaze point
    # ...

The Offline Fixation Detector in Pupil Player additionally includes the following keys:

python

    # ...
    'start_frame_index': 679  # Index of the first related frame
    'end_frame_index': 685  # Index of the last related frame
    # ...

    # ...
    'start_frame_index': 679  # Index of the first related frame
    'end_frame_index': 685  # Index of the last related frame
    # ...

Blink Messages

The online Blink Detector in Pupil Capture publishes the following notification:

python

{   # blink datum
    'topic': 'blink',
    'confidence': <float>,  # blink confidence
    'timestamp': <timestamp float>,
    'base_data': [<pupil positions>, ...]
    'type': 'onset' or 'offset'
}

{   # blink datum
    'topic': 'blink',
    'confidence': <float>,  # blink confidence
    'timestamp': <timestamp float>,
    'base_data': [<pupil positions>, ...]
    'type': 'onset' or 'offset'
}

Remote Annotations

You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation.

python

{
    'topic': "annotation",
    'label': "Hi this is my annotation 1",
    'timestamp': <pupil time>,
    'duration': 1.0,
}

{
    'topic': "annotation",
    'label': "Hi this is my annotation 1",
    'timestamp': <pupil time>,
    'duration': 1.0,
}

TIP

You can add custom fields to your annotation which will be included in the csv export.

TIP

This script demonstrates how to send remote annotations. Use this script as a starting point for your integrations.

Log Messages

The topic is logging.log_level_name (debug, info, warning, error,...). The message is a key-value mapping that contains all attributes of the python logging.record instance.

python

# message topic:
'logging.warning'
# message payload, logging record attributes as dict:
{
    'levelname': 'WARNING',
    'msg': 'Process started.',
    'name': 'eye',
    ...
}

# message topic:
'logging.warning'
# message payload, logging record attributes as dict:
{
    'levelname': 'WARNING',
    'msg': 'Process started.',
    'name': 'eye',
    ...
}

Pupil Detector Plugin Notifications

The following are notifications handled by pupil detector plugins, that can be sent to the IPC Backbone:

python

# Notification for enabling/disabling any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
{
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
}

# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for enabling/disabling any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
{
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
}

# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

In response to pupil_detector.broadcast_properties, zero or more pupil detector plugins will respond with the following messages:

python

# This message will be sent on the topic `pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>`
{
    "subject": "pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>",
    "values": <detector_properties: dict>,
}

# This message will be sent on the topic `pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>`
{
    "subject": "pupil_detector.properties.<EYE_ID>.<DETECTOR_PLUGIN_CLASS_NAME>",
    "values": <detector_properties: dict>,
}

For an example script that showcases pupil detector plugins' network API, please consult this helper script.

Writing to the IPC Backbone

You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:

python

# continued from above
import msgpack
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = msgpack.dumps(notification)
pupil_remote.send_string(topic, flags=zmq.SNDMORE)
pupil_remote.send(payload)
print(pupil_remote.recv_string())

# continued from above
import msgpack
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = msgpack.dumps(notification)
pupil_remote.send_string(topic, flags=zmq.SNDMORE)
pupil_remote.send(payload)
print(pupil_remote.recv_string())

We say reliable transport because Pupil Remote will confirm every notification we send with 'Notification received'. When we get this message we have a guarantee that the notification was received by the Pupil Core software.

If we listen to the Backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.

Writing to the Backbone directly

If you want to write messages other than notifications onto the IPC Backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.

python

# continued from above...
# Assumes pupil_remote to be a connected REQ socket

import msgpack
from time import time, sleep

pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

publisher = ctx.socket(zmq.PUB)
publisher.connect(f'tcp://{ip}:{pub_port}')

# Wait for the connection to be established...
# See Async connect in "Delivery Guarantees PUB-SUB" below
sleep(1)

notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)

publisher.send_string(topic, flags=zmq.SNDMORE)
publisher.send(payload)

# continued from above...
# Assumes pupil_remote to be a connected REQ socket

import msgpack
from time import time, sleep

pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

publisher = ctx.socket(zmq.PUB)
publisher.connect(f'tcp://{ip}:{pub_port}')

# Wait for the connection to be established...
# See Async connect in "Delivery Guarantees PUB-SUB" below
sleep(1)

notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)

publisher.send_string(topic, flags=zmq.SNDMORE)
publisher.send(payload)

Communicating with Pupil Service

This code shows how to use notifications to start the eye windows, set a calibration method and close Pupil Service:

python

import zmq, msgpack, time

# create a zmq REQ socket to talk to Pupil Service
ctx = zmq.Context()
pupil_remote = ctx.socket(zmq.REQ)
pupil_remote.connect('tcp://localhost:50020')

# convenience function
def send_recv_notification(n):
    pupil_remote.send_string(f"notify.{n['subject']}", flags=zmq.SNDMORE)
    pupil_remote.send(msgpack.dumps(n))
    return pupil_remote.recv_string()

# set start eye windows
n = {'subject':'eye_process.should_start.0','eye_id': 0, 'args':{}}
print(send_recv_notification(n))
n = {'subject':'eye_process.should_start.1','eye_id':1, 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# set calibration method to hmd calibration
n = {'subject':'start_plugin','name':'HMD_Calibration', 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# close Pupil Service
n = {'subject':'service_process.should_stop'}
print(send_recv_notification(n))

import zmq, msgpack, time

# create a zmq REQ socket to talk to Pupil Service
ctx = zmq.Context()
pupil_remote = ctx.socket(zmq.REQ)
pupil_remote.connect('tcp://localhost:50020')

# convenience function
def send_recv_notification(n):
    pupil_remote.send_string(f"notify.{n['subject']}", flags=zmq.SNDMORE)
    pupil_remote.send(msgpack.dumps(n))
    return pupil_remote.recv_string()

# set start eye windows
n = {'subject':'eye_process.should_start.0','eye_id': 0, 'args':{}}
print(send_recv_notification(n))
n = {'subject':'eye_process.should_start.1','eye_id':1, 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# set calibration method to hmd calibration
n = {'subject':'start_plugin','name':'HMD_Calibration', 'args':{}}
print(send_recv_notification(n))
time.sleep(2)

# close Pupil Service
n = {'subject':'service_process.should_stop'}
print(send_recv_notification(n))

The code demonstrates how you can listen to all notifications from Pupil Service. It requires a little helper script called zmq_tools.py.

python

from zmq_tools import *

ctx = zmq.Context()
requester = ctx.socket(zmq.REQ)
requester.connect('tcp://localhost:50020') #change ip if using remote machine

# request 'SUB_PORT' for reading data
requester.send_string('SUB_PORT')
ipc_sub_port = requester.recv_string()

monitor = Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',)) #change ip if using remote machine

while True:
    print(monitor.recv())

from zmq_tools import *

ctx = zmq.Context()
requester = ctx.socket(zmq.REQ)
requester.connect('tcp://localhost:50020') #change ip if using remote machine

# request 'SUB_PORT' for reading data
requester.send_string('SUB_PORT')
ipc_sub_port = requester.recv_string()

monitor = Msg_Receiver(ctx,'tcp://localhost:%s'%ipc_sub_port,topics=('notify.',)) #change ip if using remote machine

while True:
    print(monitor.recv())

IPC Backbone Implementation

This section provides detailed inside information about the IPC Backbone implementation. Please see specifically the subsection about delivery guarantees.

Pupil Core software uses a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors' messages. Therefore, it is the Backbone of all communication to/from and within the Pupil Core software.

TIP

Note - The main process does not do any CPU heavy work. It only runs the proxy, launches other processes and does a few other light tasks.

IPC Backbone used by Pupil Capture and Service

The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate.

Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.)
Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.

Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.

Delivery guarantees ZMQ

ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don't want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.

Messages are guaranteed to be delivered whole or not at all.
Unlike bare TCP it is ok to connect before binding.
ZMQ will try to repair broken connections in the background for us.
It will deal with a lot of low level tcp handling so we don't have to.

Delivery Guarantees PUB-SUB

ZMQ PUB SUB will make no guarantees for delivery. Reasons for not receiving messages are:

Async Connect/The Late joiner: PUB sockets drop messages before a connection has been established and topics subscribed. ZMQ connects asynchronously in the background.
The Snail: If SUB sockets do not consume delivered messages fast enough they start dropping them.
Fast close: A PUB socket may loose packages if you close it right after sending.

For more information see ZMQ Guide Chapter 5 - Advanced Pub-Sub Patterns.

TIP

In order to avoid accidentally dropping notifications in Pupil, we use a PUSH instead of an PUB socket. It acts as an intermediary for notifications and guarantees that any notification sent to the IPC Backbone, is processed and published by it.

Delivery Guarantees REQ-REP

When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.

Delivery Guarantees in general

We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.

However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.

Latency

Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests...

python

# continued from above
ts = []
for x in range(100):
    sleep(0.003) #simulate spaced requests as in real world
    t = time()
    requester.send_string('t')
    requester.recv_string()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts), max(ts))
>>> 0.000266075134277 0.000597472190857 0.00339102745056

# continued from above
ts = []
for x in range(100):
    sleep(0.003) #simulate spaced requests as in real world
    t = time()
    requester.send_string('t')
    requester.recv_string()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts), max(ts))
>>> 0.000266075134277 0.000597472190857 0.00339102745056

... and when talking directly to the IPC Backbone and waiting for the same message to appear to the subscriber:

python

# continued from above
monitor = Msg_Receiver(ctx, sub_url, topics=('notify.pingback_test',))
sleep(1.)

ts = []
for x in range(100):
    sleep(0.003)  #simulate spaced requests as in real world
    t = time()
    #notify is a method of the Msg_Dispatcher class in zmq_tools.py
    notification = {'subject':'pingback_test'}
    topic = 'notify.' + notification['subject']
    payload = serializer.dumps(notification)
    publisher.send_string(topic, flags=zmq.SNDMORE)
    publisher.send(payload)
    monitor.recv()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts) , max(ts))
>>>0.000180959701538 0.000300960540771 0.000565052032471

# continued from above
monitor = Msg_Receiver(ctx, sub_url, topics=('notify.pingback_test',))
sleep(1.)

ts = []
for x in range(100):
    sleep(0.003)  #simulate spaced requests as in real world
    t = time()
    #notify is a method of the Msg_Dispatcher class in zmq_tools.py
    notification = {'subject':'pingback_test'}
    topic = 'notify.' + notification['subject']
    payload = serializer.dumps(notification)
    publisher.send_string(topic, flags=zmq.SNDMORE)
    publisher.send(payload)
    monitor.recv()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts) , max(ts))
>>>0.000180959701538 0.000300960540771 0.000565052032471

Throughput

During a test we have run dual 120fps eye tracking with a dummy gaze mapper that turned every pupil datum into a gaze datum. This is effectively 480 messages/sec. The main process running the IPC Backbone proxy showed a cpu load of 3% on a MacBook Air (late 2012).

Artificially increasing the pupil messages by a factor 100 increases the message load to 24.000 pupil messages/sec. At this rate the gaze mapper cannot keep up but the IPC Backbone proxy runs at only 38% cpu load.

It appears ZMQ is indeed highly optimized for speed.

Network API ​

Pupil Remote ​

Pupil Groups ​

IPC Backbone ​

Reading from the IPC Backbone ​

IPC Backbone Message Format ​

Message Topics ​

Pupil and Gaze Messages ​

Notification Message ​

Fixation Messages ​

Blink Messages ​

Remote Annotations ​

Log Messages ​

Pupil Detector Plugin Notifications ​

Writing to the IPC Backbone ​

Writing to the Backbone directly ​

Communicating with Pupil Service ​

IPC Backbone Implementation ​

IPC Backbone used by Pupil Capture and Service ​

Delivery guarantees ZMQ ​

Delivery Guarantees PUB-SUB ​

Delivery Guarantees REQ-REP ​

Delivery Guarantees in general ​

Latency ​

Throughput ​

Network API

Pupil Remote

Pupil Groups

IPC Backbone

Reading from the IPC Backbone

IPC Backbone Message Format

Message Topics

Pupil and Gaze Messages

Notification Message

Fixation Messages

Blink Messages

Remote Annotations

Log Messages

Pupil Detector Plugin Notifications

Writing to the IPC Backbone

Writing to the Backbone directly

Communicating with Pupil Service

IPC Backbone Implementation

IPC Backbone used by Pupil Capture and Service

Delivery guarantees ZMQ

Delivery Guarantees PUB-SUB

Delivery Guarantees REQ-REP

Delivery Guarantees in general

Latency

Throughput