Network API

There are two different ways that you can interact with the Pupil Core Network API:

  1. Pupil Remote: Simple text-based API to remote control Pupil Core software.
  2. IPC Backbone: msgpack based API with access to realtime data.

In the sections below, we outline how the basic communication works and how you can access the different stages of the Network API.

Pupil Remote

Pupil Remote provides a simple, text-based API to remote control the Pupil Core software, as well as to access the second Network API stage (IPC Backbone). It uses ZeroMQ's REQ-REP pattern for reliable one-to-one communication.

This is how you connect to Pupil Remote using pymzq:

import zmq

ctx = zmq.Context()
pupil_remote = zmq.Socket(ctx, zmq.REQ)
pupil_remote.connect('tcp://127.0.0.1:50020')

Pupil Remote accepts requests via a REP socket, by default on port 50020. Alternatively, you can provide a custom port via the --port application argument.

After opening the REQ socket, you can send simple text messages to control Pupil Capture and Pupil Service functions:

# start recording
pupil_remote.send_string('R')
print(pupil_remote.recv_string())

sleep(5)
pupil_remote.send_string('r')
print(pupil_remote.recv_string())

For every message that you send to Pupil Remote, you need to receive the response. If you do not call recv(), Pupil Capture might become unresponsive!

This is an overview over all available Pupil Remote commands:

'R'  # start recording with auto generated session name
'R rec_name'  # start recording named "rec_name" 
'r'  # stop recording
'C'  # start currently selected calibration
'c'  # stop currently selected calibration
'T 1234.56'  # resets current Pupil time to given timestamp
't'  # get current Pupil time; returns a float as string.
'v'  # get the Pupil Core software version string

# IPC Backbone communication
'PUB_PORT'  # return the current pub port of the IPC Backbone
'SUB_PORT'  # return the current sub port of the IPC Backbone

Pupil Service does not support the creation of recordings, i.e. the R and r commands do not work with Pupil Service.

See this Python script for a full example interaction with Pupil Remote.

IPC Backbone

The IPC Backbone grants you realtime access to nearly all data generated by Pupil Capture and Pupil Service. It uses ZeroMQ's PUB-SUB pattern for one-to-many communication.

If you want to tap into the IPC Backbone you will need both the IP address and the session's unique port. You can request them from Pupil Remote:

import zmq
ctx = zmq.Context()
# The REQ talks to Pupil remote and receives the session unique IPC SUB PORT
pupil_remote = ctx.socket(zmq.REQ)

ip = 'localhost'  # If you talk to a different machine use its IP.
port = 50020  # The port defaults to 50020. Set in Pupil Capture GUI.

pupil_remote.connect(f'tcp://{ip}:{port}')

# Request 'SUB_PORT' for reading data
pupil_remote.send_string('SUB_PORT')
sub_port = pupil_remote.recv_string()

# Request 'PUB_PORT' for writing data
pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

Reading from the IPC Backbone

To start reading from the IPC Backbone, you need to subscribe to the topic of your desired data. Once the subscription was successful, you will start receiving data.

#...continued from above
# Assumes `sub_port` to be set to the current subscription port
subscriber = ctx.socket(zmq.SUB)
subscriber.connect(f'tcp://{ip}:{sub_port}')
subscriber.subscribe('gaze.')  # receive all gaze messages

# we need a serializer
import msgpack

while True:
    topic, payload = subscriber.recv_multipart()
    message = msgpack.loads(payload)
    print(f"{topic}: {message}")

See the data conventions and message format sections for details on the data format.

IPC Backbone Message Format

All messages on the IPC Backbone are multipart messages containing (at least) two message frames:

  • Frame 1 contains the topic string, e.g. pupil.0, logging.info, notify.recording.has_started

  • Frame 2 contains a msgpack encoded key-value mapping. This is the actual message. We choose msgpack as the serializer due to its efficient format (45% smaller than json, 200% faster than ujson) and because encoders exist for almost every language.

Message Topics

Messages can have any topic chosen by the user. See topics below for a list of message types used by Pupil apps.

Pupil and Gaze Messages

Pupil data is sent from the eye0 and eye1 process with the topic pupil.0 or pupil.1. Gaze mappers receive this data and publish messages with topic gaze. See the Timing & Data Conventions section for example messages for the pupil and gaze topics.

Notification Message

Pupil uses special messages called notifications to coordinate all activities. Notifications are key-value mappings with the required field subject. Subjects are grouped by categories category.command_or_statement. Example: recording.should_stop.

# message topic:
'notify.recording.should_start'
# message payload, a notification dict
{'subject':'recording.should_start', 'session_name':'my session'}

The message topic construction:

topic = f"notify.{notification['subject']}"

You should use the notification topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.

TODO: Link on_notify docs

In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more than 10 messages a second, it is probably not a notification but another kind of data. Make a custom topic instead.

import zmq
import msgpack

topic = 'your_custom_topic'
payload = {'topic': topic}

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)

# send payload using custom topic
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

The script above requires you to implement a custom Plugin to process the incoming messages. Alternatively, you can use remote annotations.

Remote Annotations

You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation.

{
    'topic': "annotation",
    'label': "Hi this is my annotation 1",
    'timestamp': <pupil time>,
    'duration': 1.0,
}

You can add custom fields to your annotation which will be included in the csv export.

This script demonstrates how to send remote annotations. Use this script as a starting point for your integrations.

Log Messages

The topic is logging.log_level_name (debug, info, warning, error,...). The message is a key-value mapping that contains all attributes of the python logging.record instance.

# message topic:
'logging.warning'
# message payload, logging record attributes as dict:
{
    'levelname': 'WARNING',
    'msg': 'Process started.',
    'name': 'eye',
    ...
}

Writing to the IPC Backbone

You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:

# continued from above
import msgpack
notification = {'subject':'recording.should_start', 'session_name':'my session'}
topic = 'notify.' + notification['subject']
payload = msgpack.dumps(notification)
pupil_remote.send_string(topic, flags=zmq.SNDMORE)
pupil_remote.send(payload)
print(pupil_remote.recv_string())

We say reliable transport because Pupil Remote will confirm every notification we send with 'Notification received'. When we get this message we have a guarantee that the notification was received by the Pupil Core software.

If we listen to the Backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.

Writing to the Backbone directly

If you want to write messages other than notifications onto the IPC Backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.

# continued from above...
# Assumes pupil_remote to be a connected REQ socket

import msgpack
from time import time, sleep

pupil_remote.send_string('PUB_PORT')
pub_port = pupil_remote.recv_string()

publisher = ctx.socket(zmq.PUB)
publisher.connect(f'tcp://{ip}:{pub_port}')

# Wait for the connection to be established...
# See Async connect in "Delivery Guarantees PUB-SUB" below
sleep(1)

notification = {'subject':'calibration.should_start'}
topic = 'notify.' + notification['subject']
payload = serializer.dumps(notification)

publisher.send_string(topic, flags=zmq.SNDMORE)
publisher.send(payload)

IPC Backbone Implementation

This section provides detailed inside information about the IPC Backbone implementation. Please see specifically the subsection about delivery guarantees.

Pupil Core software uses a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors' messages. Therefore, it is the Backbone of all communication to/from and within the Pupil Core software.

Note - The main process does not do any CPU heavy work. It only runs the proxy, launches other processes and does a few other light tasks.

IPC Backbone used by Pupil Capture and Service

The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate.

  • Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.)
  • Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.

Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.

Delivery guarantees ZMQ

ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don't want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.

  • Messages are guaranteed to be delivered whole or not at all.
  • Unlike bare TCP it is ok to connect before binding.
  • ZMQ will try to repair broken connections in the background for us.
  • It will deal with a lot of low level tcp handling so we don't have to.

Delivery Guarantees PUB-SUB

ZMQ PUB SUB will make no guarantees for delivery. Reasons for not receiving messages are:

  • Async Connect/The Late joiner: PUB sockets drop messages before a connection has been established and topics subscribed. ZMQ connects asynchronously in the background.
  • The Snail: If SUB sockets do not consume delivered messages fast enough they start dropping them.
  • Fast close: A PUB socket may loose packages if you close it right after sending.

For more information see ZMQ Guide Chapter 5 - Advanced Pub-Sub Patterns.

In order to avoid accidentally dropping notifications in Pupil, we use a PUSH instead of an PUB socket. It acts as an intermediary for notifications and guarantees that any notification sent to the IPC Backbone, is processed and published by it.

Delivery Guarantees REQ-REP

When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.

Delivery Guarantees in general

We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.

However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.

Latency

Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests...

# continued from above
ts = []
for x in range(100):
    sleep(0.003) #simulate spaced requests as in real world
    t = time()
    requester.send_string('t')
    requester.recv_string()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts), max(ts))
>>> 0.000266075134277 0.000597472190857 0.00339102745056

... and when talking directly to the IPC Backbone and waiting for the same message to appear to the subscriber:

# continued from above
monitor = Msg_Receiver(ctx, sub_url, topics=('notify.pingback_test',))
sleep(1.)

ts = []
for x in range(100):
    sleep(0.003)  #simulate spaced requests as in real world
    t = time()
    #notify is a method of the Msg_Dispatcher class in zmq_tools.py
    notification = {'subject':'pingback_test'}
    topic = 'notify.' + notification['subject']
    payload = serializer.dumps(notification)
    publisher.send_string(topic, flags=zmq.SNDMORE)
    publisher.send(payload)
    monitor.recv()
    ts.append(time()-t)
print(min(ts), sum(ts)/len(ts) , max(ts))
>>>0.000180959701538 0.000300960540771 0.000565052032471

Throughput

During a test we have run dual 120fps eye tracking with a dummy gaze mapper that turned every pupil datum into a gaze datum. This is effectively 480 messages/sec. The main process running the IPC Backbone proxy showed a cpu load of 3% on a MacBook Air (late 2012).

Artificially increasing the pupil messages by a factor 100 increases the message load to 24.000 pupil messages/sec. At this rate the gaze mapper cannot keep up but the IPC Backbone proxy runs at only 38% cpu load.

It appears ZMQ is indeed highly optimized for speed.