# Sync with External Sensors
Many experimental setups record data from multiple sensors in parallel. This data needs to be synced temporally for a joint analysis. All eye tracking data you record with Pupil Invisible is accurately timestamped using Unix timestamps in nanoseconds. This makes syncronisation with other sensors easily possible. The only requirement is that the other data also has absolute timestamps like this, i.e. including the date and exact time when every sample was recorded. Note that these differ from relative timestamps which count time since, e.g. the start of a recording.
If the sensor you are using only provides relative timestamps it is often still possible to convert them to absolute timestamps. Typically, the start time of the recording is available as an absolute timestamp. If you add the relative time to this absolute start point, you get absolute timestamps.
In this guide, you will learn how to sync datastreams with absolute timestamps using the pd.merge_asof function of Pandas. As an example, we will sync a heart rate sensor (from a Garmin Fenix3 HR running watch) with a Pupil Invisible recording.
We will also produce a visualization to show the real-time heart rate of a person jogging alongside the eye tracking video. To build this visualization, we will actually be matching three data streams: 1. Pupil Invisible’s scene video, 2. Gaze data, and 3. Heart rate.
This guide is about syncing data post hoc after you have made a recording. In some experimental setups it can be handy to sync already at recording time using Labstreaming Layer (LSL). See here for an introduction on LSL with Pupil Invisible.
If you need to sync data in real-time while recording, see the real-time API instead.
# Dependencies of this Guide
You can find all code accompanying this guide here. To run it you need to install the following dependencies:
pip install numpy pandas av fitdecode tqdm opencv-python datetime typing
The heart rate data used in the example is located in data/eye-tracking-run.FIT.
The Pupil Invisible recording used in the example is available here. Unpack it inside the data/demo-recording folder.
# Loading all Data
For the example visualization we need the scene video and gaze data from the Pupil Invisible recording, and the heart rate data.
The heart rate data can be read using the fitdecode module. For more details check out the implementation of the load_fit_data function.
The gaze data CSV file can be read using Pandas. All we need from it is the timestamps and the gaze values.
For the scene video, we initially only need its timestamps and the corresponding frame indices for matching, we don't have to touch the actual video frames yet.
import pandas as pd
from decode_fit import load_fit_data
heart_rate_path = "data/eye-tracking-run.FIT"
hr = load_fit_data(heart_rate_path)
hr = hr[["timestamp", "heart_rate"]]
gaze_path = "data/demo-recording/running_rd-4a40d94d/gaze.csv"
gaze = pd.read_csv(gaze_path)
gaze["timestamp [ns]"] = pd.to_datetime(gaze["timestamp [ns]"])
gaze = gaze[["timestamp [ns]", "gaze x [px]", "gaze y [px]"]]
world_ts_path = "data/demo-recording/running_rd-4a40d94d/world_timestamps.csv"
world_ts = pd.read_csv(world_ts_path)
world_ts = world_ts[["timestamp [ns]"]]
world_ts["frame_index"] = world_ts.index
world_ts["timestamp [ns]"] = pd.to_datetime(world_ts["timestamp [ns]"])
# Timestamp Matching
The challenge with syncing the three data streams is that while they are all timestamped, their timestamps are not identical. Every stream is sampled independently and at different rates. E.g. the gaze data is sampled at 200 Hz, while the scene video is only sampled at 30 Hz, so there are a lot more gaze samples than video frames in our recording.
For our visualization, we will need to overlay every scene video frame with a gaze circle that shows where the wearer was looking. We therefore have to decide between two options:
1. Given the timestamp of the video frame we search for the gaze sample that is closest in time and choose it for the overlay. This would imply that most of the gaze samples are not visible in the overlay, because there are more gaze samples than frames.
2. We match every single gaze sample to its closest video frame. All the gaze samples that match to the same frame are averaged and this value is used for the overlay.
For gaze data option 2 is usually better, as the averaging contributes a bit of noise reduction.
The heart rate sensor is not sampling data at regular intervals, but instead only records a new sample when the data changes. Also it is sampled much more sparsly at 1 Hz (or lower in case there are no changes to report). This changes how we need to match the data a bit. First of all, given the lower sampling rate we will now match every heart rate sample to multiple world frames. Second, instead of matching a heart rate sample to the world frames that are closest in time to it, we need to match it to world frames whose timestamps are larger-equal to the heart rate sample's timestamp. The heart rate sample is a point of change, so it is valid until the next change, but not before.
All of this can be implemented using the Pandas function pd.merge_asof. It merges two DataFrames based on indices that do not match perfectly by finding the closest matches, which is exactly what we need.
# Matching Video and Gaze
As mentioned above we will use option 2 for matching, i.e. we will match every gaze sample to the nearest existing world frame, and then calculate the mean gaze value for every world frame. This type of matching is achieved using direction="nearest". Note in the result below that the first couple of world frames do not have any matches. This is because the world camera initializes faster when starting a recording and thus records a couple seconds sooner than the eye cameras.
# Match every gaze sample to the closest world timestamp
gaze_world = pd.merge_asof(gaze, world_ts, left_on="timestamp [ns]", right_on="timestamp [ns]", direction="nearest")
# Calculate the mean gaze position for each world frame
gaze_world = gaze_world.groupby("frame_index").mean()
# Merge the gaze data with the world frame data
df = pd.merge(world_ts, gaze_world, left_on="frame_index", right_on="frame_index", how="left")
df
| timestamp [ns] | frame_index | gaze x [px] | gaze y [px] | |
|---|---|---|---|---|
| 0 | 2022-07-09 17:19:08.694000000 | 0 | NaN | NaN |
| 1 | 2022-07-09 17:19:08.744000000 | 1 | NaN | NaN |
| 2 | 2022-07-09 17:19:08.794000000 | 2 | NaN | NaN |
| 3 | 2022-07-09 17:19:08.844000000 | 3 | NaN | NaN |
| 4 | 2022-07-09 17:19:08.894000000 | 4 | NaN | NaN |
| ... | ... | ... | ... | ... |
| 74214 | 2022-07-09 18:00:53.466044444 | 74214 | 427.126000 | 880.560833 |
| 74215 | 2022-07-09 18:00:53.498311111 | 74215 | 424.746857 | 882.799000 |
| 74216 | 2022-07-09 18:00:53.534600000 | 74216 | 426.106167 | 884.010000 |
| 74217 | 2022-07-09 18:00:53.566655555 | 74217 | 427.429429 | 884.170000 |
| 74218 | 2022-07-09 18:00:53.598533333 | 74218 | 430.543800 | 885.591900 |
74219 rows × 4 columns
# Match Video and Heart Rate
Next we will match video and heart rate data. As mentioned above, we need to match every world frame to the closest heart rate sample whose timestamp is smaller than or equal to that of the world frame. This is done using direction="backwards", which can be read as ""from the world timestamp look backwards in time for the closest match". Note how the heart rate samples repeat themselves for several world frames as expected.
# Match every world frame to the closest preceding heart rate sample
world_hr = pd.merge_asof(world_ts, hr, left_on="timestamp [ns]", right_on="timestamp", direction="backward")
# We use the frame index to merge this into the data frame
# so we can drop the timestamps here
world_hr.drop(["timestamp", "timestamp [ns]"], axis=1, inplace=True)
# Merge the matched heart rate data with the previous
# data frame containing the world frame data and gaze
df = pd.merge(df, world_hr, left_on="frame_index", right_on="frame_index", how="left")
df
| timestamp [ns] | frame_index | gaze x [px] | gaze y [px] | heart_rate | |
|---|---|---|---|---|---|
| 0 | 2022-07-09 17:19:08.694000000 | 0 | NaN | NaN | NaN |
| 1 | 2022-07-09 17:19:08.744000000 | 1 | NaN | NaN | NaN |
| 2 | 2022-07-09 17:19:08.794000000 | 2 | NaN | NaN | NaN |
| 3 | 2022-07-09 17:19:08.844000000 | 3 | NaN | NaN | NaN |
| 4 | 2022-07-09 17:19:08.894000000 | 4 | NaN | NaN | NaN |
| ... | ... | ... | ... | ... | ... |
| 74214 | 2022-07-09 18:00:53.466044444 | 74214 | 427.126000 | 880.560833 | 169.0 |
| 74215 | 2022-07-09 18:00:53.498311111 | 74215 | 424.746857 | 882.799000 | 169.0 |
| 74216 | 2022-07-09 18:00:53.534600000 | 74216 | 426.106167 | 884.010000 | 169.0 |
| 74217 | 2022-07-09 18:00:53.566655555 | 74217 | 427.429429 | 884.170000 | 169.0 |
| 74218 | 2022-07-09 18:00:53.598533333 | 74218 | 430.543800 | 885.591900 | 169.0 |
74219 rows × 5 columns
# Visualization
Now that we have matched all three data streams successfully we can visualize them together. This can be done in various ways and the complexity of creating visualizations is out of scope of this guide, but we have provided an example visualization below that shows a simple gaze overlay and textual visualization of the heart rate.
Feel free to check the implementation of the make_visualization function.
from visualization import make_visualization
world_video_path = "data/demo-recording/running_rd-4a40d94d/3e2512bf_0.0-2504.94.mp4"
visualization_path = "data/visualization.mp4"
make_visualization(df, world_video_path, visualization_path)
nvenc not available h264_nvenc
100%|█████████▉| 74218/74219 [50:32<00:00, 22.36 frames/s]