Integrating PupilEXT with Psychtoolbox

[atasevac]

Is it possible to integrate PupilEXT functions with Psychtoolbox projects? For example, we would like to embed codes about the timing and types of stimuli that are presented in psychtoolbox into our pupil recordings.

[BZandi]

Of course, you can synch PupilEXT with other experimental devices or software tools like the Psychotoolbox. It is easy to do, and there is also a Matlab script from an experiment in this repository where you can see how this can be done in principle, but let me start step by step.

Let's say we plan an experiment in which we need to measure the pupil diameter in response to different coloured lights. To carry out the experiment, we have three systems: Firstly, we have PupilEXT to record the pupil diameter. Secondly, we have the luminaire for generating different light stimuli. Thirdly, we have a software with which the luminaire can be controlled to turn on different lights.

Further, let's say that the protocol of the experiment looks as follows:
2 sec. white light (Reference) -> 2 sec. blue light (Stimuli, Wavelength 450 nm) -> 2 sec. white light (Reference) -> 2 sec. red light (Stimuli, Wavelength 630 nm)

In other words, each light setting is turned on for two seconds. We assume that the luminaire can be controlled using Matlab, which is similar to your question as you want to use the Psychtoolbox in Matlab to generate different types of stimuli.

The basic idea is to run the PupilEXT measurement in the background and leave it on while you start your experimental script in Matlab or Python. Then after the experiment is finished, you manually turn off the measurement in PupilEXT. The PupilEXT software will provide a log file as *.csv, where each pupil diameter is associated with a timestamp, right? Okay, the problem is how we can sync the signalling times of the experiment, which are, in our example, the light-onset times of the light stimuli, with the log file of PupilEXT. For this, you need a sort of "Marker-tables", in which the timestamps of the light-onsets are saved, which you can use to combine the PupilEXT log with the signalling times.

Let me give you an example. Coming back to our example with the light stimuli, a Matlab script to carry out the experiment with Matlab would look as follows (see the Matlab script below). The important part is how the light onset timestamps are saved in a table, which can be used later to synch the signalling times with the PupilEXT log file.

% Order of stimuli:
% 1. 2 Seconds on: White Light (Reference),
% 2. 2 Seconds on: Blue Light (Wavelength- 450 nm),
% 3. 2 Seconds on: White Light (Reference),
% 4. 2 Seconds on: Red Light (Wavelength - 630 nm)

TurnOnTimeRef = 2;
TurnOnTimeStimuli = 2;

TrialNumber = 1;

Date = {datestr(now,'mm/dd/yyyy')};
Time = {datestr(now,'HH:MM:SS.FFF')};
Type = {'Initial'};
SpectrumLabel = {'-'};
SpectrumCode = {'-'};
CodeOfSubject = {'A'};
NameOfSubject = {'John'};
SurnameOfSubject = {'Doe'};
AgeSubject = 30;

DateExperiment = {datestr(now,'mm_dd_yyyy')};
TimeExperiment = {datestr(now,'HH_MM')};

switch TrialNumber
    case 1
        ExperimentLabel = {'Const_Luminance_Quasi'};
    otherwise
        disp('Error')
end

% Create table
T = table(Date, Time, Type, CodeOfSubject,...
    NameOfSubject, SurnameOfSubject, AgeSubject, SpectrumLabel,...
    SpectrumCode, ExperimentLabel);

StimuliLabels = {'450_nm', '630_nm'};
StimuliCodes = {'B', 'D'};
ReferenceLabels = {'Referenz_1', 'Referenz_2'};
ReferenceCode = {'A', 'C'};

Index = [1, 2]; % Order of stimuli

% Experiment begins ========================

for i = 1:size(Index,2)
    
    % 1. Reference light is turned on
    CurrentTime = datetime('now'); % Save the current time
    fprintf(cell2mat(strcat('Reference spectrum: ', ' ',  ReferenceLabels(Index(i)), ' turned on', '\n')));
    
    % 2. Make entry into the maker table at which time the reference was turned on. Use currentTime.
    T(size(T,1)+1,:)=table({datestr(now,'mm/dd/yyyy')}, {datestr(CurrentTime, 'HH:MM:SS.FFF')}, {'Begin'}, CodeOfSubject,...
        NameOfSubject, SurnameOfSubject, AgeSubject, ReferenceLabels(Index(i)), ReferenceCode(Index(i)), ExperimentLabel);
    
    % 3.Keep the reference light on
    pause(TurnOnTimeRef); % keep the reference light on
    
    % 4. Turn on stimuli
    CurrentTime = datetime('now'); % Save the current time
    fprintf(cell2mat(strcat('Stimuli Spectrum: ',' ',  StimuliLabels(Index(i)), ' turned on', '\n')))
    
    % 5. Make entry into the maker table at which time the stimuli was turned on. Use currentTime.
    T(size(T,1)+1,:)=table({datestr(now,'mm/dd/yyyy')}, {datestr(CurrentTime, 'HH:MM:SS.FFF')}, {'Begin'}, CodeOfSubject,...
        NameOfSubject, SurnameOfSubject, AgeSubject, StimuliLabels(Index(i)), StimuliCodes(Index(i)), ExperimentLabel);
    
    % Keep the stimuli time on
    pause(TurnOnTimeStimuli);
end

% The last entry in the table should be at which the the experiment is over
CurrentTime = datetime('now'); % Save the current time
T(size(T,1)+1,:)=table({datestr(now,'mm/dd/yyyy')}, {datestr(CurrentTime, 'HH:MM:SS.FFF')}, {'End'}, CodeOfSubject,...
    NameOfSubject, SurnameOfSubject, AgeSubject, {'-'}, {'-'}, ExperimentLabel);

disp('Experiment is over !!');

disp(T);
filename = strcat(CodeOfSubject, '_', NameOfSubject, '_' ,SurnameOfSubject, '_',ExperimentLabel,'.mat');

MarkerTable = T;
save(cell2mat(strcat('Marker_',DateExperiment, '_', TimeExperiment, '_', filename{1})), 'MarkerTable')

As you can see in the script, the different lights settings will be turned on, which is indicated in the script using the fprintfcommand, while every time a new light stimulus is set, the timestamp will be saved in a "Marker-table". The"Marker-table" looks according to this script as follows (see the image below).

In the "Marker-table" the different light-onset times for each light setting are saved. Since PupilEXT was running in parallel on the same computer while the Matlab script was executed for the experiment, the timestamps of the PupilEXT's log file are identical to those of the "Marker-table", meaning the counters are already synced as Matlab and PupilEXT both apply the computer's system time. The only thing you need to do is to combine these two tables using a preprocessing script.

The experiment with the different chromatic light settings was recently performed. The pupil data and the"Marker-tables" for this experiment are made available in the repository. Please go to this page, download the folder and open it in Matlab:

https://github.com/openPupil/Open-PupilEXT/tree/main/Misc/Matlab_Pupil_Data_Evaluation

The folder Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\00_Data\Marker_Tables contains the created "Marker-tables". The pupil test was carried out repeatedly six times; hence, six files exist. Furthermore, the folder Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\00_Data\Pupil_Log_Data contains the recorded pupil log files from PupilEXT. In this folder, there are five subfolders because, during the experiment, no real-time pupil data recording was done. We recorded the pupil images only. After the experiment, the pupil images were loaded into PupilEXT, and the pupil diameter was evaluated using different pupil tracking algorithms for comparing the performance. Therefore, separate subfolders exist for each algorithm, whereby each subfolder contains again six different log files (as the pupil experiment was repeated six times).

A tutorial on how to sync a "Marker-tables" with the PupilEXT log files is provided in the Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\Main.m file. In Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\Main.m, the method Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\01_Methods\PupilEXT_Preprocessing.m is used, which preprocesses the pupil data and syncs it with light-onset timestamps provided in the "Marker-tables". It is recommended to debug through the code and see how the data are synchronised step by step (especially in the Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\01_Methods\PupilEXT_Preprocessing.m file). Note that the Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\Main.m file contains a little more processing as the light stimuli (mentioned above) were presented in randomised order, which were than re-ordedered to white light -> blue light -> white light -> red light. However, the principle is simple. One result of the Open-PupilEXT\Misc\Matlab_Pupil_Data_Evaluation\Main.m is the following plot (see below).

You can see the pupil diameter values (raw data and pre-processed) in the plot for the different chromatic light stimuli, which were evaluated using the different algorithms provided in PupilEXT. The light on times are highlighted using the plot's background colour, i.e. a yellow background shows that the white light was on, and the first blue background indicates the blue light. The second blue background in the plot indicates at what time the red light was switched on. Something similar can be done with the stimulus times saved by the psychophysics toolbox.

I hope it is straightforward how to proceed, but if you have further questions, don't hesitate to ask as this is an important topic.

[atasevac]

This was extremely helpful thank you very much.

However, when recording with our single camera set up, our log files do not seem to record all of the variables that appear in the log files of the provided sample data. All of the variables with *Main_px and *Sec_px are missing.

I'm wondering what the differences are between say "diameter_px", "diameterMain_px", and "diameterSec_px"?
Is it advisable to edit the preprocessing script so that it can use log files without those variables?

Thanks!

[BZandi]

When recording in stereo mode, you have two cameras (main camera and secondary camera) with the advantage that you can measure the pupil diameter directly in millimetres. For this, the pupil diameter is tracked in the main camera's image (in pixel) and in the secondary camera's image (in pixel). Next, both pupil diameter values will be used for a transformation into world coordinates to compute the pupil diameter in absolute units (check out the triangulation page of Wikipedia) i.e., in millimetres (see image below).

In the provided Matlab script that you mentioned, there is a lot of pre-processing conducted concerning the stereo data as one needs to make sure that the pupil fit in both the main camera's image and the secondary camera's image is accurate; otherwise, the transformed pupil diameter in millimetres has not a high confidence. For this, in the Matlab script, all pupil diameter values are excluded that have a lower confidence value (lower than 1). This procedure is conducted for the estimated pupil diameter from the main camera and the secondary camera, as the prerequisite for a precise pupil diameter in mm is that the pupil fit was very accurate in both images (see image below).

Back to your question: You need to adapt the provided Matlab script when using only one camera because you naturally have no data for the main or secondary camera (only provided in stereo mode). Further, the pre-processing is much easier when using the data of a single camera as you only need to exclude the pupil diameter data of one camera.

I hope it is clear. The pre-processing is also explained in our paper:
https://www.frontiersin.org/articles/10.3389/fnins.2021.676220/full