Recommendation for Syncing Long Movies to Electrophysiology Rig

Hi there,

I'd like to play long (~10-20 minutes) videos to my monkeys while recording electrophysiology data through an Intan RHD system. I was wondering what the best solution would be for getting timestamps of when each frame of the movie played synced to my electrophysiology data.

I initially tried adding a black/white flickering stimulus to the corner of each frame which I then read into Intan through a photodiode. However, the number of frame flips I registered differed slightly from the true number of frames in the video (presumably due to a few skipped frames), so I wasn't able to ascertain the exact frame identity from the photodiode signal.

Ideally I was thinking MonkeyLogic could send a behavioral code corresponding to the frame number that I would read as a digital input on Intan. It seems like set_frame_event is the closest to accomplishing this, though my understanding is that it considers screen frames rather than frames of a movie. I saw here that you can display videos according to the screen refresh rate instead of the video's frame rate, and then I thought I could look at the timing of skipped frames to try to infer actual frame identity. Does this sound like it's on the right track, or is there a more elegant solution?

In case it's helpful, here are my very simple MovieTask.m and MovieTask_userloop.m files. I can switch to alternative MonkeyLogic framework (v1 vs v2, conditions file vs. userloop, etc.) if necessary.

hotkey('x', 'escape_screen(); assignin(''caller'',''continue_'',false);');

% Path to the video file
videoFile = 'test_vid.mp4';

% Create a VideoReader object to read video properties
v = VideoReader(videoFile);

mov = MovieGraphic(null_);
mov.List = {videoFile, [0 0] };  % movie filename
mov.info.FrameByFrame = 1;

tc = TimeCounter(mov);
tc.Duration = v.Duration * 1000 + 10000; %Length of the video plus 10 additional seconds (screen turns gray after video ends)

scene = create_scene(tc);
run_scene(scene);
idle(0);  % clear the screen

function [C,timingfile,userdefined_trialholder] = MovieTask_userloop(MLConfig,TrialRecord)

timingfile = 'MovieTask.m';
userdefined_trialholder = '';
C = [];

Please tell me about your research. There are a few reasons why what you want is not as easy as it seems. I need to know what you are trying to do, to think about how to solve the problems and determine whether it is worth my time. You can email me, if you do want to write it here.

Hi Jaewon,

Thanks a lot for your reply. I hadn't realized such a thing would be technically challenging, and after some thought I don't believe it's necessary for my research. I'm a PhD student in Anirvan Nandy's lab at Yale and am studying laminar interactions in the ventral stream (e.g. V2/V4) of marmosets, mostly in the context of a 2AFC task which is running fine. However, marmosets aren't very good at prolonged task performance but will readily watch natural videos (e.g. cartoons, nature documentaries) for long periods, so I figured I may as well record neural activity while they are watching videos after they have finished their tasks for the day. That being said, this is exploratory and not hypothesis-based, and I don't have a strong motivation for knowing such precise frame timing for my analyses.

I found that a good compromise was flickering the black square in the corner to white every 30 frames. The likelihood of a dropped frame falling precisely on the white square is quite low (and hasn't happened a single time in all of my tests thus far), and then I can just interpolate to estimate the timing of the other 29 frames. Given these are natural videos and the stimuli aren't moving very quickly, I think this level of resolution will be fine for some exploratory analyses.

I'll update you if I or anyone else in the lab comes up with a strong motivation for precise single frame timing, but I don't want to waste your time on it for now. Thanks for the insight!

Thank you for the explanation. Sending additional codes itself is not a difficult thing. What is difficult is to add a new function without making it conflict with existing ones. Things that immediately occur to me are like how to stamp the frame number when two movies with different starting points and different frame rates are presented, what if the frame number is already used by other codes, etc. Also, determining the frame number can be tricky if the movie file is incorrectly encoded and has missing frames.

People here at NIMH are also recording from subjects watching natural movies, but I have not heard that they want to stamp each frame. Maybe they, too, think a small time difference is not critical for natural movies.

I think it is a good feature to have. Let me think about how to implement it. In the meantime, check out the FrameMarker adapter. It is not suitable for stamping many frames, but may help with your issue.
https://monkeylogic.nimh.nih.gov/docs_RuntimeFunctions.html#FrameMarker

By the way, the time information of skipped frames is stored in the data file.
https://monkeylogic.nimh.nih.gov/board/read.php?2,998

----------

I have a couple of things to mention about the code that you showed. In the code, a movie was loaded by VideoReader, as well as MovieGraphic. This makes two copies of the same movie kept in memory, just to calculate its duration. If you update your NIMH ML, you can get the duration from MovieGraphic, like mov.Duration, so VideoReader is unnecessary.
https://monkeylogic.nimh.nih.gov/docs_RuntimeFunctions.html#MovieGraphic

Also, info.FrameByFrame is a method that works only for GEN TaskObjects. The MovieGraphic adapter does not have such a property, so you will get an error.