Avoiding reallocating and permuting video data

Hello all,

I'm generating a relatively large video data array as a stimulus using the 'gen' TaskObject. I'm using TrialRecord.User to store the Y*X*4*N matrix and avoid reallocating the matrix each trial, and according to the TrialRecord structure documentation, the User portion isn't written to the file structure, but my testing uncovered that it is (presumably, it's not guaranteed to be written?)

It might be something I'm doing wrong, but I'd very much like to avoid taking the latency hit of writing the buffer to the filesystem. Is there some way to avoid that, or some place in one of the userloop or timing script I can sneak the buffer out of the User field to ensure it's not written? I'd have used persistent variables in the gen function, but it doesn't look as though they're actually surviving from one trial to another.

I may have answered my own question: I've mlocked the generator function being called by the gen TaskObject and set the buffer to be persistent there.

There's still a rather high amount of latency, though. I've measured the rendering time within the userloop to be below 200ms for an ARGB video with 1000 frames at 376x376 px size; the inter-trial interval is still taking longer than it ought to; when I reuse the stimulus instead of calling it again, the latency goes away.

Is there somewhere the video stimuli from gen(...) are being copied or modified? I've already disabled "save stimulus," but it seems like the obvious performance-saving things have happened: reusing the big block of memory for each trial, implementing the video rendering as a mexed function, and measuring the time it actually spends from invoking that function to receiving a value.

By profiling the userloop script, I've determined that the main cause of the latency relates to a lot of permuting and reshaping of the input movie matrix. It's a shame, actually - while I know it's not a common use case, I'm already rendering the raw bytes of video of my stimulus, and I could give it in whatever shape or ordering is needed directly to avoid the copying and shifting if that were a supported feature of TaskObjects. I did figure there was probably something like that going on, given how unusual it is to encode ARGB planes separately instead of interleaved.

Unfortunately, this is also something that will have a large impact on how I was planning to structure the userloop and timing script; because there's no way to circumvent these two lines in mglsetproperty:

frame = bits(:,:,:,m); % Line 88
frame = flipud(reshape(permute(frame,[2 1 3 4]),[],4)'); % Line 91

I'm not above modifying MonkeyLogic myself, but this does throw a little bit of a wrench in things.

There is no time you can save, as long as you keep the video as a matrix. The delay occurs when a presentable stimulus is created out of the matrix, so reusing a stimulus that is already created does not take additional time. If you keep the matrix, however, a new stimulus has to be created from it every time and there is no time saving.

You should talk more about what is the purpose of doing this rather than how you want to implement it, since your approach may not be the best way.

I see, thank you.

A sketch of the experiment is:

1. For each trial, a custom stimulus video is generated (for which I've written a C++ library and call into it via mex, keeping the video buffer persistent from trial to trial). This video is unique per trial, and lasts somewhere in the neighborhood of ten seconds (with exact length subject to variation per trial).
2. The subject views the stimulus and completes a task, and depending on the eye tracking, the trial may terminate early. Otherwise, the task takes around 10000ms.
3. Depending on the outcome of the task, a reward may be triggered, and the parameters for further videos are altered.
4. The inter-trial period elapses, which is on the order of 1000ms (also subject to variation)

By my measurement, the 200ms rendering is followed by around 1200-1500ms of permute and flipud.

As I've thought about it, I might modify the library to add a new TaskObject. This would be rather user-unfriendly to other groups, but since I'm a lot more familiar with the graphics and video world, I would probably find it rather nice to have; Matlab does not need to understand the video or image data as such, only ferry the data from my mex component to the display.

Open to thoughts! And if I do make this little hack, I'm happy to file a pull request and write up a little documentation for it.

What do you mean by rendering? Are you referring to your mex file generating the video?

Why is the video creation time so critical? It does not seem that you need a short ITI. If you increase ITI to 2 sec, there will be no noticeable delay due to the video creation.

How do you create the stimulus? Do you assign a string like 'gen(func_name)' to the C variable in the userloop?

Also do you script in the v1 style or v2?

> What do you mean by rendering? Are you referring
> to your mex file generating the video?

Yes, exactly. The video is created programmatically, and because it's generated within a mex component, it never has to hit the filesystem, and can stay in memory during an entire session.

> Why is the video creation time so critical? It
> does not seem that you need a short ITI. If you
> increase ITI to 2 sec, there will be no noticeable
> delay due to the video creation.

A short ITI was one of the requirements of the experiment - I'm a computer scientist, so I can't really speak to the "why" of the interval, but one of the concerns my colleagues had was about the timing and latency, so I'm doing the best I can to meet their requirements.

> How do you create the stimulus? Do you assign a
> string like 'gen(func_name)' to the C variable in
> the userloop?

That's correct. I have a function for 'gen(...)' to call, then that function translates the trial parameters and calls the C++ component via the mex function. On the first run, the video array is created inside mex and returned, then on subsequent runs, the array is reused; this follows a design pattern suggested by the mex documentation, and has pretty good performance.

> Also do you script in the v1 style or v2?

I've followed v2; the actual timing script was quite intuitive, and I appreciated the work you put into things like adapter chains and trial configuration.

Realistically, what I was thinking for the workaround is that either 'gen' can receive a fourth parameter or I'll use 'raw' in place of gen to specify a TaskObject that takes pre-arranged binary data and passes it through to the underlying graphics calls. In either case, my modifications to the library aren't going to be very complicated (although I might be biased in that regard).

What do you think?

If you are writing the script for someone else, ask the person if ~2-sec ITI is okay. Typically ITI is not an object of concern, when people talk about the timing and latency.

If video data can be fed directly in the 32-bit color format, the stimulus creation time will be reduced considerably. Let me think about how to add an interface for it.

By the way, the byte stream that your mex file should provide is a uint8 vector that contains the color information of each pixel (P) like the following. The RGB order of a pixel is [B G R A].

   Frame 1        Frame 2       Frame 3
[ P111 P112 ]  [ P211 P212 ] [ P311 P312 ]
[ P121 P122 ]  [ P221 P222 ] [ P321 P322 ]

imdata = [B111 G111 R111 A111 B112 G112 R112 A112 B121 G121 R121 A121 B122 G122 R122 A122 ...  % Frame 1
          B211 G211 R211 A211 B212 G212 R212 A212 B221 G221 R221 A221 B222 G222 R222 A222 ...  % Frame 2
          B311 G311 R311 A311 B312 G312 R312 A312 B321 G321 R321 A321 B322 G322 R322 A322];    % Frame 3

I added a way to create a GEN video from a BGRA vector. Please update your NIMH ML and try the attached example.

To indicate the provided data is a BGRA vector, not a MATLAB bitmap matrix, the info struct that the GEN function returns must have the [width height] of the video in the DoNotPermute field. See the attached example for more details.

Hey, thanks! This is great, and I'll try it ASAP and let you know this afternoon - I'm on Pacific Time, but I've set aside some time to work on it today.

To your previous questions:

* I confirmed with my collaborators that ITI needs to be around 1000ms, and preferably 800ms for this experiment
* I can absolutely generate a BGRA stream - that's the ideal thing, in fact! I liked your diagram, that's exactly how I'd originally set it up, and I found it very easy to understand.
* I'll update NIMH ML and update the scripts accordingly

Really appreciate your work on this. I'll update you soon.

Thanks again!

Took me a little longer than I expected to get back to where I could test it, but this works like a charm. Very much appreciate your careful work on it, and now the same test is down around 400ms total. I suspect I can probably squeeze a little more out of some optimizations here and there that having pixels in one contiguous location in memory will enable, too!

I think we can count this a closed issue