Combination of Concurrent and Sequential

It’s a complicated case, I hope I can explain it clearly.

We are trying to program a task in which the avatar has to follow the color cue but the subject maintains the central fixation. So, I have a Single Target + Loose Hold chain to detect the eye signal, then a Sequential chain to control the appearance and disappearance of the cues (blank 300ms + the first cue 300 ms + blank 300 ms + the second cue 300 ms + blank 300 ms + two choices 400 ms), then a movie Sequential chain to control the avatar videos (center 400ms + turn to the cue 300ms + center 300ms + turn 300ms + center 300ms + turn 300ms). At last, I combine the three sequential chains by using Concurrent to make them run simultaneously. I have defined the duration for each Time Counter so that there is a short delay between the cue and the avatar's head movement, which makes the scene more natural. I expect that the adapter in each Sequential chain can run independently but actually, they will interrupt each other. For example, in the cue Sequential chain, when the first Time Counter (blank) finishes, the second Time Counter (the first cue) will not run immediately, but until the end of the first movie in the movie Sequential chain), which messes up all the duration settings.

I can’t figure out why. So, I tried another way. I combine the movie and the cue using concurrent first and then put them in the sequential chain. The structure will be movie center 300ms + conc (movie center+cue1) 100ms + conc (movie turn+cue1) 200ms + movie turn 100ms + movie center 200ms + conc (movie center+cue2) 100ms + conc (movie turn+cue2) 200ms + movie turn 100ms + movie center 200ms + conc (movie center+2 cues) 100ms + conc (movie turn+2 cues) 300ms. Now the timelines of the movie and the cue can match, but the duration of this sequential chain is longer than the predefined value. Here, the sum of all the Sequential chains should be 1900ms, but actually, it needs more than 2700 ms.

The two attachments are the schema of the two solutions. Can anyone see the problem in my logic?

Please show me your task code. From your description, I think it should have a structure like the following and I do not see any interruption between two Sequentials.

t11 = TimeCounter(null_); t11.Duration = 300;
t12 = TimeCounter(null_); t12.Duration = 300;
t13 = TimeCounter(null_); t13.Duration = 300;
t14 = TimeCounter(null_); t14.Duration = 300;
t15 = TimeCounter(null_); t15.Duration = 300;
t16 = TimeCounter(null_); t16.Duration = 400;
seq1 = Sequential(t11);
seq1.add(t12);
seq1.add(t13);
seq1.add(t14);
seq1.add(t15);
seq1.add(t16);
seq1.EventMarker = [11 12 13 14 15 16];

t21 = TimeCounter(null_); t21.Duration = 400;
t22 = TimeCounter(null_); t22.Duration = 300;
t23 = TimeCounter(null_); t23.Duration = 300;
t24 = TimeCounter(null_); t24.Duration = 300;
t25 = TimeCounter(null_); t25.Duration = 300;
t26 = TimeCounter(null_); t26.Duration = 300;
seq2 = Sequential(t21);
seq2.add(t22);
seq2.add(t23);
seq2.add(t24);
seq2.add(t25);
seq2.add(t26);
seq2.EventMarker = [21 22 23 24 25 26];

con = Concurrent(seq1);
con.add(seq2);

scene = create_scene(con);
run_scene(scene);

Here it is. The LEDOutput is a function written by our engineer to control the LEDs by Arduino. The event codes are [21 11 22 12 23 13 24 14 25 15 26 16], and the code times are
3001.28740000002
3001.54420000001
3451.66829999999
3485.01990000000
3952.08470000000
3952.34199999999
4402.45490000001
4402.71419999999
4902.84700000001
4903.10380000000
5336.52940000002
5336.78600000002

% fixation
fix1 = SingleTarget(eye_);  % fixation period
fix1.Target = fixation_point;
fix1.Threshold = fix_window;
lh1 = LooseHold(fix1);
lh1.HoldTime = 1900;
lh1.BreakTime = 50;

% color sequence first cue
seq_mov1 = MovieGraphic(null_);
seq_mov1.List = { 'central.mp4', [0 -2], 1};
seq_tl1 = TimeCounter(seq_mov1);
seq_tl1.Duration = 400;
seq_mov2 = MovieGraphic(null_);
seq_mov2.List = { 'left.mp4', [0 -2], 1};
seq_tl2 = TimeCounter(seq_mov2);
seq_tl2.Duration = 300;
seq_mov3 = MovieGraphic(null_);
seq_mov3.List = { 'central.mp4', [0 -2], 1};
seq_tl3 = TimeCounter(seq_mov3);
seq_tl3.Duration = 300;
seq_mov4 = MovieGraphic(null_);
seq_mov4.List = { 'right.mp4', [0 -2], 1};
seq_tl4 = TimeCounter(seq_mov4);
seq_tl4.Duration = 300;
seq_mov5 = MovieGraphic(null_);
seq_mov5.List = { 'central.mp4', [0 -2], 1};
seq_tl5 = TimeCounter(seq_mov5);
seq_tl5.Duration = 300;
seq_mov5 = MovieGraphic(null_);
seq_mov6.List = { 'right.mp4', [0 -2], 1};
seq_tl6 = TimeCounter(seq_mov5);
seq_tl6.Duration = 300;
cue_mseq = Sequential(seq_tl1);
cue_mseq.add(seq_tl2);
cue_mseq.add(seq_tl3);
cue_mseq.add(seq_tl4);
cue_mseq.add(seq_tl5);
cue_mseq.add(seq_tl6);
cue_mseq.EventMarker = [11 12 13 14 15 16];

blank_tl1 = TimeCounter(null_);
blank_tl1.Duration = 300;
blank_tl2 = TimeCounter(null_);
blank_tl2.Duration = 300;
blank_tl3 = TimeCounter(null_);
blank_tl3.Duration = 300;
led1 = LEDOutput(TrialRecord, null_);
led1.Target = cue_1;
led1_tl = TimeCounter(led1);
led1_tl.Duration = 300;
led2 = LEDOutput(TrialRecord, null_);
led2.Target = cue_2;
led2_tl = TimeCounter(led2);
led2_tl.Duration = 300;
opt1 = LEDOutput(TrialRecord, null_);
opt1.Target = cued_LED;
opt1_tl = TimeCounter(opt1);
opt1_tl.Duration = 400;
opt2 = LEDOutput(TrialRecord, null_);
opt2.Target = uncued_LED;
opt2_tl = TimeCounter(opt2);
opt2_tl.Duration = 400;
opts_conc = Concurrent(opt1_tl);
opts_conc.add(opt2_tl);
LED_seq = Sequential(blank_tl1);
LED_seq.add(led1_tl);
LED_seq.add(blank_tl2);
LED_seq.add(led2_tl);
LED_seq.add(blank_tl3);
LED_seq.add(opts_conc);
LED_seq.EventMarker = [21 22 23 24 25 26];

fix_conc = Concurrent(lh1);
fix_conc.add(LED_seq);
fix_conc.add(cue_mseq);
scene1 = create_scene(fix_conc, fixation_point);

Have you tested if a single LEDOutput + TimeCounter chain ends exactly after the Duration of the TimerCounter? I do not know how LEDOutput works but I can tell it is not written correctly. The constructor of an adapter can receive only one argument and it must be another adapter. NIMH ML would make an error about it, so I wonder how you could run the code.

When I ran your code without LEDOutput, the order of the eventcodes were the same as yours and the code times were as below.
264.972199999988
264.972199999988
564.983499999983
664.989200000008
865.000899999984
965.003800000005
1165.01670000000
1265.02410000001
1465.03079999999
1565.03430000001
1765.04459999998
1865.05180000000

By the way, 'seq_mov5' was used incorrectly twice at where it was supposed to be 'seq_mov6'. I marked them in red above.

Thanks for your reply. Inspired by your reply, I have run several tests, which makes me believe that it’s the movies but not the LEDs that cause the problem.
1. I tested the duration of a single or two sequential LED+TimeCounter chains, which are close to the predefined value.
2. I replaced all LED chains with simple CicrleGraphic objects in my script and run it again. It didn’t change anything.
3. I only combined the LED sequential chain with the LooseHold chain, and the event code times were alright: 3564.39890000001, 3864.65379999993, 4164.88259999994, 4465.13759999993, 4765.37459999997, 5065.61139999997;
4. However, when I combined the movie sequential chain with the LooseHold chain, something was wrong: 3651.76389999999, 4135.50550000002, 4569.18370000000, 4919.46910000002, 5353.17600000002, 5770.15810000000, 6211.45799999999.
5. I also ran the movie sequential chain alone, and the event code times were not right either.
6. Then I checked my videos, the duration of the video is longer than the defined value, so I trimmed the videos (25ms) and tested the script again (LooseHold + movie chain + LED chain). The code times were
3462.26960000058
3462.52620000087
3945.96740000088
4026.12020000015
4446.37180000063
4446.63360000050
4963.46740000081
4963.72949999932
5347.11129999960
5347.36800000064

So I'm sure that the problem is related to the videos, but I'm not sure it's the movies themselves or any hardware that causes the problem.

It appears that your reasoning is right. My concern about LEDOutput is still valid though. I guess you might have to modify the existing NIMH ML code, which could be avoided. I can take a look at it, if you want.

I used one-minute-long movies to test your code, but did not see any problem. MovieGraphic is just like any other graphic adapters and does not have a component that slows down the execution of the chain, unless the size of the movie is really huge. I cannot tell what the problem is without exmining your movie files. Can you send them to me? It would be great if you could upload them somewhere and give me a download link.

Thanks. I did another test today. I resized the videos, which makes the size of the videos much smaller. Then the event code times seem alright: 1014.67250000000
1014.92940000000
1314.91650000000
1415.00590000000
1648.52570000000
1748.60660000000
1965.45490000000
2082.21530000000
2265.69830000000
2382.46670000000
2565.95170000000
2682.71350000000

So I guess your hypothesis about the movie size is right. Here is the link to download the videos. The original (10M per video) and trimmed version (800k per video) couldn't work, but the resized videos (200k per video) work.

About the LEDOutput, I prefer to talk with our engineer first. He is the one who wrote the function. I will show him your comments. Thanks again for your help.

I got the files. Thanks.

In this case, the problem is not the file size, but the resolution of the movie. Because of the huge amount of data to transfer (roughly 1.5 GB per second), there is no way to present 4K 60-Hz videos seamlessly while doing all the things that NIMH ML does.

The resolution of your videos need to be reduced. You can still present them in the same size as before, either by decreasing the resolution of the subject screen as well or increasing the scale of MovieGraphic. And I definitely would crop out the unnecessary black background.

Thanks. I will edit the videos.

About the LEDOutput, here is the answer from the engineer: "About the LEDOutput adapter, you can explain that I had to add another input argument to the class because I needed to access the TrialRecord variable inside it which is not available from classical ML adapter. "

Of course, you added another argument because you needed it. However, it prevents NIMH ML from determining the initial conditions of scenes, so mlplayer will not work with the resulting data files.

The recommended way to add more arguments to an adapter is to add more properties and copy necessary values from TrialRecord to the properties, rather than assign the whole structure as an argument.

Thanks for your reminder. I will discuss it with the engineer.