Sync square 'triple-state' behavior

[mkrumin]

Sync square in some signals experiments is flickering between three states - white, black and background. The order seems to be random. It looks like it might just be rendered fully transparent or just behind the main texture on some frames.
Example trace ('2019-09-12_11_test', sparse noise experiment, ZMAZE stim PC, B-Scope rig, ):

The same was true running one of the ChoiceWorld Expdefs, I could turn the wheel slowly and stop with the sync square being in one of the three states.

[mkrumin]

After some more investigation the findings are:

1. It is happening a lot on slower computers
2. It is still happening on faster computers, especially when the wheel is turning fast, and the textures need to update fast. It is just not obvious by eye, as white and black get fused into gray anyway. (this becomes more apparent if you set background to be not gray, e.g. cyan)

A possible, and quite likely explanation is that the Screen('Flip') (inside the callback that flips textures from the buffer) happens in between the two lines of Screen('DrawTexture') and Screen('FillRect') (inside the callback that prepares the buffer for the next flip), as everything happens asynchronously in signals. (Not exact code that is used, just something to illustrate the idea)

A proposed solution to test is to use the following code in the callback that fills the buffer:

drawnow nocallbacks;   
Screen('DrawTexture');
Screen('FillRect');
drawnow;

The two Screen commands now should be forced to be executed one after another uninterrupted by other callbacks (hopefully).

[k1o0]

Cheers for the info. I'll look into this next week. I'll try to build a test class for SignalsExp so that we can check for this sort of thing. I'm also hesitant to make any changes to the code until we have a test.

[k1o0]

A possible, and quite likely explanation is that the Screen('Flip') (inside the callback that flips textures from the buffer) happens in between the two lines of Screen('DrawTexture') and Screen('FillRect') (inside the callback that prepares the buffer for the next flip), as everything happens asynchronously in signals. (Not exact code that is used, just something to illustrate the idea)

A proposed solution to test is to use the following code in the callback that fills the buffer:

drawnow nocallbacks;   
Screen('DrawTexture');
Screen('FillRect');
drawnow;

I'm still puzzled by this because the main loop of SignalsExp is actually synchronous. The values of signals can update any time but the drawing of the frame can only happen once per loop. Here's the (simplified) code:

if obj.StimWindowInvalid % flag set asynchronously by signals callback
  ensureWindowReady(obj); % complete any outstanding refresh, i.e. Screen('AsyncFlipEnd')
  drawFrame(obj); % draw the visual frame, i.e. 
  if ~isempty(obj.SyncBounds) % render sync rectangle
    % render sync region with next colour in cycle
    col = obj.SyncColourCycle(obj.NextSyncIdx,:);
    % render rectangle in the sync region bounds in the required colour
    Screen('FillRect', obj.StimWindowPtr, col, obj.SyncBounds);
    % cyclically increment the next sync idx
    obj.NextSyncIdx = mod(obj.NextSyncIdx, size(obj.SyncColourCycle, 1)) + 1;
  end
  % start the 'flip' of the frame onto the screen
  Screen('AsyncFlipBegin', obj.StimWindowPtr);
  obj.AsyncFlipping = true;
  obj.StimWindowInvalid = false;
end
drawnow; % allow other callbacks to execute

None of these function are associated with any event callbacks. The only think affected by the signals callbacks is the StimWindowInvalid flag.

I've written a full test suite for expServer and now I will move on to tests for SignalsExp...

[k1o0]

Adding drawnow nocallbacks after if obj.StimWindowInvalid on line 707 apparently works. I don't yet understand why.

I don't know if adding this affects performance.

[jkbhagatio]

when a visual stimulus element's show property is set to true, could this trigger a callback that fills the Screen buffer? If not, do you know of code anywhere else that fills the Screen buffer?

Because of this line

Screen('AsyncFlipBegin', obj.StimWindowPtr); in exp.SignalsExp/mainLoop, it seems possible that the screen buffer is getting filled (by a visual stimulus element?) after the sync square gets sent to the buffer, which effectively overwrites the sync square from the buffer?

[k1o0]

when a visual stimulus element's show property is set to true, could this trigger a callback that fills the Screen buffer?

When show is set to true a callback sets the StimWindowInvalid flag to true and the screen is redrawn the next run of the main loop. I can't yet see how any of this would lead to the sync square failing to get filled.

Screen('AsyncFlipBegin', obj.StimWindowPtr); in exp.SignalsExp/mainLoop, it seems possible that the screen buffer is getting filled (by a visual stimulus element?) after the sync square gets sent to the buffer, which effectively overwrites the sync square from the buffer?

If that were the case you would never see a stimulus at the same time as the sync square.

[jkbhagatio]

If that were the case you would never see a stimulus at the same time as the sync square.

I hypothesize that you would still see both so long as in the time between the sync square being drawn to buffer and that buffer being flipped to screen, nothing else is drawn to buffer. However, if something is drawn to buffer between this time, this would overwrite the sync square from the buffer. I think drawing to the buffer is happening asynchronously.

[k1o0]

Again, the main loop is synchronous. I can't see anyway in which that would happen.

Also I've tried adding drawnow nocallbacks like Peter did and for me the problem still persisted, so I don't think it's this. I will have a look at the update times from the experiment Krumin mentioned...

[k1o0]

Okay, by recreating the problem in a controlled way and spying on the function calls I can rule out Jai's hypothesis. It looks to me as though this may be a psychtoolbox issue. On my laptop the FillRect fails in a stochastic manner that may be slightly correlated with CPU load. I will try to recreate the issue outside of Signals next.

[jkbhagatio]

Great you found what is likely the cause! That is interesting/concerning though that it is a FillRect issue. Wonder if it's an issue with other Screen function calls as well...

[k1o0]

Before I post an issue to PTB, @mkrumin and @peterzh could you please give me more info about the computers on which this happens? Ideally update the graphics drivers and restart the computer before running the task, then check if it still happens. Pay attention to the command prompt; there might be informative warnings from PTB. I doubt they'll look much into it if we don't sort out any sync failures first. Also give me some info on the PTB version and GPU, OS, etc.

I've attached a script that redraws a GL texture and calls FillRect each time the mouse is moved. On my laptop the square disappears a small percentage of the time, regardless of how fast I move the mouse. NB: change the extension from .txt to .m.

screenPoke.txt

[jkbhagatio]

have you tried with other Screen function calls? e.g., using the following as Screen input args: FillArc, FillOval, FillPoly, DrawArc, DrawLine ?