Digital Video Notes
DMDX has the ability to display a streaming Digital Video multi-media file, beit AVI, MPEG, QuickTime or whatever else DirectShow supports (see TimeDX's Digital Video
help on Codecs), or it can also just use just the Streaming Audio abilities (see
below). It's probably not a good idea to use the
freesync video mode
option with digital video playback.
As with the <animation> switch all frame buffers (including the visible primary buffer)
are cleared at the end of an item when the legacy DirectDraw rendering path is
active, originally when the Direct3D renderer was in use a blank display was rendered after
the video was finished but by the time I got the Direct3D renderer (VMR9)
working properly this behavior was removed. So all Digital Video items
when using the legacy DirectDraw renderer should end with a blank frame, for example:
10 <dv> "movie" / ;
The primary reason
DMDX attempts to clear the screen after video when using the legacy DirectDraw
renderer is that the video can run on for
an indeterminate number of frames, should you wish to display normal DMDX frames
after the video a bit of extra attention will be needed to deal with either DMDX
clearing the display or the run on, for example the following item's second
"black.bmp" after the video may display
on some machines, may not on others:
+1 <ms% 500> "+" / <dv>
"../laura/neutral/608-80_l.mpg" / <ms% 500> <bmp> "black.bmp" / * <bmp> "response.bmp";
Instead that following frame will be need to be repeated,
maybe more than once in order to work on a large range of machines, for example:
+1 <ms% 500> "+" / <dv>
"../laura/neutral/608-80_l.mpg" / <ms% 50> <bmp> "black.bmp" / <ms% 50> <bmp> "black.bmp" / <ms% 400> <bmp> "black.bmp"/ * <bmp> "response.bmp";
Similarly if you wish to display the
video against a static background image the frame to display it might need to be
issued a number of times. Here one uses the <dv%
0> modifier to cancel the digital video's frame duration immediately
after the video begins and to then render the static background. The trouble with that of
course is that if you're wanting to start gathering responses after the video is
finished you'll have to have the duration of that video encoded in your items,
here we're assuming the video is 1.5 seconds long:
+1 <ms% 500> "+" / <dv>
"../laura/neutral/608-80_l.mpg" <dv% 0> / <ms% 50> <bmp> "black.bmp"/ <ms% 50> <bmp> "black.bmp"
/ <ms% 1400> <bmp> "black.bmp" / * <bmp> "response.bmp";
you want that static background image to precede the video you can display it
and then the video with a no erase switch
and use the <dv%
0> modifier to cancel the digital video's frame duration immediately
after the video begins and then render the static background again afterwards --
assuming you're using DMDX 18.104.22.168 or later and the
Direct3D renderer as trying to
render the background before the video began prior to that was fraught with difficulty
and would at best wind up with a flash of the background color prior to video
commencing (! or | or
<NoErase> were specifically
disallowed in a <dv> frame due to the weird unexpected things it did as
the playback rectangle was adjusted by the display routines to erase what was
previously on the screen and that is decidedly not what is desired). That
trailing frame re-rendering the static image is critical as it's that frame that
the code renders the video against, without it you're likely to just get a blank
display when using a no erase switch:
0 <bmp 0,0,0,0> "golftp" / ! <ml 0> <dv
-1,.1,.56,.75> "gro20017-0-005_l.mpg" <dv% 0> / <bmp 0,0,0,0> "golftp" ;
Note that support of No Erase in a Digital Video
<dv> frame is limited, normally in DMDX a
No Erase frame is copying bitmap data from one frame to another, none of that is
happening in a <dv> frame, instead
because you're using the Direct3D renderer DMDX
re-renders the previous frame's textures that are still available when it's time
to render a new video frame and if that frame doesn't exist
! in a <dv>
frame can't function. So while you can use !
at the start of an item in a normal frame to merge the previous item's display
with the next you can't do that in a <dv>
frame, what you can do however is use ! in both a normal frame and a
<dv> frame so constructs like the
0 <bmp 0,0,0,0> "golftp" ;
0 ! / ! <ml 0>
<dv -1,.1,.56,.75> "gro20017-0-005_l.mpg" <dv% 0> / ! ;
When DMDX processes an item with a Digital Video file in it
using the DirectDraw renderer it uses a completely different method than it
otherwise would to load images into the display memory. Instead of buffering any number of frames in the video card's memory ahead of time
and flipping between them at the appropriate time DMDX now only moves a frame
into video memory immediately prior to it's being displayed as DirectShow
doesn't provide more than one frame at a time. Using the Direct3D renderer
things aren't that different than they normally are where the frame's displays
are loaded into 3D textures prior to the item beginning and then rendering them
when their time is due, with <dv> frames there's additional code checking for
new video frames to be rendered on top of whatever normal DMDX display is
happening. Additionally, regardless of renderer digital video files have enormous initial latency (kind of like pre-roll time on tape machines) in the order of hundreds of milliseconds
on a lot of machines, DMDX automatically re-schedules the rest of the display queue once a digital video frame has actually started producing images to be displayed.
Multiple digital video files can be played at once (assuming you have the CPU horse power) and the frames can be combined with other frames in normal DMDX ways. The DMDX RT clock can be turned on at a particular digital video file frame number using a new clockon switch, <DVClockOn> and the duration of the DMDX frame can be set to expire when a particular digital video frame is displayed with the <DVFrameDuration> switch.
When playing back a file with the <DigitalVideo [N,N[,N,N]]> switch the first optional pair of Ns specify the top left corner location of playback, the second the bottom right
corner location (or if a centered playback region has been requested these can
be the size of the playback region). DMDX switches <X> and <Y> are overridden by these
-- indeed it would appear that any attempt to use them (or <BMPMultipliers>)
will result in a Lost Graphic message meaning the only way to position digital
video is using the parameters in the <dv> keyword). In the same way that the <X> and <Y> switches can be real and express a fraction of the screen dimensions these values can be too
with the one slight difference and that is that if the value 1 is used here it's
the full relative dimension of the screen, in
<xy> it's the pixel value 1. If neither pairs of coordinates are present the images are displayed centered on the screen in the size they are stored in the file in, if only the first pair is present the top left corners of the images are moved to that location, if both coordinates are specified the images are stretched to fit (overriding any <BMPMultipliers>). If all four values are zero the image is displayed full screen. If a -1 is used for a field the default value for that field is used, if -1 is used for either or both of the first coordinate pair the image is centered in that axis
and if the second pair of coordinates has been provided they become the size of
the playback region, if -1 is used for either or both of the second pair the original image size in that dimension is used. For example:
This would display the images near the top left corner of the screen in the original file size.
This would display the images near the top left corner of the screen as 100x100 pixel images.
This would display the images full screen.
This would display the images near the top center of the screen in the original file size.
This would display the images near the top center of the screen in the original file size width but stretched to half the screen height.
This would display the images in the center of the screen in one quarter of the screen area.
0 <bmp -1,-1,.66,.7> "border" %1 / ! <ml
0> <dv -1,-1,.6,.6> "video.mpg" <dv% 0> / ! ;
This item renders a video with border around it,
first the the border that is centered and 70% of the screen in height for a tick
before the video starts and then the video itself scaled a bit smaller at 60% of
the screen height. The border X dimension is 66% in a crude attempt to get
it the same width around the video, people can probably do some actual math to
figure out what those values should actually be...
When turning the clock on for responses at a given digital video frame with the <DVClockOn> switch the normal DMDX clock on switch should not be used as well. This switch must occur in the frame that defines the digital video file name regardless of whether another DMDX frame will be displayed by the time that digital video frame is displayed. The way this functions is that as the code is displaying video frames it takes note of the number of that video frame in the video sequence, when it sees that the video frame number is greater than or equal to the clockon frame number (and it hasn't already turned the clock on) it turns the clock on. In order to determine which video frame contains the relevant stimuli TimeDX can be used to seek around the digital video file to find a particular video frame and it's number can be noted and then entered into the item file (sorry, cursors don't exist in any generic fashion for digital video files).
If synchronization between a digital video sequence and the rest of DMDX is needed the <DVFrameDuration> switch can be used. In the same fashion as the <DVCLockOn> switch, when the specified video frame has been displayed DMDX moves to the next element in the display queue and re-schedules everything accordingly.
If display of the digital video needs to be abortable
can be used, if the display needs to be contingent upon a subject's response (or
some other condition I can't currently imagine)
can be used (see the jobstatus .eq. 7 example).
If you are doing something extreme like playing back 60 Hz HD video on
older hardware and are experiencing stuttering or outright freezing of the
playback you may have success by either forcing DMDX to track the raster in a
relaxed fashion with -auto -ez on
command line or tripping it with test mode 18
instead of relentlessly hammering the raster status functions or by using a
video mode with a slower refresh, say 24
Hz should you be able to do so. If that's not enough you could use the
freesync video mode modifier that stops DMDX
tracking the raster outright (the fact that you don't necessarily have FreeSync
or GSync display is irrelevant), YMMV however as when
freesync is active DMDX is processing the display queue every millisecond
and this extra load might outweigh any savings from not using relaxed raster
Note that if you are reusing the same video over and over (or even just
twice in a row) certain combinations of things (like using DIrect3D and VMR9 and
some particular video formats) can give VMR9 indigestion and DMDX's attempts to
reuse video resources from the previous item can fail in either freaky ways
(video hitches or only plays partially) or spectacularly (DMDX crashes).
To mitigate this DMDX 22.214.171.124 has extended the
Media Life keyword to handle a media
life of zero that flags to DMDX do not reuse the resources used to render this
To facilitate the use of large audio instruction sets a subset of the Digital Video abilities have been implemented with the <StreamingAudio> keyword. The problem with large multi-minute (and therefore multi-megabyte) audio files is that they must be read into main memory before DMDX plays them if the <wav> keyword is used, this can take appreciable amounts of time on some systems and can also cause portions of memory to be swapped out to the paging file if really large audio files are used that are a significant percentage of or larger than the RAM installed in a machine, slowing things right down. Another problem is that the file must be a plain vanilla .WAV file, no compression. Using the <StreamingAudio> keyword the audio is streamed from disk (only the next fraction of a second is read at any one time) and DMDX also uses the codecs installed on the machine to decode the file.
To use compressed audio files (say .mpg or .au files) a suitable codec must be found, see TimeDX's Digital Video help on Codecs.
The downsides are that any significant degree of precision or synchronization is lost as DirectShow is now doing all the work. The use of .WAV cursors is no longer possible, instead the millisecond accuracy of the <PlaySAFrom> and <PlaySATo> keywords is available (these are in fact synonyms for <PlayDVFrom> and <PlayDVTo>, the frame duration for a streaming audio file is one millisecond). <StreamingAudio> does not support any of the <Sound> (<wav>) keyword's <SetVisualProbe> functionality, the visual probe is always at the start of the audio file (the audio file always begins playing at the frame's scheduled time) and the duration of the frame is either the duration of the audio file or the explicit % frame duration. Nor does it support the <Volume>, <Pan>, <DVFrameDuration>, or <DVClockOn> keywords.
It is currently not possible to have the same instance of a streaming audio file playing twice (the old <wav> keyword limitation in a slightly different incarnation) as the <StreamingAudio> keyword really is designed for simple instructional purposes -- should you really need to do so then use slightly different parts of the file, say <PlaySAFrom 0> in one frame and <PlaySAFrom 1> in the other.