50 2. FIVE STORIES TO A MODEL OF VIDEO STRUCTURE
Birds or viewer reactions to the piece could be analyzed in the same manner that we have applied
to Bellour’s work. Every person who interacts with a document and commits some permanent
behavioral product of that interaction contributes to the document’s signal set for subsequent uses.
Considered from our perspective, this contribution becomes a fundamental aspect of the
setting for considering the relationship between the document/message structure and the semantic
meaning. e additional signal, for example a review, can have a signicant impact on whether a
document is accessed and on how it is evaluated for tness to a given information need. e doc-
ument is not necessarily static with the same impact on any given user; rather, it is an evolutionary
process. e concept of document as evolutionary process receives more discussion in Anderson
(2006) and Wilson (1968).
Bellour sought means to explore and represent moving image documents with the precision
already applied to verbal documents at the micro and macro levels. He sought means to go beyond
what Augst (1980b) termed the “gratuitousness and arbitrariness of impressionistic criticism.” e
digital environment oers the opportunity to do so; to enable speaking directly of the native ele-
ments such as the RGB components and their changes across time; and, to paraphrase Godard, to
confront vague ideas with precise images.
2.5 STORY FIVE: WHAT MAKES A MOVIE AND WHY DOES IT
MATTER?
Movies do not move. Essentially all movie formats are made up of
still images displayed rapidly. Each of the 16 mm frames to the left is
about the size of a ngernail. In projection, a frame is held motion-
less, a shutter opens and allows light to pass through and project an
image onto a screen, the shutter closes, another frame is pulled into
place, the shutter opens, … 24 times per second. e process of inter-
mittent motion was the invention of the Lumiére brothers in 1895.
Electronic, analog, and digital formats, while they do not
present still images observable by the naked eye, do store data in
single frame packets. e frame has been the addressable unit of
the movie since the earliest of days. e frame is a still photograph, so a movie can be said to be a
collection of still photographs. What makes a movie is something more than viewing a collection
of still images.
We opened this lecture with two quotes from highly regarded lmmakers and directors:
e dominant, all-powerful factor of the lm image is rhythm, expressing the course of time
within the frame.
Andrey Tarkovsky (1987), Sculpting in Time
51
Shot and montage are the basic elements of cinema. Montage has been established by the Soviet
lm as the nerve of cinema. To determine the nature of montage is to solve the specic problem
of cinema.
Sergei Eisenstein (1969), "A Dialectic Approach to Film Form," in Film Form
We did so in order to point out the fundamental
role of movement at the frame level as the primary com-
ponent of moving image documents. At the same time,
the two quotes bespeak a fundamental rift in thinking
about what sort of movement counts and how it ought
to be achieved. Should we to essentially leave the camera
alone while making long recordings of action in front of
the lens, as Tarkovsky argues, or should we to construct
movement (and meaning) by placing a variety of images together in rapid succession as Eisenstein
argues? We argue that the controversy is essentially a tokenizing issue—looking at the issue of
movement from a macro level. Looking at change of the video signal over time at the pixel level
essentially resolves the issue.
e frame has been the fundamental unit of pro-
duction of movies, enabling control of the viewing expe-
rience down to the fraction of a second. Eisenstein and
Vertov and most editors working in analog lm made
mechanical cuts at the frame lines; digital editors work
with pixels and timelines, but still cut at the frame level.
e frame serves as a robust means of sampling the movie
data stream and an explanation of what is a movie.
Figure 2.12: Above: top left, Eisenstein edits; right, later analog editing. Lower: left, mechanical frame
level editing mechanism; right digital frame level editing.
2.5 STORY FIVER: WHAT MAKES A MOVIE AND WHY DOES IT MATTER?
52 2. FIVE STORIES TO A MODEL OF VIDEO STRUCTURE
For some time we have been examining ways to describe lmic documents in unambigu-
ous ways, to describe the structure of a movie, to compare structures of movies, and to engineer a
robust model of moving image documents. We had made signicant progress toward these goals
combining the idea of seeing moving image documents as signal sets together with what might
broadly be called a behavioral component. is behavioral component consisted in the well-estab-
lished semiotic literature, particularly Metz, Bellour, and Augst; and the theories and practices of
behavior analysis.
Calculate RGB
histogram from
video frames
Apply a Lorenz
trnsformation
to RGB
histogram
Calculate Gini
coecient for
each
transformed
histogram
Analyze data
Extract frames
from
video source
Figure 2.13: Pixel level analysis system.
Our rst step was to step away from the debates and failures inherent in seeing the “shot”
as the unit of analysis. As Bonitzer (1977) notes, the denition of “shot” is: “endlessly bifurcated,”
essentially rendering the shot useless as a unit of analysis. We used changes in the red, green, and
blue components of every pixel in every frame of a lm sequence to nd points of discontinuity in a
lm. By itself, this approach is interesting but does not provide any particular way to nd signicant
points of discontinuity. Bellour had wrestled for some time with the notions of how lms generate
meaning; he, too, looked to signicant points of discontinuity in the signal set. In his work on the
Bodega Bay sequence from Hitchcock’s e Birds he used his highly regarded critical expertise to
determine the signicant points of discontinuity. We used Bellour’s approach to develop a compu-
tational heuristic for description of any lm—we assumed he was engaging a signal set and char-
acteristics of the signal made it possible for him/necessary for him to see points of discontinuity.
53
Our eorts replicated Bellour’s work very well and we validated the Bellourian heuristic with our
analysis of Looney Tunes lms by two dierent directors. e work with our heuristic met with
enthusiasm from lm theorists and documentalists (e.g., Buckland in Document (Re)turn: Anderson,
O’Connor, and Kearns provide a striking example of combining radically dierent qualitative and quan-
titative analytical methods in their discussion of the [Bodega Bay] sequence of Hitchcock’s e Birds (Skare
and Lund, 2007, p. 319)).
Still, a heuristic is of only limited value for dening “moving image document” and describ-
ing lms in a manner useful for classication. Our current challenge is to engage more lms and
push beyond a heuristic. We currently have RGB signal data for the frames of 60 lmic docu-
ments—Hollywood titles, experimental of various sorts, TREC (Text Retrieval Conference) test
documents, animations, TV shows, etc.
Briey, we use the same sort of signal data acquisition as in our previous work, we simply
use a dierent form of analysis. We derived RGB values for each frame (1,800 frames per minute);
posited an even distribution (as per Gini analysis); derived the area between the RGB histogram
and the line of even distribution; for each and every pair of frames we subtracted the derived area
for frame n from the derived area for frame n + 1. Plotting the dierences yielded a graphical rep-
resentation of structure, particularly points of discontinuity.
A seemingly simple shift of perspective provides another way to look at the frame- to-frame
change. If we plot the same data on a Cartesian plane with value for frame n as the X-coordinate
and the value for frame n + 1 as the Y-coordinate, we have a system in which the unit of analysis is
the CHANGE—this depends on the pixel level data stream (actually sub-pixel as R, G, B).
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8
1.0
Diagraph of Birds Diagraph of Hyde and Go Tweet
Figure 2.14: Digraphs of a feature lm and a Looney Tune cartoon.
Presenting our data in this digraphic way allows us to see a structural pattern within an
entire lm. e greater the deviation of a plotted point for any frame pair from the norm, the
2.5 STORY FIVER: WHAT MAKES A MOVIE AND WHY DOES IT MATTER?
54 2. FIVE STORIES TO A MODEL OF VIDEO STRUCTURE
greater the probability that pair bounds a point of signicant discontinuity. In examining data
with digraph we see the same frame pairs data as in our previous method, but we see them
more obviously. Also, we now have the means of constructing a formula for what constitutes a
movie—most frames would have to lie along the line, some would have to lie o the line. e art
and craft of movie making, and a way of characterizing lmic structure, lies in how many lie o
the line and by how much.
Signicance of points of discontinuity can be presented and examined in two ways. With
Bellour we have signicance dened by a recognized expert in his expert subjective viewing. With
empirical data derived from RGB values and shown to be consistent with Bellour’s expert notion
of consistency, we can dene signicance (on the whole and with some intriguing exceptions) to be
any plotted point of change falling outside one standard deviation. With diagraphic presentation
of RGB data and a much larger set of lmic documents, we have gone from heuristic to the algo-
rithmic. We can take this same data and present it in a rather dierent form—synthetic frames. It is
not too facile to say that each plotted dot in the digraph is roughly equivalent to a synthetic frame.
e data for just those pixels that are dierent between frame N1 and frame N2 can be used to
generate a viewable image that is neither of the two frames nor is it made up of some regions of one
and some regions of the other; in other words, it is synthetic. In most movies there are periods where
most of the frames are similar, although not exactly alike; then there is some signicant change. In our
frames from e Birds we see Melanie in a boat for several seconds, then we see the farmhouse she
is approaching, then we see her in the boat again. In the theatrical release of the e Birds there were
24 frames for each second of viewing time, so in a sequence of four seconds length we would see 96
frames of Melanie in the boat. Not much changes from frame to frame, but there are some changes
from frame to frame; the boat is in slightly choppy water, so the woman and the boat have slightly
dierent distances from the frame edges. ese small dierences yield what almost looks like a pencil
sketch of just the major outlines, since the watercolor remains the same, the boat color remains the
same, the hair color remains the same, and the coat color remains the same—they just shift a bit from
frame to frame. Timing is in standard format of hours: minutes: seconds: frames.
When we reach the point of change from Melanie in the boat to the farmhouse—frame X
last
(00:01:03:15) and Y
rst
(00:01:03:16), as one might expect, there are many more points of dier-
ence so the synthetic frame shows many more points than the sketched outline. en, once we are
at the dierence between frame Y
rst
(00:01:03:16) and Y
second
(00:01:03:17) the synthetic frame is
made up of only a few points of dierence; although the camera has the point of view of the woman
in the boat and the boat moves, there are small shifts from frame to frame.
What is it then that distinguishes a movie from a static still photograph or a set of static still
photographs, as in a slideshow? e narrow constraints that provide the viewer of the document the
illusion of motion and a sense of narrative in the broadest sense make the distinction. ere is a nar-
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.