The Composite Effect

Though on first inspection it may be difficult to believe, every media consumable is an artifact born of composited forms. The composite is the leviathan that directly, or indirectly, manipulates the interpretation of cultural forms, and it will only get stranger. The current status of the composite in the creation of cultural forms can be explored by first examining its use in music, the manipulation of images (still and moving), and the growing field of augmented reality. The imminent status of the composite might best be understood by observing the futures suggested by an independent filmmaker and a large manufacturing company. 

Contemporary Examples of the Composite

In The Language of New Media, Lev Manovich defines digital compositing as having a specific and well-defined meaning. Particularly, he writes that “it refers to the process of combining a number of moving image sequences, and possibly stills, into a single sequence with the help of a special compositing software…” (Manovich , Kindle Ed. 136). Here Manovich is specifically talking about the use of this term in relation to the manipulation of video. This initial definition can be described by considering the weather report on most news programming. While providing some glib observations on city life in tandem with an abridged description of the temperature, the weather-reporter is often witnessed standing in front of some form of map. In some cases this map includes animated patterns of repeating cloud formations, often indicating where there might be rain or snow. The image of this person standing in front of a map is, in fact, a lie. The truth is that this particular reporter is standing in front of an evenly lit backdrop that is either bright green or blue. This flat color can then be removed and in its place a still image or animation is added. Here we can see a simple example of a composite: the combination of a number (in this case two) of moving image sequences into a single sequence. While Manovich initially talks about this in relation to images, he later applies the same principles in talking about the DJ. 

The DJ’s art is measured by his ability to go from one track to another seamlessly. A great DJ is thus a compositor and anti-montage artist par excellence. He is able to create a perfect temporal transition from very different musical layers; and he can do this in real time, in front of a dancing crowd. (Manovich , Kindle Ed. 144)

The rise of digital authoring and distribution tools has helped to disseminate the work of the DJ beyond a single fixed set in a club. For the modern DJ, the recording studio is the computer and the distributor is the internet. The specific form born out of this type of creative act is sometimes referred to as a mash-up. While this form may be a composite of only two or three songs, many popular artists create works that are comprised of multiple samples from multiple songs. Jordan Roseman, known as DJ Earworm, describes his work by saying “what I do is take a bunch of songs apart and put them back together again in a different way. I end up with tracks called mashups.” Roseman’s work is the quintessential composite in its nature. His modular approach to disassembling music into component pieces before reassembling it into a new unified work reads as though he were quoting Manovich. In writing about the process of working with composited work Manovich notes that, “a typical new media object is put together from elements that come from difference sources, these elements need to be coordinated and adjusted to fit together” (Manovich , Kindle Ed. 138). The DJ’s appropriation and rearrangement of material in the pursuit of creating a new work is a variation of the theme of sampling and looping and has largely been met with enthusiasm. Many devoted fans of specific artists attend concerts to witness the act of live mixing and compositing. Greg Gillis, known as Girl Talk, has garnered especially high praise and support for the form.  Gillis is a composite artist. A musician whose work is entirely based in the deconstruction of established works so that he can reassemble the component pieces into something different. 

While the mash-up is quickly finding home in the realm of pop-music, arguably the most well established form of composite is now largely invisible. The manipulation of photographic imagery has become so commonplace that it is nearly invisible to most consumers. In this regard, advertising, particularly, has become the champion of the composited work. Should one open any Sunday-Super-Shopper adverts for any corporate chain the only imagery involved is entirely created out of a number of independent images paired and layered together. This pairing and stacking of visual components is now so commonplace as to seem banal. High-end advertising is especially notorious for the use of composited works. The perfect photograph of a hamburger is a fine example of this magic: the composition is carefully framed and arranged by a food photographer; later the photo is manipulated to ensure for a proper distribution of sesame seeds; another artist will color correct the image for the most appetizing shade of green in the lettuce and red in the tomato; finally a designer will move the hamburger digitally to ensure that it is being presented in an appropriately branded environment. Layer upon layer, the composited image has ceased to be a record of the actual object, and has instead become an abstract representation of what it ought to be. This is the work of composite in advertising. Perhaps a more startling example can be found in the pages of the Ikea catalog. According to the Wall Street Journal, twelve percent of the images in the Swedish furniture distributor’s catalog were not actual photographs of real objects but were instead three-dimensional computer renderings (Hansegard). To accomplish this, first a scene is created in wire-frame with 3-D modeling software. The resulting wireframe is then painted with textures, and light in order to resemble an actual room. The WSJ goes on in the article to note that in 2013 one out of four images used in the print catalog and online will be made exclusively of computer renderings (Hansegard). This is both astonishing in its execution, but also a prime example of the true power of the composite. The amalgamated form of polygons and texture are layered and manipulated until the line between real object and assembled representation becomes invisible.  

The spread of the composite into all cultural forms is further fueled by the efforts and exploration of the technology juggernaut Google. In April of 2012 Google released one of its first promotional videos for the development of “Project Glass” (Google). The video presents a montage of moments out of the day of the subject. The video is shot entirely from a first person perspective and demonstrates what life might be like if instead of looking at the screen of a mobile device, that same information was instead accessible through a heads-up display. This additional semi-transparent layer of information is often referred to as augmented reality. Google’s device, Project Glass, is a lightweight augmented reality (AR) system worn in place of a pair of glasses. The wearable device contains a touch surface for control, a camera, wireless antennas, onboard computer processor, and a prism display. The promise of AR is that it will change the existing relationship between user and screen forever. Simply put, there is no more screen as it’s conceptualized today because the screen is potentially everywhere and everything. When using this device, every visual experience is a composited image composed of the physical world and an additional layer of data. Google’s efforts with AR aren’t the first examples of the layered reality push. With the rise of smart phones as portable computing devices many applications (apps) have been developed across mobile platforms for access to additional layers of information. AR applications typically take advantage of GPS data (in order to determine physical location), gyroscopic inputs (in order to determine the orientation of the device), and Internet connectivity (to populate the field of view with data). While not the immersive experience of Project Glass, AR apps have proven to be an interesting investigation into the possibilities of the composite as rendered in real-time with only a mobile device. 

Future Implications of compositing: Constant Composited Reality

While the promises of Project Glass might be intriguing, the future is often difficult to predict. Corning, maker of the Gorilla Glass used in many touch screen devices, has its own vision of what the future might resemble and in February of 2012 released a short promotional video, “A Day made of Glass 2”, suggesting the possibilities of touch surfaces in the near future (Corning Incorporated).  In their six-minute montage of the near future nearly every surface is fabricated out of touch-sensitive transparent glass. Each interaction with a different product reveals sheets of glass that can be transparent or opaque with a single press or swipe. In Corning’s day of glass every surface suddenly becomes a window of composited information. Tablet computers are transparent sheets that act as windows into the world where a sea of data is waiting to be revealed. For Corning it is not enough that every surface might be a display, every surface should have the potential of being a composite display. While this may initially seem like flights of technological fancy, it’s worth mentioning that in the summer of 2012 the display manufacturer Samsung began showcasing its transparent LCD screens (Sidev). Corning’s vision of the future seems limited only by the current economy of cost for these displays, and in many ways lacks any real revolutionary implications about the role of the composite in the consumption of information.

Beyond Corning’s slick vision of a world crafted of transparent touch surfaces the video short made by Sight Systems provides for a more interesting suggestion of what the future of displays might look like. While the short is built around a questionable storyline, the real magic happens here in regards to the representation of an augmented reality display worn as contact lenses. Sight Systems seems to suggest a vision of AR that is not simply additionally layered information but instead approaches composited reality. Here the proposal isn’t merely that the world could contain additional layered information about text messages and weather, but rather that the world aught to be constructed according to the desires of the subject. A world of constantly composited visual representations. A world of constructed realities filled with notifications and advertisements ad nauseum. 

Closing Thoughts

The divining of specific manifestations of future technological advancements or ideological implementations is beyond reasonable conjecture. What is not unreasonable, however, is a recognition of the influence of the composite. Today’s new media artifacts are crafted realities built upon stacked films of both visual data and invisible meta-data. Digital reality, and expression is predicated upon notions and expressions of the composite, and it is not presumptions to assume that this is only the beginning. The future may well be a world of composited reality.

---

Works Cited

Corning Incorporated. “A Day Made of Glass 2. Same Day. Expanded Corning Vision.” Online Video Clip. YouTube. YouTube, 12 Feb. 2012. Web. 13 Dec. 2012. <http://www.youtube.com/watch?v=jZkHpNnXLB0>.

Eveleth, Rose. “How fake images change our memory and behavior.” bbc.com. BBC. Web. 13 Dec. 2012. 14 Dec. 2012. <http://www.bbc.com/future/story/20121213-fake-pictures-make-real-memories/1>.

illegal A.R.T. “Girl Talk.” illegal-art.net. illegal art. Web. 15 Dec. 2012. <http://illegal-art.net/girltalk/>.

Google. “Project Glass: One day….” Online Video Clip. YouTube. YouTube, 4 April, 2012. Web. 13 Dec. 2012. <http://www.youtube.com/watch?v=9c6W4CCU9M4>.

Hansegard, Jens. "IKEA's New Catalogs: Less Pine, More Pixelsc." wsj.com. Wall Street Journal. Web. 23 Aug. 2012. 13 Dec. 2012. <http://online.wsj.com/article/SB10000872396390444508504577595414031195148.html?mod=WSJEUROPE_business_LeadStoryCollection>

Manovich, Lev. The Language of New Media. Ed. Roger F. Malina. Kindle Edition. Cambridge: The MIT Press, 2001.

Roseman, Jordan. “earworm MASHUPS.” Web. 15 Dec. 2012. <http://djearworm.com/>.

SidevDisplaySystems. “L' écran transparent Samsung NL22B bientôt chez SIDEV.” Online Video Clip. YouTube. YouTube, 16 Jul. 2012. Web. 13 Dec. 2012. <http://www.youtube.com/watch?feature=player_embedded&v=rZslQZ6iMgA>.

Sight Systems. “Sight.” Online Video Clip. Vimeo. Vimeo, 24 July, 2012. Web. 13 Dec. 2012. <https://vimeo.com/46304267>.

A Review of The Language of New Media

Photo by Matthew Ragan

The Language of New Media. By Lev Manovich. Cambridge, MIT Press, 2001; 354 pp. $16.71 kindle.

What, precisely, is the right term for content that lives with its left foot in a world of traditional gestalts, and its right foot in a mire of experimental methods? How does one characterize the trends that have emerged in the making, distribution, and viewing of media? How does one broadly conceptualize and decode the impact of a digital mechanism as the primary method of cultural expression? These questions frame the narrative in The Language of New Media by Lev Manovich. Heavily constructed around the critical conventions of cinema, Manovich’s perspective works to deconstruct the modalities of today’s media with an “aim to describe and understand the logic driving the development of the language of new media” (Manovich 7). 

In his examination of (what Manovich calls) the “new media” this work looks to create “an attempt at both a record and a theory of the present” (7). Manovich helps the reader towards a working definition of new media by stating that “all new media objects, whether created from scratch on computers or converted from analog media sources, are composed of digital code; they are numerical representations” (Manovich 27). In short, his assertion is essentially that any new works created, modified, or distributed by using a computer fall into the category of new media. Further, historical analogue works that have been transferred into a digital representation also bear the title of new media. While his definition seems, at first blush, to be all encompassing, his explanation heavily relies on the concept of numerical representation. “Numerical representation turns media into computer data, thus making it programmable. And this indeed radically changes the nature of media” (Manovich 52). Here Manovich is really stating that that the numerical nature of media stored on a computer is the defining characteristic of new media. The programmable nature of this media is what makes it new. The problem with this newness is that it is not explicitly visible. A painting scanned and stored on a computer carries the same characteristics as the analogue original. How then does one differentiate between the two? Manovich’s distinction about the programmability of the numerically represented copy here becomes the key to understanding his definition. This new copy can be manipulated in a manner that the original could not. As an example: a tool like Photoshop can be used to instruct a computer to replace all of the color values of a specified hue, with a different hue; or, average color values from specified rows or columns could be fed into a synthesizer with the end result being an auditory representation of the painting. The numerically represented version of this painting can now be changed in an infinite number of ways. At the heart of how Manovich characterizes new media is its programmability – its ability to be altered and changed with a computer. In this way Manovich asks the reader to conceptualize the computer as a meta-medium for media. “No longer just an Analytical Engine, suitable only for crunching numbers … it has become a media synthesizer and manipulator” (Manovich 25). It is here, after a lengthy process of defining his terminology, that Manovich starts to push the reader into interesting intellectual territory. He argues that a computer is the mechanism for consuming new media as well as the tool for creating it, transcoding it, and distributing it. “All culture, past and present, came to be filtered through a computer, with its particular human-computer interface” (Manovich 64). It is at this junction that Manovich begins to push the reader to consider that “at the same time, the design of software and the human-computer interface reflects a larger social logic, ideology, and imagery for the contemporary society” (Manovich 118).

While Manovich goes to great lengths to help the reader see that the medium has a bias, this is not a new problem for media. The medium for every work has a bias, so what precisely is it that Manovich is trying to tell the reader? In a linguistic fashion that feels distinctly dystopian, and also strangely inviting, he slowly spells out the real problems of this initially invisible bias. 

Now interactive media asks us to click on a highlighted sentence to go to another sentence. In short, we are asked to follow pre-programed, objectively existing associations. Put differently…we are asked to mistake the structure of somebody else’s mind for our own. (Manovich 61)

The convention of using hyper-links on the web is one of the underlying principles of a flat network. Unlike hierarchical file structures on a computer that require the user to follow a specified path to locate a file, on the web any artifact can be linked to any other artifact. Manovich reminds us that someone, or something, did the work of creating those connections.  Further, those connections are not arbitrary. The author of a blog makes specific decisions about what links to include, and where to include them. Here Manovich pushes the reader to remember that at some point the relationship between those two highlighted sentences had to be constructed by another party. Another party that, presumably, had a specific agenda for how the user is going to experience that moment. The implications here are tremendously powerful for the reader, but Manovich over-reaches his assertion with a one-up-manship that is logically sloppy. 

Two sources connected through a hyperlink have equal weight; neither one dominates the other. Thus the acceptance of hyperlinking in the 1980s can be correlated with contemporary culture’s suspicion of all hierarchies, and preference for the aesthetics of collage in which radically different sources are brought together with a single cultural object. (Manovich 76)

Correlation is not causation, and without appropriate support the reader is left to question what ideological position they are being asked to adopt.

Perhaps what Manovich is really trying to communicate is the deeply integrated, and partially invisible, impact that this shift towards new media implies. 

Because new media is created on computers, distributed via computers, and stored and archived on computers, the logic of a computer can be expected to significantly influence the traditional logic of the media; that is, we may expect that the computer layer will affect the cultural layer. (Manovich 45) 

The result of this composite is a new computer culture – a blend of human and computer meanings, of traditional ways in which human culture modeled the world and the computer’s own means of representing it. (Manovich 46)

Who then is responsible for this new cultural model that is a blend of traditional and experimental modalities? This work seems to be targeted distinctly towards academics and media historians who are invested in having a sense of the broad strokes that created the new media landscape of the early 1990s. Interestingly, it seems as though an appropriate readership for this work should also include media designers, application developers, cyber-culture artists, and students of new media. This is especially true given his critique of the typical citizen’s daily mode of operation, and the role these individuals might take in reversing this trend: 

Today, the subject of the information society is engaged in even more activities during a typical day: inputting and analyzing data, running simulations, searching the Internet, playing computer games, watching streaming video, listing to music online, trading stocks, and so on. Yet in performing all of these different activities, the user in essence is always using the same few tools and commands: a computer screen and a mouse; a Web browser, a search engine; cut, paste, copy, delete, and find commands (Manovich 65).

Manovich is explicit in his belief that a computer is meta-medium media machine, a cultural transcoder that acts as a mechanism for creation, translation, and distribution. This is perhaps best represented when he says “in short, we are no longer interfacing to a computer but to a culture encoded in digital form” (Manovich 69). He is the most inspiring and the most chilling when he examines the computer as our cultural monolith. His only failing is that the critical language he predominantly adopts is squarely grounded in cinema. While this approach is not without its merits, at times it feels stretched too thinly to make an argument that’s truly compelling. One of his grossest assertions about the reach of cinema comes in his insistence that “cinematic ways of seeing the world … have become the basic means by which computer users access and interact with all cultural data” (Manovich 78). This is true insomuch that the computer interface is a montage of individual actions framed by a screen. However, this opinion seems to deny the reality of reading text as one of the primary acts of interfacing with a computer. Further, it dismisses the fact that not all text wants to be cinema. To belabor the point, not all sound or still images strive to be represented through cinema.  Provided that one accepts Manovich’s bias towards cinema as the dominant cultural form, The Language of New Media provides an inspiring examination of the historical trajectories of mediated culture, as well as a suggestion of what is likely to come. 

Works Cited

Manovich, Lev. The Language of New Media. Ed. Roger F. Malina. Kindle Edition. Cambridge: The MIT Press, 2001.