Keynote: Color is the Keyboard

Schedel’s keynote “Sounding Sight" presents various ways artists have conceived of metaphors of expression and created algorithms to transcode data and dissolve image into sound. From simple analog electro-mechanical devices through complex programs which perform calculations, process data and make logical (or even illogical) decisions Schedel will discuss and demonstrate instruments/devices/programs that inhabit Voltaire's "secret analogy between light and sound."

Keynote: Oskar Fischinger, Preserving Visual Music and Raumlichtkunst

Cindy Keefer will discuss filmmaker/artist Oskar Fischinger and CVM’s work preserving and promoting historical visual music films. Fischinger (1900-1967) is arguably the most influential visual music filmmaker, often called the Father of Visual Music. Her talk covers CVM's work preserving Fischinger’s legacy, his influence on animation and visual music, and CVM’s work restoring and distributing historical visual music films by Jordan Belson, Mary Ellen Bute, Jules Engel, Charles Dockum and others, plus related research and scholarship. She will discuss the process of restoring Fischinger’s original 1920s nitrate film to reconstruct his multiple projector live performances as the HD installation Raumlichtkunst (c.1926/2012).

Fischinger produced a series of cinematic performances in the 1920s using his abstract films, lighting effects, slides and avant-garde music. From press reviews and Fischinger’s notes, we understand these shows as attempts to create some of the first cinematic immersive environments, and as precursors to expanded cinema and 1960s light shows. This paper explores CVM’s reconstruction project translating his Raumlichtkunst live performances to an HD three-channel installation displayed in major museums. Requiring extensive research and fundraising, the multi-year project began with Fischinger’s nitrate film originals, through photochemical film restoration, digitization, color restoration and reconstruction.

The case study of Raumlichtkunst explores the scholarly and archival research, film restoration methodology, curatorial decisions regarding museum display of archival moving images, and issues with the recreation of historic live cinematic performances. Also covered are issues of translation and authenticity, and restoration vs. presentation. Raumlichtkunst was recently exhibited at Whitney Museum, New York; Tate Modern, London; Palais de Tokyo, Paris; Len Lye Centre, New Zealand; and other international venues.

Talk: Teaching Visual Music

Düsseldorf is the home of two specialized undergraduate programs offering a major in Visual Music. My talk will explore how this concentration was established and how it operates. At the Robert Schumann Conservatory, a prestigious music academy in western Germany, undergraduate students can choose to major in Visual Music. All students must pass an entrance exam on their instrument in order to be accepted into the programs. They consider themselves primarily musicians.

My professional background is in motion design for advertising, broadcast, and music videos. I started teaching motion design in 1998 at Germany’s leading film school. When I was appointed as tenured professor for Digital Visual Media in 2004 at Robert Schumann Conservatory I considered how to start a dialogue with musicians who had no exposure to, or even interest in my field. With a team of four assistant professors, one engineer, and half a dozen tutors we developed a Visual Music concentration. I will discuss the team of lecturers, their classes and approaches, and show a few exceptional projects by our students.

Prof. Ulla Barthold, a cinematographer who has worked with many artists and composers, teaches “Visual composition.” Falk Grieffenhagen, a current member of the band Kraftwerk, teaches “Visual Music Tools” like MaxMSP/Jitter and Resolume Arena. In the class of Christian Schäfer, who is an internationally renowned DJ and graphic designer, the students compose their own tracks from graphic notation systems of fellow students and design a vinyl cover for it. And last but not least, Marcus Schmickler, an internationally recognized composer, leads two seminars, one for composition in a visual music context, and the other a series of project meetings where the students discuss their final projects, e.g. multimedia installations and audiovisual performances for their final show in a public space.

I run three seminars, the first one introduces different artists from this genre, the focus of the second is in developing a concept for an audiovisual project, and the third is a series of project meetings leading up to the exams. Currently we are developing graduate seminars in visual music as part of the Transmedia Form major for our new Masters program called Sound and Reality.

John Cage and Audiovisuality

This talk examines the role of visual music and intermedia aesthetics on the career of avant-garde composer John Cage. I look at key moments of intersection in Cage’s career with filmmakers such as Oskar Fischinger, Stan Brakhage, and Stan VanDerBeek. Each informed Cage’s notion of audiovisuality in unique ways, and helps to frame a reading of Cage’s own notion of intermedia around the audiovisual. Essential to this orientation is the notion of audiovisuality, in which a concept or artwork elucidates or expands upon the phenomenological present, thus opening up one’s eyes to the very notion of existence.

Combining qualitative, historical, systematic and theoretical approaches, audiovisuology pivots from metaphorical uses of language to describe music toward the aesthetic experience of both listening and seeing. This shift from the notion of music in absolute terms is more in line with the foundations of intermedia aesthetics, where no single medium is held above another. I find this methodology helpful for approaching a topic that combines elements of historical musicological approaches to style analysis, avant-garde film studies in identity politics, medium, and defamiliarization, contemporary art theory on ontology and objecthood, and traditional film music scholarship on the relationship between sound, music, and image.

Psychedelic Light Shows of the 1960s

The light shows of the sixties were a spontaneous jam session of light and sound.  What were "light shows" and why did they appear in the 1960s? My talk will examine the culture of the sixties which led to light shows and give a brief overview of some of the major shows. In the 1960s television was having a major effect on society. It had a profound influence on the youth culture. They were the first TV generation to mature and they expected visual media. The psychedelic light show was a natural addition to music concerts.

My Electric Collage light show was produced by combining films, slides, abstract color and shapes into one "Electric Collage" of images that flowed with the music. It was described by one viewer as "a spectacular visualization of music through color and images designed to extend your musical experience." We performed at concerts and festivals with bands including Santana, Led Zeppelin, Allman Brothers, BB King, Richie Havens, Herbie Mann, Janis Joplin, Johnny Winter, Ten Years After and Jimi Hendrix, to name a few.

I’ll discuss our specific techniques, equipment and how our light show differed from the other major shows such as Joshua Light Show and Headlights. I’ll examine the interaction of the light shows with the music and audiences, as a form of live visual music.

Where Modern Art Was Headed

With the rise of instrumental music, Romanticism freed music from the bonds of its imitative, representational and supportive roles. In one way of seeing it, instrumental music provided a new language that connected directly with human emotions, a language in which the idea and its expression were one. This is what Walter Pater referred to in his oft-quoted observation that “all art constantly aspires to the condition of music” for instrumental music had obliterated the distinction between subject matter and form.

Two centuries later, painters and sculptors were possessed by a similar impulse. How might they do for the eyes what composers had done for the ears? When Wassily Kandinsky published Der Blue Reiter in 1912, he placed Goethe’s formulation of the problem right at the center of things. “For some time painting has lacked a knowledge of the harmonic base, an established and approved theory, as is found in music.” Painting had yet to find its basso continuo, the harmonic framework that supports the melody in a work.

In that same year Guillaume Apollinaire, the art world’s first blogger, defined art as “the creation of new illusions.” In the work of modern painters he saw the “most audacious school that ever existed” with new artists searching for a beauty beyond the celebration of human and other natural form. Scores of artists joined in looking to music for ideas they could apply to their art, with the goal of creating “an art that would be to painting what music is to literature (Apollinaire).” In this presentation I describe how some of them, including Kandinsky, Leopold Survage, Stanton Macdonald-Wright, Oskar Fischinger, and Karl Gerstner, marshaled color, form, harmony and rhythm to that end, and illustrate some of the implications of their insights for visual music instrument design.

XYScope

XYscope is a library I’ve developed for Processing that enables drawing vector graphics on an analog oscilloscope by converting all shapes into an audio signal, which when fed back into the 2-channel oscilloscope in XY-mode, reveals the image. This library was made open source in July 2017 and has since been adopted by artists interested in exploring the unique visual possibilities that reside in analog vector displays. This talk will outline my developments leading up to the release of XYscope, furthermore the media installations created with it since. Next to my own work, a selection from the state of the art progress taking place parallel in this field of oscilloscope drawing will be presented.

Beyond enabling the revitalization of the analog oscilloscope for creative-coded visuals, a device that enabled profound works by the visual music masters of the 1940s and 50s (Bute, Hirsh, McLaren), XYscope enables a direct symbiosis of the audible and visible mediums. When the signal for image transmission being sent to the oscilloscope is amplified, what one hears is exactly what one sees. With the image being mediated into custom waveforms, generated in real-time as oscillators, the frequency can be controlled, producing a singular hum or a varying soundscape with musical qualities.

Heated Hardware: Preserving Video Image Processing Instruments

This paper presents a critical archival analysis of the literature and methodological approaches concerning the curation, preservation, maintenance and access to video image processing instruments. Visual music artists use a breadth of mediums, from film to code, but rarely perform preservation tasks on their works, let alone the equipment and instruments they use to make their artwork which are so entwined in the concepts and aesthetics of their work. This paper focuses on the preservation of video image processing instruments, sometimes referred to as video synthesizers or video effects generators. My focus on video-based artwork and equipment comes from my own background in artmaking as well as my preservation-based interest in the complexities of the medium that has proven to be more of a timely concern for moving image archivists than any other medium.

Artists such as Nam June Paik, Dan Sandin, Denise Gallant, Steina and Woody Vasulka, Stephen Beck and Ed Tannenbaum make appearances in the paper. To support my claims on the importance of video image processing instrument preservation, I will make use of literature from the fields of media archaeology, moving image archiving, and video art history. This argument builds upon a small body of literature dedicated to archival impulses, concepts and practices surrounding video image processing instruments, of which the 2014 book The Emergence of Video Processing Tools, edited by Kathy High, Sherry Miller Hocking, and Mona Jimenez, may be considered the seminal text to date. Jussi Parikka’s book What Is Media Archaeology? will conceptually ground the issue surrounding obsolete technology within this issue. I will propose new archival methods to preserve and place conceptually within the information fields the issue of video image processing instrument conservation.

The Neural AV Mapper: An Instrument for the Performance of Abstract Visual Music

This paper presents a novel approach to the creation of live Visual Music performances using interactive machine learning (IML) techniques to map audio and visual parameters to the real time input of the performer. The issue of mapping is fundamental to the practice of audiovisual art. Visual Music artists have explored many types of mapping throughout history. For example, Father Castel’s mapping of tone to color, John Whitney’s (1980) use of Pythagorean ratios to transform musical harmonic motion into visual movement, and the free metaphoric approach favoured by artists such as Jean Piché.

As detailed by Bernardo et al. (2017), IML gives artists and non-experts access to machine learning algorithms that can be used interactively and intuitively. According to Sá (2016) there is a balance to be struck between implying a cause and effect relationship between the audio and the visuals and creating enough complexity to hide the nature of the relationship. The task of creating a mapping network between parameters that has enough complexity and flexibility to achieve this goal traditionally takes a lot of programming and manual adjustment. However, with the use of IML techniques, this large workload can be reduced and the mappings can be explored intuitively through playing with the system in real time. In this way, the algorithm takes care of the mapping so the artist can concentrate on the aesthetic result.

The technique presented in this paper relies on artistic intuition informed by cross-modal correspondences (Abbado, 1988) to create complimentary audio and visual stimuli which are then used to train a neural network to map 3D positional data to a range of audio and visual parameters in real time. A description of the original composition Ventriloquy will demonstrate the Neural AV Mapper in practice.

Sonification and Music Visualization, Abstract Narratives and Concrete Poems

Experiences and experiments exploring sonification and music visualization in abstract narrative and concrete poem forms applied to installation, performance and exhibition contexts. In one example of Sonification to be discussed, the rhythm and frequency of rain drops were translated as a musical phrase or melody and as inspiration for animation in a work We notice raindrops as they fall (Fletcher, Pollard, 2016). Algae Data Music (P. Fletcher, J. Clark, 2015). was a sonification experiment mapping and interpreting scientific data to sound and animation. Data of sea temperature changes was translated into musical pitches and animation created in response to the overall theme and flow of this resulting music. Jennifer Clark's scientific work involved, in her own words, looking at “how climate change will affect the resilience of an intertidal macroalgae (or seaweed) to increasing temperatures.” The author’s Algae Data Music was an example of Visual Music applied to bringing together the disparate fields of audiovisual media artwork and scientific data, hopefully communicating to general public and niche audiences in science, visual art, and sound art. This was an attempt to confound and interact the specialist fields of environmental science and media artwork.

Visual rhythm, hypnotic and hallucinatory abstract narrative is discussed in relation to the silent but nonetheless visual music multi-screen projection installation The Drive to Work. Circular Horizon explores the idea of responding to poetry with visual music as a unique form of poetry itself. A thread of continuity throughout all these discussed works is an idea of  “Audio -Spectatorship,” which will be shown to be central to the author's approach to abstract narrative and  the appreciation  of Visual Music. This discussion will reference the theories and writings of Vivien Sobchack, Michel Chion, Claudia Gorbman, Holly Rogers, Albertine Fox, Siri Hustveda and others.

Hybrid Analogue/Digital Computing for Live Visual Music Performance

This paper will outline recent research towards the evolution of a new personal practice for live visual music performance. Over an extended period of research and development, I have investigated a variety of analogue technologies and techniques, principally for their immediacy and accessibility and the direct relationships they can facilitate between sound and image. These have included analogue video synthesizers and oscilloscopes, a hacked '80s vector games console, lasers, and Paik/Abe-modelled Wobbulators and a host of other vintage equipment at Signal Culture in upstate New York.

These technologies were thoroughly explored by many well-known artists when they were ‘current’, and there are also increasing numbers of contemporary artists rediscovering them for their unique capabilities. As well as exhibiting specific audio and visual behaviors and qualities, they will typically facilitate a ‘hands on’ approach, where actions and their consequences can be transparently observed and read, by artist and audience alike.

The specific focus of my research is a combination of these techniques and technologies with 21st century digital computing, in the form of a complex interactive control software written in Max/MSP. Digital technology also offers many advantages, principally in terms of fine control, complex interactions with many parameters, and repeatability. I have been exploring ways in which analogue circuits can be controlled and monitored by digital systems in a complex feedback loop. This opens up the ‘best of both worlds’—it allows very precise control but also opens up new possibilities for complex and chaotic behaviors. Examples of this will be used to illustrate the talk.

James Whitney’s Turn to Pottery

"What a relief, and what a feast, to see a work where one can feel the many years of intense living, feeling, thinking. So that there is always somebody, working almost in total silence, who comes in to restore Cinema with a work that looks like it’s made by the gods."

Jonas Mekas wrote the above passage for Soho Weekly News (May 12, 1977) in response to the New York premiere of James Whitney’s film Wu Ming (1977). The film was Whitney’s second after his ‘return’ to filmmaking following years devoted to making ceramics. The quote is emblematic of the critical vein in which Whitney’s time as a potter was contemporaneously considered – mainly an anecdote furthering the biography of James Whitney (1921-1982). Between the release of Whitney’s films Lapis (1963-66) and Dwija (1973), many wrote that they believed he had retired. The “silence” to which Mekas referred was actually what Whitney coined a ‘white wait’– a period of rejuvenating one’s art by concentrating on a new or different discipline. Whitney also described his work in clay as a way of “putting behind certain ideational patterns” (Unpublished interview,1974).

This presentation outlines James Whitney’s pottery making and how he, his brother John and others thought of his craft practice during the 1960s and 1970s. Based on new research and resources, including James Whitney’s library of craft books, his pottery in the William Moritz and Angeline Pike collections, an archive of photographs and papers from Moritz, and Whitney’s unpublished interview from 1974, this study illuminates an oft-mentioned but rarely analyzed period of Whitney’s artistic practice.

The Code of the Cano

The cano is what the Shipibo, an Amazonian tribe known for its shamanic work with the plant medicine ayahuasca, call what they perceive as the vibrational pattern that inhabits and connects all life forms. In ayahuasca medicine ceremonies, during which the bitter psychotropic brew is drunk, the shamans conduct the healing work through songs called icaros. It is the songs themselves—not the medicine—that are understood to make the normally invisible cano visible, and therefore the icaros, as if, are used to sing to and align the sick or traumatized energy lines of participants with the coded, ontologically-given cano.

From the Shipibo shamanic perspective, there is no difference between the cano, the sung icaro which calls it into being and animation, or the healing it produces. It is an ancient form of “visual music” with striking implications related to, among other things, the nature of oracular vision. It is possible to examine the cano outside of ceremonies, through its traditional and contemporary presentation on the embroideries and pottery by Shipibo artists. This paper will draw on the author’s personal five-year journey with Shipibo shamanism and her study of the artwork and patterns, as well as her Western art history background and work in the contemporary art field. The paper will provide first-hand and art historical analysis, while synthesizing and recontextualizing within an art context a number of anthropological, scholarly articles, and books that have sought to understand the nature of this patterning.

In addition, the paper will relate this tradition to an epistemological search of the numerous modern “Western” artists, which in many cases attempts the elucidation of similar energies but through seemingly opposite means, which is to say by a complex patterning of interference, chance, or automatic or mechanistic tools.

A Metaphor Toolkit For Visually Mapping Musical Scores

Jack Ox will show and explain some mapping tools that she has derived from conceptual metaphor and blending theories. She has a 30-year history of translating musical scores into an ever-expanding visual language. By moving through different composers and types of musical structure her visual music language increased in scope and size.  Presently, Ox is abstracting the processes discovered by reverse engineering examples in her own work with music and other examples found in science, design, and art that were created before the theories were written down by George Lakoff, Mark Johnson, Mark Turner, and Gilles Fauconnier. Her Ph.D dissertation, Manifestations of Conceptual Metaphor and Blending Theories in Design, Art, and Science looked for instantiations of these theories, which she is now codifying into metaphor tools for other people to adapt to their own cross-modal works.

These tools not only map musical structures into a visual language, but they can be adapted to many other modes, such as from visual to sound, smell to touch, or smell to sound. Included in the talk will be descriptions of three different color systems that carry significant musical information in how they are sequentially structured: One maps harmonic movement and quality, the second maps how and where vowel sounds are produced in the human vocal tract, and the last is based on an extensive timbre system for timbre based music. Included in this presentation will be a description of how a musical score differs from a performance. If we can extract the essential processes, or algorithms, from examples in the world, we can then build and wield metaphor tools with greater precision.

In Media Res: Light as the Medium of Visual Music in Prewar American Art.

The many color-light devices imagined and created by American artists—such as Morgan Russell’s project for a Kinetic-Light Machine, Thomas Wilfred’s Clavilux, or even Charles Dockum’s Mobilcolor Projector—have all been categorized as pioneering works in the field of light art. The light devices created by those artists are more than mere machines. They are theories and ideas on art and technology and solutions found after a comparison—a modern paragone—between all the possibilities offered by media to create a visual equivalent to music.

Refusing technological determinism and modernist views over film and new media, these artists form a specific American trend in the use of technology in art. They all claimed that they had invented a new medium of expression—an independent art form that would supersede all existing media in terms of emotional effect. Following the concept used by Gene Youngblood to define the work of Thomas Wilfred, one of the most emblematic figures of that trend, early American experimenters in light could be defined as intermedia artists.

This paper will discuss the media theoretical aspects of prewar American Visual Music using the writings of Thomas Wilfred, Morgan Russell, Van Dearing Perrine and Joseph Schillinger. My ambition is to explore the reasons of their refusal to use traditional media to create Visual Music.

Norman McLaren: Audiovisual Aesthetics

Norman McLaren is one of the pioneers of Visual Music, having produced dozens of films in this area since the 1930s. McLaren is also one of the artists in Visual Music with a greater variety of aesthetic ideas for synchronization between sound and image, expressed in quite diverse ways in his vast filmography. Perhaps the most important aspect of his work is the creative audiovisual relationships that McLaren made, going beyond a mere visualization of music, arriving at transmidiatic relations between sound and image, where the pair constitutes an inseparable unity. This synchronization goes from direct correspondences where each sound corresponds to an image, to relations where what matters is the sensation that the audiovisual set causes in the spectator, in which the artist uses more sophisticated creative strategies, in an expanded and fluid way, where the elements maintain a dialogical, but not explicit relation.

In this paper, I will try to systematize some of the aesthetic solutions used in his films, demonstrating the broad creative potential of the artist. Some of his films will be used as an example for a deeper analysis of the insertion of each visual element and its relation with sound. The aesthetic aspects of his work will be analyzed mainly on the basis of Kandinsky's systematization in his book Point and Line to Plane, where he talks about the intrinsic power of form, color and sound, where each element causes a different sensation in the spectator. The analysis will also include authors such as Tom DeWitt and Brian Evans, who analyzed audiovisual elements in Visual Music works for their aesthetic appeal, as well as texts by McLaren himself that discuss technical and artistic issues he developed in his works.

Are Light Instruments Color Organs?

This paper concentrates on a moment of apparent disruption in the history of light-producing instruments, or “color organs”: the abandonment of the musical keyboard interface by Mary Hallock Greenewalt and Thomas Wilfred, two twentieth-century light artists/inventors.

Eighteenth- and nineteenth-century instruments (created by Castel, Bishop, and Rimington), regardless of whether they could produce sound, produced lights and colors by means of keyboard interfaces. The interface of these early instruments was predicated on the perception of music as a discrete series of pitches, and the transferability of each of those pitches into discrete colors.

Greenewalt and Wilfred’s new interfaces reify their rejection of color-tone analogies and their desire to create a medium-specific, autonomous light art, freed from its historical dependence on music. The slides and pedals of Greenewalt’s “Sarabet” prioritized subtle control of luminosity over shifting colors, while the slides and rotating keys on Wilfred’s “Clavilux” allowed him to create nebulous, slow-changing forms for projection. Both figures rejected the term “color organ,” but contemporaneous reception and recent histories have applied the term to their instruments regardless (Peacock 1988; Gage 1993; Moritz, 1997; Kienscherf, 2005; Elder, 2008; Farmer, 2008). I suggest three reasons for this discrepancy: the historical precedence of keyboard-based instruments for creating light effects, Greenewalt and Wilfred’s continued reference to music in describing their light art, and later historians’ desire to create a seamless pre-history of intermedia or visual music.

Lastly, I reflect on later uses of the term “color organ” to describe DIY light-bulb radio units and early disco lighting that could automatically light in response to recorded music. I argue that the long history of the term “color organ” suggests that, despite Greenewalt and Wilfred’s intervention, music remained an important stimulus for the creation of light effects and a means through which abstract light has been understood.