N I O & T H E A
R T O F I N T E R A C T I V E
NIO AND VISUAL/SOUND POETRY
When I first started interactive audio for the Web, being a writer, I was attracted to experiments with poetry. But I was interested in layering sounds, so it seemed, after some experimentation, that musical sounds were more interesting and engaging layered than words. When you layer words, they make a certain type of sense and have certain resonances, but they're quite different from musical sense and resonances which, at the moment, I find more fundamentally expressive in interactive audio.
I recorded sound poetry some time ago. It was a way into things that cannot be said in words but sometimes need saying. Written words and sentences do not have easy access to the primal or the harmonic/dissonant reveries of pure sound or the meaningful repetition, variance, trance, and pattern of the drum. Though, on the other hand, poetry has access to beauties, voices, song and thought that music can barely contain. Inscribed language and voice extend along many dimensions including the air.
As sound poetry is to spoken poetry, so is visual poetry to written poetry, often. I've been drawn for years to visual poetry, particularly lettristic visual poetry that deals in syllables and letters as opposed to words, phrases and sentences. For the above reasons, but also because in the digital realm the shapes of letters are more various than the shapes of words, which tend to be elongated rectangles. And, as a programmer, a letter is typically a continuous thing on which various transformations/animations are more visually appealing and suggestive than on whole words or sentences. Letters are characters. They have more character than words do, in some ways. And since they take up less memory than whole words, they're more amenable to smooth animations.
Which opens up the possibility of experimenting with the tone and rhythm of motion, whereas much literary animation on the Web is sufficiently jerky that it needs to depend almost completely on the conceptual dimension of the animation, the sense of the words and how they permute or the situation, rather than being able to dance with human feeling, with the gestures of the body, with the body of the body.
Sound poetry and visual poetry are often in strong relation anyway. Poets doing sound poetry realize that printed versions of their work would not exactly fit in nice stanzas, usually. And visual poets seeking to perform their works often resort to sound poetry, given the non prosy spaces in which most visual poems operate.
I wanted Nio to combine the visual, sonic, musical and interactive
as integrally as I could in aspiring to some poetic, writerly realization
THE SOFTWARE NOW AND FUTURE REVS
There's software for the desktop, such as Cakewalk (which is a multi-track recording 'studio'), Acid, and other sound studio-ware that enables synchronized layering of one sound over another and synchronized sequencing of sounds, and arbitrary rearrangement of the sounds, but Nio is one of only a few pieces of software for the Net that permit this. Additionally, Nio synchronizes audio and arbitrarily layered visual animations. It has a sense of rhythm sonically and visually, which previous work for the Web has usually lacked.
In addition to suggesting a new form of music and, really, an interactive multimedia form, since the visuals are tightly integrated with the audio, it could also be the basis of studio-ware for the Web. It combines art and the tool, art and the application, rather tightly yet the interface is very simple and mostly it's just a lot of fun, I hope. There are a few pieces on the Net I've found that move in these directions. At the bottom of the 'page' I've linked to the fine work I've found so farwhich probably is a small sample of the great work out thereI don't have any illusions that my links are exhaustive of the best work done. If you encounter interactive audio on the net that you like, please drop me a line.
Nio and similar works pose interesting interactive alternatives to the Shockwave/Beatnik music videos and mixers on the Web at Shockwave.com, Beatnik.com, or MTV, for instance. Pinch me, but I think I have a more exciting experience here, as do many of the others in the links below. Surprise: we see the best of the interactive audio work is by independent artists doing work on their own music.
Nio is also designed to read in external casts of sound icons/sounds/animations. In other words, Nio is designed to be a kind of player, like the Real Player is a player, of synchronized, interactive, dismembered media.
And it streams in such a way that even 56k modems start Nio quickly. It reveals itself gradually as more of itself streams in.
The animations are Flash vector graphics imported into Director, so they are quite small.
Nio's abilities concerning synchronization of layers of audio and sound and synchronization of sequences of sound paves the way for more interesting musical software for the Web. Without audio synch, the music you deal with, if it's interactive at all, has to be essentially arrhythmic or, if it is rhythmic, then you must stay away from many situations requiring synch, such as dealing with layers of rhythmic sound. Synch is a precondition for full-featured, interactive rhythmic music. I'm a drummer. I wanted it bad and I wanted it on the money.
Beyond the above possibilities, there are juicy possibilities for further development concerning the ability of players to both record within Web-based software and also add sounds of their own from their hard drives. This can be accomplished via the 'Audio Xtra' at updatestage.com. Also, the multi-user server, together with the 'Audio Xtra', make possible some sort of online jamming, though obviously there are bandwidth considerations in such heavy, real-time exchange of data over the Web. At very least, you can make software such that Mary Jane adds vocals today and Cliff Syringe adds bass tomorrow, that sort of thing, very useful asynchronous composition over the Web. Though probably some form of real-time jamming is also possible: full-duplex telephony already exists for the Web. Add the ability not just to hear each other but also hear the current mix and record new tracks together over the Web, and then play with them in something like Nio, and you have a rocker, I think.
That's a tool. There are also possibilities in this idea for art pieces. And live performance with an audience.
New territory and lots of it artistically, technically, and in business.
Mostly what I'm doing in this essay is discussing some of the possibilities within these three spheres. And my own approach to them.
Interactive audio is
all I'm doing now.
The Egg Word of Ur Exercising
prefers to be called Egg Word
POETICS OF SYNTHESIS
I've been working on Nio and related interactive audio pieces, collectively called Vismu (visual music), for a year. The prospect of once again working with sound and combining sound work with the interactive textual/graphical work I've done for years at vispo.com was irresistible once I found the tools that suit my needs and are capable of further expansion and precision.
I hosted and produced a literary radio program in the eighties for six years; it concentrated on audio writing, ie, material done by people with a background in writing but the material was done for recorded media. I played the work of people such as Gregory Whitehead, Susan Stone, Jay Allison, Helen Thorington, blackhumour, G. X. Juppiter Larsen, Rod Summers, and Dan Lander as well as sound poetry, and produced my own work, some of which is in the audio section of vispo.com. After I left the radio show in the late eighties, I got a computer, studied Computer Science and Math, and concentrated on programming, the page, graphics, and so forth; not much sound work. It is a pleasure to return to it.
Sound is now a big part of the net because of sound compression and streaming technology (which provides sound even to 56k modem connections), increased bandwidth for many via DSL and cable modem, etc., and also because of programs like Napster, sites like mp3.com, and the presence of radio stations on the net. Music and sound has migrated to the net in quite a big way just as writing did previously. Last but not least, multimedia tools for the Web such as Director, Beatnik, Flash, and a number of other products enable artists of all types to combine media in works that stream well to 56k modems.
It feels good to combine one's interests and abilities and means of expression, to make connections. Such synthesis is also 'natural' in digital media and on the net: the net has a way of introducing people to people and people to ideas and ideas to ideas, media to media and the digital has a way of defining media objects in similar ways, whether sound or text or image, etc., and offering ways to manipulate media objects that are similar.
Multimedia is a natural for the Web because we ourselves communicate
in multi-media. And many media can be transmitted and received, and
acted upon on the net.
NIO AND SYNTHESIS OF LITERACIES
The Web has wrought on language and image a kind of dynamic fusion. The sound recording is next up, now that it is mixing with language and image on the Web. What is recorded, whether sound or language or image, etc., is inscribed, is 'written' even if it is also 'spoken' or 'drawn' etc., and becomes subject to the same sorts of editing, self-consciously external representation, morphing among like objects, and even blending with unlike objects. What we saw with language and image will be recapitulated with sound, language, and image.
The audio writing for radio and cassette carried out by the people I mention above and others made them and their audience aware of recorded sound as inscription, as a kind of "writing on air," as Gregory Whitehead said. Whitehead's brilliant audio work, together with his poetical/analytical writings about radio and recorded sound, live bodies and "anti-bodies" inscribed in the air were and still are important to me in approaching media as an inveterate inscriber in various electronic media. Audio writing has its relations with earlier work (Burroughs's and Gysin's cut up experiments with recorded sound and paper, for instance) and with 'theory'Whitehead's background, for instance, was initially in theater and literary theory.
My own literary background involves admiration for literary work of text, sound, and image by writers like Giordano Bruno, Apollinaire, Isidore Isou, Burroughs, and Whitehead. I wrote at some length elsewhere on literary electronic media.
Nio is toward such a synthesis of literacies.
DISEMBODIED, DISMEMBERED MEDIA AND THE MIND
We know that in media like film or TV or recorded sound, sound and image are cut from their source, are disembodied via the electronic inscription. And we develop over time a certain multi-perspectival view of the images, the sounds, and what's in them, thereby. For instance, when I worked in radio, I had occasion several times to record the voices of people for their first time. Often they would say 'It doesn't sound like me' when I played the recording of their voice back to them. Of course it sounded like them to me and to others, but not to themselves because they had never heard themselves when not speaking and had a whole different internal representation of how they sounded unsupported by a 'mirror' or 'picture' of themselves. Via recorded sound and pictures, we develop a certain self-consciousness or objectivity in which we act and see/hear/think in part apart from ourselves. Perhaps when this apartness or disembodied perspective is all we can experience of ourselves, we have a form of contemporary schizophrenia.
But we recognize the value of having both the ability to live in our skins, be, fully, completely, and yet also be capable of seeing ourselves and others and things from different perspectives.
Digital media multiply the disembodied, cut the image and the sound and the text from their source like non-digital media, but cut it into giga-slices and reassemble it in unexpectedly integral yet startlingly transformed ways. The morph, the packet, the pixel, the filter...things get atomized and can be dramatically or very subtly transfigured.
But we are no longer really interested in psychedelic transformation. Instead, our sense of transformation and metaphor has responded to the often wild digital transformations possible by demanding more meaningful transformation in which the elements of the 'original' and the 'transformed' are in highly meaningful relation that both illuminate the 'original' and yet also bring it into relation with something quite different. Metaphor not simply as a figure of speech but towards a more capacious understanding of metaphor and change, process.
So, in Nio and Vismu more generally, I'm trying to synthesize and transform image, sound, and text, not simply juxtapose them. I seek some sort of critical mass to fuse them.
As they are fused, really in our own mentation, if only we could coordinate
them and compose them in unity. But the divided nature of the media
have also kept them apart in the world and in our minds.
and did OK,
ALTERNATIVE MUSIC VIDEOYou can compose specially for vismu, or use 'regular' songs. You can take a single piece of music, cut it up, and see how it layers and sequences. My experience is that some parts of songs sequence nicely whereas other parts of the song layer nicely if the recording is not too dense with instrumentation or you have access to separate tracks.
You might want to record some special tracks for the piece, in some cases, so that it's both pleasurably layerable and sequenceable, and you might want to take some parts of the song out. You take out repetitions, in particular, so long as you can use sound icons more than once, which will be the case in later versions of the software.
The animations might key on vectorizations (or whatever) of the video done for the song or combine parts of the video with other stuff, or be totally original and separate from the video. And the animations themselves, which are Flash imports into Director, could be interactive; when clicked, they could cause changes in the audio or visuals or both or do something else like open a channel to other players or whatever. What you end up with isn't a reworking of the original material, if you do it well.
However, the form of Nio and related interactive audio pieces is sufficiently rich that you can see it would be quite satisfying to create music and visuals specially for the form itself, as I have done in Nio and some other pieces. I should add, though, that the work I've done in cutting up some 'regular' songs and putting them into vismu has peaked my interest in doing much more of it.
The new piece I'm working on involves about 60 sounds. It's a 'regular'
piece of music cut up into contiguous loops. Verse Two of Nio contains
a 4x4 grid. It is quite a nice looking grid, but it is a grid and one
would like to do away with the grid altogether and create something
as appealing as the animations and the circle in Verse One, yet offers
the ability to both layer and sequence however the player wishes. There
are juicy possibilities.
DOMINANCE OF THE VISUAL
We live in a visually dominant culture. People don't generally hear very carefully, and the auditory has little status as a medium for the transmission of knowledge. When we say 'I hear that...' one is usally asking for verification, whereas when we say 'I see,' it's an acknowledgment of full understanding. Most programming languages are set up so that the visual dominates and controls the audio. In Nio, when a new sound begins playing, it causes the animations to change, which is a case of the audio controlling the visual. Director is rich in ways to have the audio control the visual even though the most prominent paradigm of Director, like Flash, is that of the movie, though the 'timeline' is also called the 'score'.
Lingo is sufficiently powerful a programming language and attentive to audio that it's possible to subvert the dominance of the visual (see the Technotes).
There isn't a case in Nio where the visuals control the audio, ie, there are no situations such that when the movie gets to a certain frame, it plays a certain sound. The reason is because you cannot achieve musically precise synchronization that way: the frame rate of the visuals is not constant enough for musically precise synchronization. The frame rate changes depending on the number of animations currently playing; the file size of the animations (it takes different amounts of time to draw different sized graphics onto the monitor); and the amount of computing going on in the background of the computer both by Shockwave and by other programs running at the same time and the operating system processes. However, the rate at which sound is delivered to the speakers is generally constant, so timing has to cue on the soundstream and timers you program in Director (see the Technotes).
NIO, ART AND THE APP
Nio is part 'tool' and part heap of art. I'm not sure that drawing a hard and fast distinction between the two serves any purpose that furthers either the art or the tech. We're familiar with literary and visual and musical generative technology, technology that generates the art 'on its own' and also with technology that is used by artists in the creation of art, and also with technology that allows 'wreaders' or 'players' to manipulate pre-existing content in digital work. The latter seem to offer different ways to read or explore the content. Such is the nature of Nio.
There are only sixteen sounds in Nio. The compositional possibilities are, I hope, engaging, but they do not compare with the sorts of pieces one can imagine involving far more sounds. The next piece I'm working on contains about 60 sounds and offers more sophisticated compositional possibilities, and sound icons will be able to be used more than once.
In Nio, one gets the sense that one is exploring rather than creating because of the limited number of sounds. That needn't be the case in interactive audio. On the other hand, you don't want sufficiently many sounds that you're basically presenting people with a musical instrument that takes quite a while to learn how to play and compose on. Not yet, anyway, until there's familiarity with the form. Maybe as I make more of these things I'll come to a grand opus or something, though, who knows?
I do plan to develop both the 'tool' aspect of interactive audio and the 'art' part. Let us say that something that's strictly a 'tool' provides only the merest scraps of the 'content' made with the tool, whereas when we think of art, we generally think of the products of tools, not a tool itself.
Though this needn't be the case, obviously, and art itself is a tool in various ways, a tool for insight and enlightenment or entertainment, etc, that we use in a personally transformative way. We have also seen tools on the Web for generating texts or images, etc., that rightly lay claim to being works of art. This is partly why I don't see much use in drawing a hard and fast distinction between tools and art.
Also, one can imagine that artists who make tools that explore or create art do so with a sense that the tool itself unfolds rather like art.
Also, the application (I mean the computer program) is in some sense one of the fundamental compositions or, as it were, literary forms possible on a computer. It would be odd to draw some line between art and the application or between the tool and art. If anything, it's territory that needs close scrutiny by artists. Computer programs people use should expand their humanity, not turn them into robots. There are both types around, it seems.
It was the conceptual artist
Joseph Kosuth who asked "
what is more 'creative' than creating
a new idea of what art is?" Art tightly integrated with the tool
and the application has not exactly been explored deeply by many.
Medieval and Renaissance
PREVIOUS PIECES AND REFLECTIONS ON SURFING
As I mentioned, I've been doing interactive audio full-time for a year now. I'd like to show you some earlier work; it's kind of an interesting progression, I think. And along the way I've run across some great work by other people, some of which blows me away, that I'd like to mention.
The first piece I did was Rude Little Song. The idea of this piece was to create a little musical instrument in which you could make a tune from the available sounds, which are the first singing vocals I recorded of my own, though there is sound poetry at vispo.com/audio. I tried to synch the vis and the audio in this piece also, but hadn't thought carefully enough about how to do that, didn't understand the problems involved in doing that.
As an instrument, Rude Little Song is, well, 'rude' or 'simple'. It can engage one's sense of rhythm, though, as you mouseover the icons to create uninterrupted rhythm and perhaps your own rude little song. The tonics are in the middle and the sounds are grouped in pairs that go together. I think it's a more interesting instrument than it would be were it to play only notes. It would be even more rude, in that case, uninterestingly so, though it would clearly have more combinatorial complexity, in a sense. But the combinatorial complexity would be mired in mediocrity. Who wants to plink out notes on a bad instrument? If interactive audio on the Web is going to have any appeal as an instrument, you have to get away from single notes and move toward situations where the player plays phrases and can also build/explore compositions, where you can improvise and also 'write' in sequences and layers of sound, and then improvise some more over that. Additionally, one can play/build the visuals, whether this amounts to building a light painting or a visual narrative or poem or a rhythmically moving thing or whatever. Check out Steve Tanza's work at Stanza for wonderfully interactive visuals, and also Squid S O U P (which is further charming to me because of its lettristic, literary dimension). Mark Napier's visuals in p-soup are also very apropos and integral to the work.
The mouse is a very limited thing to play an instrument with. But obviously it's important to use it well as a composer of interactive audio for the Web, given the prominence of the mouse. Check out electrica for excellent use of the mouseamong electrica's many other strong charms.
Yet it will also be important to eventually move beyond the mouse into the use of the keyboard, also. That opens up a different set of possibilities in which shorter sounds could be more appropriate though, even then, providing individual notes might not be a good idea because you can't bend them as naturally and with varying degrees of force depending on how hard you hit the keyboard, I don't think, though clearly you can play on the duration of the keypress. On the other hand, you have a programmable computer at your disposal, so there will be things you can do that you can't do with a typical instrument. Exploring those possibilities energetically is important rather than trying to turn it into a regular instrument. Pianographique has done some pretty amazing Shockwave work with the keyboard that turns it into not just a piano but a player of samples and also a kind of video synthesizer.
Instrumental shovelware, on the other hand, will not succeed or interest many people because it's doomed to be inferior to the instrument it attempts to emulate. I've seen pianos and guitars and turntables etc. emulated online and they are quite boring, in my opinion, unfortunately, even when technically done very well.
One needs to discover the 'phenomenology' (or unique characteristics and possibilities...) of the media/um/tools rather than turn it into some other previously existing media/um/tool.
Previous to the interactive audio work, I made pieces such as the stir fry texts, sometimes in collaboration with other Web.artists such as Talan Memmott and Brian Lennon. I also corresponded with Joseph Weizenbaum (of Eliza fame) and the critic Jerome McGann to obtain permission to use their texts in the stir frys. They seemed suitably amused. The stir frys are a kind of textual analog of the interactive audio work. The stir frys involve interactive layers of text and sequences of text just as the interactive audio work involves layers of sound/animations and sequences of these. Further, the 'wreader' does not compose texts from scratch with the stir frys, but instead explores/composes with 'content' that is already within the piece. This is true of Nio and the other interactive audio work I've done so far, though it is easy to see that both could be developed as tools alone. But I am not drawn to making tools alone.
After Rude Little Song, Helen Thorington commissioned me to do something for Turbulence.org and actually paid me to work on my passion. This sent me over the deep end. From that point, I got progressively deeper into interactive audio to the point where I quit my job in Seattle to pursue it full-time as the potential became clear to me. I have been an artist for a long time, but it wasn't until the interactive audio work that I could see not only vast new territory artistically (vispo.com has explored that for some years) but also numerous business possibilities as a developer, a producer of media, and as a writer and musician. Helen and I first met back in the eighties when we were both doing radio; she edited and published an article in EAR I wrote back then on Gregory Whitehead's work, who produced many fine pieces for Helen's New American Radio series. Nio is the culmination of an eventful year of work and change in my life, and reconnection with my former life as a disembodied voice collaborating with Helen, Gregory, and other audio writers around the world in projects that, for me, were utterly formative in my approach to media and art.
The piece I did after Rude Little Song, Prototype, you will recognize if you've seen Nio; Prototype is much like Verse Two of Nio. However, they differ considerably not only by virtue of the animations in Nio; Verse Two of Nio is a much more responsive instrument, as you will note, is considerably more robust than Prototype (which periodically resets itself), and the synchronizations of Nio are more musically precise. And Nio does not suck as much juice as Prototype did in its audio processing, by far. Also, Prototype does not stream. It's a 500k download.
After I did Prototype, I wanted to investigate two issues further and separately: layering and sequencing of sounds. The next piece, Oppen Do Down, involves only layering of sounds, and is similar under the hood to Verse One of Nio. The essential algorithms of Oppen Do Down and Verse One of Nio are discussed in the Technotes.
There have been many other shorter or yet more incomplete experiments in Vismu over the year, some of which I will finish, but the above ones are the most interesting and complete, and you can see that they all relate to Nio concerning one feature or another. You see the way that they all point toward Nio, as I suspect Nio will point to future works, though there will be considerable branching.
POSTSCRIPT (June 2001): ENO AND THE "UNFINISHED"
I came across a 1995 interview by Kevin Kelly of Brian Eno that I'd like to quote. You'll certainly see the relevance of it to the above essay, which I wrote a few months ago (April 2001). Also, it turns out that one of the main audio engineers at Macromedia, Jonathan Powers, with whom I sometimes correspond, is primarily a musician and lectured with Eno years ago.
I recall reading this interview online in 96 or 97. But I wasn't doing much audio then, never mind music. In any case, you can see the relevance of the quote to Nio and to the works linked below. I might also note that there's an Animisms section at vispo.com and has been for a few years, has been the focus of my work for a few years. That includes quite a few textual works, mostly in DHTML, and they also are "unfinished" in Eno's sense.
I would say that "unfinished" works constitute a subset of "interactive" works. Though, because all works are unfinished in the sense that meaning is highly constructed by s/he who reads/experiences, and because all works are interactiveeven booksconfusion might arise unless I add that I'm referring to digital "interactive" works here, and referring to interactivity via the mouse and keyboard (or whatever other physical devices you like).
I think of "unfinished" interactive works as 'heaps of art' or, as Eno says, works that involve "pieces of raw material, but highly evolved raw material, that has a strong flavor to it already." Not all digital interactive works are this way. The emphasis in "unfinished" works is on composition of the heap though in other ways it is already highly composed. There's some sort of app that the artist has written, often, that permits derangements and rearrangements and perhaps filtering also of the 'content', but there is some content supplied.
A word processor presents us with a tool for creating something that is not only unfinished but barely started (though as Talan Memmott points out in the Webartery thread, the resulting work is vigorously framed already, in a certain sense). One can imagine art pieces all the way along the continuum of the unstarted to the unfinished.
One of the problems, then, with the term 'interactive' is that it refers to a large assortment of types of digital work. We are currently lacking useful language to distinguish between types of interactive digital (and non-digital) works. But the terminology of the 'unstarted' versus the 'unfinished' seems useful to me in talking about works that play between being tools and 'works'. This terminology allows us to make distinctions between types of interactive works, distinctions based on how much and what sort of content is part of the work itself (or not).
LINKS TO INTERACTIVE AUDIO WORKS/SITES