I've been seeing lots of simple interactive programs popping up all over the internet that generate basic music, usually quite nice sounding, from simple interactions from the user. Otomata is one recent popular example; on the iDevices there are a variety of programs, like Bloom.
I'm just wondering how these programs work. How do they generate the music live? Are there established APIs for doing this? I'm interested in the technical aspects of how this is implemented. My knowledge is so lacking here I'm honestly not sure how to better phrase the question. Ask if anything isn't clear!
Answer
Your question can be broken down into 2 components:
1. How do I generate musical sounds and timbres, and then play that through the speaker?
This is even a 2-parter:
1.1. How do I generate any kind of sound and play it back through the speaker?
Audio data is frequently represented and processed in 32 bit floating point numbers ranging from -1.0 to 1.0, representing. Open up an audio file in Audacity and zoom in to the sample level to see individual pieces of data. When you write data either to a sound file or to an audio buffer that goes out to the sound card, you typically map these -1.0 to 1.0 values to a fixed scale (no fractional values) of 0 to 65536 (for 16 bit depth) or 0 to 16777216 (for 24 bit depth).
Once you have your buffer constructed, you feed it to your sound device. This is library and platform-specific.
Flash 10 added support for dynamically generating sound via the SampleDataEvent of the Sound class. You create a Sound object, but don't load any data into it. When you call Sound.play(), it will fire the SampleDataEvent, and the data member of SampleDataEvent will be a bytearray you can feed samples into. Read the docs for details.
XNA recently gave you the option of using DynamicSoundEffectInstance, which is similar to Flash's sampleDataEvent.
iOS seems to support AudioUnits, which are a standard way of generating sound in Cocoa (disclaimer: haven't used them myself). You could use this in a Mac application as well. (example: http://cocoawithlove.com/2010/10/ios-tone-generator-introduction-to.html)
1.2 How do I make my generated sound musical?
For a sound wave to have a sense of pitch, it must be generally periodic (obviously sound waves from instruments are not perfectly periodic, but we're talking broad strokes here). Any periodic wave can be constructed by adding together multiple waves of different frequencies and phases.
An spectrum analysis will show that the frequency with the greatest amplitude will generally be the pitch that the sound is perceived as. See this figure. To add depth, you also add waves with frequencies that are integer multiples of the base frequency (see the overtone or harmonic series).
2. How do I use my new robot zombie sounds to generate music on the fly?
This is a really open ended question, and is half of the fun of computer music. You can start somewhere as basic as selecting notes at random from a predefined scale, and then from there implement random rhythm variation as another step.
People do everything from selecting portions of music from a database of pre-written compositions (there's a really really thick book on a project to get a program to write 2-part Inventions in the style of Bach), to applying genetic algorithms to an initial interval vector and apply the results to different starting pitches.
Eventually you'll want to think about how to create larger movements of smaller, randomly generated musical components. As you start building up, this is where you can begin to insert game hooks. Maybe the angle of the mobile device can alter the range of notes, and when bad guys arrive you change the set of pitches in the scale to closer resemble a minor rather than a major scale, or adjust other properties such as attack envelopes and frequency of note events.
Timing can be tricky, because you don't want to be polling the elapsed time of your audio source, but you also want to be accurate. Relying on "sound complete" type of events will generally be much too slow. In Flash you can get away with musical timing for long sound cues if a 50ms delay is acceptable and it wouldn't be too obvious if you faded out one sound then started it again on another channel in sync with everything else. The catch is you need to check Channel.position on every frame update, unless you're manipulating your own sample data, in which case you're crunching a lot of data anyway just copying stuff to the SampleDataEvent buffer. Performance aside, just use basic ratios to figure out beat and tempo to millisecond interval conversions.
There's a whole wide world of audio programming fun out there
A book that I haven't managed to read much of yet (damn school) is The Audio Programming Book by Richard Boulanger and Victor Lazzarini. The first 60-70 pages are a basic C primer wrapped up in music-specfic examples and may get a little boring, but they get to the good stuff later on like oscillators and filters.
There's a lot of popular visual tools, libraries, and mini programming languages for creating or prototyping computer music:
The newfangled "creative computing" frameworks like Processing, OpenFrameworks, and Cinder all have some basic audio capability as well, but nowhere near what you get with these dedicated tools. A classmate of mine models HRTFs in SuperCollider to simulate environments in a basic 3d environment. Another fun thing is to pipe OSC out of Unity and let it control a MaxMSP patch.
No comments:
Post a Comment