Vector Sound Manipulation in Mathematica

An introduction to the new Audio`Music` package

Michael Pilat

Software Technology
Wolfram Research, Inc.

Initialization

Needs["Audio`Music`"]

PlayNote[sn_SoundNote]:=PlayMusic[SoundTrack[{sn}]]

PlotMusicHashFunction[MIDIEvent[t_,_,opts___]]:={"Channel","Note",t}/.{opts}

rectanglize[{{x1_,y1_},{x2_,y2_}}]:=
(
If[x2>XMAX,XMAX=x2];
If[y2<YMIN, YMIN=y2];
If[y2+1>YMAX,YMAX=y2+1];
Rectangle[{x1,y1},{x2,y2+1}]
)

PlotMusic[st_SoundTrack]:=
Block[{XMAX=0,YMIN=127,YMAX=0},
Module[{midi,events,ons,offs,hash,tmp,tpb},
midi=Audio`Music`Private`NormalizeSoundTracksToMIDI[{st},False,None];
tpb="Resolution"/.List@@Rest[midi];
events=Flatten[First/@First[midi]]/.
MIDIEvent[t_,"NoteOn",aaa___,"Velocity"->0,bbb___]:>MIDIEvent[t,"NoteOff",aaa,"Velocity"->0,bbb];
ons=Cases[events,MIDIEvent[t_,"NoteOn",___,"Note"->n_,___,"Channel"->c_,___],1];
offs=Cases[events,MIDIEvent[t_,"NoteOff",___,"Note"->n_,___,"Channel"->c_,___],1];
tmp=PlotMusicHashFunction[#]&/@ons;
tmp=tmp[[Ordering[tmp]]];
ons=Map[Reverse[Rest[#]]&,Split[tmp,(First[#]==First[#2])&],{2}];
tmp=PlotMusicHashFunction[#]&/@offs;
tmp=tmp[[Ordering[tmp]]];
offs=Map[Reverse[Rest[#]]&,Split[tmp,(First[#]==First[#2])&],{2}];
tmp=Riffle[Map[rectanglize,Transpose/@Transpose@{ons,offs},{2}],
{Blue,Red,Green,Purple,Yellow,Brown}];
Button[
Graphics[{Thickness[1/Min[600,50XMAX]],
RGBColor[.85,.85,.85],EdgeForm[{Black,Thickness[2/Min[600,50XMAX]]}],
Line[Table[{{x,YMIN-.5},{x,YMAX+.5}},{x,0,XMAX,tpb}]],
Orange,tmp},PlotRange->{{-1,XMAX+1},{YMIN-1,YMAX+1}},ImageSize->{Max[150,Min[600,50XMAX]],150},AspectRatio->150/(Max[150,Min[600,50XMAX]]),Background->White],
PlayMusic[st]
]
]
]

NoteButton[note_SoundNote]:=
Button[Style[ToString[note,InputForm],FontSize->18,FontColor->RGBColor[0,.1,.5]],PlayNote[note],Background->White]

NoteButton[SoundNote[60]]

What is "Vector Audio"?

Analogue: Vector Graphics vs. Raster Graphics

gr=Graphics[{Red,Disk[{0,0},.5],Green,Rectangle[{0,0},{1,-1}]},ImageSize->100]

ras=Rasterize[gr,ImageSize->100]

What if I want to resize the image?

Raster Version:

Graphics[First@ras,ImageSize->600]

Vector Version:

Graphics[{Red,Disk[{0,0},.5],Green,Rectangle[{0,0},{1,-1}]},ImageSize->600]

Size of the representation

ByteCount[gr]

ByteCount[ras]

"Vector" Audio

SampledSoundList[] vs

Two dimensions of "vectorness":

Time    (how fast playback occurs)

Frequency / Pitch    (transposition, etc.)

Rendering with MIDI

Musical Instrument Digital Interface

MIDI Music Association -- 1985

Events

Note On

Note Off

Controller Change ( Foot Pedal, etc. )

Meta Events

Notes

Equal-tempered tuning in semitone intervals

Notes are numbered from 0-127

Note 60 - C4 / 261.6 Hz / "Middle C"

Note 69 - A4 / 440 Hz

Specifying Notes

SoundNote

A SoundNote object specifies up to three things:

■ The note to be played
■ The duration or start and end time
■ Velocity

Single Notes

Specify a note by (MIDI) note number:

SoundNote[60]//PlayNote

Specify a note by name:

SoundNote["ASharp4"]//PlayNote

SoundNote["G♭3"]//PlayNote

Chords

Two or more notes with the same duration, played simultaneously

SoundNote[{"C4","E4","G4"}]//PlayNote

Duration

The second parameter to SoundNote controls duration:

SoundNote["C4",1/2]//PlayNote

SoundNote["C4",3]//PlayNote

Velocity

The third parameter specifies velocity:

SoundNote["C4",1,1/4]//PlayNote

SoundNote["C4",1,1/2]//PlayNote

SoundNote["C4",1,1]//PlayNote

You can specify both an attack and decay velocity:

SoundNote["C4",1,{1,.5}]//PlayNote

Why "Velocity" and "Volume"?

Velocity doesn't always imply volume, and even when it does, it doesn't necessarily apply only volume.

For example, plucking a guitar string very hard can bend the pitch (since the string stretches) and cause the string to hammer against the frets.

Sequences of Notes

SoundTrack

A Soundtrack represents a sequence of notes:

SoundTrack[{SoundNote["A4"],SoundNote["B4"],SoundNote["G4"],
SoundNote["G3"],SoundNote["D4"]}]//PlotMusic

Note that the duration of the sound track is defined by the duration of its elements!

Specifying Duration

Of course, notes can have different durations:

SoundTrack[{SoundNote["A4",1/2],SoundNote["B4",1/2],SoundNote["G4",1/2],SoundNote["G3"],SoundNote["D4"]}]//PlotMusic

Just like a SoundNote, the second parameter of SoundTrack can specify a duration:

SoundTrack[{SoundNote["A4",1/2],SoundNote["B4",1/2],SoundNote["G4",1/2],SoundNote["G3"],SoundNote["D4"]},1]//PlotMusic

Note that the contents are automatically rescaled to fit the requested duration!

SoundTrack[{SoundNote["A4",1/2],SoundNote["B4",1/2],SoundNote["G4",1/2],SoundNote["G3"],SoundNote["D4"]},7]//PlotMusic

Nesting SoundTracks

As perhaps one of the most powerful features, SoundTracks can be nested:

st=SoundTrack[{SoundNote["A4",1/2],SoundNote["B4",1/2],SoundNote["G4",1/2],SoundNote["G3"],SoundNote["D4"]}];

SoundTrack[{st,st}]//PlotMusic

PlayMusic

The helper functions PlayNote and PlotMusic above internally use PlayMusic, which is how to actually play sounds:

PlayMusic[SoundTrack[{SoundNote["C3"],SoundNote["G3"],SoundNote["C4"]}]]

SoundNotes, Continued...

Specifying explicit times

SoundNotes can have a specific start and end time, instead of duration:

SoundTrack[{SoundNote["C4",{1,2}]}]//PlotMusic

Combining different kinds of SoundNotes

SoundTrack[{SoundNote["C4"],SoundNote["E4",{.75,2}],SoundNote["G4"]}]//PlotMusic

Instruments

Specifying Instruments

SoundTracks can also specify which instrument should be used to render the notes:

SoundTrack[{SoundNote["C4",1/2],SoundNote["E4",1/2],SoundNote["G4",1/2],SoundNote["C5",1/2]},
"Instrument"->"Harpsichord"]//PlotMusic

MIDI Instruments

Instruments are named according to the General MIDI 2 (GM2) specification

Grid[Partition[Button[InstrumentNumberToName[#],
PlayMusic[SoundTrack[{SoundNote[60]},"Instrument"->#]]]&/@Range[0,127],4]]

Option Inheritance

Options[SoundTrack,"Instrument"]

Nested SoundTracks inherit their options from the parent container:

st=SoundTrack[{SoundNote["C5",1/2],SoundNote["E5",1/2],SoundNote["G5",1/2],SoundNote["C6",1/2]}];
SoundTrack[{SoundNote["C6"],st,SoundNote["C5"]},"Instrument"->"Vibraphone"]//PlotMusic

...Unless of course you explicitly specify something different:

st=SoundTrack[{SoundNote["C5",1/2],SoundNote["E5",1/2],SoundNote["G5",1/2],SoundNote["C6",1/2]},"Instrument"->"Electric Piano 1"];
SoundTrack[{SoundNote["C6"],st,SoundNote["C5"]},"Instrument"->"Vibraphone"]//PlotMusic

And if you don't specify anything, instrument 0 (Acoustic Grand Piano) is used.

SoundTracks, Continued...

(st=SoundTrack[SoundNote[#,1/2]&/@{"C4","E4","G4","C5"}])//PlotMusic

Explicit Start Time:

The second parameter to SoundTrack can be used to specify a start time instead of a duration:

SoundTrack[{st,SoundTrack[st,{1},"Instrument"->"Violin"]}]//PlotMusic

Note the duration of the embedded SoundTrack is unchanged.

Like SoundNote, this does not affect subsequent duration events:

SoundTrack[{st,SoundTrack[st,{1},"Instrument"->"Violin"],st}]//PlotMusic

Explicit Start and End Time:

The second parameter can also specify both a start time and end time:

SoundTrack[{st,SoundTrack[st,{1,4},"Instrument"->"Cello"]}]//PlotMusic

SoundTrack Options and Effects

Tempo

notes={"C4","E4","F4","G4","A4","G4","F4","E4","D4","F4","G4","A4","B4","A4","G4","F4"};
SoundTrack[SoundNote[#,1/8]&/@notes]//PlotMusic

SoundTrack[SoundNote[#,1/8]&/@notes,"Tempo"->40]//PlotMusic

SoundTrack[SoundNote[#,1/8]&/@notes,"Tempo"->400]//PlotMusic

Other Effects

Reverb

Modulation

Breath

Foot Pedal

Balance

Panning

Expression

Chorus

Sustain

Portamento

Sostenuto

Soft

Legato

Hold

Modulation

Percussion

Special Instrument — "Percussion"

You can specify "notes" in the normal syntax, but special percussion names can also be used:

SoundTrack[{SoundNote[60]},"Instrument"->"Percussion"]//PlayMusic

SoundTrack[{SoundNote["Hi Bongo"]},"Instrument"->"Percussion"]//PlayMusic

Grid[Partition[Button[#,
PlayMusic[SoundTrack[{SoundNote[#]},"Instrument"->"Percussion"]]]&/@
(Last/@Audio`Music`Private`PercussionNames),3]]

SoundTrack[{
SoundTrack[Table[SoundNote["Ride Cymbal 1",1/2],{8}],{0}],
SoundTrack[SoundNote["Acoustic Bass Drum",#]&/@{{0,1},{2,2.5},{2.5,3}},{0}],
SoundTrack[SoundNote["Acoustic Snare",#,.75]&/@{{1,2},{3,4}},{0}],
SoundTrack[Table[SoundNote["Cowbell",1,.6],{4}],{0}]
},
"Instrument"->"Percussion","Tempo"->120]//PlotMusic

Examples

Listening To Numbers

SoundTrack[SoundNote[60+2#,1/(#+1)]&/@(First[RealDigits[N[GoldenRatio,60]]]),{0},"Tempo"->60,"Instrument"->"Vibraphone"]//PlotMusic

Direct MIDI Representation

Audio`Music`Private`NormalizeSoundTrackToMIDI[SoundTrack[SoundNote/@Range[60,72]]]

Recording SoundTracks

RecordMusic

snd=RecordMusic[st=SoundTrack[SoundNote/@Range[60,72],"Tempo"->160]];

EmitSound[snd]

Head[First[snd]]

ByteCount[snd]/ByteCount[st]//N

Clear[snd];

Future Features and Directions

Rendering Options

Better synthesizers

SuperCollider / CSOUND

Support any frequency instead of only MIDI notes

Custom Instruments

Model your own in Mathematica

SoundFonts, etc.

Unified Sound Language

Mix with SampledSound

Schedule vector events with "raster" sound:

SoundTrack[{SoundNote["C4"],SoundEvent[SampledSoundList[...]]}]

Other Features:

MIDI Import to SoundTrack/SoundNote (coming soon!)

Support for more MIDI features

Pitch bend

Metadata

???

Thank You!

Questions?

Audio`Music` is available in recent Mathematica 6.0 Alpha releases  and works on all major platforms!

(Better documentation is forthcoming)