Source Code. Projects. Nerd Stuff. Art Stuff.

Drawing Machine Playtests


Posted on October 20th, by amoeboar in classes, pcomp. No Comments

I started playing around with the Minim library for Processing to see how we might be able to read sound data as input for our Drawing Machine. The Minim library is interesting, if a bit limited in how it will suit the needs of this project. Still, worth checking out for the future are the music programming and synthesis algorithm tutorials.

Following the tutorials, I created a simple sketch for taking microphone input and doing a FFT to separate out discrete components of the waveform. In this version of the program, I used BeatDetect to have Processing listen for “attack”, hoping to get a value for beats per minute. It works, but many beats are missed, especially when the “attack” isn’t so clear because a person is speaking too quickly or monotonically. You’ll see the waveform at the top, the beat detector as the green circle, and the FFT analysis at the bottom in red.

[gn_spoiler title="Sound Analyzer Source Code:" open="0" style="1"]
[gn_tabs style="3"]
[gn_tab title="soundAnalyzer2"]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
/*
 * A sound analyzer. Uses FFT algorithm to transform the signal in the time domain (sample buffer)
 * into a signal in the frequency domain (spectrum). Here, frequency and amplitude are returned.
 * BeatDetect analyzes an audio stream for beats (rhythmic onsets). This returns beats per minute.
 *
 * John Capogna, Brett Peterson, Maria Paula Saba
 * ITP, 2012
 */

import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioInput input;
FFT fft;
BeatDetect beat;

float radius;
int beginTimer = 0;

void setup() {
  size(500, 500, P3D);
  frameRate(60);
  minim = new Minim(this);
  input = minim.getLineIn(minim.STEREO, 512);
  // a beat detection object SOUND_ENERGY mode with a default sensitivity of 10 milliseconds
  beat = new BeatDetect();
  // set sensitivity
  //  beat.setSensitivity(0);
  fft = new FFT(input.bufferSize(), input.sampleRate());
}

void draw() {
  background(0);
  spectrum();
  beatsPerMinute();
}

void stop() {
  // always close Minim audio classes when you are finished with them
  input.close();
  // always stop Minim before exiting
  minim.stop();
  super.stop();
}
[/gn_tab] [gn_tab title="beatsPerMinute"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
void beatsPerMinute() {
  int beatCount = 0;
  // Analyzes samples in input buffer. Cumulative, so must be called every frame
  beat.detect(input.mix);
  ellipseMode(CENTER);
  float a = map(radius, 20, 80, 60, 255);
  fill(60, 255, 0, a);
  if ( beat.isOnset() ) {
    radius = 80;
    beatCount++;
//    println(beatCount);
  }
  ellipse(width/2, height/2, radius, radius);
  radius *= 0.95;
  if ( radius < 20 ) {
    radius = 20;
  }
  beginTimer++;
//  println(beginTimer);
}

//// Start and Stop function
//void keyPressed() {  
//  if (key == CODED) {   //test required to pick up special keys, like arrows
//     switch(keyCode) {
//       case (LEFT):
//         beginTimer++;
//       case (RIGHT):
//         break;
//     }
//  }
//  println(beginTimer);
//}
[/gn_tab] [gn_tab title="spectrum"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
void spectrum() {
  fft.forward(input.mix);
  // line
  stroke(255, 0, 0, 128);
  // draw the spectrum as a series of vertical lines
  // I multiple the value of getBand by 4
  // so that we can see the lines better
  for (int i = 0; i < fft.specSize(); i++) {
    line(i, height, i, height - fft.getBand(i)*4);

    // SO THIS IS AMPLITUDE OF EACH FREQUENCY BAND
    // COULD USE MAX VALUE AS ONE OF OUR VARIABLES
//    println(fft.getBand(i));
  }

  stroke(255);
  // I draw the waveform by connecting
  // neighbor values with a line. I multiply
  // each of the values by 50
  // because the values in the buffers are normalized
  // this means that they have values between -1 and 1.
  // If we don't scale them up our waveform
  // will look more or less like a straight line.
  float[] freqValues = input.left.toArray();
  for (int i = 0; i < input.left.size() - 1; i++) {
    line(i, 50 + freqValues[i]*50, i+1, 50 + freqValues[i+1]*50);
    //line(i, 150 + input.right.get(i)*50, i+1, 150 + input.right.get(i+1)*50);
  }
}
[/gn_tab] [/gn_tabs] [/gn_spoiler]

 

We decided that perhaps the beat detection wasn’t working in the way we wanted it to, so Brett modified the code so that it takes a recording, analyzes the waveform during playback, and returns a value for the clip length as an alternate variable. We then took our new sound analyzer and mashed it with the drawing code we had.

Here are some examples of the sketch’s output after analyzing the sound data:

[gn_spoiler title="Sound Analyzer/Drawing Source Code:" open="0" style="1"]
[gn_tabs style="3"]
[gn_tab title="soundAnalyzer7"]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
/*
 * A sound analyzer. Uses FFT algorithm to transform the signal in the time domain (sample buffer)
 * into a signal in the frequency domain (spectrum). Here, frequency and amplitude are returned.
 * BeatDetect analyzes an audio stream for beats (rhythmic onsets). This returns beats per minute.
 *
 * John Capogna, Brett Peterson, Maria Paula Saba
 * ITP, 2012
 */

import ddf.minim.*;
import ddf.minim.analysis.*;

// setup of angle, radius, position, and changing variables
float angle1, position1, r1, x1, y1;
float speed1;
float direction1;
int a;

float startMillis = 0; // current time at playing


//arrays to save the x and y position of the line to be drawn
float x[] = new float [0];
float y[] = new float [0];

Boolean drawing = false;
Boolean donePlaying = false;

Minim minim;
AudioInput input;
FFT fft;
BeatDetect beat;
Boolean recording = false;
Boolean live = true; // Used to indicate whether or not we'll call spectrum() during the draw
AudioRecorder rec; // To record from mic
AudioPlayer recorded; // The recorded file
AudioMetaData metaData; // MetaData class for recorded audio

float radius;
int beginTimer = 0;
float maxAmp = 0; // Maximum amplitude seen this sketch (Volume)
int maxPitch = 0; // Frequency band of the loudest frequency (~ Pitch)
int clipLen = 0;

PFont font;

void setup() {
  size(900, 800);
  smooth();
  frameRate(60);

  //interesting settings changes:
  // speed1 (0-10), direction1 (0-10)
  speed1 =  0;
  direction1 = 0;
  position1 = 0;
  angle1 = 0;
  r1 = 0;
  a = 1;

  minim = new Minim(this);

  // Font stuff
  font = loadFont("Edmondsans-Bold-48.vlw");
  textFont(font, 20);


  // Record from internal mic
  input = minim.getLineIn(minim.STEREO, 512);

  // Not using, for now
  // a beat detection object SOUND_ENERGY mode with a default sensitivity of 10 milliseconds
  // beat = new BeatDetect();
  // set sensitivity
  //  beat.setSensitivity(0);

  // perform an FFT on the input that was previously saved.
  fft = new FFT(input.bufferSize(), input.sampleRate());

  // load the recorded audio clip. Placed here to avoid null pointer in draw
  recorded = minim.loadFile("test.wav", 512);
}

void draw() {
  background(0);


  if (!drawing) {
    // If the recorded clip isn't currently playing, show a live view of the spectrum
    if (!recorded.isPlaying()) {
      spectrum(input.mix);
      //beatsPerMinute();
    }
    // If the recorded clip is playing, show the live view of the spectrum and analyze
    // It's done this way because the recorded clip must be playing when the fft is run.
    else {

      // Get the values of the freq bands
      float[] bands = spectrum(recorded.mix);

      // Run the analysis on the bands
      analyze(bands);

      // Print out the max values for pitch and volume, and the length of the clip
      println("loudest pitch: "+maxPitch+" volume of pitch: "+maxAmp+" Length of clip: "+clipLen);
    }
    textAlign(CENTER);

    //Display text and symbol for whether or not recording
    if (recording) {
      fill(100);
      rectMode(CORNER);
      rect(width/2-20, height/2-20, 40, 40);
      text("Recording...Click to stop", width/2, height/2-30);
    }
    else if (donePlaying) {
      fill(255, 0, 0);
      ellipse(width/2, height/2, 40, 40);
      text("Analyzing sound", width/2, height/2-30);
    }
    else {
      fill(255, 0, 0);
      ellipse(width/2, height/2, 40, 40);
      text("Click to record", width/2, height/2-30);
    }
  }

  if (donePlaying && millis() - clipLen > startMillis) {
     if (position1 >=  r1 || position1 < -r1)  a = - a;
     r1 = int(map(maxAmp, 5, 150, 0, 200));
    //r2 = int(map(maxPitch, 1, 10, 0, 100));
    //speed1 = int(map(maxAmp, 5, 150, 1, 5));
    speed1 = int(map(clipLen, 0, 8000, 0, 10));
    direction1 = int(map(maxPitch, 1, 10,  1, 5))*a;
   

    println(position1+","+r1+","+direction1);


   
    drawArt();
    drawing = true;
   
    textAlign(CORNER);
    textFont(font, 16);
    text("max. volume - size: "+r1, 50, 50);
    // text("max. volume - 1st arm speed: "+speed1,50, 70);
    text("max pitch - arm speed:  "+direction1, 330, 50);
    //text("max pitch - 2nd arm speed: "+speed2,330, 70);
    text("duration - rotation speed: "+speed1, 600, 50);
  }
}





void keyPressed() {

  if (key ==' ') {
    drawing = false;
    donePlaying = false;
  }

  if (key == 'd' ) {
    //empty arrays  
    x = new float [0];
    y = new float [0];
  }
}


// Click the mouse to start and stop recording
void mouseClicked() {
  if (!recording) {
    // Create a new audio recorder
    rec = minim.createRecorder(input, "test.wav", true);
    rec.beginRecord();     // Start recording
  }
  else {
    rec.endRecord();    // Stop recording
    rec.save();    // Save recording

    // Load the recorded file into memory
    recorded = minim.loadFile("test.wav", 512);

    // Zero out the maxAmp and maxPitch for the next recording
    maxAmp = 0;
    maxPitch = 0;

    // Get the clip length
    clipLen = recorded.getMetaData().length();

    // Playback the recorded file. The clip must be playing when the FFT is run
    recorded.play();

    // Start playing timer
    startMillis = millis();

    // perform a new FFT on the input that was previously saved.
    fft = new FFT(recorded.bufferSize(), recorded.sampleRate());

    // Toggle Done Playing switch
    donePlaying = true;
  }
  recording = !recording; // Toggle state
}

void stop() {
  // always close Minim audio classes when you are finished with them
  input.close();
  // always stop Minim before exiting
  minim.stop();
  super.stop();
}
[/gn_tab] [gn_tab title="beatsPerMinute"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
void beatsPerMinute() {
  int beatCount = 0;
  // Analyzes samples in input buffer. Cumulative, so must be called every frame
  beat.detect(input.mix);
  ellipseMode(CENTER);
  float a = map(radius, 20, 80, 60, 255);
  fill(60, 255, 0, a);
  if ( beat.isOnset() ) {
    radius = 80;
    beatCount++;
//    println(beatCount);
  }
  ellipse(width/2, height/2, radius, radius);
  radius *= 0.95;
  if ( radius < 20 ) {
    radius = 20;
  }
  beginTimer++;
//  println(beginTimer);
}

//// Start and Stop function
//void keyPressed() {  
//  if (key == CODED) {   //test required to pick up special keys, like arrows
//     switch(keyCode) {
//       case (LEFT):
//         beginTimer++;
//       case (RIGHT):
//         break;
//     }
//  }
//  println(beginTimer);
//}
[/gn_tab] [gn_tab title="drawArt"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
void drawArt() {
  translate(width/2, height/2);

  //drawing the apparatus
  pushMatrix();
  noFill();
  stroke(0);
  strokeWeight(1);
  //center
  ellipse(0, 0, 30, 30);

  //arm1
  rotate(radians (angle1));
  stroke(255);
  line(-200, 0, 200, 0);

  //arm2

  translate(position1, 0);
 // rotate(radians (angle2-45));
  rectMode(CENTER);
  stroke(255);
  rect(0,0,10,10);
  popMatrix();


//drawing the line
  //adding each point to the array
  x1 = cos(radians(angle1))*position1;
  y1 = sin(radians(angle1))*position1;

  x = append (x, x1);
  y = append (y, y1);


  //drawing a line between point and last point
  stroke(255, 0, 0);
  strokeWeight(2);

  for (int i = 1; i < x.length; i++) {
    line (x[i], y[i], x[i-1], y[i-1]);
  }


  //arm rotation
  angle1 = angle1 + speed1;
  if (angle1 == 360 || angle1 == -360) {
    angle1 = 0;
  }  
 
  position1 = position1 + direction1;
 
  translate(-width/2, -height/2);
}
[/gn_tab] [gn_tab title="spectrum"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
float[] spectrum(AudioBuffer buffer) {

  // FFT on the given mix
  fft.forward(buffer);

  // Array to hold amplitude for all freq bands
  float[] bands = new float[fft.specSize()];

  // loop over the bands and draw to make an equalizer
  for (int i = 0; i < fft.specSize(); i++) {
    fill(255);
    stroke(0);
    rect(i*10, height - fft.getBand(i)*10, 10, fft.getBand(i)*10 );

    // Put the amplitudes of all the bands into an array
    bands[i] = fft.getBand(i);
  }
  // return the array of band values to be analyzed
  return bands;
}


// analyze band values and check against the current max values
void analyze(float[] bands) {
  // Grab the amplitude of the loudest band - (Volume)
  float currMax = max(bands);       // Current loudest band this time around

  // If the current max is larger than the running max,
  // set the new max, also note which freq band is loudest (index)
  if (currMax > maxAmp) {
    maxAmp = currMax;
    // Grab the position in the array of the max amplitude - this is the freq band that is the loudest (Pitch)
    for (int i = 0; i < bands.length; i++) {
      if (bands[i] == currMax) {
        maxPitch = i;
      }
    }
  }
}
[/gn_tab] [/gn_tabs] [/gn_spoiler]

 

We did some user tests to get some feedback:

[gn_media url="http://youtu.be/sr8HsQP7lkw " width="600" height="400"]

 

Our testers gave us good feedback. One of the things we took away from it was that there was too much of a lag between when the person made their sounds and when the machine generated a drawing. This is because the sketch has to playback the recording to analyze it and return a clip length. We will want to address this. Also, people mentioned that it wasn’t completely obvious how their actions were affecting the drawings. We knew this would be a challenge, and the remedy I can think of is to map the variables more discerningly, for instance, frequency to speed (the higher the pitch, the faster the movement). Lastly, One person also mentioned that it didn’t matter how intentionally “noisy” their sounds were, the sketch always produced a pretty drawing. I think we should work more on implementing a real time “live draw” version of this project, but there is still time for that.

Our next step will be to make a physical device to match our software. We’ve started thinking about how we’re going to do that. First prototype coming soon…





Leave a Reply