Drawable stompbox

Drawable stompbox offers a more interesting and interactive way for guitarist to explore the variety of the parameters of the guitar effect world. With this instrument, people can select the guitar effect they want, then use finger to draw the parameters of your effect. Just like the diagram of time-domain and frequency-domain, this instrument can map what you’ve drawn to specific a set of number representing the amplitude and frequency information which will change the pre-written guitar effect’s parameter. You will get a lot of fun when you are trying to figure out the relationship between your drawing and the sound you heard.

Screenshot and video demo

Here is the screenshot of the Drawable Stompbox running in my iPad:

When you are drawing on iPad, you can not see the lines or patterns. The reason I made the canvas always blank is letting people hear what they have drawn instead of seeing what they have drawn.

Here is a video demo:

drawable stompbox final video from Haochuan Liu on Vimeo.

Previous version

There had been a big change of my project. At the beginning, drawable stompbox was just like a selector for guitar effects: After people wrote down the effects on a piece of paper, the webcam which was above the paper would capture what you had written into a software written in openframeworks. The software would analyze the words and do the recognition using optical character recognition (OCR). When you wrote the right words, the software will tell puredata to turn on the specific effect through OSC, you would finally hear what you’ve written when you play your guitar.

Technical Details

Here is the diagram of Drawable Stompbox:

Buttons and coordinates

I use very simple functions in OpenFrameworks to draw the buttons and get the x/y coordinates when moving fingers on the screen of iPad.

Mapping

The blue coordinates which is invisible in the real software represents amplitude (x coordinate) and time (y coordinate) information.

When you draw something on the canvas, the peak will determine the volume of the sound you will hear. The length of your drawing will determine the frequency parameter.

Communication

The software in iPad uses OSC to communicate with PureData running on the laptop. Thus, PureData can always know which effect is selected and also the values of amplitude and frequency.

Future Work

Currently when you play the guitar with using the Drawable Stompbox you still need a partner to help you draw something on the canvas of iPad to get the parameter change of the effect. It is right now just a prototype or a toy for people to practice instead of performance. The improvement of this project can be changing using people’s finger to using people’s foot. Thus, you can play the guitar and use your foot to draw the effect parameter at the same time.

Drawable Stompbox

Write down your one of your favorite guitar effects on a piece of paper, then play your guitar, you will get the sound what you’ve written down.

Here is the final diagram of this drawable stompbox:

After you write down the effects on a piece of paper, the webcam which is above the paper will capture what you’ve written into a software written in openframeworks. The software will analysis the words and do the recognition using optical character recognition (OCR). When your write the right words, the software will tell puredata to turn on the specific effect through OSC, you will finally hear what you’ve written when you play your guitar.

The source code of this software can be found here.

Here is a demo of how this drawable stompbox works.

Feedback from my final presentation:

I have got a lot of good idea and advice for my drawable stompbox as below:

1. Currently writing down a word to get the effects has no relationship with ‘drawing’. It is more like a effect selection using word recognition.

2. I was thinking of drawing simple face on the paper instead of just boring words. How about using a webcam directly to scan real people’s face, getting their emotion on their face and then find the relationship between different faces and different effects.

3. Words recognition is so hard, for there are a lot of factors to make it doesn’t work well, such as the hand-writing, the resolution of the webcam and the light of environment.

Following work:

For the following weeks, I decide to make my instrument a real drawable stompbox. I will begin with a very simple modulation:

People can simply draw the ‘wave’ like this:

From this drawing, it is easy to define and map the amplitude and the frequency.

Then I will use the ‘wave’ from the drawing to do a modulation with the original guitar signal. People can draw different type of waves to try how the sound changes.

In my milestone 3, I’ve reorganized and optimized  all the parts of my previous milestone including optical character recognition in openframeworks, communication using OSC between openframeworks and puredata, and all of the puredata effect patches for guitar.

Here is the screenshot of my drawable interface right now:

Here is the reorganized patch in puredata:

Also, I’ve applied the Levenshtein distance algorithm to improve the accuracy of the optical character recognition. For a number of tests made with this algorithm, the recognition accuracy can reach about 93%.

I am still thinking of what can I do with my drawable stompbox. For the begining, I was thinking this instrument could be a good way for people to play guitar and explore the variety of different kind of effects. I believed that using just a pen to write down the effects you want might be more interesting and interactive instead of using real stombox, or even virtual stompbox in computer. But now, I have realized that there is no way for people to use this instrument instead of a very simple controller such as a foot pedal. Also, currently just writing the words to get the effects is definitely not a drawable stompbox.

For my milestone 2, I have did a lot of experiments of audio effects in puredata. Besides very simple and common effects (gain, tremolo, distortion, delay, wah-wah) I made in milestone 1, here are the tests and demos I made for these new effects with my guitar.

Test 1: Jazz lead guitar

• Original audio

• Bass Synth

• Falling Star

• Phaser

• Reverb

• Ring Modulation

• Slow Vibrato

• Magic Delay

• Violin

• Vocoder

Test 2: Acoustic guitar

• Original audio

• Bass Synth

• Falling Star

• Phaser

• Reverb

• Ring Modulation

• Slow Vibrato

• Magic Delay

• Violin

• Vocoder

Test 3: Guitar single notes

• Original audio

• Bass Synth

• Falling Star

• Magic Delay

• Vocoder

Plan for milestone 1

The plan for milestone 1 is to build the basic system of the drawable stompbox. First, letting the webcam to capture the image of what you have drawn on paper, then the computer can recognize the words on the image. After making a pattern comparison, the openframwork will send a message to puredata via OSC. Puredata will find the right effect which is pre-written in it and add it to your audio/live input.

Here is the technical details of this system:

After writing words on a white paper, the webcam will take a picture of it and then store this photo to cache. The program in OpenFrameworks will load the photo, and turn the word on the photo to a string and store it in the program using optical character recognition via Tesseract. The string will be compared to the Pattern library to see if the stompbox you draw is in the library. If it is, then the program will send a message to let PureData enable this effect via OSC addon. You can use both audio files or live input in puredata, and puredata will add the effect what you have drawn to your sound.

Here are some OCR tests in OF:

About the pattern library on OF:

After a number of test for OCR, the accuracy is pretty high but not perfect. Thus a pattern library is needed to do a better recognition. For example, Tesseract always can not distinguish “i” and “l”, “t” and “f” , and “t” and “l”. So the library will determine what you’ve drawn is “Distortion” when the result after recognition is “Dlstrotlon”, “Disforfion” or “Dislorlion”.

Effects in PureData:

Until now, I’ve made five simple effects in PureData, which are Boost, Tremolo, Delay, Wah, and Distortion.

Here is a video demo of what I’ve done for this project:

Hybrid Instrument Final Project Milestone 1 from Haochuan Liu on Vimeo.

littleBits is a system of electronic modules that snap together with magnets. We built littleBits to break the boundaries between the products we consume and the things we make, and to make everyone into an inventor.

Each littleBit has one unique function (light, sound, sensors, buttons), and with different combinations you can make large circuits. littleBits allows you to create interactive projects without any background in engineering, programming or wiring, in a few seconds. It’s as easy as snapping LEGO bricks together. And the best part is, it’s open source!

What is littleBits? from littleBits on Vimeo.

SketchSynth: A Drawable OSC Control Surface

SketchSynth lets anyone create their own control panels with just a marker and a piece of paper. Once drawn, the controller sends Open Sound Control (OSC) messages to anything that can receive them; in this case, a simple synthesizer running in Pure Data. It’s a fun toy that also demonstrates the possibilities of adding digital interaction to sketched or otherwise non-digital interfaces.

More...

More..