My third milestone is to be able to make all the hardwares and softwares talk to each other properly, and have something interesting displayed on the screen. By this time, I am able to accomplish the task, sending the pen stroke data from Processing to 4D Systems through serial communication and have it displayed on the screen smoothly. I got a simple animation interacting with the pen strokes – a prove of concept that everything finally works together. Now, it’s time to make something interesting.

Bouncing Ball from Patt Vira

My original second milestone was to use computer vision as an alternate way to track the pen strokes. However, it took me longer to figure out how to send serial data from Processing to 4D Systems workshop, and how to draw properly on the screen. Therefore, the second week was spent mostly to figure out these problems.

I finally solved the problem from the first milestone, and was able to clear the screen after each draw function. The first video below shows a ball bouncing against two pre-drawn rectangular boundaries. The second one shows a real-time pen strokes being sent from the tablet to Processing to 4D Systems to the display.

Ball Bouncing Against Boundaries from Patt Vira on Vimeo.

Line Drawing from Patt Vira on Vimeo.

I found out from working more with the display that it unfortunately cannot handle heavy libraries, specifically box2D. Consequently, as a proof of concept, my goal is to draw some simple interaction. My next step is to be able to draw the lines from the Wacom tablet, and have the ball interacting with the these lines instead of the rectangular boundaries.

Finally, I built the final prototyping of the device for interactive music listening.

And, here is the manual for this device.

The difference between Milestone 3 and final one are that smaller circuit size and form which is more fit-table to wear. In the sound effect and feedback side, I wanted to bring more interactive situation with such as [Bonk~] and beat detector. However, the result brings too much editing to the original music, so I didn’t add to the final show.

I developed the cubes further by adding leds outputs which can gave the cubes ability to trigger one by one. I tried to improve the richness of the sound effect by using PWM, but meet problem of control the pitch of the sound.

Instead of  just copying more cubes, I add more function when create more cubes just like use slide/rotary potentiometers to control the speed and the pitch of the notes.

Music cubes from Wanfang Diao on Vimeo.
Note cubes from Wanfang Diao on Vimeo.

Note Cubes from Wanfang Diao on Vimeo.

In this milestone 2, I finished soldering electronic circuits by using micro-controller, audio amplifier and photosensors. I programed the micro-controller to read the analog input from photo sensors and trigger the speaker to make a note.

After that, I made  a hardboard box to fit the circuits.  So the first version cube has been created.

What’s more I also find a wooden plate for the mechanical part.

More demos:

Note cubes from Wanfang Diao on Vimeo.

Note Cubes from Wanfang Diao on Vimeo.

Through the Lens

Bouncing Ball from Patt Vira

Throughout the process, I have been able to get the different pieces of hardwares and the technology to talk and to work together properly. The video above shows a working prototype of a ball bouncing on the pre-written pen stroke. This is a prove of concept that everything works together. The next step is to explore other possibilities that puts meaning to the instrument through user interaction.

One application that I came up with is a maze game. The point of the game is to be able to control a ball (using LEFT, RIGHT, UP, and DOWN keys on the keyboard) along the path from the starting to the finishing points without touching a pre-drawn maze. As you can see from the series of images below, the maze can be customized to your likings.

This is one example of an application that adds meaning and interaction to the device. However, I am planning to create more applications as I become even more familiar with the instrument itself. I am also open to ideas and suggestions!

The final project goes wrong with Pin conflict on Trinket. Since I use Pin #1 to read microphone’s digital data and Pin #2 (Trinket requires “1”, which means A1, instead of “2” in code) to read analog X-axis accelerometer data,  it gets confused when I have to write the same command “PinMode(1, INPUT);” in the code to execute both data read. It leads to the failure of reading microphone and accelerometer at the same time. Annoyingly, I had to use Teensy at the very last minute instead to perform my demo. It was not robust and that good, and very preliminary. I felt sorry for the audience and reviewers that night. However, they gave me a lot of feedback and suggestions on potential revise and development. Here I sum up some key points:

1. My biggest problem is that I attempt to cover many scenarios and applications, which is so generic that confuses audiences and eventually lose its value. It fails to address the major problem it tries to figure out, or the goal for its exist. It throws abstract pictures to audience, leave alone under the situation that it cannot work.

2. The gesture seems weird since the microphone works part the role of the gesture. I would argue that the gesture is kind of the way people feel the liquid in the cup. Honestly, when I design the gesture, I find only one gesture (shake) is meaningful for people.

3. Other formations. No matter what kind of stuff I want to create and make, it should respect my motivation and its goal. So, again it goes back to “Point #1”.

I agree with most of the comments in the critique and they drive me recall my original motivation: I know cup has resonance with liquid, cup has material, people use cup, and it can be an instrument to perform music. In the past weeks, I continue to do some research how to make use of these characters and what kind of music it can generate. Here I have some answers: it can generate beats, then rhythm, so it can perform some kind of percussion performance. Besides cups, other objects also have resonance property. When I look back at these, I narrow down my scenario for Tapo and come up with a new but iterative design and development solution.

Redefine the story for TAPO

Physical objects have resonance property and specific material. Tap object gives different sound feedback and percussion experience. People are used to making rhythms by beating objects. So, why not provide a tangible way not only allowing people to make rhythms with physical objects around she/he, but also enriching the experience by some computational methods. The ultimate goal for this project is that ordinary people can make and play rhythms with everyday objects, even perform a piece of percussion performance.

Goal: Orchestrate sounds producing (sampler) and Combination of Hardware and Software.

Sampler:

Originally, planning to trigger sound files in max to play one at a time, but this approach proved to limited. Audio would need to complete finish play 808 drum sound or interrupt it prematurely to keep up with users triggering.

Ali.sampler object in max allowed for multiple sound files to be played at once and layer over tope of each other to gain a poly-morfic effect.

Code:

Start of with a keyboard for your feet that implements a drum sound based on dance step.

This week was fill with experimenting with various way to play sound files in max. Tested continuous cycles and frequencies with phasors and groove object. Adapted playback volume and file choice decision based on analog data. Attempted to choose sound based on peak jerk accelerometer data from each leg, but accelerometer data is not clean enough a fast movements to grant consistent readings. So

Establish streamline method of read data. Run each accelerometer data through a smoothing filter. The Y-axis reading for each leg was offset due to 1 g of gravity in the downward direction. To counteract this for quick readings, the derivates of each of the accelerometer data were taken, and routed to the analog peak choosing method for sound files.

Since the peak method proved unsuccessful, A smart option was utilized…… the Wekinator.

Wekinator is a Neural Network Machine learning algorithm that takes in a set number of variables and yield how any number of parameters you want to fluctuate and changes depending how you train it. The Wekinator is ideal for reading the position of certain foot position and matching them to a number for a sound file to be played. It’s limitation are that is doesn’t offer discrete numbers, which becomes haphazard when in-between positions.

Since Wekinator failed to produced desired results. Switched to Node based estimation of foot position.

Accelerometer determine acceleration in 3-directions (xyz). Gravity constantly impose a 1 g of downward force on the sensor.  Utilizing this, one can roughly determine the orientation of sensor. different step position have different orientation and leans of the shin. Utilizing this I can roughly differentiate steps in the X-Z plane of the floor. using the Node object in max set up 2D plane a specified area of different sound. Various combination of Heel or Ball press and the two shins reading triggers preset sounds. This method produce best results so far.

Combination of Hardware and Software:

Integration of Hardware and software was straightforward due to the established wireless.

A interesting trait of this instrument in that it’s Completely Concealable.