See some of the robes created by visitors on flickr

This installation was exhibited by the V&A at the Digital Shoreditch festival on Saturday 7 May 2011, Truman Brewery on Brick Lane.

http://vimeo/20323985

More information can be found at Mapping festival and 1024 Arcitecture

I tried to keep the source super minimal and as easy to read as possible. For better results, you can try running an erode on the subtracted image, do a contour analysis and smooth the edges slightly. It also helps to run it in front of an empty wall with stable lighting, and ideally a webcam which has manual exposure / shutterspeed / gain etc.

To run the source code you will need:
- processing, which can be downloaded from www.processing.org
- the ControlP5 library (to handle the UI) which can be downloaded from www.sojamo.de/libraries/controlP5/

P.S. Everytime I see this effect I keep thinking of the idclip cheat in doom!

Update: Check out the King of Pop’s version from over 3 decades ago here

Update: Anton aka Vade has made a Quartz Composer version which can be downloaded from bit.ly/5v43tM

Update: goto10 also has a sweet Quartz Composer version which can be downloaded fromwww.hybridvisuals.nl/downloads/m8-trace-quartz-composer-effect/

Update: OKGo just released an ‘official’ processing app to do the same (not related to mine, but developed by Jonathan Bobrow). Check that out at www.okgo.net/wtf_holidays.asp

The source code for the above demo is included with the library as an example to demonstrate how to use the features.

Main features include

• particles
• springs
• attractions (+ve or -ve)
• collision
• replay saving and load from disk (temporarily disabled in current alpha release)
• custom particles (extend ofxMSAParticle and add to the system)
• custom constraints (extend ofxMSAConstraint and add to the system)
• custom force fields (extend ofxMSAParticleUpdater and add to the system)
• custom drawing (extend ofxMSAParticleDrawer and add to the system)

ofxMSAPhysics requires my ofxObjCPointer v1.1+ addon which makes memory management easier. It does not require ofxVectorMath anymore as the required functionality is now in the new OF core.

You can download ofxMSAPhysics from my github repo.

List of new features (from changelog):

- a lot of new files in this one, so you’ll need to remove and re-add it to your project
- no longer requires ofxVectorMath
- attractors fully implemented
- collision (between particles) fully implemented
- you can enable/disable collision per particle, and it will collide with all other particles which have collisin enabled
- AND/OR you can globally enable or disable collision
- AND/OR you can manually create a collision constraint between any 2 (or more) specific particles
- you can set world dimenensions for optimized collision (using Zach L.’s binning code) and particle world edge collision
- particles have individual drag
- particles have individual bounce (for collision)
- particles have individual size (for collision and rendering)
- all ‘setter’ methods return the instance so you can chain them (e.g. myParticle->setMass(1)->setBonuce(0.5)->enableCollision()->makeFree(); )
- replay functionality temporarily disabled while I fix stuff
- intense testing of memory management so should be stable as a rock, lemme know if you see anything weird (turn verbose on to see whats going on)
- using the super fast inverse square root approximation (attributed to john carmack but originally from Silicon Graphics)
- lots of internal restructing and optimization

ofxMSAPhysics uses verlet integration and principles found in this article at gamasutra and binning code for collision optimization from Zach Lieberman.

You can play with an online interactive Java version here.

Download

Get the Java version (for processing) here.

Get the C++ version (for openFrameworks or Cinder) here.

It has also been ported to Flash AS3 by Eugene Zatepyakin. Get it here.

Acknowledgements

A lot of the fluid solving algorithms are from Jos Stam’s paper. I tried to wrap it up in an easy to use API to provide methods for adding & getting forces and color at any position. Other useful resources and implementations I looked at while building this lib are:
* Mike Ash (C) - http://mikeash.com/?page=pyblog/fluid-simulation-for-dummies.html
* Alexander McKenzie (Java) - http://www.multires.caltech.edu/teaching/demos/java/stablefluids.htm
* Pierluigi Pesenti (AS3 port of Alexander’s) - http://blog.oaxoa.com/2008/01/21/actionscript-3-fluids-simulation/
* Gustav Taxen (C) - http://www.nada.kth.se/~gustavt/fluids/
* Dave Wallin (C++) - http://nuigroup.com/touchlib/ (uses portions from Gustav’s)

and some other Stam solver implementations I’ve come across since writing this are:
* Chris Sugrue & Arturo Castro (C++) - http://csugrue.com/code/?p=15
* Alain Ramos (C++) - http://code.google.com/p/ofxding/downloads/list
* Mick West - http://www.gamasutra.com/view/feature/1549/practical_fluid_dynamics_part…
* Mick West - http://www.gamasutra.com/view/feature/1615/practical_fluid_dynamics_part…
* NVidia (C / Cg) - http://developer.download.nvidia.com/SDK/9.5/Samples/gpgpu_samples.html - gpgpu is the future

…and you can find a brilliant AS3 port of this library at blog.inspirit.ru/?p=248

Originally created to forward UDP TUIO (OSC) messages straight to TCP to be read from within Flash.

This application forwards all incoming UDP messages straight to TCP without touching the data, just a straight forward.(Since version 0.2.1 there is the option to prefix the size of the packet before sending the data to comply with OSC / TCP specifications). This enables applications that don’t support UDP (e.g. Flash) to receive the data. Since OSC / TUIO are generally sent via UDP, this enables Flash to receive those messages in their raw binary form.

Settings can be edited from data/settings.xml.

Source and binaries at http://code.google.com/p/udp-tcp-bridge/

Isn’t that what flosc does?

Nope. Flosc is specific to OSC, and converts the incoming data to XML and sends that over TCP. Flash then receives and parses that XML. While this is an ingenious solution, it isn’t the most efficient way to do this currently as we now have flash.net.Socket’s in flash allowing us to open a TCP Socket and read raw binary data.

Can’t I just send TUIO over TCP in CCV/tbeta?

While it’s great that CCV/tbeta sends data over TCP so flash clients can listen directly, and we can cut out the middle man (e.g. flosc), the data is still sent as XML, not as binary TUIO. In fact the data sent by CCV/tbeta isn’t TUIO at all, it’s just an XML string with the touch data.

So how can I receive this data in Flash?

Using flash.net.Socket http://livedocs.adobe.com/flex/2/langref/flash/net/Socket.html

Can I still use the NUIGroup Flash touchapi?

Yes, but you need to replace a few files. If you have a tracker sending TUIO over UDP (e.g. almost any tracker), and you use udp-tcp-bridge to forward that to TCP, the data being sent to TCP is raw TUIO, not an XML. So theNUIGroup flash touchapi cannot parse it out of the box. Thankfully there are a few replacement classes (TUIO.as, TUIOCursor.as etc.) at http://www.touchgateway.com/?p=70 which do all the nessecary binary parsing and generates the same TUIO events so you can carry on using the rest of the NUIGroup touchapi. So if you overwrite those AS3 files with the ones in that zip, you can carry on using your flash multitouch apps seamlessly.

Building the source

The application was written with openframeworks 006 and uses the addons:

• ofxNetwork
• ofxThread
• ofxXMLSettings

Also worth mentioning

flosc - Java app which forwards incoming OSC (UDP) messages to XML (over TCP).

oscar - Native max osx app that does the same as above (osc->xml).

toxy - native windows/linux app that forwards incoming TUIO to TCP in a custom binary format (as far as I understand).

touchgateway - Java app which forwards all incoming UDP to TCP. Same as what my udp-tcp-bridge does, but in Java.

** Update 27/10/08 – Version 1.1 **

Added loop mode configurable from the preferences.plist

—————–

MSAQTPlayer is a Mega Super Awesome (& mega super basic), fullscreen / multiple output Quicktime player with fast, greater-than-4096-pixels support.

I wrote this app for an installation because I needed to play a 4,800 x 600 Quicktime file across 2 outputs on my graphics card, each feeding 3 triple heads to go into 6 projectors, and I couldn’t find a single application that could do it.

Most video applications which can output across multiple video-outs cannot play files larger than the maximum size of a texture (usually 4096 pixels). And even if the movie resolution fits in a texture, they are very slow because they convert each frame to a texture and upload to the graphics card. The only application I found which could play a large file is Quicktime Pro, but that cannot output across multiple video-outs.

MSA QTPlayer is a native Cocoa app which can do this (using QTKit and OSX 10.5) .

Assumes a simple physical screen layout – all screens are side by side and same height.

—————–

MODES

The app has a ‘Stretch to fullscreen’ mode (toggle with ‘s’ key), which simply stretches the video to cover the entire desktop area. If you have 3 monitors / projectors this will stretch across all 3 monitors / projectors. It doesn’t preserve aspect-ratio, will add that later.

It also has a ‘in Front’ mode (toggle with the ‘f’ key). This brings the player window to the front of everything (including the dock and menubar).

For my installation I didn’t need the ‘stretch’ option, just set my two output resolutions to 2400×600 (3 x 800 x 600) giving a combined desktop size of 4800×600 and output straight to all 6 projectors with a 1:1 pixel mapping for best quality. The app was in ‘front’ mode, to cover the dock and menubar.

—————–

STARTUP SETTINGS

There is a ‘preferences.plist’ in the application bundle (right click on the MSAQTPlayer.app and select ‘Show package contents’, then navigate to contents >> Resources >> preferences.plist).

Double-clicking on this file should open it up with your default plist editor. If you do not have one installed you can either download one, or edit it with Text Edit – it will be ugly, but will still work.

Under Path, you can enter an absolute or relative (to the MSAQTPlayer.app) path. The default is mymovie.mov. If the movie is not found the app will just quit (see the system console for messages).

Stretch to fullscreen, Start in front, and Autoplay are all pretty self-explanatory. If you have a WYSIWYG plist editor they will appear as checkboxes, otherwise you need to type in true or false underneath them.

Save and close the file, next time the app runs it will use those settings.

—————–

KEYS

esc – quit
f – toggle ‘in Front’ mode
s – toggle ‘Stretch to fullscreen’
space – toggle play
j, k,l – control playback rate: slowdown (and play backwards) / stop / speed up (and play forwards)
home, end – goto beginning / end
left, right – frame advance back / forward
[ ] – seek backward / forward 1/10th of the movie
pgup, pgdn – goto previous / next chapter
m – toggle Mute
up, down – change volume

** WARNING **
If the app crashes while in ‘front’ mode, it might be a bit hard to get back to your desktop. So make sure your file works first while in normal mode, then you can set it to be in front.

MSAQTPlayer – The Source Code 
MSAQTPlayer – The App

“Roots” is an interactive musical/visual installation for the Brick Table tangible and multi-touch interface, where multiple people can collaborate in making generative music in a dynamic & visually responsive environment. It is a collaborative effort between myself and the Brick Table creators Jordan Hochenbaum & Owen Vallis. It will premiere at the Minitek Music + Innovation Festival September 12-14, 2008 in New York.

The essence of the interaction, is that you control parameters of a chaotic environment – which affect the behaviour of its inhabitants – which create and control music.

To breakdown very briefly without going into much detail:

• There are vinelike structures branching and wandering around on the table. They live and move in an environment governed by chaos.
• Audio is triggered and controlled entirely by how and where the branches move.
• You – the user – control various parameters of the chaotic environment. Parameters which range from introducing varying amounts of order, to simply changing certain properties to let the chaos evolve in different directions.

There are varying levels of interaction, ranging from traditional one-to-one correlations – ‘this movement I make creates that sound’, but also to more complex relationships along the lines of ‘this movement I make affects the environment in this way which sends the music into that direction where it evolves with a life of its own’. The visuals are purely generative, as is the audio, and as user you can play with the parameters of that system and watch and listen to the results…

Working with fellow techheads at Seeper, our concept was to take a 50ft tent, and convert it into a giant audio/visual instrument – all of the music, audio and visuals inside the tent are generated and controlled purely by the movements of the occupants.

The space was divided into 6 zones. Two of the zones were purely visual, this was the waiting area. Here people could dance, chill, run about and do what they pleased. Two cameras tracked their movement and applied it to the fluid/particles visuals – so people could ‘throw’ plasma balls at each other, or send colorful waves propagating around the space. The other 4 zones had the same visual interactions, but in addition were also connected to an audio system. Each of these four zones was allocated an instrument type (drums/beats/percussion, pads, bass, strings etc.), and movement within these zones would also trigger notes or beats – depending on precisely where in the zone the movement was triggered. A lot of effort went into designing the sounds and notes triggered to make sure the end result would almost always sound pleasant and not be complete cacophony.

The first psychedelic fluid/particles interaction prototype developed in processing.org:

Camera -> osc/midi interaction tests (developed in Quartz Composer):

The two concepts strung together and written in C++ with openFrameworks:

Just a reminder, this isn’t some random trippy visuals set to a piano soundtrack… I’m playing a virtual piano by waving my hands and fingers around in the air (which also moves the fluid and particles around as well)

Technical information:

The visuals start with the camera analysis – without motion there are no visuals. 6 cameras are fed into an 8-core Mac Pro and are all analysed in parallel with an optical flow motion estimation, the analysis for each camera feed running in its own thread. Once the camera analysis is complete for all cameras, the velocity vectors are stitched together and fed into a fluid simulation. Any movement the user makes, causes ‘colored dye’ to be injected into the fluid simulation, with the speed and direction of the movement being inherited by the dye. These movements also create ‘currents’ in the fluid, allowing the user to create swirls and vortices with circular movements (e.g. waving arms around), or just send waves of coloured plasma rippling across the room with a simple push of the body. These currents also allow the user to control particles as described below.

Any movement faster than a certain threshold, also creates ‘glitter’. The speed of the movement controls the number of glitter created (larger & faster movements create more glitter) – thousands of glitter can be swimming around at any one time. Once created, each glitter is independant and swims around with a basic AI – but always overpowered by the current of the fluids, so the user can herd the swarms of glitter using their arms, legs, body etc. and push them towards one another.

Even larger movements – e.g. swiftly swinging hands or head around – create ‘energy orbs’. These orbs cut through the fluid quite quickly, but can still be affected by the currents allowing the user to control the direction and speed of the energy orbs. This allows users to bounce the orbs back and forth between each other playing imaginary tennis or football.

The biggest challenge in creating an application of this scale was to structure and optimize it in a way so it could analyze upto 6 camera feeds, and run at a large enough resolution to cover the entire tent. A multiple computer approach was out of the question due to the complications of synchronising a fluid simulation across multiple PC’s, so the decision was made to go with a multi-threaded app running on an 8-core Mac Pro. The motion estimation was split into 6 threads (one for each camera), the fluid solver ran in its own thread, and the particles (glitter & orb) ran in another thread – all of these threads ran in parallel. Once all threads were finished processing their data for one frame, they exchanged their results ready for processing for the next frame (camera motion fed into fluid solver ready for next frame, fluid currents fed into particles ready for next frame etc.). This approach allowed everything to run in parallel with smooth framerates of 30fps.

In addition to controlling the visuals, motion is also what triggers the sound. The users’ motion is analyzed and broken down into a spatial grid, with imaginary pads corresponding to different notes. This information is sent over a local network to another computer running Ableton Live where all the samples and loops are stored and triggered. The trigger information is sent using OSC and is mapped on the audio computer from OSC to midi using OSCulator – which originally wasn’t designed to handle such complex polyphony, but working closely with the author we received a version which suited our needs perfectly.

Made with openFrameworks.