Post 26: What Next?

   Well, now that Mike has figured out the basics, we need to refine it, and I would like to actually participate. He used a Game Building software called Unity, that ia actually cross platform. The full Pro version costs $1500, but there is a free limited version which I downloaded and installed on my Mac. Mike sent me his project folder, and I could open it, but in the end couldn't get the Kinect to connect. He offered to loan me a windows machine, but I would rather run Windows on my Mac with Parallel Desktop. Since all I have is the old Windows XP, I need to upgrade to Windows 7, that Mike recommended over Windows 8.
   Not so simple though, as Microsoft has discontinued Windows 7, and non OEM copies are rare and expensive, over $160 on eBay. I suppose I could download the much cheaper OEM version, and buy it again if I change machine… 
   I may also have to upgrade to Parallel 9.

Post 25: Hope for the Kinect at Last!

   BIG NEWS: Corey loaned Mike a Kinect for the week end, and he got it to work and do pretty much what I was asking for in my emails to them last week:

Hi Guys,

   I am playing with the idea of a set that could be projected on at a roughly 30 degrees angle WITH A SINGLE PROJECTOR. I made a cardboard model to test it:

    And it works. My 5K could cover a 20 ft wide stage in the dark, floor and walls.

    It can be mapped with an image like this, which requires precise projector positioning and accurate alignment:

     Or I can use MadMapper with an image like this, and map each panel separately, which makes it a lot easier:

     

 Either way, this is the way it ends up looking on the model:

   As you can see, the image projected on the "dancers" is mostly of the floor(which could be interesting as long as they stay at the front of the stage).

   Now, WOULD IT NOT BE NICE if I could use the Infra Red Kinect camera to map BOTH the shape and movement of the dancers, and replace it image in image, on the fly, with either a solid color or an entirely different image. That would be like having several precisely fitted infinitely adjustable projector/spotlights following them around. I expect some kind of delay due to processing, but the movements could be kept slow.

   I can dream it up, but I honestly don't know how to do it (yet), and won't have time to learn the Processing language and the Kinect Hacks in the book I bought until after Light Dreams.

   So, do you see any way you could figure it out?

   Keeping you on your toes… Let me know what you think.

   Thanks.

JJ

  This is a neat idea.  I think it can be done without a lot of work, but there will definitely be spillover from the dancers to the ground/walls (both from latency and registration errors).

   I'll run a couple of tests to see what the issues look like; maybe we can come up with a way that they either aren't very apparent, or otherwise work into the overall projection.

Corey Shum

Technical Director, Enabling Technology Laboratory
School of Engineering, Dept of Mechanical Engineering
University of Alabama at Birmingham — Knowledge that will change your world

Corey,

  I am more concerned about the latency than the registration errors, but that in itself could create interesting effects, with the Dancer sometimes getting ahead of her projected image. 

  Because of the way the IR sensor in the Kinect works (projected dot pattern reflected towards the IR camera), the limited density of dots, and the fact that surfaces at almost 90 degrees (like the edges of a body) don't reflect the dots, there is some raggedness around the edges, but that can be minimized with a flat distant backdrop, and could be smoothed and feathered to some extent.

  They are also going to come out with a new much improved higher definition Kinect with skeletal tracking, that is also supposed to act like a green screen and key out the background:

       http://www.ign.com/articles/2013/11/07/how-next-gen-kinect-is-more-accurate-than-ever

  That would be fabulous, and I hope it comes out soon, and can be hacked for the Mac…

  Thanks for your help. Keep me posted.

JJ

PS: When we get this one figured out, I would like to find a way to harness the amazing point cloud rotating 3D imaging capability of the Kinect:

      The best software I have found for the Mac to visualize the images is cocoaKinect.:

                      http://fernlightning.com/doku.php?id=randd:kinect

       Synapse does a pretty good job at recognizing the skeleton:

                               http://synapsekinect.tumblr.com/post/6610177302/synapse

JJ,

   Corey was kind enough to loan me a Kinect over the weekend.  I played around with it and was able to create a proof of concept application.  I like your idea.  It lets you do a green screen effect without the green screen.  I wrote some code that scans through the depth data from the device.  For each pixel that belongs to the player, I replaced with the color information from the RGB camera on the device.  All other pixels were replaced with a background image. I also went ahead and added the particle effect we were talking about last time.  If you look at the screen shot, you can see that the particles are being emitted from my hand:

    The latency wasn't nearly as big of an issue as I thought it would be.  But, I wasn't moving that fast.  A dancer may see a little more.  The only major issue I ran into happened when the Kinect lost me and had to reacquire.  Sometimes when it reacquires a body, it will stop sending depth and color information from the device.  The tracking still worked though.  So, in the application, you could still see the particles but the image of me disappeared.

    BTW, we can add any 3D objects to the scene.  not just particles.  Here is a scene with me holding the ghost and PacMan models from my game in each hand.
   Thanks,

 Mike

Good Morning my Friends,

  Whaooooo!… Look at that! Super cool! That is some great news to wake up to and start a new week.

  Mike, you are a Godsend. You are the quiet man that delivers… And quickly!

  The images look great by the way, and I think you really need to do something with the Kinect either for PacMan, or another interactive game.

  My big problem is that I use a Mac, so can it work for me. I actually did manage to move an object one time with Synapse and Quartz composer, but now keep getting an error message.

  In any case, great job Mike, as always.

  Thanks a bunch to both of you.

JJ

Post 24: What Music?

   The Great thing about cooperating with Mary on this project is that we bring two totally different points of view to the game, and yet communicate and understand each other very well. I am the "Half Nerd Artist" who tends to think of visuals, images and technical possibilities afforded by the new hardware and software available today to play with. Mary is the reminder, as Myrna was in the workshop, not to lose track of "content"(I hate that word, it makes Art sound like it belongs in a box, and my greatest desire is to get out of the box!). She thinks in terms of music, rhythm, meaning, style, choreography, feelings, expression, movement.
   Which brings me to our current discussion about "music". I love "Good Music" in all genres. I was brought up on "serious" classical music mostly because my parents had gotten me in a "record of the month club" that didn't offer any rock and roll, and because my high school Music Teacher was in love with the great Russian composers. This is definitely a bit of an acquired taste, but it stuck for many many years, even though I also loved the Beatles, Ray Charles, John Lee Hooker, Cat Stevens, Leonard Cohen, as well as Georges Brassens, Jean Ferrat, and Jacques Brel. 
  There was a Music from the Andes period in the late 60's culminating in a trip to Peru and Bolivia in 72.
  After moving to the US to be with Rachel, I got into Renaissance Music in the 70's, and baroque music in the 80's. The high point was definitely hearing Mozart's Requiem in Prague in the very church where it premiered.
  In the 90's, I started to concentrate on Vivaldi's Religious and Operatic Music, almost to the point of obsession. I researched and tracked down every recording by RV numbers. I would play it loudly non stop all day when I was in the studio painting, from the Stabat Mater to Orlando Furioso. Don't mention the Four Seasons please...
   Things changed around 1999, which is when I started spending most of my time either inside on the computer, or outside building our house, which took several years. I found that either activity required a kind of concentration that tuned out music, so there was actually no point playing any. At the same time, I was restless and searching for a new way to express myself as an Artist, and the Internet offered a novel marvelous new way of doing research. No more poring through shelves of books at the library, everything became readily available from my desk with just a few clicks. It was a game changer. I was already a "hunter", I became a maniac, one site leading to another, click after click, procuring rapidly masses of information, knowledge and images… eBay changed the face of collecting, and the house filed up with wondrous old stuff from all over the world.
  My hunting included music, and I discovered all kinds of new music, mostly singers: Rokia Traoré, Souad Massi, Missia, Elena Frolova, Agnès Jaoui, Daby Balde, Etta Scollo, Yasmin Levi, Cesaria Evora, Chavela Vargas, Sopor Aeternus, Devendra Benhart, Katie Melua, Paolo Conte, Daan. I suppose the common thread is "singing from the guts".
  I became particularly impressed by the oratorio like work of Belgian composer Nicholas Lens: Flamma-Flamma, Terra-Terra, Amor Eaternus. Unfortunately, his work is becoming more minimalistic and challenging over the years, and I still consider his early Flamma-Flamma his best. 
  My work in Animation, Video and Architectural Mapping required sound tracks, and I started collecting thousands of sounds to go with the thousands of images I already had. I used Nicholas Lens work many times as a base. But what works for a sound track is often not the music you would listen casually. It has to have power, expression, rhythmic variety, originality, and the beat drives the movement of the images on the screen. A sound track doesn't have to be what we would call music, it is more like a mix of layered interesting sounds: music, sounds, noise, speech. 
   Which brings me finally to our ongoing conversation with Mary about what kind of "music/ sound" we want to work with. We saw the Koresh Dance Company the other day, and the strongest piece had no music per se, just spoken words and sounds. The track for Bridgman-Packer's last piece "Voyeur" is a very complex forever changing layering of music and sounds. I liked both.
   We definitely don't want "pretty music", nor traditional "dance music"(if I hear another Bolero, I will puke! I swear), nor boring repetitive "minimalistic music". I would like to have beautiful harmonies, but also some dissonances, a strong changing beat, some jarring unexpected sounds(to wake up the audience…), natural sounds, words. I want unexpected "things" to happen. I want the listener to be surprised, a little confused, a little shocked, never bored.

Post 23: Moving on from there

   I found this video on YouTube showing some interesting  Kinect experiments by the German Group Animatronik, a bit along the lines of what we, as well as Art and Myrna are experimenting with. I don't quite get the technical explanation at the end though…

   I also found this rather minimalistic strange performance  using a Kinect somewhere in  Japan.

   Then there is this great point cloud sculptural particle effect by Daniel Franke which actually requires 3 Kinects, as well as an accompanying "making of video". I have no idea what software they use or how they did it, but it is terrific…
   Even though it's mostly Windows, THE BLOG to follow seems to be KINECTHACKS.COM , where they have an interesting colorful "ribbon"piece created in Copenhagen:

I also found some info specific to the Mac on the music web site of John Bellona, which has links to download various Kinect Applications, as well as some technical videos:

       Kinect-Via- interface series

            Simple Kinect

 

Post 22: The Bridgman-Packer Workshop

  The workshop took place over the course of 4 nights 6 to 9 and one day 9 to 3 . It was intense and driven. Art and Myrna didn't waste a minute, gave us as much as they possibly could, and every student tried to get the most out of them. The first two nights were devoted to showing us their live video and projection techniques, practicing moves, and exploring possibilities. It was definitely hand on. Even I had to "dance" the first night!
  The group was diverse and cosmopolitan, made up of a majority of young female dancers and choreographers, and a few "tech guys". It was a good mix and worked well. The third night, people started coming up with personal projects, and we shot green screen footage. The 7 projects were refined the fourth night, and plans made for a show of works in progress on the last day of the workshop. That last day was definitely very intense and somewhat hectic, but Art and Myrna kept things moving and organized, and everybody managed to come up with more or less the piece they had in mind. The show started almost on time in front of a group of family and friends, and went quite well. I will post a video when I get it.
  Everybody then helped pack the equipment, and the day ended with a dinner party at my house.  

Post 22: Discovering Bridgman-Packer

   This year, the Alabama Dance Festival included a workshop on Dance and Video Technology led by Art Bridgman and Myrna Packer. I have to be honest and say I actually had never heard of Bridgman-Packer… They were apparently here a few years ago, but I was so wrapped up into Architectural Mapping at the time that I entirely missed the opportunity to discover them.My loss.
   As soon as I started watching their videos, I knew they were Masters at the game I was just starting to play with Mary, and resolved to "pick their brains" as much as I could. Not, mind you, to try and imitate what they do better than anybody, but to figure out our own way of using the new technologies at hand.
    The workshop has been a wonderful experience, even though I didn't get exactly what I was looking for technically. It was a workshop for dancers and choreographers, with support from the Nerds. Well, the Half Nerds with an Artistic bend… Myrna kept reminding us that technology was there ONLY in a supporting role, to help express feelings and ideas, and tell a story. I have no problem with that.
  First, Art and Myrna are absolutely amazing: charming, gracious, friendly, unassuming, energetic, stimulating, inspiring, and extremely generous with both their time and knowledge. 
  The workshop was very stimulating and inspiring for me, and I am quite sure every other participant felt the same,
  I met several very creative and original young dancers and choreographers, whom I am hoping to work with in the future, both on "Light Dreams 2014", and on our own "scAena lumInaria" still very loose Dance and Projection Project.
    We set up and practiced various live video techniques, black and green screen, both vertical and on the floor, with multiple video feed. And we got to try our own ideas. I was rather surprised by how much they are able to squeeze out of very basic  often outdated video equipment, and liked that very much. Years ago as a photographer, I always felt I could get better pictures out of an Instamatic than a lot of people did out of fancy expensive cameras with big fat prime glass. And to this day, I have never owned "professional" equipment. I would love to have fancy glass, but it's just too expensive, and I have done fine without it.

   The performance at the Museum of Art of their latest and most elaborate piece "Voyeur" was absolutely spell binding and awe inspiring. Saying it is intense and dazzling is a huge understatement. The grace of the movements, the interaction, the beauty and mystery of the set, the choice of images, the emotional and visual complexity and variety, the originality of the gestural vocabulary,the controlled eroticism, the rich colors, the sophistication, the attention to the smallest detail are all admirable and extremely stimulating. 
  They bring the still stylized visual world of Edward Hopper to life through light, but re invent it in a complex additive fragmented way. They constantly deconstruct and reconstruct the set, which becomes almost cubist at times.

 They are EVERYWHERE,large and small, live and canned, real and virtual in sometimes almost indistinguishable ways. The bodies merge into the projected images and vice versa. Thanks to carefully controlled live cameras, the spectator is made privy to their "private sexual life" in an almost uncomfortable way, and can eavesdrop on what is happening behind the zigzag wall of the set, now and then catching a glimpse of a body part through a window.

   As an aging Artist myself, I cannot help but notice and admire how fit, beautiful and sexy these "aging dancers" are, with no trace of make up or artifice.

   As Myrna said in the Q&A after the show, she has been dancing since she was seven, but is still hungry and thirsty for more, and it looks like that thirst is still very far from being quenched. I think we are going to see these guys create and invent and break new ground for a long time. As far as I am concerned, they have singlehandedly redefined modern dance, and raised the bar of creativity to new levels. Hell, they have come up with an entirely new bar!

Post 21: MORE NEW TOYS: MAKEY MAKEY AND ARDUINO

   Well, in my never-ending quest for new ways to control a computer and a projected image, this is one I have wanted to investigate since last year. So I ordered a Makey Makey touch controller:

  It connects to the computer via USB, and to conductive objects with alligator clips.
  People have used it to make fun stuff like "banana pianos":

and other things as shown in this video:
   http://www.youtube.com/watch?v=wkPt9MYqDW0
  But most of the stuff I found on Google is rather geeky and shoddily made, except for this "apple marimba":

and the 3D "musical drawing in the video above:

   I do like the idea of using graphite on paper as a touch trigger, and made a little experiment with a rough drawing that plays 'Twinkle Little Star" when you run your finger along the bottom "keys":

  Besides the 6 keys on the front that I used, the Makey Makey has 6 more connections in the back pre programmed for letters W,A,S,D,F and G, so it offers a total of 12 touch inputs.    I could make an instrument or graphite drawing playing a tune with 12 notes.

  Now if I could figure out how to also make LED lights turn on and off, I would be in business! That should be easy…

   Well, not so easy actually. The Makey Makey talks to the computer allright  but to turn lights on and off, the computer has to talk to the lights, and to do that, I will need an electronic interface called an Arduino:

  I am going to have to learn to program the Arduino to do what I want using the PROCESSING language and the Arduino software. 

  So I went ahead and ordered an Arduino Kit from Amazon, which on top of the Arduino Uno, includes a breadboard, a manual, and all king of gadgets to connect to it:

   I would really love to see a "music and light making contraption" at Light Dreams. It could be a big drawing that plays tunes and lights up when you run your finger on it, or stepping stones that play music and light up when you walk, or a fruit xylophone that projects matching images, or a set of lighted balloons under a conductive ceiling...