Augmented

Web Gestures

a tQuery way to interact with your Browser

Jerome Etienne

Who Am I

• coder, speaker, trainer
• Proud member of Web-5 association
• Javascript, Node.js, WebGL and other HTML5 games stuff

Learning Three.js

• Author of learningthreejs blog
• Lists of three.js tutorials or "WebGL for dummies"

Contacts

• @jerome_etienne on twitter, jeromeetienne on github

So what is this ?

What are Augmented Web Gestures

Steps

1. User moves in front of the camera
2. Browser analyse the images from the webcam
3. They are converted into user events

Features

• Instinctive
• Simple to Understand
• Innovative

Show, Dont tell!

Doom: a new workout for geeks!

• link to youtube video Here
• use augmented web gesture to detect the user punches
• each user punch hurt the doom monster
• here is the demo demo

"Show me the code" - linus torwald

Coding with augmented gesture

Initialize it

var aGesture	= new tQuery.AugmentedGesture();

Add Debug View

// get debug texture
var texture	= aJoystick.texture();	
// put the debug texture on top on the screen
var domElement	= texture.image;
document.body.appendChild(domElement);

Get a Gesture out of Pointers

Get Pointers Coords

aGesture.bind('update', function(pointerR, pointerL){
	console.log(pointerR, pointerL);
});

Recognize the Gestures

var gestureR	= new tQuery.GestureRecognition();
aGesture.bind('update', function(pointerR, pointerL){
	var eventR	= gestureR.update(pointerR.x, pointerR.y, canvasW, canvasH);
	// eventR === null if nothing recognize
	// eventR === 'punch' if a punch if recognize
});

Pack it with Doom Character

Get MD2 character

var character	= new tQuery.RatamahattaMD2Character();
character.attach(world);

Action from Gestures

if( eventR === 'punch' ){
    character.animation('crdeath');   // make character move
    // play a sound
    tQuery.createSound().load('eatghost.mp3', function(sound){
        sound.play();
    });
    // change the weapon
    character.setWeapon(Math.floor(Math.random()*11 % 11));
}

Not bad, what is under the hood ?

Augmented Web Gestures : Fact sheets

All open technology

All inside your web browser. pure web

Very efficient image processing

Image Processing Requirements

Fast

• Thus you got time to do other things. Like a game

Robust

• Thus it works even in harsh environement.

Easy to setup

• Thus anybody can do it wihout prior knowledge.

Comparison with alternatives

Previous work

• WII and kinect did good.
• but closed technology
• specific installation required (not webby)

Augmented Web Gesture

• costs less (dishes gloves/children toy)
• All open standards
• All in browser so no install

What are the capabilities ?

Localize Two Hands

two pointers

• two hands == two pointers
• no click for now
• demo. Look at the crosses

Props

• 2 simple elements quite distingishable
• Currently 2 kids toys. total cost 6euro
• successfull experimentation with dish gloves too

hmm what about a actual application ?

Doom: a new workout for geeks!

• link to youtube video Here
• use augmented web gesture to detect the user punches
• each user punch hurt the doom monster
• here is the demo demo

Seems promising... What's next ?

Future works

on Medical Apps

• Navigation in 3D data. Scanner
• User are non specialist
• Sanatized UI: no contact, so no germ propagation.

on Game

• tennis, golf, aerobic ? Doom will likely a lot of sport too :)
• good for elders or youngsters, as the wii did it

on other fields i dunno yet :)

That's all folks.

Question ?

Jerome Etienne