Interactions

Control your visuals with your voice, hands, and body

Transform your audio-reactive creation into a multi-input performance instrument. Connect microphone input for live vocal control, use your webcam as a dynamic texture source, and implement hand gesture recognition to manipulate your visuals in real-time.

Control visuals with your voice

1. Add a Microphone node (from the SOURCE DATA section).
2. Connect the Microphone node output to soundData input of the Frequency Split 5 node.
3. Close the Shader Editor and then enable the microphone by clicking the microphone icon in the control bar at the bottom of the Scene Viewer.
4. Start making noise!

Use the camera feed as texture

1. Add Camera node (from the SOURCE DATA section).
2. Connect the Camera node to the Texture Value node.
3. Close the Shader Editor, and then enable the camera by clicking the video icon in the control bar at the bottom of the screen.
4. Zoom in to see the camera input projected on all sides of the box.

Recognize gesture in camera feed

Instead of having the audio scale the image on the box, we will scale it based on the distance between two fingers: the thumb tip and the index finger tip.

1. Add a Gesture Data node (from the SOURCE DATA section).
2. Add two Gesture Landmark 2D nodes (from the SOURCE VALUE section).
3. Connect the Camera texture output to the Gesture Data texture input.
4. Connect the gestureData output from the Gesture Data node to the gestureData input of each of Gesture Landmark 2D nodes.
5. Modify each of the Gesture Landmark 2D nodes, so that one references Thumb Tip and the other Index Finger Tip.
6. Add a UV Distance node (from the UV OPERATIONS section), and connect one of the Gesture Landmark 2D to its uv input, and the other to its uvCenter input.
7. Connect the output from the UV Distance node to the scale input of the Texture Scale node.
8. Close the Shader Editor.
9. Enable camera in the control bar.
10. Make sure the camera sees your hand, then change hand position and open and close the thumb and index finger to see how it affects the camera image on the box.

Completed Project

Code