Creatability

Who made these?

These experiments were a collaborative effort by Jay Alan Zimmerman, Claire Kearney-Volpe, Kyle Philips, Yotam Mann, Luisa Pereira, Use All Five, and Google Creative Lab. Special thanks to Chancey Fleet, Josh Miele, James Maxson, and friends at Henry Viscardi School at The Viscardi Center, Tech Kids Unlimited, and ADAPT Community Center.

What can you do with them?

The experiments explore a diverse set of inputs, from mouse, keyboard, body, wrist, nose, or voice. You can make music by moving your face, draw using sight or sound, experience music visually, and more.

How does the body tracking work?

The body tracking feature was made using Posenet, a machine learning model that can detect key body joints in images and videos. This lets you control the experiments with your webcam, simply by moving your body.

When I turn on my webcam, are my images being sent to Google servers?

No. Your images are not being sent to any Google servers while you’re using body tracking mode in these experiments. All the image recognition is happening locally in your browser.

What devices does it work best on?

For the best experience, use Chrome on a desktop PC or Mac. Most features also work on tablets, but have not been tested on all devices. Phones are not fully supported. 

Do they work with screen readers?

We’ve worked to make the experiments work with many screen readers across different platforms, but we’re still working to improve compatibility. If you have feedback, drop us a line at creatability-support@google.com.

Why is the experience slow on my computer?

The body tracking feature works best on newer computers with faster GPUs. If you have an older computer, you can still try it, but it just might just feel a little slower and less accurate.

How can I build projects like this?

We’re working on sharing open-source code and components in this Github repository. We’ve also creating guides as starting points, like this tutorial that lets you learn how to use body tracking with P5.js.

Made something that you’d like to see featured here? Contribute your own experiment to this collection.