The Play-Doh’s the thing: Prototyping mobile augmented reality apps with paper and clay

A new mobile augmented reality (AR) tool specifically created for interaction designers was recently unveiled at the University of Michigan School of Information by its inventor, assistant professor Michael Nebeling.

Michael Nebeling

AR is a technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view.

The first design tool of its kind, ProtoAR essentially removes the technical expertise and programming skills currently required for mobile apps to make use of AR. It integrates physical and digital prototyping to simplify the mobile AR app design process.

ProtoAR allows designers to work with physical prototypes using paper and modeling clay such as Play-Doh, then quickly transition to digital prototyping using their mobile phones’ cameras. Existing tools, such as InVision, Sketch and Adobe Experience Design, do not support designing for mobile AR, Nebeling explains.

“Even experienced mobile app designers face significant challenges creating AR interfaces and 2D/3D AR content with current digital prototyping tools,” he says.

From models on paper or in Play-Doh, app developers can create a variety of potential practical AR applications using ProtoAR, which seamlessly blends physical and digital content creation.

Modeling with paper and Play-Doh

Paper and Play-Doh are used for practical reasons. Designers work on paper for initial sketching and wire framing, where making changes are quick and easy. The Play-Doh was used to produce 3D props to enable rapid prototyping and iteration of the AR content specific to the application.

“Play-Doh is key here,” Nebeling explains, “since this would otherwise require designers to have significant experience with 3D modeling tools, or limit them to AR content created and shared online by 3D artists.” To capture a quasi-3D model from Play-Doh, designers simply record a video of the object, either by moving the camera around it or by spinning it in front of the camera.

One ProtoAR application allows people remodeling their homes to overlay 3D images of selected furniture and appliances onto views of rooms where the items will go. This allows buyers to visualize how rooms will look before purchasing anything. Nebeling recently began a project with Masco Cabinetry utilizing this application.

Another application is reducing drivers’ blind spots by employing AR on windshields, a recent project by Nebeling and several UMSI students. Other apps help recreational tennis or basketball players to analyze their technique by visualizing recordings of their forehand technique in 3D. Nebeling and colleagues are working this project as part of a U-M Exercise and Sport Science Initiative (ESSI) grant.

Nebeling involved UMSI interaction design master’s students in ProtoAR’s design process. 

In four Design Jams around the AR furniture preview app in Nebeling’s Information Interaction Lab, students sketched mobile user interface (UI) wireframes and user flows on paper, modeled increasingly complex furniture with Play-Doh, and walked through their demos making use of both media.

“We wanted to learn how our students – the next generation of professional interaction designers – would use these materials in combination with ProtoAR,” Nebeling says. “We really saw the limits of each medium individually, and then the strengths as they were combined using ProtoAR.”

Students who are enthusiastic about AR but have limited technical skill could create mobile AR apps in less than an hour using ProtoAR.

Nebeling already is working on several new features he plans to add to ProtoAR.

His short-term goal is to adapt it for use in teaching new courses he will teach for UMSI’s AVMR Initiative. He’s working with U-M’s Office of Technology Transfer to explore ways of incorporating ProtoAR techniques into existing professional tools. Major vendors such as Adobe and InVision would be ideal partners, Nebeling says.

This commercial potential only reinforces, Nebeling says, “how beneficial it would be to so many if we had this kind of tool available to anyone who wants to design for the fascinating world of augmented reality.”

Nebeling will present a paper describing ProtoAR at the annual Computer-Human Interaction conference (CHI) in April in Montreal, Canada. “ProtoAR Rapid Physical-Digital Prototyping of Mobile Augmented Reality Applications” is one of three papers by Nebeling teams accepted for CHI 2018. Co-authors are Janet Nebeling, UMSI sponsored affiliate; Ao (Carl) Yu, visiting student from Tsinghua University; and Rob Rumble, UMSI master’s student.

by Sheryl James, UMSI PR Specialist
Posted February 20, 2018