Designed a multi-modal art tool for people with limb disabilities to create art in Virtual Reality
MAR 2018—NOV 2018
We started this project as an Inclusive Design Challenge by Microsoft, our collaborator for the project. The challenge was to design for inclusivity for deskless workspace tools using emerging technologies like AR, VR & AI. After studying the Microsoft Inclusive Design toolkit and exploring multiple options, my team decided to work on making Virtual Reality more inclusive by making an accessible art tool and introduce this new platform of computing to more people.
Role: I was responsible for user research, user interactions and prototyping. I facilitated the interviews with our users and was responsible for testing our prototypes. Based on the research I created the interface for our prototypes.
“VR is inaccessible to people with limb disabilities due to the use of hand controllers”
300 million people of the world's population has some sort of limb disability and due to the use of hand controllers, most VR applications are not accessible to people with these disabilities. People who have cerebral palsy, multiple sclerosis, paralysis, someone who is missing an arm, and even people with injuries are excluded from utilizing the full potential of VR.
We looked into academic research done in this space to make informed decisions while creating our solution. Some of the articles we looked into were:
We decided to analyze the current creative tools in Virtual Reality to get a sense of how people create art in VR. We also studied their interfaces to figure what are the current features, functionalities and limitations. We tried the following tools—Google Tilt Brush, Oculus Quill, Google Blocks and Oculus Medium.
We spoke with Professor Todd Bryant, a VR developer and educator, to understand the current scenario in Virtual Reality and discuss our potential solutions. We also spoke with Serap Yigit, a User Experience Researcher at Google, formerly at Microsoft Hololens, to learn about user research techniques and usability test methods for spatial computing. Accessibility expert Claire Kearney-Volpe guided us to focus on multi-modal interactions and even put us in touch with potential users at ADAPT Community Network, a pioneer in programs and services for people with disabilities. The user interviews at ADAPT gave us insights of how our target users would interact and experience VR. All these opinions were extremely important and helped us validate our design decisions. We also had discussions with the Tobii team, the industry leader in eye tracking to understand the current capabilities of the technology.
Based on our research and interviews, we decided to create a solution that involved multi modal interaction. We decided to test the use of voice, eye gestures and movements to navigate and create in our art tool.
Art Interface
Gaze Tracking
Voice
An intuitive interface to draw, record and share your work
Tracking the eyes for position and gaze point, for drawing and selection
Speech input coupled with an assistant that processes natural language
We created sketches to get a sense of the user interface and how to approach the creation of prototypes. We sketched out some ideas for the menus, drawing and selection zones. We also used 360° grid paper to get an idea of how it might look like in VR.
The idea was to build something simple which we could take to our users and test it out. We listed down the features we wanted to add to the prototype. We were looking to add common and easy to comprehend tools. We came up with a list of tools which included drawing tools, system tools and functionality tools. After this we printed out each tool icon as demonstrated below.
To showcase the vision I have of my project, I created these concept images that illustrate a future where we would be interacting with simulated plants in our personal spaces.
After that we needed to decide on the UI of the tools, we wanted to group the more frequently used tools, to the right for easy access. Based on categories we card sorted the tools and added them to the category they were most relevant in. After this card sorting we taped all the sheets to form the User Interface we wanted to test with.
When our prototype was ready, we wanted to come up with an easy solution to demonstrate gaze and gaze selection. After some quick brainstorming we concluded a laser pen would be the most optimal way to do that. This would give an idea to user where they were pointing on the paper prototype. We had the users perform certain tasks and with a highlighter and a laser pen and we role-played the art creation.
We tested the paper prototype with both accessibility experts and potential users from the NYU Ability Lab to understand if the interface is intuitive enough for the users. We learnt that people who have no experience using art tools found it difficult to navigate and understand what the tools meant.
To showcase the vision I have of my project, I created these concept images that illustrate a future where we would be interacting with simulated plants in our personal spaces.
Based on the results we got from testing we proceeded to create a high fidelity prototype using Unreal Engine. We rebuilt our information architecture by reorganizing the tool structure and finalized our interactions. We included Movement tracking, Time-based gaze selection, Teleport function, and Painting function. Creating a working prototype, which users could interact with in an immersive environment, would offer us more insights and validate our idea.
We tested the prototype with users at the ADAPT Community Network, the Axis Project at the Wheeling Forward Organization, and handsfree with some able bodied people that can potentially have limb injuries.
We incorporated feedback from the second round of user tests in this prototype. Since a lot of our users use wheelchairs and are often restricted in their travel, we added a feature to transport them to different environments. It wasn't very apparent that voice commands could be used too, so we added a visual cue to "Voice Enabled" and made it bolder. We added a name hover for each tool to improve usability and reworked the name of some of the tools for more clarity. We also included in-app onboarding and a voice assistant to help the user.
Our team also awarded a $10000 grant from NYC Media Lab for the XR Startup Bootcamp for this project. The bootcamp ran from September to November 2018 and it helped us create a business model for this application and perform customer discovery and analyze product market fit. I was primarily the product lead for the bootcamp. At the end of the bootcamp, we pitched our project at the Exploring Future Reality event in front of 300+ people where we received immense positive feedback.
This was my first VR Project and I learnt a lot about interactions in VR and how to design a tool that doesn't add to discomfort or motion sickness. VR is a truly impactful technology that can impact lives and I am looking forward to doing more work in VR.
For future iterations, we would aim to increase neck comfort and focus a bit more on ergonomics of using this tool for longer durations. I would place elements in such a way that people who cannot move their head a lot will also be able to use it effectively. I would also like to redesign the UI to improve its visual design and add more 3D elements.
This was also my first foray into accessible design, and it left a lasting impact on my design practice. I learned a lot about the challenges faced by people with disabilities. Learning about the importance of designing for inclusivity was eye-opening and having empathy for all is something I will be carrying forward to future projects.