Pen-based Interaction with Google Sheets in Mobile Virtual Reality

I’m sure for many pro Google Sheets users when you look at a Google Sheet what you see on screen is different for what you ‘see’ in your mind. Researchers from Coburg University, Microsoft Research and the University of Cambridge have potentially bridged this gap showing what Google Sheets can look like virtual reality (VR).

Whilst the video embedded above is not explicitly about Google Apps Script, I think it’s always worth keeping an eye on the horizon to see what the future might bring (you might want to skip through to 05:12 to see what I mean). You can also read the full research paper, in which the researchers explain how they used the Google Sheets API to build the experimental interface in the Unity game engine.

For Google Apps Script developers I’m sure you’ll be able to see how this solution could also benefited from the built-in SpreadsheetApp methods like getCurrentCell() and activate(), rather than using the the Sheet API, which reported as a limitation that haas reported in the research paper:

Since the Google Sheets API exposes no functionality to track client-side interactions, operations of users with the web page were tracked inside Unity. In particular, tracking of cell selection was implemented by constructing virtual-cells in the Unity space using oriented bounding boxes, and spatially position them in their corresponding places to fit the spreadsheet texture. Tracking of the pen and the Unity collision detection mechanism is used to detect whether the pen tip lies inside a certain cell.

Gesslein et al. (2020)