Microsoft’s Build 2015 conference in San Francisco was the focus of a wide variety of application development announcements. While Microsoft’s Universal Windows Platform would rate as one of the most important announcements of the conference, Microsoft’s HoloLens stole the show when it came to visual impact and astonishment. While Microsoft is being unusually tight lipped about when the HoloLens device will be made available or its price, their objective at Build 2015 was to create a spectacle and solicit some carefully orchestrated developer feedback. Although Microsoft refused to talk about what went into the development of HoloLens, they did allow us to demo it in a highly controlled setting.
Under the illusion of extraordinarily tight security that even Obama’s White House would be envious of, I was first allowed to photograph a HoloLens that was carefully suspended in a display case. After shedding all photographic, recording, and communications equipment, I was ushered through multiple sets of handlers, a maze of holding areas, a dedicated tutorial, and optical measurements before being individually escorted into a dimly lit room where I would be fitted for and begin to use a HoloLens. Microsoft’s objective was to create an aura of shock and awe about what they had just developed. In this regard they succeeded.
The purpose of the HoloLens is to project, manipulate, and annotate virtual objects (holograms) within your physical environment. The ability to bridge the gap between the physical world and virtual world is indeed an amazing accomplishment. Holograms can be either 2D or 3D. 2D holograms are resizable cards that are either static images or dynamic objects that can be manipulated by the HoloLens user. These holograms can be pinned to any surface and become a part of the physical environment to the HoloLens user. 3D holograms are even more interesting when inserted into an environment. The magic of HoloLens is that as the user navigates the physical environment, holograms are viewed with flawlessly accurate perspective. This enables the user to effectively evaluate the relationship between the hologram and the physical environment. Since a hologram superimposes a virtual object within the context of a physical environment, it is also possible to use HoloLens to see inside or through physical objects. My demo involved the construction of a new building in the middle of a city. Not only was I able to manipulate the external dimensions of the building hologram to see how it would interact in its physical setting, but where the virtual building intersected with existing physical structures - I was also able to see through walls and evaluate the relationship of existing plumbing, wiring, and construction techniques relative to the design of the new building. When inconsistencies between the physical environment and virtual design were encountered I could attach a card and record a message indicating what the issue was and what should be done to resolve it.
HoloLens operates using one set of stereo optics that monitors your physical environment and a second set of stereo optics that projects holograms to the user. These two domains are joined so that as the user looks and walks around, holograms are continuously adjusted as if they were part of the physical environment. The science behind this is admittedly complex and Microsoft deserves immense credit for this achievement. The HoloLens is controlled by gaze, gesture, and voice. Gaze (where you look) is analogous to moving a mouse and either gesture or voice commands are analogous to selecting (clicking) or typing.
As remarkable as the HoloLens is, it is still very much a work in progress. There are two significant challenges when it comes to intersecting the virtual and physical worlds. The first is the construction of the 3D objects and the second is the integration of these objects with the physical world. When it comes to the development of 3D objects, there are three important attributes to address: shape, texture, and lighting. I’ve listed these attributes in order of increasing complexity. Shape and texture are largely a function of the object model and therefore independent of how the object interacts with the physical world, with the exception of scale and orientation. However, lighting is both a function of where you pin the virtual object to the physical world and its light sources. The HoloLens is able to avoid the use of markers to scale and orient the object to the physical world by leveraging the stereo range finder optics built into the HoloLens. A marker is a physical object (typically a printed sheet of paper) that is well known to the virtual rendering software and is positioned in the physical environment to aid in addressing size and perspective when pinning a virtual object. The issue of lighting is a far more complex subject. In the physical world, lighting helps communicate the dimensionality of an object. The challenge in the virtual world is how to borrow what is understood about light sources in the physical world and then apply it to virtual objects that have been pinned to the physical world. Microsoft has not addressed this issue well and consequently many of the more complex 3D virtual objects shown to us did not integrate all that well with their physical surroundings.
The complexity of developing 3D objects cannot be underestimated. Creating a realistic 3D model is extremely expensive and this will clearly be one of the gating factors that limits the utility of the HoloLens. So the success of HoloLens is largely tied to 3D model development activities that are beyond Microsoft’s control. This situation cannot rest easily with Microsoft. While the use of 3D modeling in business is accelerating due to the improvements in cycle times and product quality, design development is currently an expensive proposition that will slow the adoption of HoloLens.
It’s hard to see HoloLens as a consumer product because there is no material portfolio of 2D and 3D objects that would encourage its use. While it’s less of a stretch to see HoloLens as a commercial product, its utility is determined more by the qualitative assessment made by the user for how the virtual and physical worlds intersect. Since Microsoft still has work to do on how 3D virtual objects are rendered in a physical world and the availability of complex 3D objects is an unknown, adoption is likely to be slow.
However, the next time I’m in the market for a new car, wouldn’t it be nice to simply put on a HoloLens, walk out into my driveway, pull up a 3D rendering of the car I’m interested in, try out different colors and options, walk around it, peer inside it, and see how it would look sitting in my driveway.