I'm in a tiny room in a basement at Microsoft headquarters in Redmond, Washington, wearing an early version of the HoloLens 2 headset. In front of me is a real ATV, which lacks a bolt. Not quite in the corner of my vision – but certainly to the side – I see a glowing indicator pointing to a bucket of right screws. I go over, bend down to take a closer look at the shelf, and grab one.
Back at the ATV hovers a holographic guide that tells me what to do and points to where the bolt is needed. After a few minutes, I successfully repaired the thing ̵
This kind of demo is becoming commonplace for tech journalists like me. However, if you read the above description carefully, you will find that three key elements of technical innovation are hidden.
Here they are: I have seen a hologram to the side because the field of view in which they can appear is much larger than before. I leaned down and was not worried that an awkward headset was moving because it was more balanced on my head. I pushed a button by pressing a button because I did not have to learn a complicated gesture to operate the HoloLens 2.
These three things may not seem very remarkable to you, but that's the point. Microsoft had to give the HoloLens a more natural feel if it really wants to get people to use it, and it does.
There's another unobtrusively remarkable thing: Although it was just a demo, I played the role of a worker, because that's the HoloLens 2 – for workers, not for consumers.
The Microsoft HoloLens 2 can now be pre-ordered for $ 3,500 and is expected to ship later this year. However, Microsoft has decided that it will only be sold to corporate customers who want to deploy the headset to their employees. At the moment, Microsoft does not even announce a version of the HoloLens 2 for the Developer Kit.
Compared to the HoloLens we saw for the first time four years ago, the second version is better in almost every important respect. It is more comfortable, has a much wider field of vision, and can better recognize real physical objects in the room. It includes new components such as the Azure Kinect sensor, an ARM processor, eye-tracking sensors and a completely different display system.
It has a couple of speakers, the visor flips up and lets see what makes your hands more accurate than before. There is an 8 megapixel front camera for video conferencing. The camera can track up to 6 degrees and also uses USB-C for charging. In short, it's full of new technology. But after four years, that should not come as a surprise.
The biggest complaint about the first HoloLens was simple: they just saw the holograms in a relatively small box right in front of you. Turn your head a little and they would disappear from view. Worse, their edges would disappear from existence, even if you stared directly at them. It was like looking at a digital world through a small rectangle.
The HoloLens 2 has a field of view that is twice as big as before. It does not fill your entire field of view – there are still clippings – but it is now big enough that you no longer feel annoyed by a mailbox. Microsoft says that every eye has the equivalent of a 2K display, but it's better to think of it as a metaphor rather than a precise specification. The exact specification is that it has a "holographic density of 47 pixels per degree", which means that the pixel density is high enough to read 8-point fonts.
Usually, when a tech product gets better specifications In these cases, it's purely technical iteration: faster processors, bigger batteries, more RAM, and so on. But this strategy would not have worked for the display of HoloLens 2. She had to get lighter, not heavier. Microsoft had to switch completely to a different kind of display technology.
Lasers and mirrors
Laser-based displays are the right choice for computers in the face. The Vaunt project from Intel used lasers, as well as the intelligent glasses from North Focals. Although Microsoft uses some of the same basic components, they have gone in a different direction and gone much further in developing their capabilities.
The lasers in the HoloLens 2 glow in a set of mirrors that oscillate as fast as 54,000 cycles per second, so that the reflected light can draw an indication. These two parts together form the basis of a display for a microelectromechanical system (MEMS). It's all tricky, but the really tricky part of a MEMS display is getting the picture you've painted into your eyeball.
One solution that companies like North have used is a holographic film on the lens where the image is directly reflected in your retina that has many disadvantages: a tiny display and a low resolution for two. The problem is, however, that the ad is aimed directly into your eye. They have to be custom-made for the North goggles, and the image may disappear completely if they are misaligned.
Microsoft does not want any of these issues, so it worked the same way as the first HoloLens: Waveguides. It's the glass parts in front of your eye that are carefully etched so they can reflect the holograms in front of your eyes. HoloLens 2 waveguides are now lighter because Microsoft uses two sandwich glass plates instead of three.
When you assemble the entire system – the lasers, the mirrors and the waveguide – you get a brighter display, a wider field of vision that does not have to be right in your eyes to work. Zulfi Alam, General Manager of Optics Engineering at Microsoft, believes Microsoft is ahead of the pack with this system and waveguides are definitely the way to go for mixed reality. "There is no competition for the next two or three years that can achieve this accuracy in the waveguides," he argues.
Want a larger field of view? Easy. Simply increase the angle of the mirrors that reflect the laser light. A larger angle means a bigger picture.
Would you like brighter pictures? Again, simply lasers, so as not to be too subtle, have light light. Of course you have to deal with the fact that waveguides lose a ton of light, but the displays I saw were set to 500 nits and looked very bright to me. Microsoft thinks it could be much brighter in the final version depending on power consumption.
Would you like to see the holograms without being customized for your headset? Just again. The waveguide does not require any special adaptation or measurement. You can just put the headset on and get started. It can also sit far enough in front of your eyes so you can wear the glasses you want comfortably.
Simple, simple, easy, right? In truth, it is devilishly complex. Microsoft had to create a completely new etching system for the waveguides. It had to find out how the light in the waveguides should be directed almost photon by photon in the right place. "We simulate every photon that comes from the laser," says Alam. The light of the laser is not just reflected. It is split in several colors and by several "pupils" in the display system and then "reconstituted" in the right places on the waveguides. "Every photon is calculated where it is expected," says Alam. That takes a lot of computing power so Microsoft had to develop a custom silicon to do all the calculations of where the photos would go.
And although alignment with the waveguide is much easier, this is not the case. It means it's perfect. That's why there are two tiny cameras on the nose bridge that are aimed at your eyeballs. This allows the HoloLens 2 to automatically measure the distance between your pupils and adjust the image accordingly. With these cameras, the HoloLens 2 can also adjust the image vertically if it is tilted or if your eyes are not perfectly even. (This is not the case.)
One of the free advantages of these cameras is that they can also scan your retina to safely log you into the HoloLens 2. After all, Windows is running and therefore supports Windows Hello. You also keep track of where you are looking, which will allow some new user interactions, which I will see below.
Then there's performance: lasers, oscillating mirrors, and custom chips that do the computing for all that needs to be chewed by the battery. But Alam tells me that, despite all this, it still manages to use less energy than the alternative. The mirrors resonate, requiring less energy to move them, as if they were the fastest metronomes. Lasers are also less lossy than LEDs, and customized silicon can be optimized for its specific task.
"Our evolution comes down to a form factor that is really glasses," says Alam, "and all of these are important steps in this direction. "
This technology is certainly impressive, but I do not want to surpass the image quality. What I used was not a finished product. I saw a tiny halo around some of the holograms, and sometimes they jumped around a bit. Most functions based on nasal bridge eye scanners have not yet been activated. Compared with the first HoloLens, I saw what I saw beyond the limit of "cool demo, which I use for 20 minutes and then upset", up to "I could see people using this for a few hours if that Software was really useful. "
But if you're using a headset for" a few hours, "it must feel good enough to come first.
How to put on the HoloLens 2: They dress like a baseball cap, turn a button on the back to tighten the headband , and then you see holograms. The end.
It's a lot less fiddly than the last HoloLens or any other face-mounted display I've ever tried. Because of all the work on the display system, you can skip the extra step to make sure you can see the picture. The body of the thing is also simpler. It is a single band that is held on the back of the head and forehead with minimal pressure. (There is an optional upper strap if required.)
All of this is nice, but it makes no sense if the headset is uncomfortable to wear. And although I've never had it for more than 20 minutes, I think it will take longer.
Microsoft has a Human Factors lab showing off its collection of dummy human heads and high-speed cameras. Carl Ledbetter, Senior Director of Design for the Microsoft Device Design Team, took me through all the prototypes and the material Microsoft has been trying to get into the final product. He explained how Microsoft experimented with different designs and materials, eventually landing on carbon fiber to save weight.
"The Reality [we have to] is suitable for children, adults, men, women and various ethnic groups around the world. Every head is different, "he says. Microsoft has a database of about 600 heads that track the shape of the skull, the depth of the eye, the size and relative position of the bridge of the nose, and other variations. Ledbetter's team attached sensors to the neck to measure the muscle load to make sure the center of gravity was correct.
The result is that the HoloLens 2 has a forgiving and more flexible fit. It's just better to accommodate the basic physical human realities. You can flip the visor up so it's out of your field of view so you can make eye contact without removing the headset. The memory foam pad resting on the forehead is removable and cleanable, and the thermals have been completely redesigned so that the heat is conducted away from the head.
That really helps, but the most important thing Microsoft did was move the center from gravity right behind your ears instead of your eyes. The HoloLens 2 is not really that much lighter than the original HoloLens. It however, feels lighter because it is more naturally balanced on your head. This balance makes a big difference. The weight is less noticeable and should strain your neck less.
Ledbetter moved the weight by moving the heaviest part in the truest sense of the word: main processor and battery are now in a module that sits on the back of the headset. The cables inside the headband run to the front of the display board and the components. Incidentally, this processor is an ARM-based Qualcomm Snapdragon 850, and that's important because it addresses another fundamental human reality: we hate it when the battery is empty and we do not mind it. An ARM processor may have a smaller processor battery.
The original HoloLens ran on an Intel processor and on Windows. Since then, Microsoft has done a lot of work for Windows to work well on ARM. These efforts are slowly being realized on laptops, but Intel is still on the agenda when raw speed is usually more important to the user than battery life. In general, there are tensions with Intel. It does not deliver the lower power chips that mobile devices demand. According to reports, Intel even had to work for Microsoft to keep the Surface Go on the chips.
So, what about HoloLens 2? Alex Kipman is responsible for the entire HoloLens project. He says that "ARM ruled in battery-powered devices. The decision of the ARM became pretty easy. If you need a battery, [it’s] it will be hard to find a product that does not run an ARM today.
When I point out that many Windows laptops run on batteries with Intel chips, it gets confused. "Intel does not even have a SoC [system on chip] for this type of battery powered product. They had one, the previous version [of the HoloLens] had Cherry Trail, which they had set, however. This decision is a breeze.
For workers, not for consumers
The HoloLens 2 is sold only to businesses, not to consumers. It is designed for what Kipman calls a "first-line worker," in car dealerships, factories, operating theaters, and field staff. It is intended for people who work with their hands and have difficulty integrating a computer or smartphone into their daily work. Kipman wants to replace the greasy Windows 2000 computer in the corner of the workspace. It's pretty much the same Google decision Google Glass has made.
"If you think of over 7 billion people in the world, people like you and me – knowledge workers – are by far the minority," he replies. For him, the workers who will use this "may be people repairing our jet engine. Maybe it's the people who are in some salesrooms. Maybe it's the doctors who operate you in an operating room.
He goes on to say that it is for "people who were neglected in some sense or had no access to technology, because PCs, tablets, phones are not really suited to these experiences. "
Good enough. This fits in perfectly with Microsoft's new focus on meeting business and enterprise needs, rather than trying to convince consumer products. When I interviewed CEO Satya Nadella last year, this was one of my souvenirs, and that's true today. As I wrote back then, it's "a different kind of Microsoft than what we're used to. It's a bit less obvious, but it has the advantage that it's much more likely to succeed. "
Also, says Kipman, even the HoloLens 2 is not good enough to be a true mass-market product. "This is the best, highest watermark of what can be achieved in mixed reality, I'm here to tell you that it's still not a consumer product," he says, continuing:
Why is not it consumer products? It's not as haunting as you want it to be. It is more than twice as intense as the previous one, [but it’s] is still not so strong that the consumer can not use the street. It's still not comfortable enough … I would say until these things are more insistent than the most immersive product, far more comfortable than the most comfortable product and at or under $ 1,000, I think people think of themselves as having this Products are ready.
Kipman says that Microsoft has not participated in the consumer type for these types of products. "We were not the company that VR hoped for. We certainly are not the company that has hired AR. And since we combined the two with the mixed reality and AI efforts, we have not mortgaged. "
That's not exactly right. We've seen many demos from Microsoft demonstrating games – including Minecraft – and other consumer applications for the HoloLens. Therefore, this step in the enterprise market is absolutely a pivot.
However, it is a pivot that belongs to Microsoft's larger enterprise strategy. And just because it's no longer positioned as a consumer product does not mean that it's not an important product – for which Microsoft seems to be engaging and developing software.
A Better Interface for Your Face
First, HoloLens users had to learn awkward gestures with names like "Air Tap" and "Bloom". These really specific hand gestures had to be made as these were the first HoloLens sensors they could recognize and understand.
The HoloLens 2 can detect and understand much more, because a new sensor array is used to read the room called Azure Kinect. "Kinect" because this is the brand for Microsoft's cameras that can scan rooms, "Azure", because apparently everything the company does today is somehow linked to its cloud service. As a further signal, this is a business product and not an Xbox add-on. on.
"HoloLens 1 is just a big web. It's like throwing a blanket over the real world, "says Kipman. "With HoloLens 2, we move from spatial mapping to semantic understanding of space. You understand what a couch is, what a person is sitting on the couch, what is the difference between a window and a wall. "
I can not say how good Kinect actually is to identify objects – Microsoft did not show it all to us – but it works theoretically because Azure Kinect sees the room at a higher resolution and because he's attached to cloud services that help him figure things out.
There is one aspect in which I can definitely say that the higher fidelity is real: it is able to recognize my hand and what makes it much easier. It can track up to 25 articulation points on both hands in space, so you no longer need to use the Air Tap gesture to interact with holograms.
In a demo I went up and down in a room and looked at different holograms that were placed on tables. When I reached my hands, a box of small handles appeared around each of the edges and corners. I could just reach in, grab the whole box and move the hologram. I could also just grab one edge to rotate it, or two to resize it. If there was a button, I could reach out and press my finger. I doubt it's accurate enough to type on a virtual QWERTY keyboard, but it's still a big step up from the first generation.
Eye tracking also plays a role in the interaction with holograms. The HoloLens 2 can recognize where you are looking and use that information as a kind of user interface. There were demos where I was staring at a small bubble to create a holographic firework, but the most useful was a car scroller. The closer I was to the bottom of the page, the faster the words rolled, but then stopped when I looked back.
I have not seen the full top-level UI, so I do not know if that will change. But one thing is absolutely not: Windows is still running. It uses the shared code in Windows OneCore. This means that you will not get a traditional Windows desktop shell. However, you can run any Universal Windows app on it. It also has the necessary drivers so you can connect a keyboard and mouse via Bluetooth if you really want to.
Chaitanya Sareen, the main group manager for Microsoft Mixed Reality, explains that they try to "make the machine work around the person and vice versa." Sareen calls this "instinctive interaction" as opposed to "intuitive" as it is different from which can replace what we already do with real objects in the world. "Is someone born saying," There will be a closing button [in the upper corner of a window] "? No, "he says. "We've learned many interfaces."
Sareen is still considering some UI details, but the goal is to use many of the natural gestures you've taken as a toddler instead of learning a whole new interface language.
Microsoft also provides developers with new software tools. One of the most important, Dynamic 365 guides, will be a mixed reality app with templates for creating instructions for fixing real things like this ATV. Other tools depend on Microsoft's cloud services. One of these is Azure Remote Rendering, which allows HoloLens compute load to load into the cloud. It exists because the HoloLens 2 can only store and render a limited kind of details, for example, for a 3D rendering of an engine locally. With remote rendering, some of the details can come from the cloud in real time, potentially displaying infinite details, allowing you to model and interact with the smallest parts of a holographic machine.
Finally there are Azure Spatial Anchors. It allows you to attach holograms to real world locations. Basically, it does not differ much from what Apple and Google already do in augmented reality: multiple devices can see and interact with the same virtual object. However, Microsoft's ambitions are much more remarkable: it wants to create the infrastructure for a number of world-scale holograms, and tools are being developed that allow developers to use that infrastructure across platforms, including iOS and Android.
For solutions that need more GPS location and object recognition only. Kipman speaks a lot about the distinction between boring conference rooms located on different floors in the same location. Tracking objects in space using optics is known to be difficult. Walk around a building in a circle and your position drifts so that the computer does not put your endpoint at the starting point. It is a bit unclear how Microsoft actually came to solve these problems, but they are actively working on it.
Alex Kipman believes we are at the abyss of the "third era of the computer". Second, there were PCs with their open ones. Secondly, phones with walled garden app stores came onto the market, and now he hopes that mixed-reality headsets will swing the pendulum back in the open as Microsoft seeks to keep HoloLens open. The HoloLens work with the cloud services of Microsoft, but would also work with other ecosystems. Kipman says the HoloLens and Azure are "loosely coupled but closely aligned."
I could do more than shame with his summary of the history of computing and point out that there are a number of outsiders who are calling for openness, but the bigger issues are: Microsoft believes mixed reality is a big one Deal is.
To understand what Microsoft's plans have been doing lately, there has been much more jargon than before. In particular, with HoloLens 2, expect much discussion about "time-to-value" (how quickly a user can do something useful after receiving a device from an employer) and "Intelligent Edge" (non-self-contained devices) connected to the cloud).
Cognitive dissonance prevails in all these conversations for ordinary consumers. On the contrary, there is a lot of hype surrounding HoloLens 2. It is now targeted at companies only. Some of it is deserved. I think the HoloLens 2 is a technical marvel. Just because it's not sold as a terminal does not mean it's not an important piece of technology either. This could change our idea of what a computer should look like.
But we're used to consumer electronics companies doing their best to put such technical marvels on shop shelves and translate that hype into gadgets in our pockets and in our heads.
For HoloLens 2, the hype is not the case about personal technology. It's only for business.