Microsoft’s new Copilot will change Office documents forever

It feels like Microsoft is slowly building on the vision it had for its Cortana assistant or even Clippy decades before. “I love that our heritage is full of products that try to adapt to people,” says Friedman. “Copilot shares some similarities with some things we’ve done in the past, but it’s far more capable, it’s humble, and it’s there to serve things up for you that help you save time.”

Microsoft has customized this Copilot system for every Office app, so there are different ways to command it. Friedman demonstrated to me how Copilot can help you write emails in Outlook, offering up short or long message drafts with options to change the tone. It even works in the mobile version of Outlook, which got me thinking about the ways this could speed up work on the go.

“Outlook mobile is the first place where we’re doing a big push,” explains Friedman. Outlook can summarize all your emails on the go, generate drafts, and generally make it easier to triage your inbox. But imagine creating entire Word documents from your phone without having to type on a tiny on-screen keyboard. “We’ll have more to talk about mobile in the coming months,” says Friedman. But you can imagine where things will go.

As impressive as Copilot is, we’ve seen the myriad ways that large language models can fail, including inserting racial or gender bias into text and simply making things up. Those traits are alarming enough in a search engine, but when you’re talking about Excel (which arguably powers the world’s economy) or your email inbox, it’s a whole different level of ethics, privacy, and data concerns.

“It gets things right a lot of the time but not all of the time,” admits Friedman. “In the user experience we do things like put in affordances to not send something until you’ve read it, or to encourage you to try again, edit, and discard.”

Microsoft also has a number of warnings inside Copilot that appear as you use it. In PowerPoint, you’ll see a message that says, “Content is generated by AI and might contain inaccuracies for sensitive material. Be sure to verify information.” Elsewhere, there are prompts that say, “AI-generated content may be incorrect.” Microsoft is trying to design the system in a way that reminds you that you’re in charge.

“We give you tools to report it when it’s wrong. We create prompt suggestions to help you write good prompts. Everything we’re doing in the user experience is to make it conversational and give you agency,” says Friedman.

We’ve seen what happens when it goes wrong in Microsoft’s Bing search engine. The AI-powered chatbot has hallucinated on multiple occasions, and Microsoft has had to put limits in place to control its outbursts. In one conversation with The Verge, Bing even claimed it spied on Microsoft’s employees through webcams on their laptops and manipulated them.

“Everything we’re learning from Bing in preview is helping us mitigate those risks,” says Friedman. “We’re applying all that learning and thinking to Copilot as well.” Microsoft is also starting off small with its Copilot rollout. It will initially be available to just 20 businesses before Microsoft opens it up to more when it’s ready. Microsoft is also starting with enterprise customers first before rolling it out to consumers.

“We feel pretty good about what we have as a starting point, but we don’t yet know if it’s performing the way we want it to and helping really empower people to accomplish their jobs,” says Friedman. “We’ll quickly iterate, we’ve been building fast, very fast. But we pause and we learn a lot and we’ll update really fast. Our plan is to move as fast as we possibly can to scale to more of enterprise in a thoughtful and responsible way and make sure the experience is awesome.”

Is Microsoft moving too fast, though? Google announced its own AI features for Gmail and Docs earlier this week, and the AI race has many experts worried that tech giants aren’t properly considering the impact of these new tools.

“In our mind we’re being thoughtfully quick,” says Friedman. “We’re being thoughtful in that we’re rolling it out to 20 customers and working side by side with them.” Despite reports of Microsoft laying off its ethics and society team that taught employees how to make AI tools responsibly, Friedman says Microsoft is growing the people working on these concerns. “In terms of our investment in ethics and AI, we’ve grown more and more ethics and AI experts in all of the product teams working on this stuff,” says Friedman. “We have to scale way bigger, so we’ve been investing more heavily and it continues to grow year over year.”

Microsoft knows Copilot isn’t perfect and that it’s going to take some time to get there. Although it impressed me during the Teams meeting recap, it could have easily confused my voice for someone else’s if I’d been using a bad microphone, or Outlook could pull out the wrong summary in an email thread. There are big challenges ahead, but Microsoft is hoping the work it does on making it easy to edit responses, correct sources, and issue feedback will ultimately improve the system.

“We know AI gets things wrong, we know it hallucinates and we know it does it confidently,” admits Friedman. “We continue to work on making it better at doing that less, but also that the user experience really empowers people and puts them in the driver’s seat.”

For all the challenges, the future of the Copilot system won’t just be text-based generation, either. Microsoft has a clear vision to use Copilot to generate images, video, and more once large language models can handle these features well.

Microsoft has already integrated OpenAI’s DALL-E model into its Designer app, which allows people to generate images based on text. Designer will also help PowerPoint select the best imagery for AI-generated slides. “We’re going to bake Designer further into Copilot so you can change things within Designer,” says Friedman. “The Designer stuff you’ve seen today is just scratching the surface. I fully suspect we’re going to use Copilot to do amazing multimedia things.”

So where else could we see Copilot appear? I asked Friedman about Windows integration. “We’re looking at all sorts of places and ways to expand [Copilot] out. I believe this is the next major wave of computing and it will change the way we work with all devices in the coming years,” says Friedman.