Overnight, Apple held their Worldwide Developer Conference and the big announcement was their entry and response to the Large Language Models and Generative AI. The past 18 months Apple had sat and watched other players like OpenAI, Google, Microsoft and Meta took the lead in delivering new services to customers leveraging.
Apple’s announcements today show a dramatic increase in capability in iOS 18, led by improvements.
While many expected Apple to spend part of their large cash balance on building our massive hardware infrastructure to train their own LLMs, it seems they’ve taken a different approach and partnered with OpenAI to deliver much of the smarts we now expect from services like ChatGPT, Gemini, Copilot and Meta.AI.
Apple’s iOS 18 will be available later this year (around September) and that will come with some serious new capabilities across iPhones, iPads and Macs.
Writing tools, powered by Apple Intelligence will be available across the OS, from email, to your notes app, to the web browser and accelerate your standard workflows, all for free.
Apple Intelligence also features generative AI for image generation. An example Apple showed was in iMessage, where one of your contacts was having a birthday. It took their profile photo and created an avatar of them with a cake and balloons in the background.
On the surface, this looks like a bit of fun and some users may have been doing this with 3rd party services already, saving out the profile photo, uploading it to a 3rd party service and asking for it to use that as a foundation to generate a new image from, using a text prompt as guidance.
By Apple offering to do this task natively, it helps speed up the task for the user. This story is repeated for many common tasks you do across your devices and apps and is really where human+AI reveals its biggest opportunities.
Apple also promoted AI-powered actions. This allows you to request the phone to do something on your behalf, that has multiple steps in your workflow and it’ll take that request, break it into chunks and perform the tasks. Some examples of this are ‘pull up the files that Joz shared with me last week’, ‘play the podcast my wife sent me the other day’.
The big difference here is that you can request it in natural language, with the parts of information you remember, taking out many of the manual steps we do dozens of time per day, this really should accelerate common workflows.
With OS-level integration, Apple Intelligence has context about you, your data and from your request, the objective. We’ll only really know how well this works once we get hands-on, but it does raise a few questions.
If you have your corporate email connected, does the request and resulting actions all get processed on-device, or does some of it get passed to Apple servers? Apple says it is doing this locally, however if you make a request for an email your boss sent you last week and have a 7 day sync on mail, does that request simply fail?
When detailing how Apple Intelligence works, they explain there are times where you want to use a larger model on the web. Apple says they have created something called Private Cloud Compute. This allows larger models to run Apple datacenters. This sounds like Apple has a copy of OpenAI’s ChatGPT 4o LLM, as Microsoft does with CoPilot. Where your requests for data exceeds what’s on your phone, it would then reach out and use the Apple Private Cloud Compute (i.e. LLM in the cloud).
I don’t think Apple did a good job of reassuring consumers or employers of when data is leaving the device and while they said users are in control, we didn’t see what those controls looked like. Is it an all or nothing toggle, or an app-by-app approval? Do they use your data to train their LLM in the cloud?
We hope Apple provides much more clarity throughout WWDC and especially before this is released to consumers.
A few years ago, voice assistants were all the rage and Siri was Apples entry into that space. Over the years the service didn’t evolve in line with competitors and certainly today, compared against the capabilities to respond to questions was a joke compared to an LLM.
Thankfully Siri is getting a big upgrade. In iOS18, you’ll use either your voice or now text input to request information from Siri. You can ask it information and you’ll know its active thanks to a new colourful UI that surrounds the display.
Your requests are contextually aware, that means it understands what’s on your screen and if you make requests, it’ll assume you’re talking about what’s on the screen at that time.
Apple demonstrated a number of use cases for the new Siri, one of the more impressive was a multi-step workflow where they say ‘add this address to his contact card’. From within the Messages app, it understood that a recent message contained an address, extracted that, identified who the sender was, looked up their contact card, edited it, and added the address and saved it.
With examples like this, it’s easy to see how this can save you time, as these processes and workflows that we do often, can be reduced from 30 seconds, to 3 seconds and through a simple voice interface. This is the power of LLMs on display for the world to see and will soon be available for free to iOS users with compatible hardware.
One of the best opportunities here is to be working on your computer, and have your phone in a stand on the desk beside you. As you have a thought, you simply ask your phone to do it and you continue on. Normally when we multi-task, we really just shift to a different task and shift back, but this actually provides the opportunity to do multiple things at once. At times the voice assistant may provide help and acceleration to what you’re working on, or it could be working on a completely different set of problems as an agent for you. The question is, how many agents can you have working for you at any given time?
If you missed WWDC 2024, you can watch it on demand below. Apple Intelligence starts at 01:04:30.
