Yes, Apple did it. It put a 3D camera on the front of the iPhone X – and the technical approach is more complex than anyone had envisioned. This is not the time-of-flight (ToF) array we thought it would be, but a combination of a ToF proximity detector and an infrared “Structured Light” camera that can either use a uniform “flood” or “dot pattern” illumination.
By using this approach, Apple is revolutionizing the consumer electronic user interface. This will have deep implications way beyond the smartphone market. Tim Cook acknowledged from the start of the introductory show for the iPhone X that this was only the beginning of a new area. As Yole Développement has forecast, this kick-starts the 3D camera market, which will grow to be worth $6B by 2022 in the report 3D Imaging and Sensing.
How does it work?
The iPhone X combines an infrared camera with a flood illuminator that projects uniform infrared light in front of the phone. It can then take images, and a face detection algorithm searches for anyone in them. In order to not be working all the time, the infrared camera is most probably linked to the infrared ToF proximity sensor, which has a different wavelength bouncing off anyone in front of the phone. This may be the signal for the camera to take a picture and if it detects a face then the iPhone X activates its dot pattern projector to take an image of the face with this dot pattern on it. It then sends both the regular and dot pattern images to the application processing unit (APU) which puts them through a neural network trained to recognize the owner and therefore unlock the phone. What is special about this approach is that no 3D image is ever computed at this point: the 3D information is contained in the dot pattern image. In order to run 3D applications, the same APU can use another algorithm which computes the depth map of the image. The iPhone X takes advantage of the massive processing power available in the A11 chip, as structured light approaches are known to be computationally intensive. The use of a neural network is the key technology that made it possible.
What is it for?
Firstly, the 3D camera is there to unlock the phone through face detection, face recognition and biometric authentication. Secondly, this is a new interface for short animated emoticon videos, a use that has coined a new word: “animojies”. With a similar goal of visual messaging, 3D “masks” can be virtually applied to someone’s face and a short video posted to social networks like Snapchat. Such an Augmented Reality (AR) application can also be applied to selfie photography, enabling software to add different 3D virtual backgrounds. Lastly, the 3D camera can be applied to photography. Automated photo editing is the main aim, as different lighting configurations can be applied, especially for portraits. The application that we expected but haven’t seen is real time video call enhancement. Due to the computational intensiveness of the approach and the power probably needed for the 3D camera, it seems that the masks are not intended to be used for long durations, as would be the case in virtual chatrooms. This might require a new generation of hardware, as it was not brought to the table at this week’s Apple presentation event.
Source: Yole Développement
Who are the winners?
The suppliers list has not changed much from what we expected. Firstly, the infrared camera, proximity ToF detector and flood illuminator seem to be treated as a single block unit. This is supplied by STMicroelectronics, along with Himax for the illuminator subsystem, and Philips Photonics and Finisar for the infrared-light vertical-cavity surface-emitting laser (VCSEL). Then, on the right hand of the speaker, the regular front-facing camera is probably supplied by Cowell, and the sensor chip by Sony. On the far right, the “dot pattern projector” is from ams subsidiary Heptagon. This last component is probably the most exotic one, as the iPhone X is the first smartphone to use it. It combines a VCSEL, probably from Lumentum or Princeton Optronics, a wafer level lens and a diffractive optical element (DOE) able to project 30,000 dots of infrared light.
What about artificial intelligence (AI)?
The introduction of AI technology into the iPhone X follows Apple’s ethos to protect user privacy. The facial recognition software emerges from deep learning techniques, trained with a billion images. Usually, these advanced algorithms, including Siri, run on the cloud on high-end computing devices which obviously needs an internet connection on the user side. In this case, and because Apple wanted to avoid sending personal data off-device, the software runs on the iPhone itself, on the “edge” of the network. This explains the need for a powerful computing chip implemented as a pair of cores in the A11 bionic processor. These two cores, which Apple calls a “neural engine”, are fully dedicated to handling this type of computation. The first computation skill benchmark tests show that this technology is far ahead of other chips on the market. Apple’s move to open its iPhone to AI is impressive, both on the software side and the dedicated hardware side. Of course, it is only the first step and a lot of detailed tuning will follow. However, because AI’s promise is fantastic in so many domains, and because of Apple’s huge influence on the market, these choices will determine the future of the whole industry.
What are the limitations? What is the next step forward?
As everyone has said, this is only the beginning of a new era, but Apple has shown how to overcome a lot of 3D cameras’ previous shortcomings using its AI software and hardware. The limitation of structured light is the computational load on the APU and the difficulties it may have in handling sunlight. As the 3D camera is not searching for faces all the time, the addition of a small proximity ToF detector to tell it when to wake up is key. The next step forward should be full ToF array cameras. According to the roadmap Yole has published this should happen before 2020. 2017 has become the year of 3D and AR for the iPhone, but for the most part it has been the year of AI. After a few years of being surpassed by competition in the specification race, Apple has finally got back to the genius it became famous for. Ten years after the iPhone’s introduction, Apple has been able to bring back Steve Jobs’ spirit into a very pricy and attractive product.
In 1999 Pierre Cambou joined the imaging industry. He had earned an Engineering degree from Université de Technologie de Compiègne in parallel to a Master of Science from Virginia Tech in 1998. More recently he graduated from Grenoble Ecole de Management’s MBA. Pierre took several positions at Thomson TCS which became Atmel Grenoble in 2001 and e2v Semiconductors in 2006. In 2012 he founded the start-up Vence Innovation (now called Irlynx) in order to bring to market a disruptive Man to Machine interaction technology. He joined Yole Développement, the “More than Moore” market research and strategy consulting company, as Imaging Activity Leader in 2014.
Guillaume Girardin works as a Market & Technology Analyst for MEMS devices and technologies at Yole Développement, the “More than Moore” market research and strategy consulting company. Guillaume holds a Ph.D. In Physics and Nanotechnology from Claude Bernard University Lyon 1 and a M.Sc. in Technology and Innovation Management from EM Lyon School of Business.
As a Software & Market Analyst, Dr. Yohann Tschudi is a member of the MEMS & Sensors business unit at Yole Développement (Yole). Yohann is daily working with Yole’s analysts to identify, understand and analyze the role of the software parts within any semiconductor products, from the machine code to the highest level of algorithms. Market segments especially analyzed by Yohann include big data analysis algorithms, deep/machine learning, genetic algorithms, all coming from Artificial Intelligence (IA) technologies.
After his thesis at CERN (Geneva, Switzerland) in particle physics, Yohann developed a dedicated software for fluid mechanics and thermodynamics applications. Afterwards, he served during 2 years at the University of Miami (FL, United-States) as a research scientist in the radiation oncology department. He was involved in cancer auto-detection and characterization projects using AI methods based on images from Magnetic Resonance Imaging (MRI). During his research career, Yohann has authored and co-authored more than 10 relevant papers.
Yohann has a PhD in High Energy Physics and a master degree in Physical Sciences from Claude Bernard University (Lyon, France).
Beyond its traditional medical and industrial markets, 3D imaging & sensing is ready to conquer consumer and automotive, with an exponential growth pattern starting from $1.3B in 2016 and reaching $9B in 2022. Get more here.
3D imaging, gas sensing and autofocus will fuel the IR light source market to reach US$3.8B in 2027. Get more here.