Optical salons: two AI apps on the iPad, one that measures for lenses 46% faster and one that styles the frames

The challenge

Placing markers by hand made every measurement slow and open to error. The chain also knew the old app’s optical calculations were wrong somewhere, but not where.

The new app had to run only on the 8th-generation iPads the salons already owned, and send data in the format the large players of the optical industry require.

App 1: measuring for lenses

• A review of the old app: the largest errors sat in the frame face form angle, one of the key values for fitting lenses.
• A new calculation method, worked out with scientists from Wrocław University of Science and Technology and reviewed by a German optometrist. It cuts cylinder placement errors by up to 2 mm.
• Real-time detection of markers and face dimensions on photos taken in the salon, with machine learning models in a Flutter app.
• Five months of optimization, so the analysis runs without waiting on 8th-generation iPads.
• Data exchange in the format the large players of the optical industry require.

How it works

The salon worker only holds the iPad. The app finds the markers and the dimensions of the face on the photo, and calculates the values needed to make the lenses.

Faces vary: asymmetry, scars, beards, drooping eyelids, posture and distance from the camera, across the chain’s whole catalogue of frames. Thousands of tests on real faces taught the image recognition models what matters on a photo and what to ignore.

Results

• 100% of the measurement automated: no markers placed by hand.
• 50 s → 27 s per measurement, 46% faster.
• Cylinder placement errors cut by up to 2 mm.

App 2: the AI stylist

• The consultant takes a photo, and the app removes the background, fixes the light and adds a subtle retouch.
• AI measures the face: its shape, the eyes and their color, the brows and their shape, the hairline and the beauty type, read by dozens of AI agents working together.
• Only then does it choose the best frames, starting with the ones on that salon’s shelves, and renders the customer wearing them, photorealistically, with outfits to switch.
• Two modes: a consultation, where the consultant walks the customer through each feature and corrects the AI in one tap, and a fast track straight to the fitting room.
• Every styling saved with the customer’s profile in the CRM, and a summary with the frames and the offer sent to the customer after the visit.

Under the hood

• Hundreds of 3D landmarks on the face and an estimate of the head’s pose, so the frames sit right even when the customer turns their head.
• Segmentation of hair, skin and background: it removes the background and finds the hairline, which decides how high a frame can go.
• Color read after white balance correction, so the warm light of a salon does not turn blue eyes green or change the skin’s undertone.
• Every frame in the catalog described by its geometry, from lens width and bridge to temple length, and matched to the face measured in millimeters, with the know-how from our measurement app.
• Physically based rendering of the frames: reflections in the lenses, shadows on the nose and cheeks, and hair or ears covering the temples where they would in real life.
• A native iPad app working with our AI backend: the quick steps run on the iPad, the heavy ones on the server, and the consultant sees the result while the customer is still in the chair.

Timeline

• 2022: work on the measurement app starts.
• 2023: the measurement app goes live in the salons.
• Recently: the second app, the AI stylist.

What comes next

The next version drops markers altogether: the app reads the face, eyes and frame in real time and takes every measurement needed to make the glasses.