The mirror with a vision for style

Imagine you’re out shopping for clothes. Those moments in the fitting room, when you try to figure out whether an outfit looks right on you, can be tough. You’ve got to decide whether the clothes fit for a start, but you also have to judge whether they fit your personal style. You might like something on the rack, but is it really you? A team of researchers is designing a smart mirror that can help you make your decision, but teaching computers how to see stylishly is no easy task.

Online in real life

The team, based in the USA and Japan, began by noticing a difference between buying clothes in a shop and buying them online. While it’s impossible to try on clothes you buy over the web, one advantage to online shopping is that it’s a lot easier to find recommendations. Click on a pair of jeans, for example, and you’re likely to see a few other pairs that are similar, plus some recommendations on what shirts might go with them. Plus, if you want, you can ask to see the most popular items – maybe you’d been looking at skinny jeans when everyone else was buying flares. The researchers wanted to find a way to combine the ability to try on your clothes with the recommendations a shopper can get online. And while we’re at it, wouldn’t it be nice to compare one outfit with another?

That’s how they came up with the Responsive Mirror. Stand in front of it and in the middle you’ll see a mirror like any other. But next to it are two display units. One shows you all the other clothes you’ve tried on, so you can compare the different looks. The other shows you other looks that people have put together, and allows you to choose between seeing similar outfits to yours, or something different if you think you’re in a bit of a style rut. As a bonus, the Responsive Mirror doesn’t just sense what clothes you’re wearing. It can sense your pose when you look in it, and show you other outfits in the same position you’re standing in.

Get the look

This means that the researchers have had to teach the mirror how to see. That’s no easy task – we take vision for granted, but computers find it a lot harder. First, the mirror needed some eyes, so they hooked a computer-controlled camera up to it. (We should probably point out that the mirror and camera setup isn’t in the same room where you get changed, so you don’t need to worry about your modesty.) Next, the mirror needed some vision programming. There were two big things the mirror needed to do: recognise the clothes a shopper is wearing, and how they are posed when they look in the mirror.

To realise just how tough it would be for a computer to match clothing items, think about how many characteristics a shirt might have. Is it a smart shirt, a casual t-shirt, or somewhere in between like a polo shirt? What colour is it? Does it have cuffs? A solid colour or stripes? And how thick are those stripes, by the way? With all those different possibilities, recognising the shirt and finding similar (or different) examples is a tall order. Oh, and not only that, but in order to detect the shirt in the first place the computer has to figure out where the person’s torso is.

Fashionable dinosaurs

Here’s how the computer figures out where in its field of vision the person is: the same way the dinosaurs in Jurassic Park did. They detect motion. If you find the moving part of the image, that’s where the person is. The computer draws a box around the whole person, and then uses the proportions of the human body to draw a reasonably accurate box around the torso, and that’s where the shirt is. See how complicated this vision stuff can get?

Next the computer needs to find out the details about the shirt, so it knows what else to recommend. Colour is an easy one. The texture is a bit more difficult, but the computer can take a small area of the shirt and try to find similar patches in other shirts, so that’s not too bad. How can it recognise whether a shirt is formal or casual, though? That’s a very complex distinction, and one that you might think only a human could tell. Fortunately, the clue is mainly in the collar. Smart shirts have collars with lots of pointy corners, but casual shirts don’t. So if you can train the computer to look for ‘pointyness’ around the top of the shirt, you’ve got a big clue. And it turns out you can! The maths of pointy corners has already been figured out, and it’s been used to detect edges in computer images for more than 20 years. So by applying the pointy corner detector to a shirt, suddenly the computer can tell whether it’s likely to go with a lounge suit, or just with lounging around.

Posing for the camera

On top of matching your outfit, the computer has to detect your pose in the mirror. This, too, is like putting together a puzzle. The first part, finding out which direction you’re facing, isn’t too tough. There’s a second camera that takes an overhead view of your body. Since your shoulders give you a sort of oval shape from overhead, the computer assumes that you’re facing the same way that the oval is facing. The second part, matching your exact pose, is more difficult. The camera in front of you, the same one that checks you for different colours and pointy collars, can see your outline. It would be easy for the computer to tell the difference between a normal pose and a really exaggerated one, but people usually do more subtle poses. There’s not much difference in the outline of a person standing straight in front of the mirror, and another person, say, crossing their arms and twisting around a bit. But if you combine the views of the two cameras in front and above, so you know the person’s outline and which direction they are facing, you can get it right. That way the Responsive Mirror can show you people wearing similar clothes in similar poses to you.

Standing in front of the Responsive Mirror, a shopper can see other outfits they’ve worn, and compare them to looks that other people have put together. That makes real-world shopping more like online shopping, and could help you walk out of the store a more satisfied customer. Upscale shops like Prada in New York already have similar systems, but as time goes on, who knows – you might be using it on your high street as well. You’ll have an extra secret though: you’ll know just how much complex computer vision goes into the mirror’s magic.