The basics of modern AI—how does it work and will it destroy society this year?
Today’s AI is technically “weak”—but it’s complex and can have a great societal impact.
AI, or artificial intelligence, is huge right now. “Unsolvable” problems are being solved, billions of dollars are being invested, and Microsoft even hired Common to tell you how great its AI is with spoken word poetry. Yikes.
As with any new technology, it can be hard to cut through the hype. I spent years doing research in robotics and UAVs and “AI,” but even I’ve had a hard time keeping up. In recent years I’ve spent a lot of time learning to answer even some of the most basic questions like:
- What are people talking about when they say AI?
- What’s the difference between AI, machine learning, and deep learning?
- What’s so great about deep learning?
- What kind of formerly hard problems are now easily solvable, and what’s still hard?
I know I’m not alone in wondering these things. So if you’ve been wondering what the AI excitement is all about at the most basic level, it’s time for a little peek behind the curtain. If you’re an AI expert who reads NIPS papers for fun, there won’t be much new for you here—but we all look forward to your clarifications and corrections in the comments.
What is AI?
There’s an old joke in computer science that goes like this: what’s the difference between AI and automation? Well, automation is what we can do with computers, and AI is what we wish we could do. As soon as we figure out how to do something, it stops being AI and starts being automation.
That joke exists because, even today, AI isn’t well defined—artificial intelligence simply isn’t a technical term. If you were to look it up on Wikipedia, AI is “intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and other animals.” That’s about as vague as you can get.
Generally, there are two kinds of AI: strong AI and weak AI. Strong AI is what most people might be thinking of when they hear AI—some god-like omniscient intelligence like Skynet or Hal 9000 that’s capable of general reasoning and human-like intelligence while surpassing human capabilities.
Weak AIs are highly specialized algorithms designed to answer specific, useful questions in narrowly defined problem domains. A really good chess-playing program, for example, fits this category. The same goes for software that’s really accurate in adjusting insurance premiums. These AI setups are impressive in their own way but very limited overall.
Hollywood aside, today we aren’t anywhere close to strong AI. Right now, all AI is weak AI, and most researchers in the field agree that the techniques we’ve come up with to make really great weak AIs probably won’t get us to Strong AI.
So AI currently represents more of a marketing term than a technical one. The reason companies are touting their “AIs” as opposed to “automation” is because they want to invoke the image of the Hollywood AIs in the public’s mind. But… that’s not completely wrong. If we’re being gracious, companies may simply be trying to say that, even though we’re nowhere near strong AI, the weak AIs of today are considerably more capable than those of only a few years ago.
Any marketing instincts aside, that’s actually true. In certain areas, in fact, there has been a steep change in capability in machines, and that’s largely because of the two other buzzwords you hear a lot: machine learning and deep learning.
Machine learning
Machine learning is a particular way of creating machine intelligence. Let’s say you wanted to launch a rocket and predict where it will go. This is, in the grand scheme of things, not that hard: gravity is pretty well understood and you can write down the equations and work out where it will go based on a few variables like speed and starting position.
But this gets unwieldy when you’re looking at something where the rules aren’t as clear and well-known. Say you want a computer to look at pictures and you want to know if any of them shows a picture of a cat. How do you write down rules to describe what every possible combination of whiskers and cat ears looks like from every possible angle?
The machine learning approach is well-known by now: instead of trying to write down the rules, you build a system that can figure out its own set of internalized rules after being shown a lot of examples. Instead of trying to describe cats, you would just show your AI a lot of pictures of cats and let it figure out what is and is not a cat.
This is perfect for our present world. A system that learns its own rules from data can be improved by more data. And if there’s one thing we’ve gotten really good at as a species, it’s generating, storing, and managing a lot of data. Want to be better at recognizing cats? The Internet is generating millions of examples as we speak.
The ever-increasing tide of data is one part of why machine learning algorithms have been blowing up. The other part has to do with how to use the data.
With machine learning, besides the data there are two other, related questions:
- How do I remember what I’ve learned? On a computer, how do I store and represent the relationships and rules I’ve extracted from the example data?
- How do I do the learning? How do I modify the representation I’ve stored in response to new examples and get better?
In other words, what’s the thing that’s actually doing the learning from all this data?
In machine learning, the computational representation of the learning that you store is called the model. The kind of model you use has huge effects: it determines how your AI learns, what kind of data it can learn from, and what kind of questions you can ask of it.
Let’s take a look at a really simple example to see what I mean. Say we’re shopping for figs at the grocery store, and we want to make a machine learning AI that tells us when they’re ripe. This should be pretty easy, because with figs it’s basically the softer they are, the sweeter they are.
We could choose some samples of ripe and unripe fruits, see how sweet they are, then put them on a graph and fit a line. This line is our model.
Look at that! The line implicitly captures the idea of “the softer it is, the sweeter it is” without us having to write it down. Our baby AI doesn’t know anything about sugar content or how fruits ripen, but it can predict how sweet a fruit will be by squeezing it.
How do we train our model to make it better? We can collect some more samples and do another line fit to get more accurate predictions (as we did in the second image above).
Problems become immediately evident. We’ve been training our fig AI on nice grocery store figs so far, but what happens if we dump it in a fig orchard? All of a sudden, not only is there ripe fruit, there’s also rotten fruit. They’re super soft, but they’re definitely not good to eat.
What do we do? Well, it’s a machine learning model, so we can just feed it new data, right?
As the first image below shows, in this case we’d get a completely nonsense result. A line simply isn’t a good way to capture what happens when fruit gets too ripe. Our model no longer fits the underlying structure of the data.
Instead, we have to make a change and use a better, more complex model—maybe a parabola or something similar is a good fit. That tweak causes training to get more complicated, because fitting these curves requires more complicated math than fitting a line.
This is a pretty silly example, but it shows you how the kind of model you choose determines the learning you can do. With figs, the data is simple so your models can be simple. But if you’re trying to learn something more complex, you need more complex models. Just as no amount of data would let the line-fit model capture how rotten fruit behaves, there’s no way to do a simple curve that fits to a pile of images and get a computer vision algorithm.
The challenge of machine learning, then, is in creating and choosing the right models for the right problems. We need a model that is sophisticated enough to capture really complicated relationships and structure but simple enough that we work with it and train it. So even though the Internet, smartphones, and so on have made tremendous amounts of data available to train on, we still need the right models to take advantage of this data.
And that’s precisely where deep learning comes in.
Deep learning
Deep learning is machine learning that uses a particular type of model: deep neural networks.
Neural networks are a type of machine learning model that uses a structure similar to neurons in the brain to perform computations and make predictions. Neurons in neural nets are organized in layers: each layer performs a set of simple computations and passes on the answer to the next.
Stacking layers lets you perform more complex computations. A simple network with a few layers of neurons is enough to replicate the line or parabola we used above. Deep neural networks are neural networks with lots and lots of layers—dozens or even hundreds, hence the name. With this many layers, you can create incredibly powerful models.
This representational power is a huge part of why deep neural networks have been so popular recently. They are able to learn all kinds of complexities without having to have a human researcher specify the rules, and this has let us create algorithms to solve all kinds of problems computers were bad at before.
But there’s another aspect of neural networks that has contributed to their success: training.
A model’s “memory” is a set of numerical parameters that govern how it generates answers to the questions it’s asked. So training a model means tweaking those parameters so that your model produces the best answers it can.
With our fig model, we were finding the equation of a line. This is a simple regression problem, and there are formulas that can give you the answer in a single step.
This is trickier with more complicated models. A line and a parabola are easily represented with a few numbers, but a deep neural net could easily have millions of parameters, and the dataset it’s being trained on could run into the millions of examples as well. There’s no one-step analytical solution.
Luckily, as the Internet might say, there is one weird trick: you can start with a bad neural network and then improve it by making incremental tweaks.
Training a machine learning model in this way is like giving a student a series of tests. Each time you get a grade by comparing what the model thinks the answers are with the “correct” answers in your training data. From there, you make an improvement and then test again.
So how do you know what parameters to tweak, and how much? Neural networks have a nice property where, for many types of learning problems, not only can you get a test score, you can also compute exactly how much the test score will change in response to changing each parameter (math insert: the score is your value function, and with neural networks and most value functions you can easily compute the gradient of your value function with respect to the parameter space).
So now, you know exactly which way to adjust the parameters to increase the score, and you can keep tweaking in successive steps in better and better “directions” until you come to a point where you can’t improve anymore. This is often called hill-climbing because it’s, well, like climbing a hill: if you just keep heading uphill, you’ll eventually come to a peak.
This makes it easy to improve a neural network. If you have a network with a good structure, you don’t have to start from scratch if you have new data. You can start with the parameters you had, and you can then retrain with new data. Your network will get incrementally better. Some of the most notable AI today—from Facebook’s cat image recognition to the stuff Amazon (likely) uses to enable cashier-less grocery shopping—build upon this simple fact.
This is the key to another reason deep learning has spread so far and so fast: hill-climbing makes it easy to take neural networks that have been trained for one purpose and adapt them to another different but similar purpose. Did you train an AI that’s pretty good at spotting cats? You can use it to train an AI that spots dogs or giraffes without having to start from scratch. Start with the cat AI, score it on finding dogs, and then hill-climb to get better!
This is why AI has had a sudden step-change in capabilities in the past five or six years. A few pieces have come together in, dare I say it, synergistic fashion: the Internet has generated huge amounts of data to train on. Computing, especially parallel computing with graphics processing units (GPUs), has made it feasible to work with these huge datasets. Finally, deep neural networks have made it possible to take advantage of these datasets to create tremendously powerful machine learning models.
And all that means some things that used to be really hard are now pretty easy.
So what can we do now? Object recognition
Probably the area where deep learning has had the deepest (forgive me) and most immediate impact is in computer vision—in particular, recognizing objects in pictures. A few years ago, this XKCD comic was a perfect encapsulation of the state of the art.
Today, recognizing birds and even specific types of birds is a trivial task that could be done by a motivated high school student. What’s changed?
The idea behind recognizing objects visually is easy to describe but hard to execute: complicated objects are built up from collections of simpler objects, which are in turn composed of simpler shapes and lines. Faces are made of eyes and noses and mouths, for instance, which are in turn made of circles and lines and so on.
Recognizing faces then becomes a matter of recognizing patterns in which eyes and mouths are arranged, which might require recognizing eye and mouth shapes from lines and circles. It’s turtles all the way down.
These patterns are called features, and until deep learning came along, recognition was a matter of coming up with features manually and programming computers to look for them. For example, there’s a famed face detection algorithm called Viola-Jones that (conceptually) relies on the fact that the brow and nose are usually lighter than the pits of the eyes, so they form a bright, T-shaped pattern with two dark spots. The algorithm essentially works by looking for these T-shapes.
Viola-Jones works well and is blazingly fast, and it’s the basis for the face detectors you see in cheap cameras and the like. But obviously, not every object you want to recognize lends itself to this kind of reduction, and people came up with ever more complex and lower-level patterns to look for. It took teams of PhDs to get vision algorithms to work right, and they were brittle and prone to failure.
The big breakthrough came with deep learning, in particular a kind of neural network called a convolutional neural network. Convolutional neural networks, or CNNs, are deep networks with a particular structure that was inspired by the visual cortex of the mammalian brain. Ars’ Timothy Lee recently published an excellent in-depth exploration on the subject of CNNs that I highly recommend reading for more technical flavor. But I’m going to touch a little bit on the conceptual reasoning here.
This structure of CNNs allows them to learn their own hierarchy of lines and patterns to recognize objects instead of having a PhD spend years figuring out what the right features are. For example, a CNN trained on faces would learn its own internal representations for lines and circles, which are aggregated to eyes and ears and noses, and so on.
CNNs work amazingly well for computer vision, and soon researchers were able to train them to perform well in all sorts of visual recognition tasks, like picking cats out of a picture or identifying pedestrians in a self-driving car’s cameras.
While that’s great, there’s another reason CNNs have spread so fast and so widely: how easily they can be adapted. Remember hill-climbing? If our high school student wanted to recognize a particular bird, they can take one of the many open source vision networks out there and train it on their own dataset without having any real understanding of how the underlying math works.
Naturally, there are even some ways you can extend this further.
Who dat? (Facial recognition)
Let’s say you want to train a network that recognizes not just faces in general, but one face in particular. Now, you could train a network to recognize a particular, specific person, and then another, and so on. But training networks takes time, and it means you’d have to retrain the network for each new person. Yikes.
Instead, what you do is start with a network that has been trained to recognize faces in general. It will have neurons that are now conditioned to recognize all of the structures of faces: eyes, ears, mouths, and so on. Next, you simply change the output: instead of having it recognize specific faces, you have it spit out a description of the face in the form of hundreds of numbers that might capture the curve of the nose or the shape of the eyes and so on. The network can do this because it already “knows” what the components of a face are.
Of course, you’re not specifying any of that directly. Instead, you train a network by showing it sets of faces and then comparing the outputs. You also train it so that it will give descriptors for images of the same face that are close to each other and descriptors for different faces that are far apart. To put it more mathematically, you train the network to create a mapping from images of faces into a point in a feature space where cartesian distance between points can be used to determine similarity.
Once you’ve trained this network, recognition is pretty easy. You take your original face and get its description. Then you take a new face and compare the description the network spits out with your original, and if they’re close enough you say they’re the same face. You’ve now gone from a network that can recognize one face to something that can be used to recognize any face!
This kind of structural flexibility is another reason deep neural networks are so useful. A huge variety of machine learning models for computer vision have been developed, and while they’ve been expanded in all different directions many of their basic structures are based on early convolutional networks like Alexnet and Resnet.
I have even anecdotally heard of people using vision networks on time-series data of sensor measurements. Instead of coming up with a custom network to analyze the data stream, they trained an open source neural network for vision to literally look at the shapes of lines on graphs.
This kind of flexibility is great, but not infinite. So to solve some other problems, other types of networks are necessary.
What’d you say? (Speech recognition)
Image recognition and computer vision aren’t the only areas where AI has had a renaissance. Another area where computers have gotten a lot better is in speech—especially when it comes to transcribing the sound of spoken human speech into words.
The basic idea with speech recognition is actually kind of similar in principle to vision: recognize complicated things as collections of simpler features. In the case of speech, recognizing sentences and phrases comes from recognizing words, which comes from recognizing sequences of syllables, or more accurately, phonemes. So when someone says “Bond, James Bond” we’re really listening for BON+DUH+JAY+MMS+BON+DUH.
In vision, features are organized spatially, which is what the structure of convolutional networks is meant to capture. In sound, these features are instead organized in time. People can speak slowly or quickly, without clear starting and stopping points. We want a model that can listen to sounds as they come, as a human would, rather than waiting and looking at complete sentences. Unlike in physics, we can’t quite just say space and time are the same and leave it at that.
Recognizing individual syllables is pretty easy, but syllables in isolation are tricky. “Hello there” can sound a lot like “hell no they’re,” for example. So for any given sequence of sounds, there’s usually several combinations of syllables that might have been actually spoken.
To clear that up, what we need is to be able to look at sequences in context. If I hear some sounds, is it more likely the person said “hello there dear” or “hell no they’re deer?” That’s where machine learning comes in again. With a large enough sample set of spoken words, you can learn what the most likely phrases are. And the more examples you have, the better you can get.
To do this, people use something called Recurrent Neural Networks, or RNNs. In most neural networks, like for example the CNNs for computer vision, the connections between neurons flow in one direction from input to output (mathematically, they’re directed acyclic graphs). In a recurrent neural network, the outputs of neurons can be fed back into neurons in their own layer, into themselves, or even further upstream. This allows recurrent neural networks to possess memory. (If you’re familiar with binary logic, it’s similar to the way flip-flops work.)
A CNN is one-shot: feed it an image and it spits out some kind of description. An RNN instead maintains some internal memory of what it’s been given before and spits out responses based on what it’s already seen plus what it’s currently seeing.
This memory property of RNNs enables them to not only “listen” to syllables as they come in one after another. It allows the network to learn what kind of syllables come together to form a word and also how likely certain extended sequences are.
Using RNNs, it’s possible to get really good transcription of human speech—to the point that by some measures of transcription accuracy, computers can now perform better than humans. Of course, sounds aren’t the only places where sequences show up. These days, RNNs are also used to identify sequences of movements to recognize actions in video.
Show me your best moves (Deep Fakes and Generative Networks)
So far, we’ve been talking about machine learning models for recognition: tell me what’s in this image, tell me what this person said. But these models can do more than that—today’s deep learning models can also be used to create content.
This is what comes into play when people talk a lot about “Deep Fakes”—incredibly realistic fake videos and images created using deep learning. A while back, a German TV personality generated a major political controversy when he created a video that seemed to show the Greek finance minister giving Germany the middle finger. That video took a team of editors working for a TV program, but now we’re looking at a world where that can be done in minutes by anyone with access to a mid-tier gaming computer.
It’s heavy stuff but not all doom and gloom—above is my personal favorite example of this kind of technology.
This team built a model that can take video of dance moves from one person and create video of someone else doing those same moves, magically imbuing them with expert-level ability. The associated paper is well worth a read.
You can imagine that, with all the techniques we’ve discussed, it’s possible to train a network that takes an image of a dancer and tells you where their arms and legs are. And if that’s the case, then clearly at some level the network has learned how to relate pixels in an image to the position of limbs of a person. Given that, unlike an actual biological brain, a neural net is just data residing on a computer, it surely must be possible to take this data and go the other way—to get pixels from a limb position.
Start with a network that extracts poses of people in images
The machine learning models that do this kind of thing are called generative models. Up until now, the kinds of models we’ve been talking about are called discriminative models. You can think of the difference in this way: a discriminator model for cats looks at pictures and discriminates between pictures with cats and those without. A generative model generates pictures of cats given, say, a description of what kind of cat.
Generative models to “draw” pictures of objects are built using the same CNN structures as the models we trained to recognize those objects. And these generative models can be trained in much the same way as other machine learning models.
But the tricky thing here is coming up with the “score” used to train them. Training a discriminator model has easy right and wrong answers, like if it tells you a cat was a dog. But how do you grade whether a drawing of a cat is good or not, or how accurate it is?
This is where, if you’re a conspiratorial, our-future-is-doomed kind of person, it gets pretty scary. You see, the best way we’ve come up with to train generative networks isn’t to do it ourselves. Instead, we use another neural network.
The technique is called Generative Adversarial Networks or GANs. You have two neural networks that are working against each other: you have a network that is trying to generate the fakes, for example taking a dancer’s arm and leg positions and drawing a new dancer. Then there’s another network that is trained to try to tell the difference between real and fake examples using a bunch of examples of real dancers.
The two networks are then set against each other in a competitive game. That’s where the “Adversarial” part of the name comes in. The generative network tries to make convincing fakes, and the discriminator network tries to figure out what’s real and what’s not.
In the case of the dancing video, the training process involved creating a separate discriminator network that did have an easy yes/no answer. It would look at an image of a person, plus a description of limb positions, and then decide if the image was a “real” original image or one drawn by the generative model.
Over repeated rounds of training, both models got better and better. It’s like pairing an expert jewelry forger against an expert appraiser—by squaring off against a capable opponent each gets stronger and smarter. Finally, when the models got good enough, the generative model could be taken and used on its own.
Generative models, once trained, can be really useful in creating content. For example, they can make up pictures of faces (which can then be used to train face detection and other algorithms), or they can do the job of creating backgrounds for video games.
Getting this to work correctly requires a good amount of tuning and tweaking, but fundamentally the human’s role is that of an arbiter. It’s the AIs that are working against each other to make the primary improvements.
So, can I expect Skynet and Hal 9000 soon?
Every nature documentary has a bit at the end where they talk about how all of the majestic beauty they’ve just showed you is going up in smoke because Human Beings Are Terrible. And in a similar vein, I think every responsible discussion around AI should include a bit about both its limitations and social consequences.
[Editor’s note: This is that bit.]
First, let’s re-emphasize AI’s current limitations: if there’s one thought I hope you’ve taken away from reading this, it’s that the success of machine learning or AI is critically dependent on the models we choose to represent the learning. If humans build these networks poorly or use skewed materials for training, suddenly such biases can play out in very public ways.
Deep neural networks are immensely flexible and powerful, but they’re not magic. Despite the fact that you might be using deep neural networks for both RNNs and CNNs, their underlying structure is very different and so far must still be pre-defined by people. So while you can take a CNN for cars and retrain it to recognize birds, you couldn’t take that model and retrain it to understand speech today.
To put it in more human terms, it’s like we’ve figured out how the visual cortex and the auditory cortex work, but we have absolutely no idea how the cerebral cortex works or even really where to start.
That means we’re probably not going to see Hollywood-style God-like artificial intelligence any time soon. But that doesn’t mean AI in its current incarnation can’t and won’t have profound societal impacts.
We often think about AI “replacing us” with a vision of robots literally doing our jobs, but it’s not going to shake out in quite that way. Look at radiology, for example: with the advances in computer vision, people sometimes talk about AI replacing radiologists. We probably won’t ever get to the point where there’s zero human radiologists. But a very possible future is one where, out of 100 radiologists now, AI lets the top 5 or 10 of them do the job of all the rest. If such a scenario plays out, where does that leave the other 90 or so doctors?
Even if the modern incarnation of AI falls far short of its most enthusiastic cheerleaders, it’s going to leave behind lasting consequences. These are problems that we will all have to sort out, so perhaps getting a handle on the basics is the least we can all do to start.
Haomiao Huang is the CTO and co-founder of Kuna, making home security smart and cloud-connected. He built self-driving cars during his undergraduate years at Caltech and, as part of his Ph.D. research at Stanford, pioneered the aerodynamics and control of multi-rotor UAVs. He is deeply grateful to have opportunities to share his love of robotics, computer vision, machine learning, and sensor networks with the Ars community.
Listing image: George Rinhart/Corbis via Getty Images
