Most computer vision task could use more data.

And so data augmentation is one of the techniques that is

often used to improve the performance of computer vision systems.

I think that computer vision is a pretty complicated task.

You have to input this image,

all these pixels and then figure out what is in this picture.

And it seems like you need to learn the decently complicated function to do that.

And in practice, there almost all competing visions task having more data will help.

This is unlike some other domains where sometimes you can get enough data,

they don't feel as much pressure to get even more data.

But I think today, this data computer vision is that,

for the majority of computer vision problems,

we feel like we just can't get enough data.

And this is not true for all applications of machine learning,

but it does feel like it's true for computer vision.

So, what that means is that when you're training in computer vision model,

often data augmentation will help.

And this is true whether you're using transfer learning or

using someone else's pre-trained ways to start,

or whether you're trying to train something yourself from scratch.

Let's take a look at the common data augmentation that is in computer vision.

Perhaps the simplest data augmentation method is mirroring on the vertical axis,

where if you have this example in your training set,

you flip it horizontally to get that image on the right.

And for most computer vision task,

if the left picture is a cat then mirroring it is though a cat.

And if the mirroring operation

preserves whatever you're trying to recognize in the picture,

this would be a good data augmentation technique to use.

Another commonly used technique is random cropping.

So given this dataset,

let's pick a few random crops.

So you might pick that,

and take that crop or you might take that, to that crop,

take this, take that crop and so this

gives you different examples to feed in your training sample,

sort of different random crops of your datasets.

So random cropping isn't a perfect data augmentation.

What if you randomly end up taking that crop which will look much like

a cat but in practice and worthwhile so long as

your random crops are reasonably large subsets of the actual image.

So, mirroring and random cropping are frequently used and in theory,

you could also use things like rotation,

shearing of the image,

so that's if you do this to the image,

distort it that way,

introduce various forms of local warping and so on.

And there's really no harm with trying all of these things as well,

although in practice they seem to be used a bit less,

or perhaps because of their complexity.

The second type of data augmentation that is commonly used is color shifting.

So, given a picture like this,

let's say you add to the R,

G and B channels different distortions.

In this example, we are adding to

the red and blue channels and subtracting from the green channel.

So, red and blue make purple.

So, this makes the whole image a bit more purpley and that

creates a distorted image for training set.

For illustration purposes, I'm making

somewhat dramatic changes to the colors and practice,

you draw R, G and B from some distribution that could be quite small as well.

But what you do is take different values of R,

G, and B and use them to distort the color channels.

So, in the second example,

we are making a less red,

and more green and more blue,

so that turns our image a bit more yellowish.

And here, we are making it much more blue,

just a tiny little bit longer.

But in practice, the values R,

G and B, are drawn from some probability distribution.

And the motivation for this is that if maybe the sunlight was

a bit yellow or maybe the in-goal illumination was a bit more yellow,

that could easily change the color of an image,

but the identity of the cat or the identity of the content,

the label y, just still stay the same.

And so introducing these color distortions or by doing color shifting,

this makes your learning algorithm more robust to changes in the colors of your images.

Just a comment for the advanced learners in this course,

that is okay if you don't understand what I'm about to say when using red.

There are different ways to sample R, G, and B.

One of the ways to implement color distortion uses an algorithm called PCA.

This is called Principles Component Analysis,

which I talked about in the

ml-class.org Machine Learning Course on Coursera.

But the details of this are actually given in the AlexNet paper,

and sometimes called PCA Color Augmentation.

But the rough idea at the time PCA Color Augmentation is for example,

if your image is mainly purple,

if it mainly has red and blue tints,

and very little green,

then PCA Color Augmentation,

will add and subtract a lot to red and blue,

where it balance [inaudible] all the greens,

so kind of keeps the overall color of the tint the same.

If you didn't understand any of this, don't worry about it.

But if you can search online for that,

you can and if you want to read about the details of it in the AlexNet paper,

and you can also find some open source implementations of the PCA Color Augmentation,

and just use that.

So, you might have your training data stored in a hard disk and uses symbol,

this round bucket symbol to represent your hard disk.

And if you have a small training set,

you can do almost anything and you'll be okay.

But the very last training set and this is how people will often implement it,

which is you might have a CPU thread that is constantly loading images of your hard disk.

So, you have this stream of images coming in from your hard disk.

And what you can do is use maybe a CPU thread to implement the distortions,

yet the random cropping,

or the color shifting, or the mirroring,

but for each image,

you might then end up with some distorted version of it.

So, let's see this image,

I'm going to mirror it and if you also implement colors distortion and so on.

And if this image ends up being color shifted,

so you end up with some different colored cat.

And so your CPU thread is constantly loading data as well as implementing

whether the distortions are needed to form a batch or really many batches of data.

And this data is then constantly passed to some other thread or some other process for

implementing training and this could be done on the CPU or really

increasingly on the GPU if you have a large neural network to train.

And so, a pretty common way of implementing

data augmentation is to really have one thread,

almost four threads, that is

responsible for loading the data and implementing distortions,

and then passing that to some other thread or

some other process that then does the training.

And often, this and this,

can run in parallel.

So, that's it for data augmentation.

And similar to other parts of training a deep neural network,

the data augmentation process also has a few hyperparameters such as how much

color shifting do you implement and exactly what parameters you use for random cropping?

So, similar to elsewhere in computer vision,

a good place to get started might be to use

someone else's open source implementation for how they use data augmentation.

But of course, if you want to capture more in variances,

then you think someone else's open source implementation isn't,

it might be reasonable also to use hyperparameters yourself.

So with that, I hope that you're going to use data augmentation,

to get your computer vision applications to work better.

Data Augmentation

Convolutional Neural Networks

Meet the Instructors