Here’s another slightly different scenario.

You’ve trained a product recommendation model based on users’ click and purchase behavior on your ecommerce site.

On Black Friday, your server responsible for transactions and payments goes down whilst the web server remains up and running, so the model thinks that no one who clicks is buying anything.

It’s impossible to have models unlearn things that have already been learned but one thing you can do

is roll back the model’s state to a time prior to the data pollution

Of course, in order to do this, you will need infrastructure that

automatically creates and saves models as well as their meta information.

Here’s another scenario. You’ve trained a static product recommendation model which alone will determine which products users see when they are on the home page and when they are viewing individual products.

The model works by using purchasing behavior of other users.

After deploying it, user session time and conversion rate initially increase.

But, in the months that follow the release of the model, conversion rate and user session time steadily decline to slightly below the levels they were at before the launch of the model.

What went wrong?

Well, your model is not updating to

new users,

new products,

and new patterns in user preference.

Because the model only knows about

your older products,

it continues to recommend them long after they’ve fallen out of favor.

Ultimately, users simply ignored the recommendations altogether,

and made do with the site’s search functionality.

This “cold start” problem is common for this sort of recommendation model.

We’ll talk more about recommendation systems later in the course.

The solution here is to

dynamically retrain your model on newer data,

and also to understand the limits of your model.

Here’s one other scenario. You’ve deployed a statically-trained fraud detection model and its performance starts off good

but quickly degrades.

What’s gone wrong here?

In adversarial environments, where one party is trying to beat another, it’s particularly important to dynamically retrain the model, to keep up with the most recent strategies.