Understanding Anomaly Detection Using Autoencoders In Python

Introduction

Anomaly Detection (or outlier detection) is getting more attention in Machine Learning because of its relevance to various fields ranging from fraud detection, to system health monitoring, to detecting irregular patterns in health conditions. Autoencoders, a type of neural network, are growing popular for Anomaly Detection tasks due to their ability to learn efficient representations of data.

What is an Autoencoder?

Autoencoders are neural networks that aim to copy their inputs to their outputs. They work by compressing the input into a latent-space representation, and then reconstructing the output from this representation.

######
            ----> Encoding ----> Latent Space Representation ----> Decoding ----> Output
######

In this blog post, we will be demonstrating how to implement Anomaly Detection using Autoencoders in Python.

Python Implementation of Autoencoders

Let's use a simple dense layer autoencoder on the popular Iris dataset for anomaly detection. We will be using the tensorflow and keras libraries in Python.

Import the necessary libraries

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split 
from sklearn.preprocessing import StandardScaler 
from tensorflow.keras.models import Model 
from tensorflow.keras.layers import Input, Dense 
from tensorflow.keras import regularizers

Load the dataset

from sklearn import datasets

iris = datasets.load_iris()
iris_data = iris.data
df = pd.DataFrame(data=iris_data, columns=iris.feature_names) 

Now, partition this data into train and test data. Standardize it because autoencoders require data that live on a similar scale.

train, test = train_test_split(df, test_size=0.2) 

scaler = StandardScaler() 
scaler.fit(train) 

train = pd.DataFrame(scaler.transform(train), columns=train.columns, index=train.index) 
test = pd.DataFrame(scaler.transform(test), columns=test.columns, index=test.index)

Define the Autoencoder

input_layer = Input(shape=(train.shape[1],)) 
encoded = Dense(100, activation='tanh', activity_regularizer=regularizers.l1(10e-5))(input_layer)
encoded = Dense(50, activation='relu')(encoded)

decoded = Dense(50, activation='tanh')(encoded)
decoded = Dense(100, activation='tanh')(decoded)

output_layer = Dense(train.shape[1], activation='relu')(decoded)

autoencoder = Model(input_layer, output_layer)
autoencoder.compile(optimizer="adadelta", loss="mse")

Fit the model to the training data.

autoencoder.fit(train, train, 
                batch_size = 256, epochs = 10, 
                shuffle = True, validation_split = 0.20)

That’s it! You have built an Autoencoder for Anomaly Detection. The low dimensional space built by the first half of the Autoencoder allows for efficient data representation that helps in detecting any irregularities in new unseen data.

This code should serve as a streamlined process to implement and simplify the workings of autoencoders for anomaly detection for beginners in Machine Learning. Keep experimenting with layers, activation functions and the architecture of the autoencoder for deeper understanding. Happy computing!