Support Vector Machine

1- Introduction

Support Vector Machine is most used algorithms for classification and regression in Artificial Intelligence world. Support Vector Machine is an implementation for the problem of pattern recognition, for the problem of regression, and for the problem of learning a ranking algorithm.

Support Vector Machine’s algorithms have many application areas. These are used for classification of images, text categorization, handwriting recognition, face detection and cancer detection. In addition, mask detection is used for COVID-19 nowadays.

2- Support Vector Machine

Support Vector Machine (SVM) which is a classifier derived from statistical learning theory by Vapnik and Chervonenkis, is a supervised Machine Learning method for classification and regression. It can solve linear and non-linear problems and can lead to high performance in many practical problems. Understanding its working principle is easy. Separates data into classes with a line or plane. Currently considered one of the most efficient family of the algorithms in Machine Learning.

There are 2 types of SVM:

Linearly Separable and Non-Linearly Separable

2.1- Linearly Separable

Using the training data set, support vectors are plotted for the data set that can be linearly separated as shown in figure 1. Too many support vectors can be plotted in between to datasets but finding optimal support vector’s location is very important. The closest point between classes is chosen as the support vector. The distance which is between two support vectors is called as margin. SVM produces number of decision margins where the best margin is identified by using perceptron algorithm. Also, hyperplane is drawn into between two support vectors, and it is “margin/2” distance from to two support vectors.

Datasets are usually never linearly separable like figure 1 in the real project so 100% effectively classified by a hyperplane will never be met. In such cases, Soft Margin Method is used. Properties of this method allow SVM to make a certain number of mistakes as shown as figure 2 and margin as wide as possible so that accuracy rate of model is increased.

2.2- Non-Linearly Separable

Datasets like figure 3 that cannot be separated linearly or have a lot of noise when separated by soft margin are called non-linear support vector machine. Kernel Method is used for these and similar data sets. The aim of this method that when the dataset is inseparable in the current dimensions, add another dimension to separate data as shown as figure 4. The most used types of Kernel Methods are Polynomial and Radial Basis Function (RBF).

2.2.1- Polynomial Kernel

Polynomial kernel is a part of non-linearly separable and widely used for Speech Recognition. Example of polynomial kernel is as given as figure 5.

The Polynomial kernel is formulated as:

2.2.2- Radial Basis Function

Also known as a Gaussian Kernel, Radial Basis Function (RBF) kernels are often used in Computer Vision. Given RBF example of figure 6 and figure 7 are two example that describe the importance of the γ variable in the formula.

The RBF Kernel function is formulated as: