# A Practical Guide to Support Vector Classi

Topics: Support vector machine, Machine learning, Data set Pages: 14 (3972 words) Published: April 19, 2013
A Practical Guide to Support Vector Classiﬁcation
Chih-Wei Hsu, Chih-Chung Chang, and Chih-Jen Lin Department of Computer Science National Taiwan University, Taipei 106, Taiwan http://www.csie.ntu.edu.tw/~cjlin Initial version: 2003 Last updated: April 15, 2010 Abstract The support vector machine (SVM) is a popular classiﬁcation technique. However, beginners who are not familiar with SVM often get unsatisfactory results since they miss some easy but signiﬁcant steps. In this guide, we propose a simple procedure which usually gives reasonable results.

1

Introduction

SVMs (Support Vector Machines) are a useful technique for data classiﬁcation. Although SVM is considered easier to use than Neural Networks, users not familiar with it often get unsatisfactory results at ﬁrst. Here we outline a “cookbook” approach which usually gives reasonable results. Note that this guide is not for SVM researchers nor do we guarantee you will achieve the highest accuracy. Also, we do not intend to solve challenging or diﬃcult problems. Our purpose is to give SVM novices a recipe for rapidly obtaining acceptable results. Although users do not need to understand the underlying theory behind SVM, we brieﬂy introduce the basics necessary for explaining our procedure. A classiﬁcation task usually involves separating data into training and testing sets. Each instance in the training set contains one “target value” (i.e. the class labels) and several “attributes” (i.e. the features or observed variables). The goal of SVM is to produce a model (based on the training data) which predicts the target values of the test data given only the test data attributes. Given a training set of instance-label pairs (xi , yi ), i = 1, . . . , l where xi ∈ Rn and y ∈ {1, −1}l , the support vector machines (SVM) (Boser et al., 1992; Cortes and Vapnik, 1995) require the solution of the following optimization problem: min 1 T w w+C 2 l

w,b,ξ

ξi
i=1

subject to

yi (wT φ(xi ) + b) ≥ 1 − ξi , ξi ≥ 0. 1

(1)

Table 1: Problem characteristics and performance comparisons. Applications #training #testing #features #classes Accuracy data data by users 3,089 391 1,243 4,000 04 41 4 20 21 2 3 2 Accuracy by our procedure 75.2% 96.9% 36% 85.2% 4.88% 87.8%

Astroparticle1 Bioinformatics2 Vehicle3

Here training vectors xi are mapped into a higher (maybe inﬁnite) dimensional space by the function φ. SVM ﬁnds a linear separating hyperplane with the maximal margin in this higher dimensional space. C > 0 is the penalty parameter of the error term. Furthermore, K(xi , xj ) ≡ φ(xi )T φ(xj ) is called the kernel function. Though new kernels are being proposed by researchers, beginners may ﬁnd in SVM books the following four basic kernels: • linear: K(xi , xj ) = xT xj . i • polynomial: K(xi , xj ) = (γxi T xj + r)d , γ > 0. • radial basis function (RBF): K(xi , xj ) = exp(−γ xi − xj 2 ), γ > 0. • sigmoid: K(xi , xj ) = tanh(γxi T xj + r). Here, γ, r, and d are kernel parameters.

1.1

Real-World Examples

Table 1 presents some real-world examples. These data sets are supplied by our users who could not obtain reasonable accuracy in the beginning. Using the procedure illustrated in this guide, we help them to achieve better performance. Details are in Appendix A. These data sets are at http://www.csie.ntu.edu.tw/~cjlin/papers/guide/ data/ Courtesy of Jan Conrad from Uppsala University, Sweden. Courtesy of Cory Spencer from Simon Fraser University, Canada (Gardy et al., 2003). 3 Courtesy of a user from Germany. 4 As there are no testing data, cross-validation instead of testing accuracy is presented here. Details of cross-validation are in Section 3.2. 2 1

2

1.2

Proposed Procedure

Many beginners use the following procedure now: • Transform data to the format of an SVM package • Randomly try a few kernels and parameters • Test We propose that beginners try the following procedure ﬁrst: • Transform data to the format of an SVM package • Conduct...