Support Vector Machinery for Infinite Ensemble Learning
- Creators
- Lin, Hsuan-Tien
- Li, Ling
Abstract
Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of some base hypotheses. Nevertheless, most existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. Thus, it is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of hypotheses. In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on the support vector machine (SVM). The framework can output an infinite and nonsparse ensemble through embedding infinitely many hypotheses into an SVM kernel. We use the framework to derive two novel kernels, the stump kernel and the perceptron kernel. The stump kernel embodies infinitely many decision stumps, and the perceptron kernel embodies infinitely many perceptrons. We also show that the Laplacian radial basis function kernel embodies infinitely many decision trees, and can thus be explained through infinite ensemble learning. Experimental results show that SVM with these kernels is superior to boosting with the same base hypothesis set. In addition, SVM with the stump kernel or the perceptron kernel performs similarly to SVM with the Gaussian radial basis function kernel, but enjoys the benefit of faster parameter selection. These properties make the novel kernels favorable choices in practice.
Additional Information
© 2008 Hsuan-Tien Lin and Ling Li. We thank Yaser Abu-Mostafa, Amrit Pratap, Kai-Min Chung, and the anonymous reviewers for valuable suggestions. Most of the work was done in 2005, in which Hsuan-Tien Lin was supported by the Caltech Center for Neuromorphic Systems Engineering under the US NSF Cooperative Agreement EEC-9402726, and Ling Li was sponsored by the Caltech SISL Graduate Fellowship.Attached Files
Published - LINjmlr08.pdf
Files
Name | Size | Download all |
---|---|---|
md5:24f10baf8abea710d0f8aa4962fe9569
|
284.3 kB | Preview Download |
Additional details
- Eprint ID
- 11153
- Resolver ID
- CaltechAUTHORS:LINjmlr08
- Caltech Center for Neuromorphic Systems Engineering
- EEC-9402726
- National Science Foundation Cooperative Agreement
- Caltech SISL Graduate Fellowship
- Created
-
2008-07-18Created from EPrint's datestamp field
- Updated
-
2019-10-03Created from EPrint's last_modified field