The VC-Dimension versus the Statistical Capacity of Multilayer Networks
- Creators
- Ji, Chuanyi
- Psaltis, Demetri
Abstract
A general relationship is developed between the VC-dimension and the statistical lower epsilon-capacity which shows that the VC-dimension can be lower bounded (in order) by the statistical lower epsilon-capacity of a network trained with random samples. This relationship explains quantitatively how generalization takes place after memorization, and relates the concept of generalization (consistency) with the capacity of the optimal classifier over a class of classifiers with the same structure and the capacity of the Bayesian classifier. Furthermore, it provides a general methodology to evaluate a lower bound for the VC-dimension of feedforward multilayer neural networks. This general methodology is applied to two types of networks which are important for hardware implementations: two layer (N - 2L - 1) networks with binary weights, integer thresholds for the hidden units and zero threshold for the output unit, and a single neuron ((N - 1) networks) with binary weigths and a zero threshold. Specifically, we obtain O(W/lnL)≤ d_2 ≤ O(W), and d_1 ~ O(N). Here W is the total number of weights of the (N - 2L - 1) networks. d_1 and d_2 represent the VC-dimensions for the (N - 1) and (N - 2L - 1) networks respectively.
Additional Information
© 1992 Morgan Kaufmann. The authors would like to thank Yaser Abu-Mostafa and David Haussler for helpful discussions. The support of AFOSR and DARPA is gratefully acknowledgedAttached Files
Published - 481-the-vc-dimension-versus-the-statistical-capacity-of-multilayer-networks.pdf
Files
Name | Size | Download all |
---|---|---|
md5:44b6b5edd9e997c07ff7f0ec6ac54d45
|
1.6 MB | Preview Download |
Additional details
- Eprint ID
- 63859
- Resolver ID
- CaltechAUTHORS:20160121-163657790
- Air Force Office of Scientific Research (AFOSR)
- Defense Advanced Research Projects Agency (DARPA)
- Created
-
2016-01-22Created from EPrint's datestamp field
- Updated
-
2019-10-03Created from EPrint's last_modified field
- Series Name
- Advances in Neural Information Processing Systems
- Series Volume or Issue Number
- 4