A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

Katsuyuki HAGIWARA

doi:10.1587/transinf.E97.D.98

A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection

Katsuyuki HAGIWARA

Full Text Views

0

Share
Cite this

Summary :

Regularized forward selection is viewed as a method for obtaining a sparse representation in a nonparametric regression problem. In regularized forward selection, regression output is represented by a weighted sum of several significant basis functions that are selected from among a large number of candidates by using a greedy training procedure in terms of a regularized cost function and applying an appropriate model selection method. In this paper, we propose a model selection method in regularized forward selection. For the purpose, we focus on the reduction of a cost function, which is brought by appending a new basis function in a greedy training procedure. We first clarify a bias and variance decomposition of the cost reduction and then derive a probabilistic upper bound for the variance of the cost reduction under some conditions. The derived upper bound reflects an essential feature of the greedy training procedure; i.e., it selects a basis function which maximally reduces the cost function. We then propose a thresholding method for determining significant basis functions by applying the derived upper bound as a threshold level and effectively combining it with the leave-one-out cross validation method. Several numerical experiments show that generalization performance of the proposed method is comparable to that of the other methods while the number of basis functions selected by the proposed method is greatly smaller than by the other methods. We can therefore say that the proposed method is able to yield a sparse representation while keeping a relatively good generalization performance. Moreover, our method has an advantage that it is free from a selection of a regularization parameter.

Publication: IEICE TRANSACTIONS on Information Vol.E97-D No.1 pp.98-106

Publication Date: 2014/01/01

Publicized

Online ISSN: 1745-1361

DOI: 10.1587/transinf.E97.D.98

Type of Manuscript: PAPER

Category: Artificial Intelligence, Data Mining

Cite this

Copy

Katsuyuki HAGIWARA, "A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection" in IEICE TRANSACTIONS on Information, vol. E97-D, no. 1, pp. 98-106, January 2014, doi: 10.1587/transinf.E97.D.98.
Abstract: Regularized forward selection is viewed as a method for obtaining a sparse representation in a nonparametric regression problem. In regularized forward selection, regression output is represented by a weighted sum of several significant basis functions that are selected from among a large number of candidates by using a greedy training procedure in terms of a regularized cost function and applying an appropriate model selection method. In this paper, we propose a model selection method in regularized forward selection. For the purpose, we focus on the reduction of a cost function, which is brought by appending a new basis function in a greedy training procedure. We first clarify a bias and variance decomposition of the cost reduction and then derive a probabilistic upper bound for the variance of the cost reduction under some conditions. The derived upper bound reflects an essential feature of the greedy training procedure; i.e., it selects a basis function which maximally reduces the cost function. We then propose a thresholding method for determining significant basis functions by applying the derived upper bound as a threshold level and effectively combining it with the leave-one-out cross validation method. Several numerical experiments show that generalization performance of the proposed method is comparable to that of the other methods while the number of basis functions selected by the proposed method is greatly smaller than by the other methods. We can therefore say that the proposed method is able to yield a sparse representation while keeping a relatively good generalization performance. Moreover, our method has an advantage that it is free from a selection of a regularization parameter.
URL: https://globals.ieice.org/en_transactions/information/10.1587/transinf.E97.D.98/_p

Copy

@ARTICLE{e97-d_1_98,
author={Katsuyuki HAGIWARA, },
journal={IEICE TRANSACTIONS on Information},
title={A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection},
year={2014},
volume={E97-D},
number={1},
pages={98-106},
abstract={Regularized forward selection is viewed as a method for obtaining a sparse representation in a nonparametric regression problem. In regularized forward selection, regression output is represented by a weighted sum of several significant basis functions that are selected from among a large number of candidates by using a greedy training procedure in terms of a regularized cost function and applying an appropriate model selection method. In this paper, we propose a model selection method in regularized forward selection. For the purpose, we focus on the reduction of a cost function, which is brought by appending a new basis function in a greedy training procedure. We first clarify a bias and variance decomposition of the cost reduction and then derive a probabilistic upper bound for the variance of the cost reduction under some conditions. The derived upper bound reflects an essential feature of the greedy training procedure; i.e., it selects a basis function which maximally reduces the cost function. We then propose a thresholding method for determining significant basis functions by applying the derived upper bound as a threshold level and effectively combining it with the leave-one-out cross validation method. Several numerical experiments show that generalization performance of the proposed method is comparable to that of the other methods while the number of basis functions selected by the proposed method is greatly smaller than by the other methods. We can therefore say that the proposed method is able to yield a sparse representation while keeping a relatively good generalization performance. Moreover, our method has an advantage that it is free from a selection of a regularization parameter.},
keywords={},
doi={10.1587/transinf.E97.D.98},
ISSN={1745-1361},
month={January},}

Copy

TY - JOUR
TI - A Sparse Modeling Method Based on Reduction of Cost Function in Regularized Forward Selection
T2 - IEICE TRANSACTIONS on Information
SP - 98
EP - 106
AU - Katsuyuki HAGIWARA
PY - 2014
DO - 10.1587/transinf.E97.D.98
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E97-D
IS - 1
JA - IEICE TRANSACTIONS on Information
Y1 - January 2014
AB - Regularized forward selection is viewed as a method for obtaining a sparse representation in a nonparametric regression problem. In regularized forward selection, regression output is represented by a weighted sum of several significant basis functions that are selected from among a large number of candidates by using a greedy training procedure in terms of a regularized cost function and applying an appropriate model selection method. In this paper, we propose a model selection method in regularized forward selection. For the purpose, we focus on the reduction of a cost function, which is brought by appending a new basis function in a greedy training procedure. We first clarify a bias and variance decomposition of the cost reduction and then derive a probabilistic upper bound for the variance of the cost reduction under some conditions. The derived upper bound reflects an essential feature of the greedy training procedure; i.e., it selects a basis function which maximally reduces the cost function. We then propose a thresholding method for determining significant basis functions by applying the derived upper bound as a threshold level and effectively combining it with the leave-one-out cross validation method. Several numerical experiments show that generalization performance of the proposed method is comparable to that of the other methods while the number of basis functions selected by the proposed method is greatly smaller than by the other methods. We can therefore say that the proposed method is able to yield a sparse representation while keeping a relatively good generalization performance. Moreover, our method has an advantage that it is free from a selection of a regularization parameter.
ER -