Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution

Shuqiong WU; Hiroshi NAGAHASHI

doi:10.1587/transinf.2015EDP7069

Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution

Shuqiong WU, Hiroshi NAGAHASHI

Full Text Views

0

Share
Cite this

Summary :

Gentle AdaBoost is widely used in object detection and pattern recognition due to its efficiency and stability. To focus on instances with small margins, Gentle AdaBoost assigns larger weights to these instances during the training. However, misclassification of small-margin instances can still occur, which will cause the weights of these instances to become larger and larger. Eventually, several large-weight instances might dominate the whole data distribution, encouraging Gentle AdaBoost to choose weak hypotheses that fit only these instances in the late training phase. This phenomenon, known as “classifier distortion”, degrades the generalization error and can easily lead to overfitting since the deviation of all selected weak hypotheses is increased by the late-selected ones. To solve this problem, we propose a new variant which we call “Penalized AdaBoost”. In each iteration, our approach not only penalizes the misclassification of instances with small margins but also restrains the weight increase for instances with minimal margins. Our method performs better than Gentle AdaBoost because it avoids the “classifier distortion” effectively. Experiments show that our method achieves far lower generalization errors and a similar training speed compared with Gentle AdaBoost.

Publication: IEICE TRANSACTIONS on Information Vol.E98-D No.11 pp.1906-1915

Publication Date: 2015/11/01

Publicized: 2015/08/13

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2015EDP7069

Type of Manuscript: PAPER

Category: Artificial Intelligence, Data Mining

Authors

Shuqiong WU
Tokyo Institute of Technology
Hiroshi NAGAHASHI
Tokyo Institute of Technology

Keyword

Gentle AdaBoost, generalization error, machine learning, modest AdaBoost, training error

Cite this

Copy

Shuqiong WU, Hiroshi NAGAHASHI, "Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution" in IEICE TRANSACTIONS on Information, vol. E98-D, no. 11, pp. 1906-1915, November 2015, doi: 10.1587/transinf.2015EDP7069.
Abstract: Gentle AdaBoost is widely used in object detection and pattern recognition due to its efficiency and stability. To focus on instances with small margins, Gentle AdaBoost assigns larger weights to these instances during the training. However, misclassification of small-margin instances can still occur, which will cause the weights of these instances to become larger and larger. Eventually, several large-weight instances might dominate the whole data distribution, encouraging Gentle AdaBoost to choose weak hypotheses that fit only these instances in the late training phase. This phenomenon, known as “classifier distortion”, degrades the generalization error and can easily lead to overfitting since the deviation of all selected weak hypotheses is increased by the late-selected ones. To solve this problem, we propose a new variant which we call “Penalized AdaBoost”. In each iteration, our approach not only penalizes the misclassification of instances with small margins but also restrains the weight increase for instances with minimal margins. Our method performs better than Gentle AdaBoost because it avoids the “classifier distortion” effectively. Experiments show that our method achieves far lower generalization errors and a similar training speed compared with Gentle AdaBoost.
URL: https://globals.ieice.org/en_transactions/information/10.1587/transinf.2015EDP7069/_p

Copy

@ARTICLE{e98-d_11_1906,
author={Shuqiong WU, Hiroshi NAGAHASHI, },
journal={IEICE TRANSACTIONS on Information},
title={Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution},
year={2015},
volume={E98-D},
number={11},
pages={1906-1915},
abstract={Gentle AdaBoost is widely used in object detection and pattern recognition due to its efficiency and stability. To focus on instances with small margins, Gentle AdaBoost assigns larger weights to these instances during the training. However, misclassification of small-margin instances can still occur, which will cause the weights of these instances to become larger and larger. Eventually, several large-weight instances might dominate the whole data distribution, encouraging Gentle AdaBoost to choose weak hypotheses that fit only these instances in the late training phase. This phenomenon, known as “classifier distortion”, degrades the generalization error and can easily lead to overfitting since the deviation of all selected weak hypotheses is increased by the late-selected ones. To solve this problem, we propose a new variant which we call “Penalized AdaBoost”. In each iteration, our approach not only penalizes the misclassification of instances with small margins but also restrains the weight increase for instances with minimal margins. Our method performs better than Gentle AdaBoost because it avoids the “classifier distortion” effectively. Experiments show that our method achieves far lower generalization errors and a similar training speed compared with Gentle AdaBoost.},
keywords={},
doi={10.1587/transinf.2015EDP7069},
ISSN={1745-1361},
month={November},}

Copy

TY - JOUR
TI - Penalized AdaBoost: Improving the Generalization Error of Gentle AdaBoost through a Margin Distribution
T2 - IEICE TRANSACTIONS on Information
SP - 1906
EP - 1915
AU - Shuqiong WU
AU - Hiroshi NAGAHASHI
PY - 2015
DO - 10.1587/transinf.2015EDP7069
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E98-D
IS - 11
JA - IEICE TRANSACTIONS on Information
Y1 - November 2015
AB - Gentle AdaBoost is widely used in object detection and pattern recognition due to its efficiency and stability. To focus on instances with small margins, Gentle AdaBoost assigns larger weights to these instances during the training. However, misclassification of small-margin instances can still occur, which will cause the weights of these instances to become larger and larger. Eventually, several large-weight instances might dominate the whole data distribution, encouraging Gentle AdaBoost to choose weak hypotheses that fit only these instances in the late training phase. This phenomenon, known as “classifier distortion”, degrades the generalization error and can easily lead to overfitting since the deviation of all selected weak hypotheses is increased by the late-selected ones. To solve this problem, we propose a new variant which we call “Penalized AdaBoost”. In each iteration, our approach not only penalizes the misclassification of instances with small margins but also restrains the weight increase for instances with minimal margins. Our method performs better than Gentle AdaBoost because it avoids the “classifier distortion” effectively. Experiments show that our method achieves far lower generalization errors and a similar training speed compared with Gentle AdaBoost.
ER -