Neural Network Training Algorithm with Positive Correlation

Md. SHAHJAHAN; Kazuyuki MURASE

doi:10.1093/ietisy/e88-d.10.2399

Neural Network Training Algorithm with Positive Correlation

Md. SHAHJAHAN, Kazuyuki MURASE

Full Text Views

0

Share
Cite this

Summary :

In this paper, we present a learning approach, positive correlation learning (PCL), that creates a multilayer neural network with good generalization ability. A correlation function is added to the standard error function of back propagation learning, and the error function is minimized by a steepest-descent method. During training, all the unnecessary units in the hidden layer are correlated with necessary ones in a positive sense. PCL can therefore create positively correlated activities of hidden units in response to input patterns. We show that PCL can reduce the information on the input patterns and decay the weights, which lead to improved generalization ability. Here, the information is defined with respect to hidden unit activity since the hidden unit plays a crucial role in storing the information on the input patterns. That is, as previously proposed, the information is defined by the difference between the uncertainty of the hidden unit at the initial stage of learning and the uncertainty of the hidden unit at the final stage of learning. After deriving new weight update rules for the PCL, we applied this method to several standard benchmark classification problems such as breast cancer, diabetes and glass identification problems. Experimental results confirmed that the PCL produces positively correlated hidden units and reduces significantly the amount of information, resulting improved generalization ability.

Publication: IEICE TRANSACTIONS on Information Vol.E88-D No.10 pp.2399-2409

Publication Date: 2005/10/01

Publicized

Online ISSN

DOI: 10.1093/ietisy/e88-d.10.2399

Type of Manuscript: PAPER

Category: Biocybernetics, Neurocomputing

Cite this

Copy

Md. SHAHJAHAN, Kazuyuki MURASE, "Neural Network Training Algorithm with Positive Correlation" in IEICE TRANSACTIONS on Information, vol. E88-D, no. 10, pp. 2399-2409, October 2005, doi: 10.1093/ietisy/e88-d.10.2399.
Abstract: In this paper, we present a learning approach, positive correlation learning (PCL), that creates a multilayer neural network with good generalization ability. A correlation function is added to the standard error function of back propagation learning, and the error function is minimized by a steepest-descent method. During training, all the unnecessary units in the hidden layer are correlated with necessary ones in a positive sense. PCL can therefore create positively correlated activities of hidden units in response to input patterns. We show that PCL can reduce the information on the input patterns and decay the weights, which lead to improved generalization ability. Here, the information is defined with respect to hidden unit activity since the hidden unit plays a crucial role in storing the information on the input patterns. That is, as previously proposed, the information is defined by the difference between the uncertainty of the hidden unit at the initial stage of learning and the uncertainty of the hidden unit at the final stage of learning. After deriving new weight update rules for the PCL, we applied this method to several standard benchmark classification problems such as breast cancer, diabetes and glass identification problems. Experimental results confirmed that the PCL produces positively correlated hidden units and reduces significantly the amount of information, resulting improved generalization ability.
URL: https://globals.ieice.org/en_transactions/information/10.1093/ietisy/e88-d.10.2399/_p

Copy

@ARTICLE{e88-d_10_2399,
author={Md. SHAHJAHAN, Kazuyuki MURASE, },
journal={IEICE TRANSACTIONS on Information},
title={Neural Network Training Algorithm with Positive Correlation},
year={2005},
volume={E88-D},
number={10},
pages={2399-2409},
abstract={In this paper, we present a learning approach, positive correlation learning (PCL), that creates a multilayer neural network with good generalization ability. A correlation function is added to the standard error function of back propagation learning, and the error function is minimized by a steepest-descent method. During training, all the unnecessary units in the hidden layer are correlated with necessary ones in a positive sense. PCL can therefore create positively correlated activities of hidden units in response to input patterns. We show that PCL can reduce the information on the input patterns and decay the weights, which lead to improved generalization ability. Here, the information is defined with respect to hidden unit activity since the hidden unit plays a crucial role in storing the information on the input patterns. That is, as previously proposed, the information is defined by the difference between the uncertainty of the hidden unit at the initial stage of learning and the uncertainty of the hidden unit at the final stage of learning. After deriving new weight update rules for the PCL, we applied this method to several standard benchmark classification problems such as breast cancer, diabetes and glass identification problems. Experimental results confirmed that the PCL produces positively correlated hidden units and reduces significantly the amount of information, resulting improved generalization ability.},
keywords={},
doi={10.1093/ietisy/e88-d.10.2399},
ISSN={},
month={October},}

Copy

TY - JOUR
TI - Neural Network Training Algorithm with Positive Correlation
T2 - IEICE TRANSACTIONS on Information
SP - 2399
EP - 2409
AU - Md. SHAHJAHAN
AU - Kazuyuki MURASE
PY - 2005
DO - 10.1093/ietisy/e88-d.10.2399
JO - IEICE TRANSACTIONS on Information
SN -
VL - E88-D
IS - 10
JA - IEICE TRANSACTIONS on Information
Y1 - October 2005
AB - In this paper, we present a learning approach, positive correlation learning (PCL), that creates a multilayer neural network with good generalization ability. A correlation function is added to the standard error function of back propagation learning, and the error function is minimized by a steepest-descent method. During training, all the unnecessary units in the hidden layer are correlated with necessary ones in a positive sense. PCL can therefore create positively correlated activities of hidden units in response to input patterns. We show that PCL can reduce the information on the input patterns and decay the weights, which lead to improved generalization ability. Here, the information is defined with respect to hidden unit activity since the hidden unit plays a crucial role in storing the information on the input patterns. That is, as previously proposed, the information is defined by the difference between the uncertainty of the hidden unit at the initial stage of learning and the uncertainty of the hidden unit at the final stage of learning. After deriving new weight update rules for the PCL, we applied this method to several standard benchmark classification problems such as breast cancer, diabetes and glass identification problems. Experimental results confirmed that the PCL produces positively correlated hidden units and reduces significantly the amount of information, resulting improved generalization ability.
ER -