An orthonormal basis adaptation method for function approximation was developed and applied to reinforcement learning with multi-dimensional continuous state space. First, a basis used for linear function approximation of a control function is set to an orthonormal basis. Next, basis elements with small activities are replaced with other candidate elements as learning progresses. As this replacement is repeated, the number of basis elements with large activities increases. Example chaos control problems for multiple logistic maps were solved, demonstrating that the method for adapting an orthonormal basis can modify a basis while holding the orthonormality in accordance with changes in the environment to improve the performance of reinforcement learning and to eliminate the adverse effects of redundant noisy states.

This paper introduces the concept of an extended pseudo-biorthogonal basis" (EPBOB), which is a generalization of the concepts of an orthonormal (OB), a biorthonormal (BOB), a pseudo-orthogonal (POB), and a pseudo-biorthogonal (PBOB) bases. Let HN be a subspace of a Hilbert space H. The concept of EPBOB says that we can always construct a set of 2M (MN) elements of H but not necessarily all in HN such that like BOB any element f in HN can be expressed by fMΣm=1(f,φ*m)φm. For a better understanding and a wide application of EPBOB, this paper provides their characterization and shows how they preserve the formalism of BOB. It also shows how to construct them.