ion is a higher order cognitive ability that facilitates the production of rules that are independent of their associations.In standard reinforcement learning it is often expedient to directly associate situations (states) with actions in order to maximise the environmental reward signal.This may lead to problems including a lack of generalisation and not utilising higher order patterns in complex domains.Thus standard Q-learning has been developed to include models or genetics-based search (Learning Classifier Systems), which improve learning speeds and generality.In order to extend reinforcement learning techniques to higher-order rules, abstraction is considered here.The process of abstraction can be likened to Information Processing Theory (a branch of Learning Theory) (Miller, 1956), which suggests that humans have the ability to recognize patterns in data and chunk these patterns into meaningful units.The individual patterns do not necessarily remain in a memory store due to the holistic nature of the individual patterns.However, the chunks of meaningful information remain, and become a basic element of all subsequent analyses.The need for abstraction arose from the data-mining of rules in the steel industry through application of the genetics-based machine learning technique of Learning Classifier Systems (Holland, 1975), which utilise a Q-learning type update for reinforcement learning.It was noted that many rules had similar patterns.For example, there were many rules of the type 'if side guide setting < width, then poor quality product' due to different product widths.This resulted in a rule-base that was unnecessarily hard to interpret and slow to learn.The initial development of the abstraction method was based on the known problem of Connect4 due to its vast search space, temporal nature and available patterns.The contribution of this chapter is that the novel method of abstraction is described and shown to be effective on a large search space test problem.Abstraction enabled higher order rules to be learned from base knowledge, which mimic important aspects of human cognition.Tests showed that the abstracted rules were more compact, had greater utility and assisted in developmental learning.The emergence of abstracted rules corresponded with escaping from local minima that would have otherwise trapped basic reinforcement learning techniques, such as standard Q-learning.

In reinforcement learning problems it has been considered that neither exploitation nor exploration can be pursued exclusively without failing at the task. The optimal balance between exploring and exploiting changes as the training progresses due to the increasing amount of learnt knowledge. This shift in balance is not known a priori so an autonomous online adjustment is sought. Human beings manage this balance through logic and explorations based on feedback from the environment. The XCS learning classifier system uses a fixed explore/exploit balance, but does keep multiple statistics about its performance and interaction in an environment. Utilising these statistics in a non-linear manner, autonomous adjustment of the explore/exploit balance was achieved. This resulted in reduced exploration in simple environments, which could increase with the complexity of the problem domain. It also prevented unsuccessful 'loop' exploit trials and suggests a method of dynamic choice in goal setting.

Abstraction is a higher order cognitive ability that facilitates the production of rules that are independent of their associations. Experience from real-world data-mining has shown the need for such higher level rules. The game of Connect 4 is both multistep and complex, so standard Q-learning and Learning Classifier Systems perform poorly. The introduction of a novel Abstraction algorithm into an LCS is shown to improve performance in the evolution of playing strategies.