換刀機(jī)械手設(shè)計(jì)帶CAD圖,機(jī)械手,設(shè)計(jì),CAD Machine Learning and AgentsPiotr J edrzejowiczChair of Information Systems, Gdynia Maritime University,Morska 83, 81-225 Gdynia, Polandpjam.gdynia.plAbstract. The paper reviews current research results integratingmachine learning and agent technologies. Although complementary solu-tions from both fields are discussed the focus is on using agent technologyin the field of machine learning with a particular interest on applyingagent-based solutions to supervised learning. The paper contains a shortreview of applications, in which machine learning methods have beenused to support agent learning capabilities. This is followed by a corre-sponding review of machine learning methods and tools in which agenttechnology plays an important role. Final part gives a more detailed de-scription of some example machine learning models and solutions wherethe asynchronous team of agents paradigm has been implemented to sup-port the machine learning methods and which have been developed bythe author and his research group.1IntroductionContemporary definition sees machine learning as a discipline that is concernedwith the design and development of algorithms that allow computers to learnbehaviors based on empirical data. Data can be seen as examples that illustraterelations between observed objects. A major focus of machine learning researchis to automatically learn to recognize complex patterns and make intelligent de-cisions based on data. Parallel to recent developments in the field of machinelearning, mainly as a result of convergence of many technologies within computerscience such as object-oriented programming, distributed computing and artifi-cial life, the agent technology has emerged. An agent is understood here as anypiece of software that is designed to use intelligence to automatically carry outan assigned task, mainly retrieving and delivering information. Tweedale andco-authors 62 outline an abridged history of agents as a guide for the readerto understand the trends and directions of future agent design. This descriptionincludes how agent technologies have developed using increasingly sophisticatedtechniques. It also indicates the transition of formal programming languages intoobject-oriented programming and how this transition facilitated a correspondingshift from scripted agents (bots) to agent-oriented designs which is best exempli-fied by multiple agent systems (MAS). A MAS tries to solve complex problemswith entities called agents, using their collaborative and autonomous properties38. Basic MAS properties are listed in 34.J. OShea et al. (Eds.): KES-AMSTA 2011, LNAI 6682, pp. 215, 2011.c? Springer-Verlag Berlin Heidelberg 2011Machine Learning and Agents3During the last decade developments in the fields of machine learning andagent technologies have, in some respect, become complementary and researchersfrom both fields have seen ample opportunities to profit from solutions proposedby each other. Several agent-based frameworks that utilize machine learning forintelligent decision support have been recently reported. Learning is increasinglybeing seen as a key ability of agents, and research into learning agent technology,such as reinforcement learning and supervised or unsupervised learning has pro-duced many valuable applications. In this paper the focus is however on usingagent technology in the field of machine learning with a particular interest onapplying agent-based solutions to supervised learning. Supervised learning is themachine learning task of inducing a function from training data which is a setof training examples. In supervised learning, each example is a pair consistingof an input object (typically a vector) and a desired output value (also calledthe supervisory signal or class label). A supervised learning algorithm analyzesthe training data and produces an induced function, which is called a classifierif the output is discrete, or a regression function if the output is continuous.The inferred function should predict the correct output value for any valid inputobject. This requires the learning algorithm to generalize from the training datato unseen situations.There are several ways the machine learning algorithm can profit from ap-plying agent technology. Among them the following will be addressed in thispaper: There are numerous machine learning techniques where parallelization canspeed-up or even enable learning. Using a set of agents may, in such circum-stances, increase efficiency of learning. Several machine learning techniques directly rely on the collective compu-tational intelligence paradigm, where a synergetic effect is expected fromcombining efforts of various program agents. There is a class of machine learning problems known as the distributed ma-chine learning. In the distributed learning a set of agents working in thedistributed sites can be used to produce some local level solutions indepen-dently and in parallel. Later on local level solutions are combined into aglobal solution.The paper is organized as follows. Section 2 contains a short review of appli-cations, in which machine learning methods have been used to support agentlearning capabilities. Section 3 offers a corresponding review of machine learn-ing methods and tools in which agent technology plays an important role. Section4 gives more detailed description of some example machine learning models andsolutions where the agent paradigm has been implemented and which have beendeveloped by the author and his research group. Finally, conclusions contain sug-gestions on future research and possible deeper integration of machine learningand agent technology.4P. J edrzejowicz2Learning AgentsProbably the most often used approach to provide agents with learning capabil-ities is the reinforcement learning. An excellent survey of multiagent reinforce-ment learning can be found in 10. As it was pointed out by Sutton and Barto57 reinforcement learning is learning what to do - how to map sitations to ac-tions so as to maximize a numerical reward signal. The learner is not told whichactions to take, as in most forms of machine learning, but instead must discoverwhich actions yield the most reward. In describing properties of the reinforce-ment learning the authors directly refer to the notion of agent. In their view alearning agent must be able to sense the state of the environment and must beable to take actions that affect the state. The agent also must have goal or goalsrelating to the state of the environment 57.Theoretical developments in the field of learning agents focus mostly onmethodologies and requirements for constructing multiagent systems with learn-ing capabilities. Connection of the theory of automata with the multiagent re-inforcement learning is explored in 45. Shoham et.al. 53 claim that the areaof learning in multi-agent systems is today one of the most fertile grounds forinteraction between game theory and artificial intelligence. In 41 challengesmotivated by engineering applications and the potential appeal of multi-agentlearning to meet these challenges are discussed.Symeonidis et.al. 58 present an approach that takes the relevant limitationsand considerations into account and provides a gateway on the way data miningtechniques can be employed in order to augment agent intelligence. This workdemonstrates how the extracted knowledge can be used for the formulation ini-tially, and the improvement, in the long run, of agent reasoning. Preux et.al. 47present MAABAC, a generic model for building adaptive agents: they learn newbehaviors by interacting with their environment. Agents adapt their behavior byway of reinforcement learning, namely temporal difference methods. The paper52 presents a systematic approach to introduce machine learning in the designand implementation phases of a software agent. It also presents an incrementalimplementation process for building asynchronous and distributed agents, whichsupports the combination of machine learning strategies. Rosaci in 51 proposesa complete MAS architecture, called connectionist learning and inter-ontologysimilarities (CILIOS), for supporting agent mutual monitoring.In 42 the concepts of stigmergy and entropy are imported into learning au-tomata based multi-agent systems with the purpose of providing a simple frame-work for interaction and coordination in multi-agent systems and speeding upthe learning process. Another extension was proposed in 8. The authors suggesta merging, and hence an extension, of two recent learning methods, utility-basedlearning and strategic or adversarial learning. Utility-based learning brings tothe forefront the learners utility function during induction. Strategic learninganticipates strategic activity in the induction process when the instances areintelligent agents such as in classification problems involving people or orga-nizations. The resulting merged model is called the principal-agent learning.Loizos 39 argues that when sensing its environment, an agent often receivesMachine Learning and Agents5information that only partially describes the current state of affairs. The agentthen attempts to predict what it has not sensed, by using other pieces of infor-mation available through its sensors. Machine learning techniques can naturallyaid this task, by providing the agent with the rules to be used for making thesepredictions. For this to happen, however, learning algorithms need to be de-veloped that can deal with missing information in the learning examples in aprincipled manner, and without the need for external supervision. It is shownthat the Probably Approximately Correct semantics can be extended to dealwith missing information during both the learning and the evaluation phase.Numerous reinforcement learning applications have been recently reported inthe literature. Some interesting examples include a proposal of reinforcementlearning for agent-based production scheduling 63, a case-based reinforcementlearning algorithm (CRL) for dynamic inventory control in a multi-agent supply-chain system 35. Supervised learning techniques have been also applied to sup-port agents learning capabilities. In 65, a support vector machine (SVM) basedmultiagent ensemble learning approach is proposed for credit risk evaluation.Different SVM learning paradigms with much dissimilarity are constructed asintelligent agents for credit risk evaluation. Multiple individual SVM agents aretrained using training subsets. In the final stage, all individual results producedby multiple SVM agents in the previous stage are aggregated into an ensembleresult.3Agent-Based Machine LearningRecently, several machine learning solutions and techniques have been reportedto rely on applying agent technologies. They belong to the two broad classes -universal one and dedicated to particular applications. Solutions and techniquesbelonging to the first class involve applications of the multi agent systems, in-cluding A-Teams and the population-based methods. This section contains areview of some recent universal and dedicated solutions with the exception ofthose based on the A-Team paradigm. Machine learning solutions using the A-Team paradigm are discussed in a detailed manner in Section 4.3.1Universal Solutions and TechniquesAs it has been observed in 40 industry, science, and commerce fields often needto analyze very large datasets maintained over geographically distributed sitesby using the computational power of distributed systems. The Grid can playa significant role in providing an effective computational infrastructure supportfor this kind of data mining. Similarly, the advent of multi-agent systems hasbrought us a new paradigm for the development of complex distributed applica-tions. Through a combination of these two techniques an Agent Grid IntelligentPlatform and an integrated toolkit VAStudio used as a testbed were proposed.Using grid platform as a testbed was also suggested in 50. The author presentsa parallel learning method for agents with an actor-critic architecture based on6P. J edrzejowiczartificial neural networks. The agents have multiple modules, where the modulescan learn in parallel to further increase learning speed. Each module solves a sub-problem and receives its own separate reward signal with all modules trainedconcurrently. The method is used on a grid world navigation task showing thatparallel learning can significantly reduce learning time.Kitakoshi et al.36 describe an on-line reinforcement learning system thatadapts to environmental changes using a mixture of Bayesian networks. Ma-chine learning approaches, such as those using reinforcement learning meth-ods and stochastic models, have been used to acquire behavior appropriate toenvironments characterized by uncertainty. The results of several experimentsdemonstrated that an agent using the proposed system can flexibly adapt tovarious kinds of environmental changes.Gifford in his Ph.D. dissertation 24 advocates an approach focused on theeffects of sharing knowledge and collaboration of multiple heterogeneous, intel-ligent agents (hardware or software) which work together to learn a task. Aseach agent employs a different machine learning technique, the system consistsof multiple knowledge sources and their respective heterogeneous knowledge rep-resentations. Experiments have been performed that vary the team compositionin terms of machine learning algorithms and learning strategies employed bythe agents. General findings from these experiments suggest that constructinga team of classifiers using a heterogeneous mixture of homogeneous teams ispreferred.Quteishat et.al. 49 proposed a neural network-based multi-agent classifiersystem using the trust, negotiation, and communication reasoning model. Themain contribution of this work is that a novel trust measurement method, basedon the recognition and rejection rates, was suggested.Several important methods can be grouped under the umbrella of the col-lective or collaborative learning. In 27 it was shown show how EvolutionaryDynamics (ED) can be used as a model for Qlearning in stochastic games. Anal-ysis of the evolutionary stable strategies and attractors of the derived ED fromthe Reinforcement Learning (RL) application then predict the desired parame-ters for RL in Multi-Agent Systems (MASs) to achieve Nash equilibriums withhigh utility. Secondly, it was shown how the derived fine tuning of parame-ter settings from the ED can support application of the COllective INtelligence(COIN) framework. COIN is a proved engineering approach for learning of coop-erative tasks in MASs. In 26 authors propose a collaborative machine learningframework to exploit inter-user similarities. More specifically, they present akernel-based learning architecture that generalizes the well-known Support Vec-tor Machine learning approach by enriching content descriptors with inter-usercorrelations.Another umbrella covers learning classifier systems introduced by Holland28 which use simple agents representing set of rules as a solution to a ma-chine learning problem. A Pittsburgh-type LCS has a populations of separaterule sets, where the genetic algorithm recombines and reproduces the best ofthese rule sets. In a Michigan-style LCS there is only a single population andMachine Learning and Agents7the algorithms action focuses on selecting the best classifiers within that ruleset. Analysis of the properties of LCSs, comparison of several proposed variantsand overview of the state of the art can be found in 4, 5, 64 and 9. Use-ful extension of the LCS concept was proposed in 55. This paper introducesa new variety of learning classifier system (LCS), called MILCS, which utilizesmutual information as fitness feedback. Unlike most LCSs, MILCS is specificallydesigned for supervised learning. Yet another extension introduces a mechanismfor recognizing a current situation by determining a boundary between self andothers, and investigates its capability through interaction with an agent 59. Anintegration of several cognitively inspired anticipation and anticipatory learn-ing mechanisms in an autonomous agent architecture, the Learning IntelligentDistribution Agent (LIDA) system was proposed in 44.Ensemble techniques have proved to be very successful in boosting the perfor-mance of several types of machine learning methods. In 6 authors illustrate itsusefulness in combination with GAssist, a Pittsburgh-style Learning ClassifierSystem. Effective and competitive ensembles constructed from simple agentsrepresented by expression trees induced using Gene Expression Programminghave been proposed in 32 and 33. Their approach has been tested using sev-eral ensemble constructing techniques including AdaBoost learning, voting poolof classifiers, incremental learning, cluster based learning, mass functions basedlearning and meta-learning.Agent technology seems to be a natural tool for the distributed systems. Com-bining approaches to distributed learning with agent technology is consideredas the promising and at the same time challenging problem in the distributedlearning research 37. In 67 an agent paradigm was proposed as a tool for in-tegration of different techniques into an effective strategy of learning from data.The proposed hybrid learning system integrates basic components of the learn-ing process. Data pre-processing, selection, transformation and induction of thelearning and post-learning models are carried out by a set of agents cooperatingduring the task execution. Several agent-based architectures have already beenproposed to solve the distributed learning problems. It is usually assumed thateach site can have one or more associated agents, processing the local data andcommunicating the results to other agents that control and manage the knowl-edge discovery process. Examples include Papyrus 48, MALE 54, ANIMALS56 and MALEF 61. In 2 EMADS, a hybrid peer-to-peer agent based systemcomprising a collection of the collaborating agents distributed across a network,was described.3.2Dedicated Solutions and TechniquesIn the machine learning literature numerous applications solving particular ma-chine learning problem type or task where agent technology have played animportant, even if supporting, role have been recently reported. In this shortreview the focus is on some example cases where agent technology has been usedin an innovative manner.8P. J edrzejowiczFan et.al. 23 have developed a two-stage model for personalized and in-telligent information routing of online news. At the first stage, persistent userqueries are extracted from rated documents based on Robertson s Selection Value(RSV). At the second stage, genetic programming is applied to discover the op-timal ranking function for individual user. Pazzani and Billsus 46 developed alearning information agent called Syskill and Webert which could learn a userprofile for the identification of interesting web documents. A separate user profilewas created for each individual information topic. Web documents were repre-sented as Boolean feature vectors, and each feature had a binary value indicatingif a particular keyword appeared in the document or not. Feature selection wasconducted based on Expected Information Gain which tends to select wordsappearing more frequently in positive documents. The classification mechanismof Syskill and Webert was based on a naive Bayesian classifier. The paper 3focuses on symbolic transducers and recurrent neural preference machines tosupport the task of mining and classifying textual information. These encodingsymbolic transducers and learning neural preference machines can be seen asindependent agents, each one tackling the same task in a different manner.Jansen 30 discusses various ways in which mobile agents could be applied toproblem of detecting and responding to intrusions. Abraham et.al. 1 proposeda distributed Intrusion Detection System (IDS) consisting of several IDS over alarge network, all