自走連續(xù)振動(dòng)式紅棗收獲機(jī)設(shè)計(jì),連續(xù),振動(dòng),紅棗,收獲,設(shè)計(jì) Research PaperDesign and control of an apple harvesting robotZhao De-An, Lv Jidong, Ji Wei*, Zhang Ying, Chen YuSchool of Electrical and Information Engineering, Jiangsu University, XueFu Road No.301, Zhenjiang, Jiangsu Province 212013, PR Chinaa r t i c l e i n f oArticle history:Received 9 February 2011Received in revised form4 July 2011Accepted 17 July 2011Published online 6 August 2011A robotic device consisting of a manipulator, end-effector and image-based vision servocontrol system was developed for harvesting apple. The manipulator with 5 DOF PRRRPstructure was geometrically optimised to provide quasi-linear behaviour and to simplify thecontrol strategy. The spoon-shaped end-effector with the pneumatic actuated gripper wasdesigned to satisfy the requirements for harvesting apple. The harvesting robot autono-mouslyperformeditsharvestingtaskusingavision-basedmodule.Byusingasupportvectormachine with radial basis function, the fruit recognition algorithm was developed to detectand locate the apple in the trees automatically. The control system, including industrialcomputer and AC servo driver, conducted the manipulator and the end-effector as itapproached and picked the apples. The effectiveness of the prototype robot device wasconfirmed by laboratory tests and field experiments in an open field. The success rate ofappleharvestingwas77%,andtheaverageharvestingtimewasapproximately15sperapple.Crown Copyright 2011 Published by Elsevier Ltd on behalf of IAgrE. All rights reserved.1.IntroductionIn China, with the rapid development of the rural economyand the continuous adjustment of planting structures, fruitcultivation areas, such as apple, citrus and pear, havereached 8-9 million ha since 1993, accounting for one-quarterof the total fruit cultivation area in the world. However, fruitharvesting tasks, which take 50%e70% of the total workinghours, still depend on manual labour (Xu & Zhang, 2004.Harvesting is expected to be automated because the farmingpopulation is gradually decreasing in China. In addition,since the fruit trees are tall, harvesting work has to be con-ducted using step ladders, which makes manual harvestingdangerous and inefficient. Therefore, there is a strong desireto mechanise and automate harvesting. Mechanical har-vestingexperimentshavebeenperformedontheassumptionof once-over harvesting in some areas, but exploitation ofthis strategy is not yet widespread (Hancock, 1999). Selectiveharvesting, which is commonly used, requires sophisticatedrobotic technology. In short, it is necessary to design anintelligent robot with human-like perceptive capabilities. Forinstance, the machine needs to detect fruit, calculate theposition of the fruit and then pick it without damaging thepericarp or the fruit tree.Research on fruit harvesting robots took place in the 1980s.Kawamura, Namikawa, Fujiura, and Ura (1984) first developeda fruit-harvesting robot for orchards. Later, Grand, Rabatel,Pellenc, Journeau, and Aldon (1987), developed an apple-harvesting robot. Since then, their pioneering studies werefollowed by many research papers covering several aspects(e.g., ;Edan, Rogozin, Flash, & Miles, 2000; Foglia & Reina, 2006;Hwang&Kim,2003;Kondo&Ting,1998;Muscato,Prestifilippo, Abbate, & Ivan, 2005; Sakai, Osuka, Maekawa, &Umeda, 2007, 2008; Sarig, 1993; Van Henten, Hemming, VanTuijl, Kornet, Meuleman, 2002). In addition, several relevantstudiesonagriculturalrobotsingreenhouseshavebeencarried* Corresponding author. Tel.: 86 511 82028322; fax: 86 511 82028322.E-mail address: (J. Wei).Available at journal homepage: engineering 110 (2011) 112e1221537-5110/$ e see front matter Crown Copyright 2011 Published by Elsevier Ltd on behalf of IAgrE. All rights reserved.doi:10.1016/j.biosystemseng.2011.07.005out; for instance, tomato harvesting (Monta et al., 1998),cucumber harvesting (Van Henten, Van Tuijl, Hemming,Kornet,Bontsema&VanOs,2003),cherryharvesting(Tanigaki, Fujiura, Akase, & Imagawa, 2008), strawberry har-vesting (Hayashi et al., 2010). However, most of the fruit har-vesting robots discussed in the literature are not currentlymanufacturedorsold.Instead,theyremainintheresearchanddevelopment stages. To this end, it is important to supportfurtherresearch anddevelopment toimprove the performanceand reduce the initial set-up costs of these robots.Based on the concepts above, this study intends to developand evaluate a competitive low price device for automaticharvesting, i.e., an apple-harvesting robot. Firstly, a detaileddescription on the components of the robot including themanipulator,theend-effectorandtheimage-basedvisionservocontrol system is described. Secondly, the geometrically opti-misation of the manipulator to gain a quasi-linear behaviourand simplify the control strategy is described. Thirdly, the end-effectorwiththepneumaticactuatedgripperdesignedtosatisfythe requirements for harvesting apple is described. Based onthis design, the harvesting robot autonomously performs itsharvesting task using a vision-based module to detect andlocate the apple in the trees, and control system conducts themanipulator and the end-effector to approach and pick apple.To verify the validity of the developed harvesting robot, thelaboratory tests and field experiments in an open field wereperformed. The experimental results are the important contri-bution of this paper.The paper is organised as follows: in section 2 the maincomponents of the robot are presented in detail, i.e., themanipulator, the end-effector and the image-based visionservo control system, respectively; in section 3 the experi-mental results are discussed to show the feasibility of therobot system proposed; finally, in section 4 conclusions aredrawn and suggestions for future research are made.2.Material and methods2.1.Mechanical structure of apple harvesting robotA prototype model of the apple harvesting robot is designedfor both efficiency and cost effectiveness. It mainly consists ofan autonomous vehicle, a 5 degree of freedom (DOF) manip-ulator, an end-effector, the sensors, the vision system andcontrol system. The mechanical structure of fruit harvestingrobot self-developed in this paper is shown in Fig. 1.2.1.1.The autonomous mobile vehicleA crawler type mobile platform was selected as the mobilevehicle. It carried the power supplies, pneumatic pump,electronic hardware for data acquisition and control, and themanipulator with the end-effector for cutting the fruit. Globalposition system (GPS) technology was used for autonomousnavigation of the mobile vehicle, whose typical speed was1.5 ms?1.2.1.2.The manipulatorCompared with other structures, as described in Sakai,Michihisa, Osuka, and Umeda (2008), joint structure is effec-tive for any position and orientation in three-dimensionalspace. The operation of a harvesting robot is a random largespace distribution, where a lot of obstacles may exist aroundthe robot. A joint manipulator with multi-degrees of freedomhas an arbitrary curve fitting function. It is therefore easy toavoid obstacles by operating the corresponding joints whenthe end-effector reaches the object position. Therefore,aharvestingrobotmanipulatorwith5DOFprismatic-revolute-revolute-revolute-prismatic (PRRRP) structure to bemounted on autonomous mobile vehicle was designed. Thefirst DOF was used for uplifting the whole manipulator. TheNomenclatureSymbolsCR, CRU, CU, CLU, CLAvoidance sensors numberXc, Yc, ZcThe camera coordinates axesXo, Yo, ZoRobot coordinates axesL1, L2, L3Lengths of waist, major arm and minor armq1;q2;q3Joint angles of waist, major arm and minor arm.u, vImage plane coordinates horizontal and verticalaxesuo, voImage centre coordinatexg, ygProjection centre coordinate of target fruitex, eyThe difference of target fruit image featurebetween xg, ygand uo, voM ? NImage plane pixels of video camerajexmaxj;jeymaxj Maximum of ex and eyDq1; Dq2; Dq3Joint deviationangles of waist, major armandminor armk1, k2Control parameters of armsDd The angle to adjust for the movement of a pixel withunit of degree per pixel.AbbreviationsACAlternating CurrentA/DAnalog, DigitalCCDCharge Coupled DevicesD/ADigital, AnalogDCDirect Current.DOFDegree of FreedomGPSGlobal Position SystemHISHue, Intensity, SaturationIBVSImage-Based Vision ServoPBVSPosition-Based Vision ServoPRRRPPrismatic Revolute Revolute Revolute PrismaticRBFRadial Basis FunctionRSTRotation Scale, TranslationSVMSupport Vector MachineUSBUniversal Serial BusVFWVideo for Windowsbiosystems engineering 110 (2011) 112e122113middle three DOF were for rotation, among which, the seconddriving arm was designed to rotate around the waist, and thethird and fourth ones were rotation axes to move the terminaloperator up and down. This DOF allowed the end-effector tomove towards an arbitrary direction in the work space. Thefifth, and last, DOF was flexible and used for elongation, whichmade the end-effector reach the target location according tothe robot control commands, thus achieving the harvesting offruit (Zhao, Zhao, & Ji, 2009; Zhao, Zhao, & Shen, 2009). Thediscussion above shows that 5 DOF manipulator designedshould be sufficient to perform the harvest operation. Themechanical structure of the manipulator is shown in Fig. 2.The lifting of manipulator was performed by the pump-driven lifting platform, which was able to cope with thespecial circumstances of tall fruit crops. The rotary joints andflexible joints were driven by servo motors. Motion parame-ters of the robot manipulator mechanical structure are shownin Table 1.2.1.3.The end-effectorThe mechanism of end-effector is determined by operationand biological characteristics of the target object. The opera-tion objects of harvesting robot are mainly spherical fruit suchas apple. A spoon-shaped end-effector (shown in Fig. 3) isdesigned according to biological characteristics of sphericalfruit, which are picked by means of cutting off the stalk.The end-effector contained the following parts: a gripper tograsp the fruit and an electric cutting device to separate thefruit from the stalk. The opening and closing of end-effectorgripper was determined by some pneumatic devices, whosequick action, fast response characteristics were suitable forthe switching control of the end-effector. Pressure trans-mission was a transferring mode using compressed gaspressure to achieve energy transference. The apple stalk wassevered by an electric cutter installed in the side of grippermechanism. When the fruit was grasped, the direct current(DC) motors transmited power by flexible wire to drive thecutter rotating around the gripper,cutting off the stalk in frontof end-effector at any position.Fig. 1 e Schematic diagram of the fruit harvesting robot.Fig. 2 e Photograph of the manipulator.biosystems engineering 110 (2011) 112e1221142.2.The sensorsThe non-structural and uncertain features of the operatingenvironment, and the individual differences and randomnature of the operating objects, determines that fruit har-vesting robots should have intelligent sensibility to theircomplex environment (Edan et al., 2000; Zhao, Zhao, & Ji, 2009;Zhao, Zhao, & Shen, 2009). During the process of clamping thefruit, the biological characteristics of fruit including its thinand fragile pericarp put a high demand on grasping force ofend-effector (Monta, 1998). It required sensors to control thegrasping force accurately. In addition, the rotation of arm, itstraveling position and accurate capture also required thesensors to detect and locate fruit (Jiang, Cai, & Liu, 2005; Qiao,Wu, & Zhu, 1999). Furthermore, in order to avoid damagingequipment, causing injury and failing to pick fruit, collisionavoidance of the arm also needs sensors to perceive theoperating environment effectively.2.2.1.The sensors on end-effectorThe layout of sensors on end-effector, which includes a visionsensor, a position sensor, a collision sensor and a pressuresensor, is shown in Fig. 4. The vision sensor, which uses high-pixel colour charge coupled devices (CCD) video camera withuniversal serial bus (USB) interface and the video for windows(VFW) capture technology to form image acquisition system,plays a key role in completing image acquisition, fruit searchand recognition. To obtain a wide visible-field and not influ-encedby end-effector, the position of the vision sensoris in aneye-in-hand mode. In Fig. 4, it can be seen that there is thephotoelectricpositionsensorwithtwo pairsofinfrareddoublephotoelectric cells. In addition, the switch position sensorwhich was usually used to limit for electric cutting knife wasalso mounted on the position sensor. The arm began decel-eration when the end-effector moved towards the target fruitguided by the vision sensor and the first pair of photodiodeswas obscured by the fruit in the holder. The arm stopped andthe gripper clamped fruit when the two pairs of photocellswere obscured. At this point, both the pressure and collisionsensors adoptedforce sensitive resistance. When the pressuresensor on the gripper felt a certain pressure, the electric cutterrotated and cuts off pedicel. The cutter stopped working whenthe switch position sensor operated. The collision sensor wasused for obstacle avoidance during the process of harvesting.Analogue signals derived from the force sensitive resistanceand infrared photoelectric tubes are usually incompatiblewith the data acquisition module inside industrial computer.Therefore, they require modulation before transmission to thedata acquisition module. Fig. 5 shows the sensors signalmodulation circuit.2.2.2.The Sensor on manipulator for collision avoidanceControl of the angle of the rotating joints and position controlof the flexible joints was fulfilled using 8 Hall sensors,installed on the rotation joints of waist, the major arm, theminor arm and both ends of flexible joints. In the workingenvironment, the movement space of minor arm was wide;Fig. 3 e Photograph of the end-effector.PositionSensorPressureSensorVisionSensorCollisionSensorFig. 4 e Layout of sensors on end-effector.Fig. 5 e Sensors signal modulation circuit.Table 1 e Motion parameters of manipulator mechanicalstructure.Joint MotionparametersLift platform0 me0.8 mRotation joint of waist?180?e180?Rotation joint of major arm?80?e80?Rotation joint of minor arm?80?e80?Flexible joint0 me0.8 mbiosystems engineering 110 (2011) 112e122115and the probability of collision with obstacles was high.Therefore, the collision sensor was fixed in the minor arm todetect obstacles. Five groups of micro switches were fixed ondifferent positions in the minor arm to obtain real-timeinformation from obstacles. Noting that software program-ming processes signals conveniently, the five groups ofavoidance sensors were designated CR、CRU、CU、CLU、CLin accordance with their position. The distribution of theminor arm collision avoidance sensors is shown in Fig. 6.2.3.The vision systemsFor the vision system of the apple harvesting robot, the keyingredient was the image processing method that recognisedand located the fruit. It affects the robots dependability andalso determines its ability to directly, quickly and accuratelyrecognise in the fruit real time (Bulanon, Kataoka, & Okamoto,2004). However, in the earlier research (Bulanon, Kataoka, &Ota, 2002; Liu, Zhang, & Yang, 2008; Plebe & Grasso, 2001;Zhao, Yang, & Liu, 2004), there exist some unsolved issuessuch as low accuracy rate and time consumption, which tosome extent restricted the real-time and multitasking abilityof the apple harvesting robot in the natural environment.To overcome these shortcomings, a real-time automaticrecognition vision system consisting of a colour CCD cameraforcapturingoriginalappleimagesandanindustrialcomputer for processing images to recognise and locate thefruit was developed. Since the Fuji apples are the mostpopular in China, our research focused on this variety.The recognition and location procedure is as follows.Firstly, due to the natural environment and the imageacquisition device used, the original unprocessed apple imageinevitably includes noise that influences its quality. A vectormedianfilter wasappliedtoimageenhancementpre-processing. It can not only remove noise effectively andhighlights the apple fruit in foreground, but it also maintainsgood image edges.Secondly, most apple images acquired in the naturalconditionsusuallyincludebranchesandleaveswhichcomplicate matters. By using only a conventional imagesegmentation algorithm, it was difficult to achieve anticipatedeffect. Based on hue histogram statistics from the hue,intensity and saturation (HIS) model, the double thresholdand region growing method was employed to develop animage segmentation algorithm for identifying apple fruitfrom complex background. The chromaticity component isirrelevant when lightness is extracted and this avoided theinfluence of different illumination levels on the images. Thealgorithm was simple, and required little processing time.The apple features were extracted to determine the spatiallocation, and provide corresponding motion parameters forarm. For colour feature extraction, the chroma componentshue and saturation, are usually extracted as colour featuresfor recognition. However, in our study, apple fruit, branchesand leaves havespecific shapes, and their differences in shapeare large. Therefore, the shape feature is important in appleobject recognition. The selected rule of shape features wasbased on invariance in rotation, scale and translation (RST).Taking account of characteristics of apple fruit images,circular variance, variance ellipse, tightness, ratio betweenperimeter and square area were used to describe the outlineshape features of apple. These four feature vectors wereextracted as shape features. After the calculation of the cor-responding eigenvalues, they were used as feature vectors ofeach sample and used for training and classification.Finally, a new classification algorithm based on supportvector machine was constructed to recognise the apple fruit.Simulation and experiment shows that the support vectormachine (SVM) method with radial basis function (RBF) kernelfunction based on both colour features and shape featureswas found to be the best for apple recognition. Details of thealgorithm can be found in Wang, Zhao, Ji, Tu, and Zhang(2009).2.4.The control systemThe hardware structure is shown in Fig. 7. At the centre of thecontrol system was the host computer, which integrates thecontrol interface and all of software modules to control thewhole system. The sensor signal acquisition system andimage acquisition system constituted the input section whichwas used to collect external environment information for theFig. 6 e Layout of sensors on minor arm.Servodrivers14Incrementalphotoelectricencoder14Cutter of theend-ffectorUSB interfacePosition limitedsensorDrive motorfor cutterAirpumpCollision sensorInfrared sensorCCD Vision sensorElectricvalveSignalmodulationcircuitGripper of theend-ffectorAC Servomotors andload joints14Data acquisition moduleHost control computerRS232/RS422convertersFig. 7 e Hardware structure of apple harvesting robotcontrol system.biosystems engineering 110 (2011) 112e122116harvesting robot. The output section included a servo drivenmotor, air pump and end-effector.2.4.1.Host computerA Kintek KP-6420i (Kintek Electronics Co., Ltd., Miaoli Hsien,Taiwan, China) industrial computer with Intel Pentium41.7 GHz processor and 512 M memory was selected as the hostcontrol computer, which was responsible for collecting wholesensor signals, processing images online, calculating theinverse kinematics of manipulator and completing the controlalgorithm. The host computer transmitted instructions to thealternating current (AC) servo driver through a serial port tocontrol the joint motors of waist and arms. HighTek HK-5108(Shenzhen FangXingLiuTong Industrial Co., Ltd., Shenzhen,China) RS-232/RS-422 converters were chosen for serialcommunicationfunctions.Adataacquisitionmoduleinstalled inside host computer