Eyes for an agricultural robot: AI system identifies weeds in apple orchards

UNIVERSITY PARK, Pa. — Weed control is essential in apple orchards because weeds compete with trees for nutrients, water and sunlight, which can reduce fruit yields. However, physically removing weeds is not only labor-intensive, but it also can damage soil structure and tree roots. Using chemical sprays to kill weeds can lead to other problems, such as pollution, herbicide resistance and excess chemical residues on apples. Another option called precision weed management — detecting and measuring weeds with high accuracy then applying small amounts of herbicide to control them efficiently — can help farmers avoid wasting chemicals or causing injury to crops or the environment, according to a team of researchers at Penn State. To help growers achieve such precise management, the researchers are developing an automated, robotic weed-management system.

The researchers reported on an early step in that process in the December issue of Computers and Electronics in Agriculture: an AI machine vision model they developed that can accurately find, outline, interpret and estimate the density of weeds in apple orchards. The system, intended to guide the eventual robotic precision sprayer, uses a machine vision innovation that allows a side-view camera to detect and identify weeds for treatment — even weeds that are partially obscured.

“In complex environments like apple orchards, it is difficult to develop weed-detection mechanisms because the tree canopy and low branches block the view from above, precluding traditional top-view camera systems, like drones, because they can’t clearly see the weeds on the ground,” said team leader and study senior author Long He, associate professor of agricultural and biological engineering. His research group in the College of Agricultural Sciences has been studying and developing robotic precision agricultural systems over the last decade. “A side-view camera can help, but weeds might be partially visible or hidden behind untargeted objects or tree trunks. This causes problems such as misidentifying weeds or losing track of a weed in real time.”

To overcome those challenges, study first author Lawrence Arthur, doctoral candidate in the Department of Agricultural and Biological Engineering led the team in improving a commercially available deep-learning model for the machine vision computer program. The model was already capable of fast object detection and segmentation, meaning it can find the weed and outline its exact shape, pixel by pixel.

To make it better, the researchers added a module that helps the model “pay attention” to the most important image features while suppressing irrelevant feature information in the scene. This innovation improved accuracy when parts of weeds are hidden or hard to distinguish, He explained. Also, they integrated a tracking algorithm with a filtering mechanism for more effective weed tracking. The algorithm preserves weed identity across video frames and prevents counting the same weed multiple times, He said. This allows the system to track weeds when they disappear briefly due to being blocked by the apple trees or even other weeds.

The study data was collected at Penn State’s Fruit Research and Extension Center in Biglerville and nearby apple orchards. Weed species included dandelion, common sow thistle, horseweed and Carolina horsenettle. The researchers took high-resolution photos of these weeds to form the dataset for training and testing the AI model.

The model achieved high accuracy and recall for weed detection, making it suitable for automated weed management in orchards,” He pointed out. It achieved 84.9% average precision for detecting segmentation and 83.6% average precision for localization. These numbers indicate strong accuracy for finding weeds and outlining them.

In tracking images across frames, the model scored 82% in multiple object tracking accuracy — meaning it achieved high accuracy in tracking multiple weeds; 78% in multiple object tracking precision — meaning it achieved good precision in estimating weed positions; and 88% in identification score — showing a strong ability to correctly maintain weed identities across video frames. Finally, the model only recorded six identity switches in the study, He noted, meaning the model rarely confuses one weed for another as it tracks them.

The research is a big step toward automated, precise weed control in agriculture, according to He.

“By combining better detection and stronger tracking with added density estimation, the model we developed provides more accurate, consistent weed detection, even in difficult orchard conditions,” he said. “By providing actionable data for site-specific weed management, this approach will improve herbicide efficiency and reduce waste.”

Contributing to the research were Caio Brunharo, assistant professor of weed science; Paul Heinemann, professor of agricultural and biological engineering; Magni Hussain, assistant research professor of electronics, instrumentation and control systems; and Sadjad Mahnan, graduate assistant in the Department of Agricultural and Biological Engineering.

This research was partially supported by the U.S. Department of Agriculture’s National Institute of Food and Agriculture, Pennsylvania Department of Agriculture, and the State Horticultural Association of Pennsylvania. 

At Penn State, researchers are solving real problems that impact the health, safety and quality of life of people across the commonwealth, the nation and around the world.    

For decades, federal support for research has fueled innovation that makes our country safer, our industries more competitive and our economy stronger. Recent federal funding cuts threaten this progress.    

Learn more about the implications of federal funding cuts to our future at Research or Regress.