[en] Recent deep learning methods have allowed important steps forward in the automatic detection of wheat ears in the field. Nevertheless, it was still lacking a method able to both count and segment the ears, validated at all the development stages from heading to maturity. Moreover, the critical step of converting the ear count in an image to an ear density, i.e. a number of ears per square metre in the field, has been widely ignored by most of the previous studies. For this research, wheat RGB images have been acquired from heading to maturity in two field trials displaying contrasted fertilisation scenarios. An unsupervised learning approach on the YOLOv5 model, as well as the cutting-edge DeepMAC segmentation method were exploited to develop a wheat ear counting and segmentation pipeline that necessitated only a limited amount of labelling work for the training. An additional label set including all the development stages was built for validation. The average F1 score of ear bounding box detection was 0.93 and the average F1 score of segmentation was 0.86. To convert the ear counts to ear densities, a second RGB camera was used so that the distance between the cameras and the ears could be measured by stereovision.
That distance was exploited to compute the image footprint at ear level, and thus divide the number of ears by this footprint to get the ear density. The obtained ear densities were coherent regarding the fertilisation scenarios but, for a same fertilisation, differences were observed between acquisition dates. This highlights that the measurement was not able to retrieve absolute ear densities for all the development stages and conditions. The deep learning measurement considered the most reliable outperformed observations from three human operators.
Disciplines :
Computer science
Author, co-author :
Dandrifosse, Sébastien ✱
Ennadifi, Elias ✱; Université de Mons - UMONS > Faculté Polytechnique > Service Information, Signal et Intelligence artificielle
Carlier, Alexis; ULiège - University of Liège [BE] > TERRA Teaching and Research Centre, Gembloux Agro-Bio Tech > Biosystems Dynamics and Exchanges
Gosselin, Bernard ; Université de Mons - UMONS > Faculté Polytechnique > Service Information, Signal et Intelligence artificielle
Dumont, Benjamin; ULiège - University of Liège [BE] > TERRA Teaching and Research Centre, Gembloux Agro-Bio > Plant Sciences
Mercatoris, Benoît; ULiège - University of Liège [BE] > TERRA Teaching and Research Centre, Gembloux Agro-Bio Tech > Biosystems Dynamics and Exchanges
✱ These authors have contributed equally to this work.
Language :
English
Title :
Deep Learning for Wheat Ear Segmentation and Ear Density Measurement: From Heading to Maturity
Publication date :
06 July 2022
Journal title :
Computers and Electronics in Agriculture
ISSN :
0168-1699
eISSN :
1872-7107
Publisher :
Elsevier, Netherlands
Peer reviewed :
Peer Reviewed verified by ORBi
Research unit :
F105 - Information, Signal et Intelligence artificielle
Research institute :
R450 - Institut NUMEDIART pour les Technologies des Arts Numériques
Name of the research project :
3970 - PHENWHEAT - Caractérisation de la dynamique de croissance de cultures de froment d’hiver au moyen d’une plateforme de phénotypage par proxidétection en conditions variables de stress biotique et abiotique - Région wallonne
Alharbi, N., Zhou, J., Wang, W., Automatic Counting of Wheat Spikes from Wheat Growth Images, in. Proceedings of the 7th International Conference on Pattern Recognition Applications and Methods, 2018, 346–355, 10.5220/0006580403460355.
Birodkar, V., Lu, Z., Li, S., Rathod, V., Huang, J., 2021. The surprising impact of mask-head architecture on novel class segmentation. arXiv abs/2104.0.
Bradski, G., Kaehler, A., Learning OpenCV. O'Reilly Media Inc, Newton, MA, USA, 2008, 10.1109/MRA.2009.933612.
Cao, L., Zhang, X., Pu, J., Xu, S., Cai, X., Li, Z., 2020. The Field Wheat Count Based on the Efficientdet Algorithm. In: Proc. 2020 IEEE 3rd Int. Conf. Inf. Syst. Comput. Aided Educ. ICISCAE 2020 557–561. https://doi.org/10.1109/ICISCAE51034.2020.9236918.
Carlier, A., Dandrifosse, S., Dumont, B., Mercatoris, B., Wheat Ear Segmentation Based on a Multisensor System and Superpixel Classification. Plant Phenom. 2022 (2022), 1–10 10.34133/2022/9841985.
Chopinet, B., Cointault, F., 2006. Colour-texture image analysis for in-field wheat head counting. In: Second International Symposium on Communications, Control, and Signal Processing (ISCCSP 2006). pp. 13–15.
Cointault, F., Guerin, D., Guillemin, J.P., Chopinet, B., In-field triticum aestivum ear counting using colour-texture image analysis. New Zeal. J. Crop Hortic. Sci. 36 (2008), 117–130, 10.1080/01140670809510227.
Cointault, F., Journaux, L., Miteran, O., Destain, M.F., Improvements of image processing for wheat ear counting. Agricultural and Biosystems Engineering for a Sustainable World. International Conference on Agricultural Engineering, 2008.
Cointault, F., Journaux, L., Rabatel, G., Germain, C., Ooms, D., Destain, M.-F., Gorretta, N., Grenier, G., Lavialle, O., Marin, A., Texture, Color and Frequential Proxy-Detection Image Processing for Crop Characterization in a Context of Precision Agriculture. Aflakpui, G., (eds.) Agricultural Science, 2012, INTech, London, 49–70.
Dandrifosse, S., Bouvry, A., Leemans, V., Dumont, B., Mercatoris, B., Imaging wheat canopy through stereo vision : overcoming the challenges of the laboratory to field transition for morphological features extraction. Front. Plant Sci. 11 (2020), 1–15, 10.3389/fpls.2020.00096.
David, E., Madec, S., Sadeghi-Tehran, P., Aasen, H., Zheng, B., Liu, S., Kirchgessner, N., Ishikawa, G., Nagasawa, K., Badhon, M.A., Pozniak, C., de Solan, B., Hund, A., Chapman, S.C., Baret, F., Stavness, I., Guo, W., Global Wheat Head Detection (GWHD) dataset: a large and diverse dataset of high resolution RGB labelled images to develop and benchmark wheat head detection methods. Plant Phenom. 2020 (2020), 1–12 10.34133/2020/3521852.
David, E., Serouart, M., Smith, D., Madec, S., Velumani, K., Liu, S., Wang, X., Pinto, F., Shafiee, S., Tahir, I.S.A., Tsujimoto, H., Nasuda, S., Zheng, B., Kirchgessner, N., Aasen, H., Hund, A., Sadhegi-Tehran, P., Nagasawa, K., Ishikawa, G., Dandrifosse, S., Carlier, A., Dumont, B., Mercatoris, B., Evers, B., Kuroki, K., Wang, H., Ishii, M., Badhon, M.A., Pozniak, C., LeBauer, D.S., Lillemo, M., Poland, J., Chapman, S., de Solan, B., Baret, F., Stavness, I., Guo, W., Global Wheat Head Detection 2021: An Improved Dataset for Benchmarking Wheat Head Detection Methods. Plant Phenom. 2021 (2021), 1–9 10.34133/2021/9846158.
Fernandez-Gallego, J.A., Kefauver, S.C., Gutiérrez, N.A., Nieto-Taladriz, M.T., Araus, J.L., Wheat ear counting in-field conditions: High throughput and low-cost approach using RGB images. Plant Methods 14 (2018), 1–12, 10.1186/s13007-018-0289-4.
Fernandez-Gallego, J.A., Lootens, P., Borra-Serrano, I., Derycke, V., Haesaert, G., Roldán-Ruiz, I., Araus, J.L., Kefauver, S.C., Automatic wheat ear counting using machine learning based on RGB UAV imagery. Plant J. 103 (2020), 1603–1613, 10.1111/tpj.14799.
Fourati, F., Mseddi, W.S., Attia, R., Wheat Head Detection using Deep, Semi-Supervised and Ensemble Learning. Can. J. Remote Sens. 47:2 (2021), 198–208.
Germain, C., Rousseaud, R., Grenier, G., Non destructive counting of wheat ear with picture analysis. Fifth International Conference on Image Processing and Its Applications. IET, 1995, 435–439, 10.1049/cp:19950696.
Gong, B., Ergu, D., Cai, Y., Ma, B., Real-Time Detection for Wheat Head Applying Deep Neural Network. Sensors, 21, 2020, 191, 10.3390/s21010191.
Grbović, Z., Panić, M., Marko, O., Brdar, S., Crnojević, V., 2019. Wheat Ear Detection in RGB and Thermal Images Using Deep Neural Networks. In: International Conference on Machine Learning and Data Mining, MLDM 2019. ibai, Fockendorf, Germany.
Hasan, M.M., Chopin, J.P., Laga, H., Miklavcic, S.J., Detection and analysis of wheat spikes using Convolutional Neural Networks. Plant Methods 14 (2018), 1–13, 10.1186/s13007-018-0366-8.
Hirschmüller, H., Stereo Processing by Semi-Global Matching and Mutual Information. IEEE Transactions On Pattern Analysis and Machine Intelligence, 2007, IEEE, Manhattan, NY, USA, 328–341.
Li, J., Li, C., Fei, S., Ma, C., Chen, W., Ding, F., Wang, Y., Li, Y., Shi, J., Xiao, Z., Wheat ear recognition based on retinanet and transfer learning. Sensors, 21(14), 2021, 4845.
Li, W., Jiang, J., Weiss, M., Madec, S., Tison, F., Philippe, B., Comar, A., Baret, F., Impact of the reproductive organs on crop BRDF as observed from a UAV. Remote Sens. Environ., 259, 2021, 112433.
Lu, H., Cao, Z., TasselNetV2+: A Fast Implementation for High-Throughput Plant Counting From High-Resolution RGB Imagery. Front. Plant Sci. 11 (2020), 1–15, 10.3389/fpls.2020.541960.
Ma, J., Li, Y., Du, K., Zheng, F., Zhang, L., Gong, Z., Jiao, W., Segmenting ears of winter wheat at flowering stage using digital images and deep learning. Comput. Electron. Agric., 168, 2020, 105159, 10.1016/j.compag.2019.105159.
Ma, J., Li, Y., Liu, H., Du, K., Zheng, F., Wu, Y., Zhang, L., Improving segmentation accuracy for ears of winter wheat at flowering stage by semantic segmentation. Comput. Electron. Agric., 176, 2020, 105662, 10.1016/j.compag.2020.105662.
Madec, S., Jin, X., Lu, H., De Solan, B., Liu, S., Duyme, F., Heritier, E., Baret, F., Ear density estimation from high resolution RGB imagery using deep learning technique. Agric. For. Meteorol. 264 (2019), 225–234, 10.1016/j.agrformet.2018.10.013.
Meier, U., 2001. Growth Stages of Mono and Dicotyledonous Plants. BBCH Monograph 2nd Edition. Federal Biological Research Centre for Agriculture and Forestry, Quedlinburg, Germany. https://doi.org/10.5073/20180906-074619.
Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.N., Fast Global Image Smoothing Based on Weighted Least Squares. IEEE Trans. Image Process. 23 (2014), 5638–5653, 10.1109/TIP.2014.2366600.
Oscarson, P., The strategy of the wheat plant in acclimating growth and grain production to nitrogen availability. J. Exp. Bot. 51 (2000), 1921–1929, 10.1093/jexbot/51.352.1921.
Pound, M.P., Atkinson, J.A., Wells, D.M., Pridmore, T.P., French, A.P., 2017. Deep learning for multi-task plant phenotyping. In: Proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017. pp. 2055–2063. https://doi.org/10.1109/ICCVW.2017.241.
Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016. You only look once: Unified, real-time object detection. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2016-Decem, 779–788. https://doi.org/10.1109/CVPR.2016.91.
Ren, S., He, K., Girshick, R., Sun, J., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 39 (2017), 1137–1149, 10.1109/TPAMI.2016.2577031.
Sadeghi-Tehran, P., Virlet, N., Ampe, E.M., Reyns, P., Hawkesford, M.J., DeepCount: In-Field Automatic Quantification of Wheat Spikes Using Simple Linear Iterative Clustering and Deep Convolutional Neural Networks. Front. Plant Sci. 10 (2019), 1–16, 10.3389/fpls.2019.01176.
Saeys, W., Lenaerts, B., Craessaerts, G., De Baerdemaeker, J., Estimation of the crop density of small grains using LiDAR sensors. Biosyst. Eng. 102 (2009), 22–30, 10.1016/j.biosystemseng.2008.10.003.
Solovyev, R., Wang, W., Gabruseva, T., Weighted boxes fusion: Ensembling boxes from different object detection models. Image Vis. Comput., 107, 2021, 104117.
Su, W.H., Zhang, J., Yang, C., Page, R., Szinyei, T., Hirsch, C.D., Steffenson, B.J., Automatic evaluation of wheat resistance to fusarium head blight using dual mask-rcnn deep learning frameworks in computer vision. Remote Sens. 13 (2021), 1–20, 10.3390/rs13010026.
Tan, C., Zhang, P., Zhang, Y., Zhou, X., Wang, Z., Du, Y., Mao, W., Li, W., Wang, D., Guo, W., Rapid Recognition of Field-Grown Wheat Spikes Based on a Superpixel Segmentation Algorithm Using Digital Images. Front. Plant Sci. 11 (2020), 1–13, 10.3389/fpls.2020.00259.
Vallat, R., Pingouin: statistics in Python. J. Open Source Softw., 3, 2018, 1026 10.21105/joss.01026.
Velumani, K., Oude Elberink, S., Yang, M.Y., Baret, F., Wheat Ear Detection in Plots by Segmenting Mobile Laser Scanner Data. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 4 (2017), 149–156, 10.5194/isprs-annals-IV-2-W4-149-2017.
Wang, D., Fu, Y., Yang, G., Yang, X., Liang, D., Zhou, C., Zhang, N., Wu, H., Zhang, D., Combined Use of FCN and Harris Corner Detection for Counting Wheat Ears in Field Conditions. IEEE Access 7 (2019), 178930–178941, 10.1109/ACCESS.2019.2958831.
Wang, Y., Qin, Y., Cui, J., Occlusion Robust Wheat Ear Counting Algorithm Based on Deep Learning. Front. Plant Sci. 12 (2021), 1–14, 10.3389/fpls.2021.645899.
Wu, Y., Hu, Y., Li, L., BTWD: Bag of Tricks for Wheat Detection. 2020, Springer International Publishing, New-York, NY, USA, 450–460.
Xiong, H., Cao, Z., Lu, H., Madec, S., Liu, L., Shen, C., TasselNetv2: In-field counting of wheat spikes with context-augmented local regression networks. Plant Methods, 15, 2019, 10.1186/s13007-019-0537-2.
Xiong, X., Duan, L., Liu, L., Tu, H., Yang, P., Wu, D., Chen, G., Xiong, L., Yang, W., Liu, Q., Panicle-SEG: A robust image segmentation method for rice panicles in the field based on deep learning and superpixel optimization. Plant Methods 13 (2017), 1–15, 10.1186/s13007-017-0254-7.
Yang, B., Gao, Z., Gao, Y., Zhu, Y., Rapid Detection and Counting of Wheat Ears in the Field Using YOLOv4 with Attention Module. Agronomy 11 (2021), 1–17.
Yang, Y., Huang, X., Cao, L., Chen, L., Huang, K., 2019. Field Wheat Ears Count Based on YOLOv3, in: Proceedings - 2019 International Conference on Artificial Intelligence and Advanced Manufacturing, AIAM 2019. IEEE, Manhattan, NY, USA, pp. 444–448. https://doi.org/10.1109/AIAM48774.2019.00094.
Zhao, J., Zhang, X., Yan, J., Qiu, X., Yao, X., Tian, Y., Zhu, Y., Cao, W., A Wheat Spike Detection Method in UAV Images Based on Improved YOLOv5. Remote Sens. 13 (2021), 1–16.
Zhou, C., Liang, D., Yang, X., Xu, B.o., Yang, G., Recognition of wheat spike from field based phenotype platform using multi-sensor fusion and improved maximum entropy segmentation algorithms. Remote Sens., 10(2), 2018, 246.
Zhou, C., Liang, D., Yang, X., Yang, H., Yue, J., Yang, G., Wheat ears counting in field conditions based on multi-feature optimization and TWSVM. Front. Plant Sci., 9, 2018, 10.3389/fpls.2018.01024.
