Taxonomy of FPGA-Based Topologies for Future Deep Learning Architectures

[en] Deep learning, the fastest growing segment of Artificial Neural Network (ANN), has led to the emergence of many machine learning applications and their implementation across multiple platforms such as CPUs, GPUs and reconfigurable hardware (Field-Programmable Gate Arrays or FPGAs). However, inspired by the structure and function of ANNs, large-scale deep learning topologies require a considerable amount of parallel processing, memory resources, high throughput and significant processing power. Consequently, in the context of real time hardware systems, it is crucial to find the right trade-off between performance, energy efficiency, fast development, and cost. Although limited in size and resources, several approaches have showed that FPGAs provide a good starting point for the development of future deep learning implementation architectures. Through this paper, we briefly review recent work related to the implementation of deep learning algorithms in FPGAs. We will analyze and compare the design requirements and features of existing topologies to finally propose development strategies and implementation architectures for better use of FPGA-based deep learning topologies. In this context, we will examine the frameworks used in these studies, which will allow testing a lot of topologies to finally arrive at the best implementation alternatives in terms of performance and energy efficiency. Keywords: Deep Learning, Framework, Optimized implementation, FPGA.

Research center :

CRTI - Centre de Recherche en Technologie de l'Information

Disciplines :

Computer science
Electrical & electronics engineering
Physics
Library & information sciences

Author, co-author :

BLAIECH, Ahmed Ghazi

BEN KHALIFA, Khaled

Valderrama, Carlos ; Université de Mons > Faculté Polytechnique > Service d'Electronique et Microélectronique

FERNANDES, Marcelo A.C.

BEDOUI, Mohamed Hedi

Language :

English

Title :

Taxonomy of FPGA-Based Topologies for Future Deep Learning Architectures

Publication date :

23 January 2019

Journal title :

Journal of Systems Architecture

ISSN :

1383-7621

Publisher :

Elsevier, Netherlands

Peer reviewed :

Peer reviewed

Research unit :

F109 - Electronique et Microélectronique

Research institute :

R300 - Institut de Recherche en Technologies de l'Information et Sciences de l'Informatique
R450 - Institut NUMEDIART pour les Technologies des Arts Numériques

Available on ORBi UMONS :

since 24 January 2019

Statistics

Number of views

46 (1 by UMONS)

Number of downloads

0 (0 by UMONS)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Graves, A., Mohamed, A.-r., Hinton, G., Speech recognition with deep recurrent neural networks. Proceedings of the IEEE International Conference on ICASSP, 2013, 6645–6649 http://doi.org/10.1109/ICASSP.2013.6638947.
Sutskever, I., Vinyals, O., Le, Q.V., Sequence to sequence learning with neural networks. Proceedings of the Advances in Neural information Processing Systems, 2014, 3104–3112 https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf.
Byeon, W., Liwicki, M., Breuel, T.M., Scene analysis by mid-level attribute learning using 2d LSTM networks and an application to web-image tagging. Pattern Recognit. Lett. vol. 63 (2015), 23–29 https://doi.org/10.1016/j.patrec.2015.06.003.
Socher, R., Huval, B., Bath, B., Manning, C.D., Ng, A., Convolutional-recursive deep learning for 3d object classification. Proceedings of the Advances in Neural Information Processing Systems, 2012, 665–673 https://papers.nips.cc/paper/4773-convolutional-recursive-deep-learning-for-3d-object-classification.pdf.
Yan, Z., Zhan, Y., Zhang, S., Metaxas, D., Zhou, X.S., Multi-instance multi-stage deep learning for medical image recognition book chapter in Deep Learning for Medical Image Analysis. 2017, 83–104 https://doi.org/10.1016/b978-0-12-12-810408-8.00006-7.
Xu, J., Zhou, C., Lang, B., Liu, Q., Deep learning for histopathological image analysis: towards computerized diagnosis on cancers. Deep Learning and Convolutional Neural Networks for Medical Image Computing, 2017, 73–95 http://doi.org/10.1007/978-3-319-42999-1_6.
Skourt, B.A., Hassani, A.El, Majda, A., Lung CT image segmentation using deep neural networks. Procedia Comput. Sci. vol. 127 (2018), 109–113, 10.1016/j.procs.2018.01.104.
A. Kalinovsky, V. Kovalev, Lung image segmentation using deep learning methods and convolutional neural networks, Proceedings of the XIII International Conference on Pattern Recognition and Information Processing (PRIP) At: Minsk, Belarus. http://elib.bsu.by/bitstream/123456789/158557/1/Kallinovsky_Kovalev.pdf.
Carneiro, G., Nascimento, J., Andrew, P., Bradley, deep learning models for classifying mammogram exams containing unregistered multi-view images and segmentation maps of lesions. book chapter from Deep Learning for Medical Image Analysis, 2016 https://doi.org/10.1016/B978-0-12-810408-8.00019-5.
Zhang, W., Lu, Z., Wu, Z., Wu, J., Zou, H., Huang, S., Toy-IoT-oriented data-driven CDN performance evaluation model with deep learning. J. Syst. Archit. vol. 88 (2018), 13–22, 10.1016/j.sysarc.2018.05.005.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L., ImageNet: a large-scale hierarchical image database. Proceedings of the CVPR09, 2009 https://doi.org/10.1109/CVPR.2009.5206848.
Krizhevsky, A., One Weird Trick For Parallelizing Convolutional Neural Networks. 2014 in arXiv: 1404.5997.
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E., cuDNN: Efficient Primitives For Deep Learning. 2014 in arXiv: 1410.0759.
Dundar, A., Jin, J., Gokhale, V., Martini, B., Culurciello, E., Memory access optimized routing scheme for deep networks on a mobile coprocessor. Proceedings of the HPEC, 2014, 1–6 2014 https://doi.org/10.1109/HPEC.2014.7040963.
Intel® Movidius™ Myriad™ X VPU, Ultimate performance at ultra-low power, https://www.movidius.com/myriadx.
Han, S., et al. EIE: efficient inference engine on compressed deep neural network. Proceedings of the ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 2016, 234–254 https://doi.org/10.1145/3007787.3001163.
Li, Z., Wang, Y., Zhi, T., Chen, T., A survey of neural network accelerators. Frontiers of Computer Science 11:5 (2017), 746–761, 10.1007/s11704-016-6159-1.
Lacey, G., Taylor, G., Areibi, S., Deep learning on FPGAs: past, present, and future. 2016 in arXiv: 1602.04283.
Guo, K., Zeng, S., Yu, J., Wang, Y., Yang, H., A survey of FPGA Based neural network accelerator. in arXiv https://arxiv.org/pdf/1712.08934.pdf, 2018.
Abdelouahab, K., Pelcat, M., Sérot, J., Berry, F., Accelerating CNN inference on FPGAs: a survey. in arXiv https://arxiv.org/pdf/1806.01683.pdf, 2018.
Daugman, J., How iris recognition works. IEEE Trans. Circuits Syst. Video Technol. 14 (2004), 21–30.
Bigun, J., Alonso-Fernandez, F., Hofbauer, H., Uhl, A., Experimental analysis regarding the influence of iris segmentation on the recognition rate. IET Biom. 5:3 (2016), 200–211.
Krizhevsky, A., Hinton, G., Learning Multiple Layers Of Features From Tiny Images. 2009.
Wang, C., Yu, Q., Gong, L., Li, X., Xie, Y., Zhou, X., DLAU: a scalable deep learning accelerator unit on FPGA. Proceedings of the IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Volume 36, Issue 3, March 2017, 10.1109/TCAD.2016.2587683 March.
LeCun, L.B., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324 https://doi.org/10.1109/5.726791.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y., Reading digits in natural images with unsupervised feature learning. Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011 http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf.
Chang, A.X.M., Martini, B., Culurciello, E., Recurrent neural networks hardware implementation on FPGA. Int. J. Adv. Res. Electr. Electron. Instrum. Eng., 5(1), 2016 January https://pdfs.semanticscholar.org/1a27/a30724ed0409791db99d96551d341807faed.pdf.
A. Menegola, M. Fornaciali, R. Pires, S. Avila, E. Valle, Towards Automated Melanoma Screening: Exploring Transfer Learning Schemes, Published 2016 in ArXiv: 1609.01228.
Razzak, M.I., Naz, S., Zaib, A., Deep learning for medical image processing: overview, challenges and the future. Classification in Bioapps, 2017, 323–350 https://doi.org/10.1007/978-3-319-65981-7_12.
Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Proceedings of the Neural Information Processing Systems Conference, 2012, 1097–1105 https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf.
Jarrett, K., Kavukcuoglu, K., Ranzato, M.A., LeCun, Y., What is the best multi-stage architecture for object recognition. Proceedings of the 12th IEEE International Conference on Computer Vision, 2009, 2146–2153 https://doi.org/10.1109/ICCV.2009.5459469.
Farabet, C., Martini, B., Akselrod, P., Talay, S., LeCun, Y., Cu- lurciello, E., Hardware accelerated convolutional neural networks for synthetic vision systems. Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), 2010, 257–260 https://doi.org/10.1109/ISCAS.2010.5537908.
Xilinx Inc. Vivado Design Suite. http://www.xilinx.com/products/design-tools/vivado.html.
Intel Quartus Prime Software Suite, https://www.intel.com/content/www/us/en/software/programmable/quartus-prime/overview.html.
Suda, N., Chandra, V., Dasika, G., Mohanty, A., Ma, Y., Vrudhula, S., Seo, J.-S., Cao, Y., Throughput-optimized opencl-based FPGA accelerator for large-scale convolutional neural networks. Proceedings of the ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, ACM, 2016.
Xiao, Q., Liang, Y., Lu, L., Yan, S., Tai, Y.-W., Exploring heterogeneous algorithms for accelerating deep convolutional neural networks on FPGAs. Proceedings of the 54th Annual Design Automation Conference DAC '17, 2017, 10.1145/3061639.3062244.
AWS Cloud: Xilinx FPGAs in world's largest cloud, https://www.xilinx.com/products/design-tools/acceleration-zone/aws.html.
Adaptive machine learning acceleration, https://www.xilinx.com/applications/megatrends/machine-learning.html.
Mipsology: the future of FPGA-based machine learning, https://www.xilinx.com/support/documentation/product-briefs/mipsology-aws-f1.pdf.
DiCecco, R., Lacey, G., Vasiljevic, J., Chow, P., Taylor, G., Areibi, S., Caffeinated FPGAs: FPGA framework for convolutional neural networks. Proceedings of the International Conference on Field-Programmable Technology (FPT), 2016, 7–9 Dec. 2016, Xi'an, China. doi: 10.1109/FPT.2016.7929549.
Ma, Y., Suda, N., Cao, Y., Vrudhula, S., Seo, J.S., ALAMO: FPGA acceleration of deep learning algorithms with a modularized RTL compiler. Integr. VLSI J., 2018, 10.1016/j.vlsi.2017.12.009.
Morcel, R., Akkary, H., Hajj, H., Saghir, M., Keshavamurthy, A., Khanna, R., Artail, H., Minimalist design for accelerating convolutional neural networks for low-end FPGA platforms. Proceedings of the IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2017, IEEE, 196 196.
Posewsky, T., Ziener, D., A flexible FPGA-based inference architecture for pruned deep neural networks. Proceedings of the ARCS, 2018, 311–323 https://doi.org/10.1007/978-3-319-77610-1_23.
Sharma, H., Park, J., Mahajan, D., Amaro, E., Kim, J.K., Shao, C., Mishra, A., Esmaeilzadeh, H., From high-level deep neural models to FPGAs. Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016, IEEE, 1–12 https://doi.org/10.1109/MICRO.2016.7783720.
H. Zeng, R. Chen, C. Zhang, and V. Prasanna, A framework for generating high throughput CNN implementations on FPGAs, In Proceedings of the ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA '18). ACM, New York, NY, USA, 117–126. doi: 10.1145/3174243.3174265.
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H., MobileNets: efficient convolutional neural networks for mobile vision applications. 2017 in arXiv: 1704.04861.
Zhang, C., Fang, Z., Zhou, P., Pan, P., Cong, J., Caffeine: towards uniformed representation and acceleration for deep convolutional neural networks. Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2016, 1–8 https://doi.org/10.1145/2966986.2967011.
Y. Wang, J. Xu, Y. Han, H. Li, and X. Li, DeepBurning: automatic generation of FPGA-based learning accelerators for the neural network family, In Proceedings of the 53nd ACM/EDAC/IEEE Design Automation Conference (DAC), 1–6. https://doi.org/10.1145/2897937.2898002.
Zhu, M., Liu, L., Wang, C., Xie, Y., CNNLab: a novel parallel framework for neural networks using GPU and FPGA – a practical study with trade-off analysis. distributed, parallel, and cluster computing (cs.DC) https://arxiv.org/pdf/1606.06234.pdf, 2016.
Solazzo, A., Sozzo, E.D., Rose, I.D., Silvestri, M.D., Gianluca, C, Durelli, Santambrogio, M.D, Hardware design automation of convolutional neural networks. Proceedings of the IEEE Computer Society Annual Symposium on VLSI (ISVLSI), 2016, 224–229, 10.1109/isvlsi.2016.101.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Going Deeper with Convolutions. 2014 in arXiv: 1409.4842.
Simonyan, K., Zisserman, A., Very deep convolutional networks for large scale visual recognition., 2015 in arXiv: 1409.1556.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y., OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks. 2013 in arXiv: 1312.6229.
Umuroglu, Y., Fraser, N.J., Gambardella, G., Blott, M., Leong, P., Jahre, M., Vissers, K., FINN: A Framework for Fast, Scalable Binarized Neural Network Inference. Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays FPGA'17, Monterey, California, USA, 2017, 65–74, 10.1145/3020078.3021744 February 22–24.
Fujii, T., Sato, S., Nakahara, H., Motomura, M., An FPGA realization of a deep convolutional neural network using a threshold neuron pruning. Proceedings of the ARC LNCS, 10216, 2017, 268–280, 10.1007/978-3-319-56258-2_23.
Lin, D.D., Talathi, S.S., Annapureddy, V.S., Fixed Point Quantization of Deep Convolutional Networks. 2015 in arXiv: 1511.06393.
Judd, P., Albericio, J., Hetherington, T., Aamodt, T., Jerger, N.E., Urtasun, R., Moshovos, A., Proteus: Exploiting precision variability in deep neural networks. Parallel Comput. vol. 73 (2018), 40–51, 10.1016/j.parco.2017.05.003 ISSN 0167-8191.
Lai, L., Suda, N., Chandra, V., Deep Convolutional Neural Network Inference with Floating-Point Weights and Fixed-Point Activations. 2017 in arXiv: 1703.03073.
Li, Y., Liu, Z., Xu, K., Yu, H., A GPU-outperforming FPGA accelerator architecture for binary convolutional neural networks. ACM J. Emerg. Technol. Comput. Syst., 14(2), 2018, 10.1145/3154839 18.
Han, S., Mao, H., Dally, W.J., Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. 2015 in arXiv: 1510.00149.
Shafique, M., Sampaio, F., Zatt, B., Bampi, S., Henkel, J., Resilience-driven STT-RAM cache architecture for approximate computing. Proceedings of the Workshop on Approximate Computing (AC), Paderborn, Germany, 2015 Oct. 15–16. https://pdfs.semanticscholar.org/8b32/d0227449b04ecbcdd082f76a5956969bdfe5.pdf.
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S, Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X., TensorFlow: large-scale machine learning on heterogeneous systems. http://tensorflow.org/, 2015 Software available from tensorflow.org.
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T., Caffe: Convolutional Architecture For Fast Feature Embedding. 2014 in arXiv: 1408.5093.
High, R., The Era of Cognitive Systems: An Inside Look at IBM Watson and How it Works. 2012, IBM Corporation, Redbooks, Armonk Google Scholar.
Canis, A., Choi, J., Aldham, M., Zhang, V., Kammoona, A., Czajkowski, T., Brown, S.D, Anderson, J.H., LegUp: an open-source high level synthesis tool for FPGA-based processor/accelerator systems. ACM Trans. Embed. Comput. Syst., 13(2, Article 24), Sept. 2013 27 pages https://doi.org/10.1145/2514740.
S. Chintala. Convnet-benchmarks. [Online]. Available: https://github.com/soumith/convnet-benchmarks.
Dundar, A., Jin, J., Martini, B., Culurciello, E., Embedded streaming deep neural networks accelerator with applications. IEEE Trans. Neural Netw. Learn. Syst., 28(7), 2016, 10.1109/TNNLS.2016.2545298.
Collobert, R., Farabet, C., Kavukcuoglu, K., Torch7: a matlab-like environment for machine learning. Proceedings of the BigLearn NIPS Workshop, 2011 no. EPFL-CONF-192376 https://pdfs.semanticscholar.org/6074/c1108997e0c1f97dc3c199323a162ffe978d.pdf.
Ortega-Zamorano, F., Jerez, J.M., Gómez, I., Franco, L., Deep neural network architecture implementation on FPGAS using a layer multiplexing scheme. proceedings of the 13th International Conference Distributed Computing and Artificial Intelligence, 2016, 79–86 https://doi.org/:10.1007/978-3-319-40162-1_9.
Alwani, M., Chen, H., Ferdman, M., Milder, P., Fused-layer CNN accelerators. Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016, 10.1109/MICRO.2016.7783725.
Chainer: a powerful, flexible, and intuitive framework of neural networks. http://chainer.org/.
Liang, S., Yin, S., Liu, L., Luk, W., Wei, S., FP-BNN: binarized neural network on FPGA. Neurocomputing, 2017, 10.1016/j.neucom.2017.09.046.
Chen, T., Moreau, T., Jiang, Z., Zheng, L., Cowan, E.YanMeghan, Shen, H., Wang, L., Hu, Y., Ceze, L., Guestrin, C., Krishnamurthy, A., TVM: An automated end-to-end optimizing compiler for deep learning. 20 May https://arxiv.org/pdf/1802.04799.pdf, 2018.
Lin, M., Chen, Q., Yan, S., Network in network., 2014 arXiv: 1312.4400.
Wang, Y., Ding, C., Li, Z., Yuan, G., Liao, S., Ma, X., Yuan, B., Qian, X., Tang, J., Qiu, Q., Lin, X., Towards ultra-high performance and energy efficiency of deep learning systems: an algorithm-hardware co-optimization framework. 18 Feb https://arxiv.org/pdf/1802.06402.pdf, 2018.
Yazdanbakhsh, A., Brzozowski, M., Khaleghi, B., Ghodrati, S., Samadi, K., Kim, N.S., Esmaeilzadeh, H., FlexiGAN: an end-to-end solution for fpga acceleration of generative adversarial networks. Appears in the Proceedings of the 26th IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2018 https://www.cc.gatech.edu/∼hadi/doc/paper/2018-fccm-flexigan.pdf.
Noronha, D.H., Leong, P.H.W., Wilton, S.J.E., Kibo: an open-source fixed-point tool-kit for training and inference in fpga-based deep learning networks. Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops, 2018, 10.1109/IPDPSW.2018.00034 21–25 May.
Ding, W., Huang, Z., Huang, Z., Tian, L., Wang, H., Feng, S., Designing efficient accelerator of depthwise separable convolutional neural network on FPGA. J. Syst. Archit., 2018, 10.1016/j.sysarc.2018.12.008 available online 28 December.
Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Comput. 9:8 (1997), 1735–1780 http://doi.org/10.1162/neco.1997.9.8.1735.
He, K., Zhang, X., Ren, S., Sun, J., Identity Mappings in Deep Residual Networks. 2016 in arXiv: 1603.05027.
Wu and al., J., Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling. Proceedings of the NIPS, 2016, 82–90 https://jiajunwu.com/papers/3dgan_nips.pdf.
W.R. Tan and al., ArtGAN: Artwork Synthesis with Conditional Categorial GANs, 2017. in arXiv: 1702.03410.