Vol. 8 No. 1 (2022): Special Issue on Embedded Systems for Computer Vision
Special Issue on Embedded Systems for Computer Vision

Introduction to the Special Issue on Embedded Systems for Computer Vision

Samarjit Chakraborty
The University of North Carolina (UNC) at Chapel Hill, US
Qing Rao
Momenta Europe GmbH, Stuttgart, Germany

Published 2022-11-16

Keywords

  • Embedded systems,
  • Computer vision,
  • Cyber-physical systems,
  • Computer architecture

How to Cite

[1]
Chakraborty, S. and Rao, Q. 2022. Introduction to the Special Issue on Embedded Systems for Computer Vision. Leibniz Transactions on Embedded Systems. 8, 1 (Nov. 2022), 00:1–00:8. DOI:https://doi.org/10.4230/LITES.8.1.0.

Abstract

We provide a broad overview of some of the current research directions at the intersection of embedded systems and computer vision, in addition to introducing the papers appearing in this special issue. Work at this intersection is steadily growing in importance, especially in the context of autonomous and cyber-physical systems design. Vision-based perception is almost a mandatory component in any autonomous system, but also adds myriad challenges like, how to efficiently implement vision processing algorithms on resource-constrained embedded architectures, and how to verify the functional and timing correctness of these algorithms. Computer vision is also crucial in implementing various smart functionality like security, e.g., using facial recognition, or monitoring events or traffic patterns. Some of these applications are reviewed in this introductory article. The remaining articles featured in this special issue dive into more depth on a few of them.

References

  1. Tanya Amert and James H. Anderson. Cupid^rt: Detecting improper GPU usage in real-time applications. In 24th IEEE International Symposium on Real-Time Distributed Computing (ISORC), 2021.
  2. Tanya Amert, Michael Balszun, Martin Geier, F. Donelson Smith, James H. Anderson, and Samarjit Chakraborty. Timing-predictable vision processing for autonomous systems. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021.
  3. Tanya Amert, Zelin Tong, Sergey Voronov, Joshua Bakita, F. Donelson Smith, and James H. Anderson. Timewall: Enabling time partitioning for real-time multicore+accelerator platforms. In 42nd IEEE Real-Time Systems Symposium (RTSS), 2021.
  4. Michael Balszun, Martin Geier, and Samarjit Chakraborty. Predictable vision for autonomous systems. In 23rd IEEE International Symposium on Real-Time Distributed Computing (ISORC), 2020.
  5. Wanli Chang and Samarjit Chakraborty. Resource-aware automotive control systems design: A cyber-physical systems approach. Found. Trends Electron. Des. Autom., 10(4):249-369, 2016.
  6. Wanli Chang, Debayan Roy, Shuai Zhao, Anuradha Annaswamy, and Samarjit Chakraborty. Cps-oriented modeling and control of traffic signals using adaptive back pressure. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2020.
  7. Sayandip De, Sajid Mohamed, Konstantinos Bimpisidis, Dip Goswami, Twan Basten, and Henk Corporaal. Approximation trade offs in an image-based control system. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2020.
  8. Sayandip De, Sajid Mohamed, Dip Goswami, and Henk Corporaal. Approximation-aware design of an image-based control system. IEEE Access, 8:174568-174586, 2020.
  9. Glenn A. Elliott, Bryan C. Ward, and James H. Anderson. GPUSync: A framework for real-time GPU management. In 34th IEEE Real-Time Systems Symposium (RTSS), 2013.
  10. Glenn A. Elliott, Kecheng Yang, and James H. Anderson. Supporting real-time computer vision workloads using OpenVX on Multicore+GPU platforms. In IEEE Real-Time Systems Symposium (RTSS), 2015.
  11. Martin Geier, Marian Brändle, and Samarjit Chakraborty. Insert & save: Energy optimization in IP core integration for FPGA-based real-time systems. In 27th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2021.
  12. Martin Geier, Marian Brändle, Dominik Faller, and Samarjit Chakraborty. Debugging FPGA-accelerated real-time systems. In IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2020.
  13. Martin Geier, Dominik Faller, Marian Brändle, and Samarjit Chakraborty. Cost-effective energy monitoring of a Zynq-based real-time system including dual Gigabit Ethernet. In 27th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2019.
  14. Martin Geier, Florian Pitzl, and Samarjit Chakraborty. GigE vision data acquisition for visual servoing using SG/DMA proxying. In 14th ACM/IEEE Symposium on Embedded Systems for Real-Time Multimedia (ESTIMedia), 2016.
  15. Abhinav Goel, Caleb Tung, Xiao Hu, George K. Thiruvathukal, James C. Davis, and Yung-Hsiang Lu. Efficient computer vision on edge devices with pipeline-parallel hierarchical neural networks. In 27th Asia and South Pacific Design Automation Conference (ASP-DAC), 2022.
  16. Kees Goossens, Arnaldo Azevedo, Karthik Chandrasekar, Manil Dev Gomony, Sven Goossens, Martijn Koedam, Yonghui Li, Davit Mirzoyan, Anca Mariana Molnos, Ashkan Beyranvand Nejad, Andrew Nelson, and Shubhendu Sinha. Virtual execution platforms for mixed-time-criticality systems: the compsoc architecture and design flow. SIGBED Rev., 10(3):23-34, 2013.
  17. Dip Goswami, Reinhard Schneider, and Samarjit Chakraborty. Re-engineering cyber-physical control applications for hybrid communication protocols. In Design, Automation and Test in Europe (DATE), 2011.
  18. Dip Goswami, Reinhard Schneider, and Samarjit Chakraborty. Relaxing signal delay constraints in distributed embedded controllers. IEEE Trans. Control. Syst. Technol., 22(6):2337-2345, 2014.
  19. Clara Hobbs, Debayan Roy, Parasara Sridhar Duggirala, F. Donelson Smith, Soheil Samii, James H. Anderson, and Samarjit Chakraborty. Perception computing-aware controller synthesis for autonomous systems. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021.
  20. Omar W Ibraheem, Arif Irwansyah, Jens Hagemeyer, Mario Porrmann, and Ulrich Rueckert. A resource-efficient multi-camera GiGE vision IP core for embedded vision processing platforms. In IEEE International Conference on ReConFigurable Computing and FPGAs (ReConFig), 2015.
  21. Chaitanya Jugade, Daniel Hartgers, Phan Dúc Anh, Sajid Mohamed, Mojtaba Haghi, Dip Goswami, Andrew Nelson, Gijs van der Veen, and Kees Goossens. An evaluation framework for vision-in-the-loop motion control systems. In 27th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 2022.
  22. Anis Koubaa, Adel Ammar, Anas Kanhouch, and Yasser AlHabashi. Cloud versus edge deployment strategies of real-time face recognition inference. IEEE Transactions on Network Science and Engineering, 9(1):143-160, 2021.
  23. Friedrich Kruber, Eduardo Sánchez Morales, Samarjit Chakraborty, and Michael Botsch. Vehicle position estimation with aerial imagery from unmanned aerial vehicles. In IEEE Intelligent Vehicles Symposium (IV), 2020.
  24. Friedrich Kruber, Jonas Wurst, Eduardo Sánchez Morales, Samarjit Chakraborty, and Michael Botsch. Unsupervised and supervised learning with the random forest algorithm for traffic scenario clustering and classification. In IEEE Intelligent Vehicles Symposium (IV), 2019.
  25. Zhen Liu, Wenjie Lin, Xinpeng Li, Qing Rao, Ting Jiang, Mingyan Han, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. Adnet: Attention-guided deformable convolutional network for high dynamic range imaging. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
  26. Dipan Kumar Mandal, Jagadeesh Sankaran, Akshay Gupta, Kyle Castille, Shraddha Gondkar, Sanmati Kamath, Pooja Sundar, and Alan Phipps. An embedded vision engine (EVE) for automotive vision processing. In IEEE International Symposium on Circuits and Systems (ISCAS), 2014.
  27. Alejandro Masrur, Sebastian Drössler, Thomas Pfeuffer, and Samarjit Chakraborty. VM-based real-time services for automotive control applications. In 16th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), 2010.
  28. Mahmoud Méribout, Asma Baobaid, Mohamed Ould-Khaoua, Varun Kumar Tiwari, and Juan Pablo Pena. State of art iot and edge embedded systems for real-time machine vision applications. IEEE Access, 10:58287-58301, 2022.
  29. Seifeddine Messaoud, Soulef Bouaafia, Amna Maraoui, Ahmed Chiheb Ammari, Lazhar Khriji, and Mohsen Machhout. Deep convolutional neural networks-based hardware-software on-chip system for computer vision application. Comput. Electr. Eng., 98:107671, 2022.
  30. Sajid Mohamed, Dip Goswami, Vishak Nathan, Raghu Rajappa, and Twan Basten. A scenario- and platform-aware design flow for image-based control systems. Microprocess. Microsystems, 75:103037, 2020.
  31. Sajid Mohamed, Nilay Saraf, Daniele Bernardini, Dip Goswami, Twan Basten, and Alberto Bemporad. Adaptive predictive control for pipelined multiprocessor image-based control systems considering workload variations. In 59th IEEE Conference on Decision and Control (CDC), 2020.
  32. Burhan Ahmad Mudassar, Priyabrata Saha, Yun Long, Mohammad Faisal Amir, Evan Gebhardt, Taesik Na, Jong Hwan Ko, Marilyn Wolf, and Saibal Mukhopadhyay. CAMEL: an adaptive camera with embedded machine learning-based sensor parameter control. IEEE J. Emerg. Sel. Topics Circuits Syst., 9(3):498-508, 2019.
  33. The Khronos Group, OpenVX: Portable, power efficient vision processing. URL: https://www.khronos.org/openvx/.
  34. Nathan Otterness and James H. Anderson. AMD GPUs as an alternative to NVIDIA for supporting real-time workloads. In 32nd Euromicro Conference on Real-Time Systems (ECRTS), volume 165 of LIPIcs, pages 10:1-10:23. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020.
  35. Qing Rao and Samarjit Chakraborty. Efficient lossless compression for depth information in traffic scenarios. Multim. Syst., 25(4):293-306, 2019.
  36. Qing Rao and Samarjit Chakraborty. In-vehicle object-level 3D reconstruction of traffic scenes. IEEE Trans. Intell. Transp. Syst., 22(12):7747-7759, 2021.
  37. Qing Rao, Christian Grünler, Markus Hammori, and Samarjit Chakraborty. Design methods for augmented reality in-vehicle infotainment systems. In 51st Annual Design Automation Conference (DAC), 2014.
  38. Qing Rao, Christian Grünler, Markus Hammori, and Samarjit Chakraborty. Stixel on the bus: An efficient lossless compression scheme for depth information in traffic scenarios. In 20th Anniversary International Conference of MultiMedia Modeling (MMM), volume 8325 of Lecture Notes in Computer Science. Springer, 2014.
  39. Qing Rao, Lars Krüger, and Klaus Dietmayer. Monocular 3D shape reconstruction using deep neural networks. In IEEE Intelligent Vehicles Symposium (IV), 2016.
  40. Qing Rao, Tobias Tropper, Christian Grünler, Markus Hammori, and Samarjit Chakraborty. AR-IVI - implementation of in-vehicle augmented reality. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2014.
  41. Debayan Roy, Clara Hobbs, James H. Anderson, Marco Caccamo, and Samarjit Chakraborty. Timing debugging for cyber-physical systems. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021.
  42. Debayan Roy, Licong Zhang, Wanli Chang, Sanjoy K. Mitter, and Samarjit Chakraborty. Semantics-preserving cosynthesis of cyber-physical systems. Proc. IEEE, 106(1):171-200, 2018.
  43. Kruttidipta Samal, Hemant Kumawat, Priyabrata Saha, Marilyn Wolf, and Saibal Mukhopadhyay. Task-driven rgb-lidar fusion for object tracking in resource-efficient autonomous system. IEEE Trans. Intell. Veh., 7(1):102-112, 2022.
  44. Kruttidipta Samal, Marilyn Wolf, and Saibal Mukhopadhyay. Closed-loop approach to perception in autonomous system. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021.
  45. Róbinson Medina Sánchez, Juan Valencia, Sander Stuijk, Dip Goswami, and Twan Basten. Designing a controller with image-based pipelined sensing and additive uncertainties. ACM Trans. Cyber Phys. Syst., 3(3):33:1-33:26, 2019.
  46. Sebastian Schmidt, Qing Rao, Julian Tatsch, and Alois C. Knoll. Advanced active learning strategies for object detection. In IEEE Intelligent Vehicles Symposium (IV), 2020.
  47. Reinhard Schneider, Dip Goswami, Alejandro Masrur, Martin Becker, and Samarjit Chakraborty. Multi-layered scheduling of mixed-criticality cyber-physical systems. J. Syst. Archit., 59(10-D):1215-1230, 2013.
  48. Shreejith Shanker, Philipp Mundhenk, Andreas Ettner, Suhaib A. Fahmy, Sebastian Steinhorst, Martin Lukasiewycz, and Samarjit Chakraborty. VEGa: A high performance vehicular ethernet gateway on hybrid FPGA. IEEE Trans. Computers, 66(10):1790-1803, 2017.
  49. Ponnan Suresh, Saravanakumar Umathurai, Celestine Iwendi, Senthilkumar Mohan, and Gautam Srivastava. Field-programmable gate arrays in a low power vision system. Comput. Electr. Eng., 90:106996, 2021.
  50. George K. Thiruvathukal and Yung-Hsiang Lu. Efficient computer vision for embedded systems. Computer, 55(4):15-19, 2022.
  51. Caleb Tung, Abhinav Goel, Fischer Bordwell, Nick Eliopoulos, Xiao Hu, Yung-Hsiang Lu, and George K. Thiruvathukal. Why accuracy is not enough: The need for consistency in object detection. IEEE Multim., 29(3):8-16, 2022.
  52. R. Udendhran, M. Balamurugan, Annamalai Suresh, and R. Varatharajan. Enhancing image processing architecture using deep learning for embedded vision systems. Microprocess. Microsystems, 76, 2020.
  53. Zhilu Wang, Chao Huang, Yixuan Wang, Clara Hobbs, Samarjit Chakraborty, and Qi Zhu. Bounding perception neural network uncertainty for safe control of autonomous systems. In Design, Automation & Test in Europe Conference & Exhibition (DATE), 2021.
  54. Peter Waszecki, Philipp Mundhenk, Sebastian Steinhorst, Martin Lukasiewycz, Ramesh Karri, and Samarjit Chakraborty. Automotive electrical and electronic architecture security via distributed in-vehicle traffic monitoring. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst., 36(11):1790-1803, 2017.
  55. Ming Yang, Tanya Amert, Kecheng Yang, Nathan Otterness, James H. Anderson, F. Donelson Smith, and Shige Wang. Making openvx really "real time". In IEEE Real-Time Systems Symposium (RTSS), 2018.
  56. Ming Yang, Shige Wang, Joshua Bakita, Thanh Vu, F. Donelson Smith, James H. Anderson, and Jan-Michael Frahm. Re-thinking CNN frameworks for time-sensitive autonomous-driving applications: Addressing an industrial challenge. In 25th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2019.
  57. Anand Yeolekar, Ravindra Metta, Clara Hobbs, and Samarjit Chakraborty. Checking scheduling-induced violations of control safety properties. In 20th International Symposium on Automated Technology for Verification and Analysis (ATVA), volume 13505 of Lecture Notes in Computer Science. Springer, 2022.
  58. Junxing Zhang, Shuo Yang, Chunjuan Bo, and Zhiyuan Zhang. Vehicle logo detection based on deep convolutional networks. Comput. Electr. Eng., 90:107004, 2021.