International Journal of Engineering Insights: (2023) Vol. 1, Nro.1, Regular Paper

https://doi.org/10.61961/injei.v1i1.3

Fast FPGA-Based Image Feature Extraction for Data Fusion

in Autonomous Vehicles.

Jeremias Gaia · Eugenio Orosco · Francisco Rossomando · Carlos Soria

Received: 14 May 2023 / Accepted: 10 August 2023

Abstract: Computer vision plays a critical role in many

applications, particularly in the domain of autonomous

vehicles. To achieve high-level image processing tasks

rate image feature extraction, specifically designed to

meet the constraints of data fusion in autonomous ve-

hicles. The system computes a set of generic statistical

image features, including contrast, homogeneity, and

entropy, and is implemented on two Xilinx FPGA plat-

forms - an Alveo U200 Data Center Accelerator Card

and a Zynq UltraScale+ MPSoC ZCU104 Evaluation

Kit. Experimental results show that the proposed sys-

tem achieves high-speed image feature extraction with

low latency, making it well-suited for use in autonomous

1 Introduction

Images are used as inputs in a variety of systems [1,

2,3] , and techniques for extracting information from

them are the foundation of many applications such as

simultaneous localization and mapping (SLAM) [4], au-

tonomous driving safety [5], and human-robot inter-

action [6]. However, achieving high-speed image pro-

Jeremias Gaia · Eugenio Orosco

Francisco Rossomando · Carlos Soria

Instituto de Autom´atica

cessing for these applications is computationally expen-

sive for standard CPUs due to time constraints and

large image sizes. To address this challenge, Field Pro-

computational load and, as a result, latency [8].

Ethernet, USB, etc., at high speeds. In the context of

data fusion, the availability of such a device enables the

creation of a communication-centric platform between

connected sensors [9]. Additionally, by utilizing paral-

lel structures, complex calculations can be performed

significantly faster.

This paper presents a high-speed FPGA-based im-

plementation for computing a set of common statistical

image features that is performance-optimized by ex-

ploiting FPGAs’ capacity to integrate sequential and

tion 2 contains related research articles, expressions used

to determine different statistical image features are pre-

sented in Section 3, and Section 4.1 describes the pro-

posed FPGA-based system. Experimentation results are

shown in Section 4.2, followed by the conclusions in Sec-

tion 5.

2 Related work

In conventional image processing applications, software

developers commonly utilize the recognized OpenCV

library [10], which uses the large amount of hardware

developed numerous algorithms to optimize image pro-

cessing operations in FPGAs, such as color to gray-level

conversion [11], histogram construction [12] and edge

detection [13], among others.

Histograms are the first layer of statistical informa-

tion that can be extracted directly from grayscale image

pixels. Younis et. al. proposed in [12] a histogram con-

struction hardware implementation. Their system re-

ceived grayscale images (pre-processed with MATLAB)

as input for a finite state machine to perform the his-

put. Recently, filtering techniques for event-based cam-

eras such as a background activity filter and a mask

filter, were also implemented on FPGA [14].

Similar to our work, Siddiqui et. al. [15] proposed

an FPGA-based soft processor called Image Processing

Processor (IPPro) for general purpose image process-

ing. The authors designed and tested an instantiable

component, since they focus on multi-core operation.

However, the source code is not available.

The computation of Graylevel Co-occurrence Ma-

occurrence matrix for each orientation.

Hardware description languages use the Register-

transfer-level (RTL) abstraction to create high-level cir-

cuit representations, which can be used to derive lower-

level representations and actual wiring [18]. Recently,

Xilinx company has developed the xfOpenCV library

[19], which provides a set of performance-optimized ker-

nels for Xilinx FPGAs and SoCs that enable the trans-

lation of common computer vision operations from se-

quential software processing to parallel hardware pro-

since the image’s pixels cover the entire range of gray levels,

in contrast to a narrow and peaky histogram that indicates a

low contrast environment.

3.1 Histogram-based measures

Histograms offer a lot of information about the image.

Take as example the different histogram variations pre-

sented in Fig. 1. A broad histogram denotes a high con-

trast picture since the image’s pixels cover the entire

range of gray levels, in contrast to a narrow and peaky

histogram that indicates a low contrast environment.

Introduced by Tripathi et. al. in [20], these metrics

are very useful to assess image contrast. HFM can be

expressed as the ratio of the geometric mean of the

image histogram h(x) to the arithmetic mean of h(x)

(Eq. (1)).

3rd and the 1st quartile of the cumulative histogram.

The 1st and 3rd quartile means the histogram bins at

which cumulative histogram has 25% and 75% of the

function, λ is a control parameter to adjust the map-

ping behaviour and ˆg

information at pixel i. Note that g

a pixel range of [0, 255] and a normalization factor of

1/255. After the estimation of g

, the γ sharpness mea-

sure can be obtained with Eq (7):

tained by manipulating this matrix. We refer the reader

to [22] for further details.

This work considers four textural features: contrast,

In order to further understand the following equa-

tions, notation is provided.

– P refers to the graylevel co-ocurrence matrix.

– p(i, j) is the (i, j)th entry in the normalized graylevel

co-ocurrence matrix.

=

g

Also known as inverse difference moment (IMC)

g

g

g

=

(13)

g

It is noteworthy that the contrast and entropy mea-

be quickly interpreted. However, as Haralick et.al. stated

in [22], not all of these features can be easily understood

from a human perspective. This is the case of the An-

interpreted as a measure of the energy of the image,

but also as a second homogeneity measure. Correlation

metrics report the presence of linear dependencies in

the image. In the case of the IMC measure, by combin-

ing different correlations it provides a reference of the

a Processing System (PS) block and a Programmable

Logic (PL) Block. The first normally contains an em-

bedded processor, while the second contains designed

hardware [23]. Both blocks are carefully employed in

this article, with the PL being used for the majority of

the calculations and the PS being used for operations

that cannot be parallelized.

Compilation directives can severely affect the sys-

tems performance. The DATAFLOW directive for instance,

plays a critical role in enabling the smooth flow of in-

all throughput of the system. Without the use of the

a PetaLinux distribution [24], reading an input image.

Subsequently, the image is transformed from RGB to

blocks: one responsible for generating the histogram

and its associated features, and the other for generat-

ing the magnitude image and its related metrics. Con-

currently, the GLCM-dependent textural features are

computed.

Once the parallel computations are completed, re-

sults are written into a buffer to return to the PS. Since

xfOpenCV library V2022.1 was installed along with the

OpenCV V4.4.0 library. GCC 7, G++ 7 and CMake

3.16 were used for compilation.

4.2 Results

Scale+ MPSoC ZCU104 Evaluation Kit [25] and an

parison purposes using a computer with an Intel Core

gray conversion, histogram calculations, Haralick fea-

tures extraction, etc. Our proposal, on the other hand,

is able to perform multiple computations at the same

system to related research in Table 1. Results displayed

in this table are those reported by the original authors

in their publications.

In terms of histogram computation, our system is

compared to [12]. The AlveoU200 implementation of

ously. This condition explains the fact that our mea-

sured times for the different tasks are the same.

Si´eler et. al. [16] proposed a system for Haralick

features extraction implemented on two Virtex boards.

Table 1 shows that for a 512x512 image the Virtex-

XC5VLX50T board performs better than the Virtex-

XCV2000 implementation. For a larger image , our pro-

posed system outperforms [16] work in this task, with

the exception of estimating one feature less. Finally, the

work by Atitallah et. al. [17] needs 5.8 ms to calculate

crease with the image size, even for the largest image,

they perform in under 3 ms. However, the performance

K. Rafferty, and D. Crookes, “Fpga-based processor ac-

celeration for image processing applications,” Journal of

Imaging, 2019.

16. L. Si´eler, C. Tanougast, and A. Bouridane, “A scalable

and embedded fpga architecture for efficient computation

of grey level co-occurrence matrices and haralick textures

features,” Microprocessors and Microsystems, 2010.

17. M. A. B. Atitallah, R. Kachouri, and H. Mnif, “A new

fpga accelerator based on circular buffer unit per orien-

tation for a fast and optimised glcm and texture feature

Kweon, “Camera exposure control for robust robot vi-

sion with noise-aware image quality assessment,” in 2019

IEEE/RSJ Int. Conf. on Intelligent Robots and Systems

(IROS). IEEE, 2019.

22. R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Tex-

tural features for image classification,” IEEE Transac-

tions on systems, man, and cybernetics, 1973.

23. Y. Nitta, S. Tamura, H. Yugen, and H. Takase, “Zytle-

bot: Fpga integrated development platform for ros based

autonomous mobile robot,” in 2019 Int. Conf. on Field-

Programmable Technology (ICFPT). IEEE, 2019.

24. Xilinx, “Petalinux tools,” https://www.xilinx.

com/products/design-tools/embedded-software/

alveo/u200.html, 2022, [Online; accessed 21-February-

2023].

License

Copyright (2023) Jeremias Gaia, Eugenio Orosco,

Francisco Rossomando, and Carlos Soria.

This text is protected under an international Cre-

ative Commons 4.0 license.

You are free to share, copy, and redistribute the ma-

terial in any medium or format — and adapt the docu-

ment — remix, transform, and build upon the material

proval of your use of the work.

License summary - Full text of the license