Extension API support for VivadoAccelerator backend

[returnwellbeing]

HI, first of all, I want to say thank you!
thanks to this awesome framework, I can do my research.

I am trying to use Extension API for VivadoAccelerator(not Vivado) backend. Because I'm planning to deploy the IP block on FPGA and use PYNQ's overlay to solve the communication between PS & PL.

I read the documentation, https://fastmachinelearning.org/hls4ml/advanced/extension.html# and wrote the super simple class, PReverse. It means the pytorch version of KReverse in the documentation :)

• PReverse gets 20 elements as input
• PReverse puts 20 elements in reversed order as output

When I used the Vivado backend, everything is ok!! The hls model's output is almost same with the output of pytorch
But the problem is occured, when i changed the backed to VivadoAccelerator :( not implemented fuction, convert_data<result_t, float, N_IN>(output_axi_t [20], float*&) is called in myproject_bridge.cpp and the implementation is expected to be in firmware/nnet_utils/nnet_helpers.h

myproject_bridge.cpp: In function 'void myproject_float(float*, float*)':
myproject_bridge.cpp:63:46: error: no matching function for call to 'convert_data<result_t, float, N_IN>(output_axi_t [20], float*&)'
   63 |     nnet::convert_data<result_t, float, N_IN>(layer2_out_ap, layer2_out);
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from myproject_bridge.cpp:5:
firmware/nnet_utils/nnet_helpers.h:139:59: note: candidate: 'void nnet::convert_data(srcType*, dstType*) [with srcType = ap_fixed<32, 6>; dstType = float; long unsigned int SIZE = 20]'
  139 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, dstType *dst) {

full error log

2024-09-02 19:16:52.760553: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-09-02 19:16:53.302448: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
myproject_bridge.cpp: In function 'void myproject_float(float*, float*)':
myproject_bridge.cpp:63:46: error: no matching function for call to 'convert_data<result_t, float, N_IN>(output_axi_t [20], float*&)'
   63 |     nnet::convert_data<result_t, float, N_IN>(layer2_out_ap, layer2_out);
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from myproject_bridge.cpp:5:
firmware/nnet_utils/nnet_helpers.h:139:59: note: candidate: 'void nnet::convert_data(srcType*, dstType*) [with srcType = ap_fixed<32, 6>; dstType = float; long unsigned int SIZE = 20]'
  139 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, dstType *dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:139:81: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'ap_fixed<32, 6>*'
  139 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, dstType *dst) {
      |                                                                        ~~~~~~~~~^~~
firmware/nnet_utils/nnet_helpers.h:145:59: note: candidate: 'void nnet::convert_data(srcType*, hls::stream<dstType>&) [with srcType = ap_fixed<32, 6>; dstType = float; long unsigned int SIZE = 20]'
  145 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, hls::stream<dstType> &dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:145:81: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'ap_fixed<32, 6>*'
  145 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, hls::stream<dstType> &dst) {
      |                                                                        ~~~~~~~~~^~~
firmware/nnet_utils/nnet_helpers.h:155:59: note: candidate: 'void nnet::convert_data(hls::stream<srcType>&, dstType*) [with srcType = ap_fixed<32, 6>; dstType = float; long unsigned int SIZE = 20]'
  155 | template <class srcType, class dstType, size_t SIZE> void convert_data(hls::stream<srcType> &src, dstType *dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:155:94: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'hls::stream<ap_fixed<32, 6> >&'
  155 | template <class srcType, class dstType, size_t SIZE> void convert_data(hls::stream<srcType> &src, dstType *dst) {
      |                                                                        ~~~~~~~~~~~~~~~~~~~~~~^~~
myproject_bridge.cpp: In function 'void myproject_double(double*, double*)':
myproject_bridge.cpp:77:47: error: no matching function for call to 'convert_data<result_t, double, N_IN>(output_axi_t [20], double*&)'
   77 |     nnet::convert_data<result_t, double, N_IN>(layer2_out_ap, layer2_out);
      |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:139:59: note: candidate: 'void nnet::convert_data(srcType*, dstType*) [with srcType = ap_fixed<32, 6>; dstType = double; long unsigned int SIZE = 20]'
  139 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, dstType *dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:139:81: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'ap_fixed<32, 6>*'
  139 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, dstType *dst) {
      |                                                                        ~~~~~~~~~^~~
firmware/nnet_utils/nnet_helpers.h:145:59: note: candidate: 'void nnet::convert_data(srcType*, hls::stream<dstType>&) [with srcType = ap_fixed<32, 6>; dstType = double; long unsigned int SIZE = 20]'
  145 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, hls::stream<dstType> &dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:145:81: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'ap_fixed<32, 6>*'
  145 | template <class srcType, class dstType, size_t SIZE> void convert_data(srcType *src, hls::stream<dstType> &dst) {
      |                                                                        ~~~~~~~~~^~~
firmware/nnet_utils/nnet_helpers.h:155:59: note: candidate: 'void nnet::convert_data(hls::stream<srcType>&, dstType*) [with srcType = ap_fixed<32, 6>; dstType = double; long unsigned int SIZE = 20]'
  155 | template <class srcType, class dstType, size_t SIZE> void convert_data(hls::stream<srcType> &src, dstType *dst) {
      |                                                           ^~~~~~~~~~~~
firmware/nnet_utils/nnet_helpers.h:155:94: note:   no known conversion for argument 1 from 'output_axi_t [20]' {aka 'out_struct [20]'} to 'hls::stream<ap_fixed<32, 6> >&'
  155 | template <class srcType, class dstType, size_t SIZE> void convert_data(hls::stream<srcType> &src, dstType *dst) {
      |                                                                        ~~~~~~~~~~~~~~~~~~~~~~^~~
/home/test5/miniconda3/envs/hls4ml-tutorial/bin/../lib/gcc/x86_64-conda-linux-gnu/12.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find myproject_bridge.o: No such file or directory
collect2: error: ld returned 1 exit status
graph():
    %input_1 : [num_users=1] = placeholder[target=input]
    %flip : [num_users=1] = call_function[target=torch.flip](args = (%input_1,), kwargs = {dims: [-1]})
    return flip
Interpreting Model ...
Topology:
Layer name: flip, layer type: HReverse, input shape: [[None, 20]]
Creating HLS model
WARNING: Changing pipeline style to "dataflow".
WARNING: You set a Part that does not correspond to the Board you specified. The correct Part is now set.
Writing HLS project
Done
Traceback (most recent call last):
  File "/home/test5/test.special/hls4ml/contrib/preverse/preverse.py", line 156, in <module>
    main()
  File "/home/test5/test.special/hls4ml/contrib/preverse/preverse.py", line 143, in main
    hmodel.compile()
  File "/home/test5/miniconda3/envs/hls4ml-tutorial/lib/python3.10/site-packages/hls4ml/model/graph.py", line 668, in compile
    self._compile()
  File "/home/test5/miniconda3/envs/hls4ml-tutorial/lib/python3.10/site-packages/hls4ml/model/graph.py", line 687, in _compile
    self._top_function_lib = ctypes.cdll.LoadLibrary(lib_name)
  File "/home/test5/miniconda3/envs/hls4ml-tutorial/lib/python3.10/ctypes/__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "/home/test5/miniconda3/envs/hls4ml-tutorial/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: hls4mlprj_preverse/firmware/myproject-Bd685B9e.so: cannot open shared object file: No such file or directory

Any comment will be great help to me. Please give me some advice to solve the problem. Or where should I look more to solve this problem? :)

[returnwellbeing]

Hi, I found the reason of the error and I want to share my experience.
I wrote my class using the example in documentation and specially descripting hls4ml's layer is same.

# hls4ml layer implementation
class HReverse(hls4ml.model.layers.Layer):
    '''hls4ml implementation of a hypothetical custom layer'''

    def initialize(self):
        inp = self.get_input_variable()
        shape = inp.shape
        dims = inp.dim_names
        self.add_output_variable(shape, dims)

Because the input and output have the same shape in reverse operation, you can see that the input's shape and dims are used when adding the output variable. Actually, dims are strings that describe the dimension of an array like N_INPUT_1_1, N_IN, N_OUTPUT_1_2, N_OUT...

To start with the solution, a separate dims for output should be described.

Solution

# dims = inp.dim_names
dims = [f'N_OUTPUT_{i}_{self.index}' for i in range(1, len(shape) + 1)]

Details

myproject_bridge.cpp is wrote by hls4ml/writer/vivado_accelerator_writer.py using fout()
So if there are some errors in writing the file. compling with g++ can report error as we can see above.

There are some logics that checks & replace & write in https://github.com/fastmachinelearning/hls4ml/blob/main/hls4ml/writer/vivado_accelerator_writer.py#L343-L360
Specially, the lines 354~357 caused the error of my case.

            elif inp.size_cpp() in line or inp.name in line or inp.type.name in line:
                newline = line.replace(inp.size_cpp(), 'N_IN').replace(inp.type.name, 'input_axi_t')
            elif out.size_cpp() in line or out.name in line or out.type.name in line:
                newline = line.replace(out.size_cpp(), 'N_OUT').replace(out.type.name, 'output_axi_t')

Every conditions are connected with else-if. so there is only one chance that can replace the single line.
The dims of output is supposed to be replaced in 357 line. but if it had input's dims it would be replaced in 355 line. This cause the abnormal code in myproject_bridge.cpp :)