Troubleshooting VCU118 design issues in 2026.2

NOEL-V
1

Hello everyone,

We are currently experiencing several issues with the new VCU118 design provided in the 2026.2 release and were wondering if anyone has encountered something similar.

For context, we have a working design for the same FPGA based on the 2024.4 release. After migrating to the 2026.2 release, we started seeing the following problems:

  • L2-Lite enabled: We are unable to connect using GRMON (v4.1.2 64-bit evaluation version). GRMON reports:

    ERROR! AMBA plug&play not found!

  • L2 disabled: GRMON detects the CPU correctly, but execution immediately fails with a Load access fault. What makes this particularly confusing is that the exception occurs at PC = 0x00000002, even though compressed instructions are disabled and no load instruction is being executed at that point.

    By single-stepping through the execution, we observed that execution unexpectedly jumps to the ROM address before the crash. The trace is shown below:

grmon4> step 50
  0x0000000000000000: 0000b197  auipc   gp, 0xb             <_start+0>
  0x0000000000000000: 0000b197  auipc   gp, 0xb             <_start+0>
  0x00000000c0000000: 00000093  li      ra, 0             
  0x00000000c0000004: 00000113  li      sp, 0             
  0x00000000c0000008: 00000193  li      gp, 0             
  0x00000000c000000c: 00000213  li      tp, 0             
  0x00000000c0000010: 00000293  li      t0, 0             
  0x00000000c0000014: 00000313  li      t1, 0             
  0x00000000c0000018: 00000393  li      t2, 0             
  0x00000000c000001c: 00000413  li      s0, 0             
  0x00000000c0000020: 00000493  li      s1, 0             
  0x00000000c0000024: 00000513  li      a0, 0             
  0x00000000c0000028: 00000593  li      a1, 0             
  0x00000000c000002c: 00000613  li      a2, 0             
  0x00000000c0000030: 00000693  li      a3, 0             
  0x00000000c0000034: 00000713  li      a4, 0             
  0x00000000c0000038: 00000793  li      a5, 0             
  0x00000000c000003c: 00000813  li      a6, 0             
  0x00000000c0000040: 00000893  li      a7, 0             
  0x00000000c0000044: 00000913  li      s2, 0             
  0x00000000c0000048: 00000993  li      s3, 0             
  0x00000000c000004c: 00000a13  li      s4, 0             
  0x00000000c0000050: 00000a93  li      s5, 0             
  0x00000000c0000054: 00000b13  li      s6, 0             
  0x00000000c0000058: 00000b93  li      s7, 0             
  0x00000000c000005c: 00000c13  li      s8, 0             
  0x00000000c0000060: 00000c93  li      s9, 0             
  0x00000000c0000064: 00000d13  li      s10, 0            
  0x00000000c0000068: 00000d93  li      s11, 0            
  0x00000000c000006c: 00000e13  li      t3, 0             
  0x00000000c0000070: 00000e93  li      t4, 0             
  0x00000000c0000074: 00000f13  li      t5, 0             
  0x00000000c0000078: 00000f93  li      t6, 0             
  0x00000000c000007c: 00010137  lui     sp, 0x10          
  0x00000000c0000080: 00000297  auipc   t0, 0x0           
  0x00000000c0000084: 02428293  addi    t0, t0, 36        
  0x00000000c0000088: 30529073  csrw    mtvec, t0         
  0x00000000c000008c: 0000100f  fence.i zero, zero, 0x0   
  0x00000000c0000090: f1402573  csrr    a0, mhartid       
  0x00000000c0000094: 00100593  li      a1, 1             
  0x00000000c0000098: 00b57063  bgeu    a0, a1, 0xc0000098
  0x00000000c000009c: 00000413  li      s0, 0             
  0x00000000c00000a0: 00040067  jalr    zero, s0          
  0x0000000000000000: 0000b197  auipc   gp, 0xb             <_start+0>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b         <_start+2>

grmon4> cont
  Error mode  (5, Load access fault)
  0x0000000000000002: 4801b197  auipc   gp, 0x4801b  <_start+2>
  SIGHUP

After further debugging, we found that the jump to the ROM address is actually the result of an exception rather than its cause.

Immediately after executing the first instruction, we observe:

grmon4> step
  0x0000000000000000: 0000b197  auipc   gp, 0xb  <_start+0>

grmon4> reg mcause mtval mepc
     mcause = 5 (0x0000000000000005) 
      mtval = 2147483664 (0x0000000080000010) 
       mepc = 0 (0x0000000000000000)

  • To move from the provided reference design to our previous working setup, we replaced ahb2axi_mig4_7series with axi_mig4_7series. With this change, execution no longer jumps to the ROM address, but instead appears to hang inside __bcc_con_outbyte.

    Interestingly, the behavior depends on the workload:

    • Small matrix sizes complete successfully.
    • Larger matrices consistently hang.
    • Some intermediate sizes occasionally complete, although not for the full benchmark (which performs multiple runs).

    This exact software binary executes correctly on our 2024.4 design. The execution trace is shown below:

grmon4> step 5
  0x0000000000000000: 0000b197  auipc   gp, 0xb       <_start+0>
  0x0000000000000004: 48018193  addi    gp, gp, 1152  <_start+4>
  0x0000000000000008: 6299      lui     t0, 0x6       <_start+8>
  0x000000000000000a: 3002a073  csrs    mstatus, t0   <_start+10>
  0x000000000000000e: 00301073  csrw    fcsr, zero    <_start+14>

grmon4> cont

** DEVICE INFORMATION **
 - CPU: NOEL-V (64-bits)
  > Optimized implementation

** TEST INFORMATION **
Matrix Multiplication:
 - Input matrix A (8-bit): 32 x 32
 - Input matrix B (8-bit): 32 x 32
 - Output matrix (32-bit): 32 x 32

** TEST EXECUTION **

  Interrupted!
  0x0000000000009f8c: 2007f793  andi    a5, a5, 512  <__bcc_con_outbyte+62>
  SIGINT

Has anyone experienced similar issues when using the new noelv-xilinx-vcu118 design in the 2026.2 release, or have any suggestions on what might be causing this behavior? Any pointers or debugging ideas would be greatly appreciated.

Thank you in advance for your help.
Marc

2

Find here the output of running info sys for this design:

grmon4> info sys
  cpu0       Frontgrade Gaisler  NOEL-V RISC-V Processor    
             AHB Master 0
  greth0     Frontgrade Gaisler  GR Ethernet MAC    
             AHB Master 1
             APB: ff984000 - ff984100
             IRQ: 5
             edcl ip 192.168.0.51, buffer 2 kbyte
  dm0        Frontgrade Gaisler  RISC-V Debug Module    
             AHB Master 9
             AHB Master 7
             AHB: e0000000 - e1000000
             AHB: f8400000 - f8500000
             hart0: ISA rv64imafdcbh, Modes M S U VS VU, SV48
                    i, m, a, f, d, c, h, shlcofideleg, smcdeleg, smcntrpmf
                    smcsrind, smdbltrp, smepmp, smpmpmt, smrnmi, smstateen
                    sscofpmf, sscsrind, ssdbltrp, sstc, svadu, svinval
                    svnapot, svpbmt, svrsw60t59b, zaamo, zalrsc, zba, zbb
                    zbc, zbkb, zbkc, zbkx, zbs, zca, zcb, zcd, zcmop, zicbom
                    zicfilp, zicfiss, zicntr, zicond, zifencei, zihpm, zimop
                    zkt
                    Stack pointer 0x7ffffff0
                    icache 4 * 4 kB, 32 B/line, lru,
                    dcache 4 * 4 kB, 32 B/line, lru
                    11 triggers,
                    itrace 32 lines
  ahbuart0   Frontgrade Gaisler  AHB Debug UART    
             AHB Master 0
             APB: ff986000 - ff986100
             Baudrate 115200, AHB frequency 100.00 MHz
  ahbjtag0   Frontgrade Gaisler  JTAG Debug Link    
             AHB Master 1
  edcl0      Frontgrade Gaisler  EDCL master interface    
             AHB Master 2
  apbmst0    Frontgrade Gaisler  AHB/APB Bridge    
             AHB: ff900000 - ffa00000
  mig0       Frontgrade Gaisler  Xilinx MIG Controller    
             AHB: 00000000 - 80000000
             SDRAM: 2048 Mbyte
  ahbrom0    Frontgrade Gaisler  Generic AHB ROM    
             AHB: c0000000 - e0000000
             32-bit ROM: 512 MB @ 0xc0000000
  apbmst1    Frontgrade Gaisler  AHB/APB Bridge    
             AHB: ff400000 - ff500000
  apbmst2    Frontgrade Gaisler  AHB/APB Bridge    
             AHB: ff500000 - ff600000
  rvintctrl0 Frontgrade Gaisler  RISC-V Interrupt Controllers    
             AHB: b0000000 - b0100000
             AHB: fc000000 - fc400000
             USR: 00000009
             Backend: clint0 plic0
  uart0      Frontgrade Gaisler  Generic UART    
             APB: ff900000 - ff900100
             IRQ: 1
             Baudrate 38343
             FIFO debug mode available
             16550 interface available
  gptimer0   Frontgrade Gaisler  Modular Timer Unit    
             APB: ff908000 - ff908100
             IRQ: 2
             16-bit scaler, 2 * 32-bit timers, divisor 100
  version0   Frontgrade Gaisler  Version and Revision Register    
             APB: ff981000 - ff981100
             Version 1024, Revision 140
  ahbstat0   Frontgrade Gaisler  AHB Status Register    
             APB: ff982000 - ff982100
             IRQ: 4
             non-correctable read error of size 2 by master 0 at 0x80000000
  gpio0      Frontgrade Gaisler  General Purpose I/O port    
             APB: ff983000 - ff983100
  clint0     Frontgrade Gaisler  RISC-V ACLINT    
             AHB: b0000000 - b0100000
  plic0      Frontgrade Gaisler  RISC-V PLIC    
             AHB: fc000000 - fc400000
             AHB: b0018100 - b0018180
             4 contexts, 63 interrupt sources, 7 max priority
  cpu0 greth0 dm0 ahbuart0 ahbjtag0 edcl0 apbmst0 mig0 ahbrom0 apbmst1 apbmst2 rvintctrl0 uart0 gptimer0 version0 ahbstat0 gpio0 clint0 plic0
3

Hi again,

After further investigating, I found that the issue is related to apbuart_dual.
Replacing it with the regular apbuart completely fixes the problem.

Comparing with the old design, I also noticed that GRMON initializes the UART differently:

  • apbuart: CONTROL = 0x800A0003 (TE and RE enabled)
  • apbuart_dual: CONTROL = 0x800A0000 (TE and RE disabled)

If I manually enable TE/RE before starting the program:

wmem 0xff900008 0x800A0003

and then use cont, the benchmark runs successfully without hanging.

Is there any reason why apbuart_dual is initialized with TE/RE disabled? Is there a recommended way to have these bits enabled automatically? Should this be handled by GRMON during initialization, or is it expected that the application/BCC library enables them?

4

Hello Marc,

The TE/RE change is to be able to support 3rd party 16550 drivers. It is expected that the application/BCC library enables them if you want to use the APBUART API. Future versions of BCC will do this automatically.

To support old software you can use these GRMON commands to start the application:
reset
set uart0::ctrl 3
go