How can I find out where the routing congestion is?

[oharboe]

Any tips?

I'm trying a larger version & different version of The-OpenROAD-Project/OpenROAD-flow-scripts#743 and it fails in global routing.

[INFO GRT-0101] Running extra iterations to remove overflow.
[WARNING GRT-0227] Reached 20 congestion iterations with less than 15% of reduction between iterations.
[INFO GRT-0197] Via related to pin nodes: 1126254
[INFO GRT-0198] Via related Steiner nodes: 39
[INFO GRT-0199] Via filling finished.
[INFO GRT-0111] Final number of vias: 2236520
[INFO GRT-0112] Final usage 3D: 15578444
[ERROR GRT-0118] Routing congestion too high. Check the congestion heatmap in the GUI.
Error: global_route.tcl, 19 GRT-0118

The most recent .odb file to be written is 4_cts.odb, but when I load it and look at the routing congestion map, I don't see anything suspicious. Such arrays have a lot of routing to do, but the routing is simple conceptually: point to point between the array elements.

I'm learning a few tricks with OpenROAD-flow-scripts:

1. read documentation, if you can find it.
2. read the OpenROAD-flow-scripts/flow/*.tcl script for the step. It is normally quite short. From here you can find out what reports are being saved and also what to search for in other examples.

Looking at congestion.rpt, it is full of these. There's nothing in that area, just filler cells. Sometimes overflow is as high as 20.

violation type: Horizontal congestion
	srcs: 
	congestion information: capacity:16 usage:17 overflow:1
	bbox = ( 1544.4, 45.9 ) - ( 1544.94, 46.44) on Layer -

[maliberty]

The violations are related to routing congestion so they can occur over fill cells. Do you not see congested areas in the heatmap? I'm not sure what "anything suspicious" means.

Also you could play with the layer derates in grt. Reducing them makes it more likely you'll get past grt but also more likely you'll have trouble in drt. We set them more conservatively in the PDK.

[oharboe]

The violations are related to routing congestion so they can occur over fill cells.
Do you not see congested areas in the heatmap?

No.

I'm not sure what "anything suspicious" means.

There is nothing special about the area that is listed in the congestion.rpt. That area should be a walk in the park...

Also you could play with the layer derates in grt. Reducing them makes it more likely you'll get past
grt but also more likely you'll have trouble in drt. We set them more conservatively in the PDK.

What are "derates"?

Try something like changing from 0.5 to 0.25 or 0.8?

If I understand https://openroad.readthedocs.io/en/latest/main/src/grt/README.html correctly, 0.25 is easier on global routing, but harder on detailed routing.

set_global_routing_layer_adjustment $env(MIN_ROUTING_LAYER)-$env(MAX_ROUTING_LAYER) 0.25
set_routing_layers -signal $env(MIN_ROUTING_LAYER)-$env(MAX_ROUTING_LAYER)

Hmm.... I wonder how to go about adjusting this. There are a LOT of nets and adjustments can be made for areas and even specific nets. The turnaround times are slow(hours). Without a strong mental model of what the problem is and how the global router works and with the very limited feedback, I don't see how to converge on a solution easily.

I would imagine that the autotuner can adjust some parameters, haven't looked into that, but there are a lot of parameters to adjust. It would be truely impressive if the autotuner can figure out how to tweak the all the parameters down to the area and net level...

[maliberty]

Something doesn't make sense. It fails to route due to congestion but you don't see any in the congestion map. If you load the congestion report in the drc viewer you should be able to see where the congestion is.

[oharboe]

Nice! I didn't know that the DRC viewer could load in the .rpt files :-)

I see what the problem is: I once heard a quip that you can't get an autorouter to route a straight line...

What's going on is that there is a macro at the right edge of the design. The pins the right edge of that macro should go a short distance horizontally to the right edge of the design. There's a ton of vertical and horizontal congestion whereas, intuitively, I'd expect there to be no congestion, just hortizontal pin to pin mapping.

[maliberty]

Can you show a local image of the area? If you click the underlined heat map you'll get an options dialog. You can investigate on what layers/directions the congestion is happening in.

[oharboe]

Hmm... something more is going on. It is still failing with:

[ERROR GRT-0118] Routing congestion too high. Check the congestion heatmap in the GUI.
Error: global_route.tcl, 19 GRT-0118

[rovinski]

Are any buffers inserted between the macros and the ports? That could easily blow up the routing.

It's also not necessarily required, but is usually helpful, if the macro pins and port pins are aligned to the routing grid. Although that likely wouldn't show up until detailed routing.

[oharboe]

Are any buffers inserted between the macros and the ports? That could easily blow up the routing.

Yes, that can happen, which is why I try to reduce the extranous area to the right and at the top of the array.

However, I think that the PDN is overlaying the pins at the top. I have found that I need to align macros and the size of the design so that top and right edge does not fall on the PDN grid.

It's also not necessarily required, but is usually helpful, if the macro pins and port pins are aligned to the routing grid. Although that likely wouldn't show up until detailed routing.

How can I ensure that? I'm using constraints and I specify minimum distance in tracks.

[rovinski]

Yes, that can happen, which is why I try to reduce the extranous area to the right and at the top of the array.

Buffers can be inserted for more reasons than just long wires - see my other comment.

However, I think that the PDN is overlaying the pins at the top. I have found that I need to align macros and the size of the design so that top and right edge does not fall on the PDN grid.

I'm not exactly sure what you mean here. Are you using 0 core margin? Are the pins in the same layer as the PDN? The PDN can only block pins if it's in the same layer and next to the pins.

How can I ensure that? I'm using constraints and I specify minimum distance in tracks.

The simple way is to turn on the track guides in the GUI and see if the macro pins are lined up center-to-center. If not, adjust the position of the macro so that it is. If the macro was created using OpenROAD, then there's a good chance it's already fine.

[oharboe]

Are these the "track guides in the GUI"?

Not really easy to make anything of that...

The macro was created using OpenROAD, but I placed it manually using MACRO_PLACEMENT environment variable in config.mk.

I didn't change the PDN layers or pin layers, it is default for asap7.

[oharboe]

I'm going to investigate something else... Could it be that the PDN is overlaying the pins at the top of the macro?

Here the blue line (from PDN) is not overlaying the pins at the top of the macro.

The vertical PDN lines go all the way to the top, but the horizontal not all the way to the right edge. That's an assymetry that I find puzzling.

The screenshot below is the upper right corner of the design I'm tinkering with:

The routing congestion heat map shows that there's a lot of routing at the top (orange). There's nothing there...

[oharboe]

Tweaking the size so that the power grid horizontal/vertical doesn't fall at the upper and right edge of the macro, didn't help. Still get routing congestion.

[rovinski]

I don't see anything concerning in the first two images. The PDN is on a different layer from the pins and is not significantly dense. If the PDN were a problem, it would show up in the heatmap. It likely wouldn't cause any problems until detailed routing.

Are you sure there are no cells whatsoever in the orange area? What % density does the orange represent?

[oharboe]

Are you sure there are no cells whatsoever in the orange area?

Yes. The only thing that's in the orange area are fillers, except just underneath the pins, where there are buffers & hold, but those are in a V shape just underneath the pins.

What % density does the orange represent?

I don't see a legend to explain the colors. I assume orange/red is more dense than yellow/greenish.

What is curious is that there's no such strip of orange on the other edges, even if these edges are nearly symmetrical.

[oharboe]

I scaled down the design (fewer wires) and then I can see gui_route.

Although there's only horizontal routes between the inner macro's right edge and the top level design's right edge, there's this spagetti just before the input/outputs on the right side...

No wonder this doesn't work if you scale up the number of wires.

I wonder what is going on here...

zoom in:

set sdc_version 2.0
create_clock [get_ports clock] -period 4000 -waveform {0 2000}

set clk_name  clock
set clk_port_name clock
set clk_period 250
set clk_io_pct 0.2

set clk_port [get_ports $clk_port_name]

set non_clock_inputs [lsearch -inline -all -not -exact [all_inputs] $clk_port]

set_input_delay  [expr $clk_period * $clk_io_pct] -clock $clk_name $non_clock_inputs
set_output_delay [expr $clk_period * $clk_io_pct] -clock $clk_name [all_outputs]

[maliberty]

So what are those signals in the 'spagetti'? Why are they routing over the macro?

[oharboe]

These are pins in an inner macro that are exposed directly to the pins of the outer macro.

[oharboe]

You can see the same thing going on in this example:

The-OpenROAD-Project/OpenROAD-flow-scripts#743

[rovinski]

From your zoomed in screenshot, it looks like a bunch of setup and hold buffers are being inserted. This makes me think of a few things:

1. The pins might be high load or the wires have high delay for some reason.
2. There are bad constraints between the macros and/or the I/O pins
3. There is a deep clock tree, which means that there is significant skew between the I/O pins and the macros.

It would be helpful to check the clock tree latency vs. your clock period. If there is a large clock tree latency, it can create an I/O constraint that is impossible to meet.

[oharboe]

I'll study, but regarding point 1 above, how is the load of a macro specified? How can the macro know what load will be placed on its pins?

I'm not yet working with a top level design, these are just macros part of a bigger design for now.

[oharboe]

@rovinski @maliberty I've polished up a test case The-OpenROAD-Project/OpenROAD-flow-scripts#764.

This array is open source and big enough to exhibit the problems, though less pronounced, seen above that will appear as you go from from mock-array-big to a bigger version, yet it will build in a more reasonable amount of time.

In this pull request, I increase the datapath from 8 to 64 bits, at which point global routing fails: The-OpenROAD-Project/OpenROAD-flow-scripts#774

[maliberty]

I looked at The-OpenROAD-Project/OpenROAD-flow-scripts#764. I see a path like:

The input arrives at 50ps (20% of your 250ps clock period). The capture clock path is at nearly 200ps with 44ps of hold from the liberty:

>>> report_checks -through hold2693/A -path_delay min -format full_clock_expanded
Startpoint: io_insVertical_0_2[4] (input port clocked by clock)
Endpoint: ces_2_7 (rising edge-triggered flip-flop clocked by clock)
Path Group: clock
Path Type: min

  Delay    Time   Description
---------------------------------------------------------
   0.00    0.00   clock clock (rise edge)
   0.00    0.00   clock network delay (propagated)
  50.00   50.00 ^ input external delay
   0.00   50.00 ^ io_insVertical_0_2[4] (in)
  10.10   60.10 ^ hold3918/Y (BUFx2_ASAP7_75t_R)
  11.76   71.86 ^ hold2691/Y (BUFx2_ASAP7_75t_R)
  11.77   83.63 ^ hold3919/Y (BUFx2_ASAP7_75t_R)
  11.61   95.24 ^ hold1087/Y (BUFx2_ASAP7_75t_R)
  11.64  106.88 ^ hold2692/Y (BUFx2_ASAP7_75t_R)
  11.81  118.69 ^ hold245/Y (BUFx2_ASAP7_75t_R)
  11.74  130.43 ^ hold2693/Y (BUFx2_ASAP7_75t_R)
  11.65  142.08 ^ hold1088/Y (BUFx2_ASAP7_75t_R)
  11.75  153.83 ^ hold2694/Y (BUFx2_ASAP7_75t_R)
  11.95  165.78 ^ input149/Y (BUFx2_ASAP7_75t_R)
  12.04  177.82 ^ hold2695/Y (BUFx2_ASAP7_75t_R)
  12.02  189.84 ^ hold1089/Y (BUFx2_ASAP7_75t_R)
  12.10  201.94 ^ hold2696/Y (BUFx2_ASAP7_75t_R)
  12.11  214.05 ^ hold246/Y (BUFx2_ASAP7_75t_R)
  12.07  226.12 ^ hold2697/Y (BUFx2_ASAP7_75t_R)
  11.89  238.01 ^ hold1090/Y (BUFx2_ASAP7_75t_R)
  12.17  250.19 ^ hold2698/Y (BUFx2_ASAP7_75t_R)
   0.05  250.24 ^ ces_2_7/io_ins_1[4] (Element)
         250.24   data arrival time

   0.00    0.00   clock clock (rise edge)
   0.00    0.00   clock source latency
   0.00    0.00 ^ clock (in)
  16.72   16.72 ^ wire1/Y (BUFx12f_ASAP7_75t_R)
  74.94   91.66 ^ clkbuf_0_clock/Y (BUFx4_ASAP7_75t_R)
  51.85  143.51 ^ clkbuf_2_1_0_clock/Y (BUFx4_ASAP7_75t_R)
  44.16  187.67 ^ clkbuf_3_3__f_clock/Y (BUFx4_ASAP7_75t_R)
   8.97  196.64 ^ ces_2_7/clock (Element)
   0.00  196.64   clock reconvergence pessimism
  44.22  240.86   library hold time
         240.86   data required time
---------------------------------------------------------
         240.86   data required time
        -250.24   data arrival time
---------------------------------------------------------
           9.39   slack (MET)

So you'll need 200ps of hold buffering to make this pass. BUFx2 is ~12ps so you need a good number to meet your hold constraint. I don't see anything wrong here other than inputs that arrive quite early relative to the macro's requirements.

The clock path doesn't look unreasonable:

[oharboe]

I tried bumping the clock to 2 GHz, but I still get lots of hold cells on the inputs:

[maliberty]

There is no reason to think that will help. If anything it will make it worse since your input arrival time is 20% of the clock period.

[rovinski]

@oharboe so this means that it is (mostly) case 3 from this comment.

In simple terms, you have a deep clock tree (200ps capture clock out of a 250ps period) but your I/O constraints are only 50ps. You are effectively telling the tool "This design has a deep clock tree, but it is getting inputs from a design with a shallower clock tree". In order to meet timing, the tools have to insert ~150ps of delay, hence all the hold buffers.

This is an important time to consider whether your I/O constraints are valid. Question to ask yourself: Are your I/Os to this design going to have a similar clock tree latency as this design, or are they going to be different? Design partitioning and constraining can be a complex process.

[oharboe]

Thank you. I've only been working on RTL code on FPGA tools where the constraints file were already set up by someone else, so to try out OpenROAD I cobbled together a constraints file from what i could find.

What I want at this point is a "forgiving" .sdc file, such that I can get the flow set up and then start working on one concern at the time.

Working on one concern at the time while having the entire flow set up is a more software like approach. From my limited experience, hardware tends to be, necessarily, a more waterfall approach where you first build a solid foundation, rather than putting up the entire house and improving it piecemeal.

Then again, part of OpenROAD's goal is to allow tapping into the larger pool of software engineers to mitigate the shortage of hardware engineers... :-)

[oharboe]

Doubling the clock period removed all the hold cells congestion.

Solved. :-)

[maliberty]

The most forgiving would be to put no i/o timings at all aside from clock. Then only paths internal to the block would be checked. Another option would be to give different min/max timings where they are both forgiving.

[oharboe]

👍 I added the IO timings as the flow would fail without them. dont recall the issue number, but I recall you clarified the error message.