CM4 Lite freeze during s2idle (USB pmdomain powered down)

[lategoodbye]

Describe the bug

Hi,
I took the first steps of s2idle support on CM4 lite (decided against RPi 4 B because of PCIe/VL805 complexity). I got it working for the most critical interfaces so far (Debug UART, USB, HDMI, emmc2, genet). But I needed to make a ugly hack, otherwise the full system freeze/never wake up.

Can you make any sense of it?
Do dwc2 and xhci on BCM2711 share the same power domain?

Here is my current working branch: https://github.com/lategoodbye/linux-dev/commits/v6.13-pm/

Steps to reproduce the behaviour

sudo su
echo enabled > /sys/class/tty/ttyS1/power/wakeup
echo freeze > /sys/power/state
# wait some seconds
# press key on console

alternative with PM_TEST enabled:

sudo su
echo enabled > /sys/class/tty/ttyS1/power/wakeup
echo platform > /sys/power/pm_test
echo freeze > /sys/power/state
# wait some seconds

Device (s)

Raspberry Pi CM4 Lite

System

This is a Mainline kernel 6.13-rc1 with arm64/defconfig as kernel config.

Logs

No response

Additional context

settings in config.txt:
device_tree=bcm2711-rpi-cm4-io.dtb
enable_uart=1
arm_64bit=1

[cm4]
otg_mode=1

[lategoodbye]

Okay, it seems that i solved the mystery behind this issue. I added more debug messages in order to see what's going on:

root@raspberrypi:/sys/power# echo freeze > state
[   70.724347] xhci_suspend finished
[   70.727730] xhci_plat_suspend finished
[   70.755624] bcm2835-power bcm2835-power: Power grafx off
[   70.761127]  USB: Set power to 0
# suspend finished & wakeup via debug UART
[   74.653040]  USB: Failed to set power to 1 (-110)
[   74.657875] bcm2835-power bcm2835-power: Power grafx on
[   74.663182] bcm2835-power bcm2835-power: Power v3d on
[   75.676474] raspberrypi-clk soc:firmware:clocks: Failed to get fw-clk-core frequency: -110
[   76.700469] raspberrypi-clk soc:firmware:clocks: Failed to get fw-clk-core frequency: -110
[   77.724470] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-core frequency: -110
[   78.748470] raspberrypi-clk soc:firmware:clocks: Failed to get fw-clk-core frequency: -110
[   79.772471] raspberrypi-clk soc:firmware:clocks: Failed to get fw-clk-core frequency: -110
[   80.796469] raspberrypi-clk soc:firmware:clocks: Failed to get fw-clk-core frequency: -110
[   81.820469] raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-core frequency: -110

So the issue occurs during resume and it seems the VC4 firmware crashes, because all firmware tags run into a timeout.

After a lot of trial and error, i had the idea to use the bcm2835-power instead of raspberrypi-power:

diff --git a/arch/arm/boot/dts/broadcom/bcm2711-rpi.dtsi b/arch/arm/boot/dts/broadcom/bcm2711-rpi.dtsi
index 6bf4241fe3b7..7aa7b4b5b8fc 100644
--- a/arch/arm/boot/dts/broadcom/bcm2711-rpi.dtsi
+++ b/arch/arm/boot/dts/broadcom/bcm2711-rpi.dtsi
@@ -101,7 +101,3 @@ &v3d {
 &vchiq {
 	interrupts = <GIC_SPI 34 IRQ_TYPE_LEVEL_HIGH>;
 };
-
-&xhci {
-	power-domains = <&power RPI_POWER_DOMAIN_USB>;
-};
diff --git a/arch/arm/boot/dts/broadcom/bcm2711.dtsi b/arch/arm/boot/dts/broadcom/bcm2711.dtsi
index e4e42af21ef3..5eaec6c6a1df 100644
--- a/arch/arm/boot/dts/broadcom/bcm2711.dtsi
+++ b/arch/arm/boot/dts/broadcom/bcm2711.dtsi
@@ -610,6 +610,7 @@ xhci: usb@7e9c0000 {
 			#address-cells = <1>;
 			#size-cells = <0>;
 			interrupts = <GIC_SPI 176 IRQ_TYPE_LEVEL_HIGH>;
+			power-domains = <&pm BCM2835_POWER_DOMAIN_USB>;
 			/* DWC2 and this IP block share the same USB PHY,
 			 * enabling both at the same time results in lockups.
 			 * So keep this node disabled and let the bootloader

and this works :-)

root@raspberrypi:/sys/power# echo freeze > state
[  116.131104] xhci_suspend finished
[  116.134493] xhci_plat_suspend finished
[  116.141287] bcm2835-power bcm2835-power: Power grafx off
[  116.146923] bcm2835-power bcm2835-power: Power usb off
[  116.152146] bcm2835-power bcm2835-power: Power peri_image off
[  116.157983] bcm2835-power bcm2835-power: Power image off
[  123.809645] bcm2835-power bcm2835-power: Power image on
[  123.809742] bcm2835-power bcm2835-power: Power grafx on
[  123.814971] bcm2835-power bcm2835-power: Power peri_image on
[  123.814983] bcm2835-power bcm2835-power: Power usb on
[  123.820282] bcm2835-power bcm2835-power: Power v3d on
[  123.838944] xhci_plat_resume started
[  123.842589] xhci_resume started

Would still be nice to get a confirmation about the solution.