Firmware 0.4.0 randomly goes into ULPM?

Ever since upgrading to firmware 0.4.0 (I used yellow_wp77xx_0.4.0.spk, but creating it myself in my updated leaf workspace with the mangOH git repo has the same problem), my mangOH Yellow is randomly turning itself off (I think it may be going into ULPM?).

Without me doing anything, this suddenly shows up in the console:

[ 2913.703514] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0 
[ 2913.709474] i2c-msm-v2 78b8000.i2c: NACK: slave not responding, ensure its powered: msgs(n:1 cur:0 tx) bc(rx:0 tx:22) mode:FIFO slv_addr:0x3a MSTR_STS:0x0c1300c8 OPER:0x00000090
[ 2913.725733] i2c-msm-v2 78b8000.i2c: NACK: slave not responding, ensure its powered: msgs(n:1 cur:0 tx) bc(rx:0 tx:22) mode:FIFO slv_addr:0x3a MSTR_STS:0x0c1300c8 OPER:0x00000090
[ 2913.749748] pm_set_mcu_ulpm_enable: wakeup source setup mask=0x2
rcK: Executing run_K_scripts... 
Stopping linkmon: no linkmon found; none killed
done
[ 2913.952860] gbam_disconnect_work: gbam_disconnect_work: Bam channel is not opened
[ 2913.971689] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister ecm0
[ 2914.030545] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister ecm0

…
And continues just like a ULPM shutdown. The last stuff printed is:

[ 2932.753645] reboot: Power down
[ 2932.755740] Failed to disable secure wdog debug: -4
[ 2932.761129] Calling SCM to disable SPMI PMI

Any ideas about what might be causing this? I don’t have any custom apps loaded on the module. Just the stock apps. (Which is really strange, because helloYellow typically prevents ULPM shutdowns…)

I have tested with waking on ADC3 voltages earlier, but I never got random shutdowns like this until I upgraded to 0.4.0.

Is your USB port providing enough power?
Seems it like it happens 50 minutes into your unit being on? Is that a correct understanding.
Are you using Octave or non-Octave fw?

I’m using non-Octave firmware. I haven’t noticed any problems with the USB port power, but I do also have a LiPo battery hooked up.

You’re correct about the timing. I went back onto my console and it seems to consistently happen about 48 minutes into the unit being powered on. I hadn’t noticed this until you mentioned it. Here are all of the historical “random” shutdown messages in my console log:

[ 2924.286987] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0
[ 2912.118214] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0
[ 2910.696398] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0
[ 2923.614657] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0
[ 2913.703514] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0

Now I’m curious if anyone else will see this after leaving it on that long…

Edit: Just happened again:

[ 2910.861493] swimcu_pm_wusrc_config: check statep->gpio_pin_mask 0x0

I have my unit running

[ 4134.122985] swimcu_gpio_irq_event_handle: Re-enabled irq 0 type A for MCU GPIO 0
[ 4134.923798] i2c-msm-v2 78b8000.i2c: NACK: slave not responding, ensure its powered: msgs(n:1 cur:0 tx) bc(rx:0 tx:2) mode:FIFO slv_addr:0x3a MSTR_STS:0x0c1300c8 OPER:0x00000090
[ 4134.940455] i2c-msm-v2 78b8000.i2c: NACK: slave not responding, ensure its powered: msgs(n:1 cur:0 tx) bc(rx:0 tx:2) mode:FIFO slv_addr:0x3a MSTR_STS:0x0c1300c8 OPER:0x00000090
[ 4136.127609] swimcu_gpio_irq_event_handle: Re-enabled irq 0 type A for MCU GPIO 0

Note, no dropouts for me
Do you have the hologram SIM connected?

Hmm…weird. Yes, I have the Hologram SIM inserted (external SIM slot 1 selected, APN = hologram). cm radio reports that I’m attached to AT&T’s LTE network. I don’t have a data session connected (cm data does not report a connection). I’ll try tomorrow with the internal SIM selected to see if the SIM is related to this problem at all.

Is there anything else I need to do in order to fully reset my board to factory defaults? I’ve been flashing the 0.4.0 spk with swiflash, and also doing legato stop ; rm -rf /mnt/flash/legato/* ; reboot to ensure that nothing remains from previous updates.

That should be fine.
I am wondering if Hologram kicks your module off the network after 48 minutes or so that causes the issue.
Anyways, let me know how that goes.

I’ve confirmed that it’s the Hologram card. At some point I’ll double check and see if it happens in 0.3.0 too.

With internal SIM selected and iot.swir APN, no data session, it stays up forever.
With external SIM selected and hologram APN, no data session, it goes to ULPM after 48 minutes.

Next I will see what happens if I leave the Hologram data session open. In practice I will have a data session open at all times, so this may be a nonissue. It still concerns me though…why would my module being kicked off the network send it into ULPM? It feels like nothing should be able to force me into ULPM if I don’t want to go to it. It’s overriding helloYellow’s block on ULPM – if I try to go to ULPM manually without stopping helloYellow, it won’t let me. So why is this able to?

Interesting…is the SIM forcing yellow to go to PSM or EDRX mode?

I think you might be onto something! I did some digging, and I think I’m starting to understand. With the radio off, it never shuts off, even if the SIM is present. With the radio on, it shuts off. Check this out:

AT+CPSMS?
+CPSMS:1,,,"00011000","01001000"

“01001000” is the requested active time, and I traced through various documents to discover that it really means:

Bits 7-5 = 010 = value is incremented in multiples of decihours
Bits 4-0 = 01000 = 8

8 decihours = 48 minutes

Is this a value that the SIM can control? It seems different from the WP77xx documented default. Obviously the simple solution is to clear this…I’m just trying to understand how it got configured that way in the first place. Is this something that is controlled by any of the sample apps?

It looks like it’s as simple as just setting this to 0 and the module remembers the setting, so I can probably just do this from an app on every boot and call it good? I’m just paranoid about what turned it on in the first place.

I don’t know if this is helpful at all, but I thought I’d mention it. I think I understand what is setting up CPSMS incorrectly. First of all on firmware 0.4.0, I verified that CPSMS was set to 0:

AT+CPSMS?
+CPSMS:0,,,"01100000","00000000"

Then, I tried these commands:

app stop helloYellow
pmtool bootOn adc 3 2000 400 1800
pmtool shutdown

…and it didn’t shut down. It printed “Initiated shutdown of MDM” and then nothing. Then I went back in and checked CPSMS:

AT+CPSMS?
+CPSMS:1,,,"00011000","01001000"

So it seems that pmtool is what is setting up CPSMS incorrectly. If I set it back to 0, then it immediately goes to ULPM:

AT+CPSMS=0

Is it possible that something changed recently causing this? I saw another thread talking about this issue of pmtool enabling PSM and preventing entry to ULPM. Is it something related to me being on LTE now? I keep blaming the firmware 0.4.0 version for everything, but I also keep forgetting that I wasn’t able to use LTE before 0.4.0.

Note that when I’m not connected at all, the CPSMS thing doesn’t happen. So it seems that when you’re connected over 2G or LTE, pmtool shutdown is setting up PSM mode for some reason. And if you’re connected over LTE, it prevents ULPM from operating. I think this might be a bug?

this is more a modem firmware topic and not related to 0.4.0 yellow firmware. I will need to ask a different team for more info on this.

1 Like