MangOH Red WP7603 - I2C arbitration lost

mangOH Red
1

Hi,

We are seeing some rare instances of I2C peripherals becoming inaccessible. The logs show I2C: ARB_LOST messages. Our application uses the battery charger, fuel gauge and accelerometer. Power cycling the board seems to be the only way to recover from this. Reset doesn’t help. Any idea why this could occur? Is it possible to recover from this in the application or drivers?

Logs

Dec 12 18:33:45 swi-mdm9x28 user.info swiapp: result : -3        ,Init step :1   ,qmiErrorCode: 0
Dec 12 18:33:45 swi-mdm9x28 user.info swiapp:   Sleeping...
Dec 12 18:33:46 swi-mdm9x28 user.err time_daemon_mdm:[599]: genoff_modem_qmi_service_request: qmi_client_get_service_list returned -2num_services 0
Dec 12 18:33:47 swi-mdm9x28 user.warn Legato: -WRN- | _UNKNOWN_[645]/framework T=main | LE_FILENAME daemon_Daemonize() 120 | Timed out after waiting 5000 ms for indication from child.
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.785333] Starting psmd: done
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.826205] PSM: Found config file ARGV /data/psm
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.826269] PSM: PSMD Start
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.833489] PSM: server_config_path /data/psm/server_config
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.845065] PSM: Path for PSM configuration: /data/psm
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.849348] PSM: modem_flag_path /data/psm/modem_flag
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.849868] PSM: backup_nv_path /data/psm/psm_nv_backup.txt
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.856582] PSM: Load modem flag: 1
Dec 12 18:33:47 swi-mdm9x28 user.info kernel: [   14.856651] subsys-restart: __subsystem_get(): Changing subsys fw_name to modem
Dec 12 18:33:47 swi-mdm9x28 user.info kernel: [   14.879532] pil-q6v5-mss 4080000.qcom,mss: modem: loading from 0x82a00000 to 0x87800000
Dec 12 18:33:47 swi-mdm9x28 user.warn kernel: [   14.914160] pil-q6v5-mss 4080000.qcom,mss: Debug policy not present - msadp. Continue.
Dec 12 18:33:47 swi-mdm9x28 user.info kernel: [   14.921757] pil-q6v5-mss 4080000.qcom,mss: Loading MBA and DP (if present) from 0x8f600000 to 0x8f700000 size 100000
Dec 12 18:33:47 swi-mdm9x28 user.info kernel: [   15.051771] pil-q6v5-mss 4080000.qcom,mss: MBA boot done
Dec 12 18:33:48 swi-mdm9x28 user.err time_daemon_mdm:[599]: genoff_modem_qmi_service_request: qmi_client_get_service_list returned -2num_services 0
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.450168] mangoh red mangoh red: mapping i2c switch
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.450443] pca954x 4-0071: GPIO lookup for consumer reset
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.450458] pca954x 4-0071: using lookup tables for GPIO lookup
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.450470] pca954x 4-0071: lookup for GPIO reset failed
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.451304] i2c i2c-4: Added multiplexed i2c bus 5
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.452609] i2c i2c-4: Added multiplexed i2c bus 6
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.453067] i2c i2c-4: Added multiplexed i2c bus 7
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.453449] i2c i2c-4: Added multiplexed i2c bus 8
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.453464] pca954x 4-0071: registered 4 multiplexed busses for I2C switch pca9546
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.453489] mangoh red mangoh red: mapping gpio expander
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.454592] mdm9607-asoc-snd soc:sound-wm8944: Defering mdm_asoc_machine_probe, q6_state 0
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.470744] gpiochip_find_base: found new base at 793
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.470961] gpiochip_add: registered GPIOs 793 to 808 on device: sx1509q
Dec 12 18:33:49 swi-mdm9x28 user.info kernel: [   16.471292] led led.0: led_probe(): probe
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.474960] mangoh red mangoh red: mapping accelerometer
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.502583] mdm9607-asoc-snd soc:sound-wm8944: Defering mdm_asoc_machine_probe, q6_state 0
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.567158] mangoh red mangoh red: mapping bmp280 pressure sensor
Dec 12 18:33:49 swi-mdm9x28 user.warn kernel: [   16.567519] 4-0076 supply vddd not found, using dummy regulator
Dec 12 18:33:49 swi-mdm9x28 user.warn kernel: [   16.567581] 4-0076 supply vdda not found, using dummy regulator
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.569646] bmp280 4-0076: GPIO lookup for consumer reset
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.569658] bmp280 4-0076: using lookup tables for GPIO lookup
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.569670] bmp280 4-0076: lookup for GPIO reset failed
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.571872] mangoh red mangoh red: mapping bq24296 driver
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.572409] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:2 cur:0 tx) bc(rx:1 tx:1) mode:FIFO slv_addr:0x6b MSTR_STS:0x081321d0 OPER:0x00000010
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.583976] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:1) mode:FIFO slv_addr:0x71 MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.596769] bq24190-charger 6-006b: Hardware init failed
Dec 12 18:33:49 swi-mdm9x28 user.warn kernel: [   16.601604] bq24190-charger: probe of 6-006b failed with error -2
Dec 12 18:33:49 swi-mdm9x28 user.debug kernel: [   16.601629] mangoh red mangoh red: mapping ltc2942 battery gauge
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.601914] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:1) mode:FIFO slv_addr:0x71 MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:49 swi-mdm9x28 user.err kernel: [   16.615339] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:1) mode:FIFO slv_addr:0x71 MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:49 swi-mdm9x28 user.err Legato: =ERR= | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1507 | Could not stat file at '/sys/bus/i2c/devices/6-0064/power_supply/LTC2942/charge_counter'. No such file or directory
Dec 12 18:33:49 swi-mdm9x28 user.err Legato: =ERR= | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1624 | Failed to create link at '/sys/bus/i2c/devices/6-0064/power_supply/LTC2942/charge_counter' in app 'batteryService'.
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_accel_scale' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068/i
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_accel_x_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068/i
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_accel_y_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068/i
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_accel_z_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068/i
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_anglvel_scale' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_anglvel_x_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_anglvel_y_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0068/iio:device0/in_anglvel_z_raw' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0068
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0076/iio:device1/in_pressure_input' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-007
Dec 12 18:33:50 swi-mdm9x28 user.info Legato:  INFO | supervisor[653]/supervisor T=main | app.c CreateFileLink() 1619 | Created file link '/sys/bus/i2c/devices/4-0076/iio:device1/in_temp_input' to '/legato/systems/current/appsWriteable/tcuDevice/sys/devices/i2c-0/0-0076/ii
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.730407] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.764658] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.800241] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.839559] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.870057] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:53 swi-mdm9x28 user.err kernel: [   20.906800] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:18) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:56 swi-mdm9x28 user.err kernel: [   24.929656] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:57 swi-mdm9x28 user.err kernel: [   25.083331] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:57 swi-mdm9x28 user.err kernel: [   25.279956] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:57 swi-mdm9x28 user.err kernel: [   25.372795] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:57 swi-mdm9x28 user.err kernel: [   25.474942] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:33:57 swi-mdm9x28 user.err kernel: [   25.575546] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:14) mode:FIFO slv_addr:0x3a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:01 swi-mdm9x28 user.err kernel: [   28.383826] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:01 swi-mdm9x28 user.err kernel: [   28.588714] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:01 swi-mdm9x28 user.err kernel: [   28.610316] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:03 swi-mdm9x28 user.err kernel: [   29.756019] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:03 swi-mdm9x28 user.err kernel: [   29.891781] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 18:34:03 swi-mdm9x28 user.err kernel: [   29.965972] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:1 cur:0 tx) bc(rx:0 tx:4) mode:FIFO slv_addr:0x1a MSTR_STS:0x081321d0 OPER:0x00000090
Dec 12 19:02:03 swi-mdm9x28 user.err kernel: [ 1710.338694] i2c-msm-v2 78b8000.i2c: ARB_LOST: msgs(n:2 cur:0 tx) bc(rx:2 tx:1) mode:FIFO slv_addr:0x68 MSTR_STS:0x081321d0 OPER:0x00000010
2

I haven’t seen this before. Is there any extra hardware attached to your mangOH Red? Does this appear at random or is there a pattern that you have noticed?

3

No there is no extra hardware attached to our board. It is an off the shelf mangOH Red with a minor mod to supply power through the RPi header.

It seems random at the moment. If there is a pattern, we haven’t observed one yet. We suspect that it seems to occur more on units which have had their battery drained, but we’ve also seen it occur on units that have had a good battery all along. I’m running more tests to find a pattern, as it is quite rare. But when it does happen we’re left with units stranded in the field with no way to recover except physically power cycling them.

Edit: Sorry I was not clear on what you meant by extra hardware. We have a Talon Wifi/Bluetooth IoT card in the IoT slot.

4

We haven’t been able to establish a pattern after a month of observation. It occurs at random. Any thoughts on why this could happen or how it can be recovered from without a physical reset?

5

Hello Ktanikel, did you find any method to remotely solve the problem without power reset?

6

No, we’ve deployed a solution where our application tries to detect this problem if any of the I2C devices stop responding and reboots. However, rebooting alone doesn’t always seem to fix the problem so we’re still grappling with it. Is anyone aware of a way to reset the I2C bus alone without resetting?