Possible hardware fault

Started by jameswhiteman

jameswhiteman

Hi Alain,
BACKGROUND
Currently we have a board with the nocan chip talking to our main processor. This talks back to the nocan hat, running on a pi. We have been field testing for the last two weeks with no issues until today.
PROBLEM
Today I went to upload to our chip and I got ;
# 'nocanc upload' failed, Upload failed
Investigation showed nocand running. However nocanc list-nodes returned nothing. I could view the output from our main chip via serial and it was running with out fault.
HYPOTHESIS
The nocan chip failed, is there a command where we can reboot the nocan chip? As the device is in a field 100 + k's away

Other than that absolutely loving the nocan system, uploading binaries over the air TICK!!

Much thanks,

James

Alain

Hi James,

How many nodes you have? If you have two nodes (or more) and none of them are listed with nocanc list-nodes it could simply be that the cable got unplugged.

Here's a list of things I would try.

1) power off and on the NoCAN network

Type nocanc power off followed by nocanc power on.

2) Restart nocand on the raspberry pi

Make sure you have the latest version of nocand (currently 0.1.13) and if necessary download the latest version from https://www.omzlo.com/downloads/nocand-linux-arm.tar.gz

It's also good to use the latest version of nocanc. You can check the version of the tools by typing nocand version and nocand version.

3) Run nocand server manually and check out the output, which should look like this:

2019/02/19 09:53:41 DEBUG No logs will be saved to file.
2019/02/19 09:53:41 INFO nocand version Undefined
2019/02/19 09:53:41 INFO Listening for clients at [::]:4242
2019/02/19 09:53:41 INFO Connected to driver using SPI interface at 250000 bps
2019/02/19 09:53:41 INFO Reseting driver
2019/02/19 09:53:41 DEBUG+ (0) SPI SEND 3: 010200 (SPI_OP_RESET)
2019/02/19 09:53:41 DEBUG+ (0) SPI RECV 3: 18a000
2019/02/19 09:53:41 DEBUG+ Waiting for TX line to be HIGH
2019/02/19 09:53:44 DEBUG+ TX line is HIGH
2019/02/19 09:53:44 DEBUG+ (1) SPI SEND 19: 02000000000000000000000000000000000000 (SPI_OP_DEVICE_INFO)
2019/02/19 09:53:44 DEBUG+ (1) SPI RECV 19: 0043414e300001090010001743434334353520
2019/02/19 09:53:44 INFO Firmware version 0.1, signature='CAN0', chip id=090010001743434334353520
2019/02/19 09:53:44 INFO Driver signature verified.
2019/02/19 09:53:44 INFO Successfully connected to PiMaster.
2019/02/19 09:53:44 DEBUG+ (2) SPI SEND 2: 0401 (SPI_OP_SET_POWER)
2019/02/19 09:53:44 DEBUG+ (2) SPI RECV 2: 00a0
2019/02/19 09:53:44 DEBUG+ (3) SPI SEND 11: 0300000000000000000000 (SPI_OP_POWER_LEVEL)
2019/02/19 09:53:44 DEBUG+ (3) SPI RECV 11: 80400000000000fd051006
2019/02/19 09:53:44 INFO Loaded node cache file /home/pi/.nocand/cache with 19 entries
2019/02/19 09:53:44 DEBUG+ Driver voltage=0.0, current sense=0 (~ 0 mA), reference voltage=3.34, status(40)=+powered.
2019/02/19 09:53:54 DEBUG+ (4) SPI SEND 11: 0300000000000000000000 (SPI_OP_POWER_LEVEL)
2019/02/19 09:53:54 DEBUG+ (4) SPI RECV 11: 18400050050400fe051006
2019/02/19 09:53:54 DEBUG+ Driver voltage=12.1, current sense=4 (~ 3 mA), reference voltage=3.34, status(40)=+powered.
2019/02/19 09:54:04 DEBUG+ (5) SPI SEND 11: 0300000000000000000000 (SPI_OP_POWER_LEVEL)
2019/02/19 09:54:04 DEBUG+ (5) SPI RECV 11: 1840004d050200fe051006
...

The lines saying DEBUG+ Waiting for TX line to be HIGH followed by DEBUG+ TX line is HIGH indicate that the PiMaster reseted correctly and that things are working well on the firmware side.

Next, observe the lines Driver voltage=12.1, current sense=4 (~ 3 mA), reference voltage=3.34, status(40)=+powered. :

  • If Driver voltage does not correspond approximately to the voltage provided to the NoCAN network, something is wrong on the power side.
  • If current sense is 0, it could be that the NoCAN network cable is unplugged, though if your Driver voltage is high enough (e.g. 24V) and you only have one node, you might get 0 here as well in some cases.

4) Perform a full hardware reset of the PiMaster HAT, with the wiringpi utility.

gpio mode 25 out
gpio write 25 0
gpio write 25 1
gpio write 25 in

If all fails, please send us a full log file of nocand running at <support(at)omzlo.com>. You can generate a log file with the --log-file option:

nocand server --log-file="/tmp/nocand.logs"

This will store the log file in /tmp/nocand.logs.

Alain

jameswhiteman

Hi Alain,

Thank you for you quick and detailed response. I didn't know of nocanc power off I shall definitely try that next time. Does that cycle the power on the chips?
You are correct I should have posted what version I was running at the header.
I'm running v0.1.13 both nocanc and nocand
nocand is set up as a service and I restarted it a few times, with no errors, I even have a pub event I fire to the nocand server that is monitored on our board and spits it out on Serial, sending to the nocand server gave no errors. However the minicom showed no response other than the general running messages of the board, hence leading me to believe it was a nocan comms failure.
There is only one node, which is our board, although there are 3 nodes in the cache, left over from testing other nodes. The node that is left and running is node 3.
I did do nocanc monitor at the time and had no issue with the voltage.

current nocanc monitor returns ;

EVENT   bus-power-status-update-event   #6      Driver voltage=13.7, current sense=0, reference voltage=3.35, status(40)=+powered.

unfortunately I didn't grab the logs I will know for next time
journalctl -u nocand.service did return the boot from this afternoon

Feb 20 15:17:11 raspberrypi nocand[747]: 2019/02/20 15:17:11 DEBUG+ Waiting for TX line to be HIGH
Feb 20 15:17:12 raspberrypi nocand[747]: 2019/02/20 15:17:12 INFO Created new client 0 (127.0.0.1:53906)
Feb 20 15:17:12 raspberrypi nocand[747]: 2019/02/20 15:17:12 DEBUG+ Processing event client-auth-event(2) from client 0 (127.0.0.1:53906), with 24 bytes of payload

all looks well, as it should as the system is running fine.
Like I say we have been running rock solid in the field for at-least 2+ weeks, with constant uploads, the only issues we are having is to do with our own code, this was the first time I had a issue with nocan comms.
Touch wood it was a random one off
Appreciate your time and when our product does launch I will add it to the forum projects page.

Cheers,

James

Alain

Good to know it work out well. Thanks for the feedback, it's always useful. We'll try to add better documentation for the noncanc options.