USB Issue Troubleshooting

Hardware related problems and workarounds

Re: USB Issue Troubleshooting

Postby timhavens » Wed Dec 03, 2014 10:30 pm

9600,

I did this exact thing this morning, and in fact the network worked fine (and the usb after some fiddling around)...some power cycles and such.

So 'it was working' and now after I 'shutdown -h now' then powered it down, and disconnected all cables from Parallella the USB has been back to giving me the -99 errors.

Just as I was typing this message I was trying something else...with the same hdmi kern/fpga installed. I powered the box down and connected my powered usb hub, only this time I didn't power the hub. SO power to the barrel connector on the Parallella and no power connected to my powered hub. And I see this:

"Welcome to Linaro 14.04 (GNU/Linux 3.12.0-g0bc9c3a-dirty armv7l)

* Documentation: https://wiki.linaro.org/
Last login: Wed Dec 3 22:17:53 2014 from 192.168.3.12
linaro@linaro-nano:~$ lsusb
Bus 001 Device 003: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
Bus 001 Device 002: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub"

So now I plug in an RTL (tv) dongle and lsusb says:

linaro@linaro-nano:~$ lsusb
Bus 001 Device 004: ID 0bda:2838 Realtek Semiconductor Corp. RTL2838 DVB-T
Bus 001 Device 003: ID 1a40:0101 Terminus Technology Inc. 4-Port HUB
Bus 001 Device 002: ID 1a40:0201 Terminus Technology Inc. FE 2.1 7-port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Notice the Dongle shows up. This is still with the powered usb hub (UN-POWERED).

I've heard of usb ports that don't like to see any return voltage coming from powered hubs at init time. (dunno why really, seems some things are ok with that and some aren't)....

Anyway...that's where I am with this now.

1.) the network has started up every time with the HDMI kern/fpga.
2.) the usb has started up 2 times now with the same as above.

Hope this helps? And thanks for you response.
timhavens
 
Posts: 6
Joined: Wed Dec 03, 2014 2:05 am

Re: USB Issue Troubleshooting

Postby timhavens » Wed Dec 03, 2014 10:35 pm

By the way, I'd been seeing constant respawns mentioning hvc0.

I also went into /etc/securetty and remarked out 'hvc0' and 'hvc1' which I don't think are required (but I could be mistaken). The respawns have stopped after a hard reboot. (ie. shutdown, power cycled).

Those respawns where happened a few times a second prior to this change.

In the back of my mind I was wondering if these were interfering with usb init...so figured it was worth a shot to remark them out and try.
timhavens
 
Posts: 6
Joined: Wed Dec 03, 2014 2:05 am

Re: USB Issue Troubleshooting

Postby 9600 » Thu Dec 04, 2014 8:58 am

Thanks for the feedback, it's much appreciated, and great to hear that you have resolved the USB issue!

As for the USB powered hub, that is strange. I would have thought that if it is compliant it would not send power back up to the host.

I had to look up what hvc devices are for and since it turns out it's Xen hypervisor console, I certainly shouldn't have thought we'd need them. And if they were trying to write to the serial console as USB initialises, I guess it's possible they could effect it, as it becomes unstable when the serial console is enabled (as is the case with the default headless kernel/devicetree, but not the HDMI one, which has console on UART disabled). Although it could be a red herring, but good to get rid of errors in any case!

Regards,

Andrew
Andrew Back (a.k.a. 9600 / carrierdetect)
User avatar
9600
 
Posts: 997
Joined: Mon Dec 17, 2012 3:25 am

Re: USB Issue Troubleshooting

Postby timhavens » Thu Dec 04, 2014 5:49 pm

Andrew,

Thanks for your comments and ideas. I wouldn't say that I've "resolved the USB issue" it's still picky about how things are started up, and I'm trying to narrow down a working sequence vs. a sequence that doesn't work.
timhavens
 
Posts: 6
Joined: Wed Dec 03, 2014 2:05 am

Re: USB Issue Troubleshooting

Postby zmc » Fri Jan 16, 2015 8:31 pm

My personal experience, hope it helps someone.

I've had my parallella since they were made publically available, and have been using it as my primary workstation for almost as long. I have had some usb issues intermitantly, but usually a powercycle fixes it.

Here's what was connected to my 1.5A powered compliant USB hub

Power to parallella
Power to hacked 12v mini powersupply fan (running with 5v is quiet, and keeps the parallella quite cool, even in PR)
Power to mini-hdmi to VGA converter
5 SanDisk Cruzer USB flash drives (LVM Raid, and storing almost all my filesystem)
SoundBlaster X-Fi Go Sound Card
Fujitsu FI Flatbed/ADF Scanner
Generic USB Bluetooth PHY
Model M Keyboard
Logitech Wired G-series Mouse

Occasionally I connect a Sony PS Eye webcam/microphone, some other flash drives, or an Arduino or 2.

The parallella sometimes crashes the USB bus when starting X, and periodically in a non load dependent way.
When the bus crashes, my filesystems are unavailable, so are my input devices, and it's generally a trainwreck.

I was in the middle of doing something I cared about, and the bus crashed on me, so this problem has come into the crosshairs.

To eliminate the power-to-parallella aspect, I created a 5v power source to the barrel connector, rated for 10A instead of powering it from the 1.5A rated hub. This worsened the problem! The USB would crash often right after the filesystems were mounted (about the time the framebuffer sets the consolefont)

I then created an alternate 500ma power source for the fan and VGA adapter. This made the problem even worse still. The kernel would initialize the hub, and the usb bus would immediately crash. Warm resets, and plugging and unplugging the usb and hdmi cables would help a little but it was nearly impossible to reach a login with working usb. Main power source to parallella was tested to min 5.03v max 5.04v.

At this point the kernel was suggesting that I had a bad cable, so I tested with various cables, 2 different micro-usb to usb otg adapters, another powered hub, an unpowered hub, and finally just a keyboard attached with the same behavior.

Then I started backtracking. I moved the fan and hdmi adapter back to the powered hub, and again powered the parallella from the powered hub as well, and the system regained the usable stability I was used to.

The fact that unplugging and replugging the HDMI cable helps somehow, and that my least stable configuration was with the HDMI adapter on a seperate (underpowered) powersource, led me to believe that the fpga was getting confused with some kind of voltage on the HDMI port.

I started testing epiphany tests, I could get simple tests to run on the epiphany (adding numbers in a loop), but more complicated tasks (matmul-16) would hang.

My next step is to move the VGA adapter to the 10A powersource, to see what happens, but I think that somehow when my VGA adapter is underpowered it causes some problem in the Zync, and perhaps we have been looking at only part of the problem, since mostly USB users use HDMI as well.

I'll let you know what I find out. If anyone has any insight into this avenue of investigation, please let me know.
zmc
 
Posts: 24
Joined: Thu Jul 03, 2014 10:01 pm

Re: USB Issue Troubleshooting

Postby nharrington » Mon Apr 06, 2015 7:02 pm

How to report a USB problem:
* Failure mode (sometimes or always?)
40.52% - 0.54% variation over 4+ parallella's and thousands of tests

* Headless/hdmi configuration
Headless

* FPGA/Linux/Ubuntu version being used
ftp://ftp.parallella.org/ubuntu/dists/t ... 0.1.img.gz

* Fan used?
No

* SKU#
SKU P1601-DK03

* Power supply used
Supplied in Parallella box, as ordered from amazon.com

* Recommended: A picture of your USB setup (as shown here)
Please see attachment below. There are no usb peripherals connected for these tests.

Here's the code we use to perform these tests:
https://github.com/nharringtonwasatch/Foreman

Adapteva Parallella Screener (The Foreman) instructions:

Wasatch Photonics uses Adapteva Parallella SKU P1601-DK03's as an embedded linux
system. Core capabilities required include a USB bus that is available at least
83% of the time on boot up. A screener was created to allow us to find the
incoming Parallella boards that meet this level of usb availability. This setup
consists of 4 Parallella DC adapters connected to a Phidgets software controlled
relay. This relay turns on the devices for 35 seconds, issues a 'lsusb' command
on each device, and stores the status of the usb availability. The device is
then turned off for 75 seconds, then the process is repeated. This will produce
summary results like:

Code: Select all
04:4F:8B:00:2C:6E Pass: 1553, Fail: 1058  (55.56% failure rate)
04:4F:8B:00:2C:7C Pass: 610,  Fail: 1999  (86.42% failure rate)
04:4F:8B:00:2C:72 Pass: 2596, Fail: 14    ( 1.23% failure rate)

(These are from: http://wasatchcookbook.com/static/orig_results.html)

You can see live results here:
http://wasatchcookbook.com/static/foreman_results.html

Your feedback is greatly appreciated - what can we do to increase the level of acceptable
boards that make it through the screening process?

Updated picture to show BeagleBone Black control computer with an actual picture of George Foreman
Attachments
Foreman_With_Display_800x600.jpg
Foreman_With_Display_800x600.jpg (321.55 KiB) Viewed 33677 times
nharrington
 
Posts: 2
Joined: Mon Apr 06, 2015 6:16 pm

Re: USB Issue Troubleshooting

Postby nharrington » Mon Apr 13, 2015 3:05 pm

Here's an update on the total tests run so far:
Code: Select all
Parallella MAC     Pass   Fail   Percent Fail
04:4f:8b:00:2c:7c   172,  3391:  95.17
04:4f:8b:00:2c:6a  4846,   186:   3.70
04:4f:8b:00:2c:6e  2417,  2659:  52.38
04:4f:8b:00:2c:72  2164,    13:   0.60
04:4f:8b:00:2c:77     0,    31: 100.00
04:4f:8b:00:2c:76  1362,  3018:  68.90
nharrington
 
Posts: 2
Joined: Mon Apr 06, 2015 6:16 pm

Re: USB Issue Troubleshooting

Postby ajtravis » Mon Apr 13, 2015 4:02 pm

nharrington wrote:Here's an update on the total tests run so far:
Code: Select all
Parallella MAC     Pass   Fail   Percent Fail
04:4f:8b:00:2c:7c   172,  3391:  95.17
04:4f:8b:00:2c:6a  4846,   186:   3.70
04:4f:8b:00:2c:6e  2417,  2659:  52.38
04:4f:8b:00:2c:72  2164,    13:   0.60
04:4f:8b:00:2c:77     0,    31: 100.00
04:4f:8b:00:2c:76  1362,  3018:  68.90

Hi, nharrington.

Yours is a depressingly familiar story, but it's also reassuring because it's clear that I'm not the only one who has demonstrated the unreliability of the Parallella USB using objective tests. It's perplexing that Adapteva 'fail' to reproduce these USB failures themselves on boards that I and other people have on boards returned to Adapteva under RMA or in Adapteva's own QC pipeline prior to shipping the boards. I'm not unsympathetic to Adapteva, because it must be extremely frustrating to see reports that the Parallella USB is so unreliable when, according to them, most of the boards returned do not have unreliable USB ports.

However, enough of us have struggled with USB problems for long enough to know that it is not our fault that the Parallella USB doesn't initialise reliably: It seems to be a fundamental problem with the board and/or it's pre-boot environment. I've read all the posts about the USB problems and I realise that Adapteva have invested a lot of time trying to locate the problem, but there are a lot of people, including me, who bought these boards expecting USB to work reliably.

Indeed, I didn't think USB would be an issue when I bought the boards and many of the ideas I had about what I would be able to do with the board are now abandoned. Not just because of the USB problems, but also because of the conspicuous lack of e-Link cables to create a mesh-connected Epiphany cluster. Talk on the forums about connecting Parallella boards via GbE (Gigabit Ethernet) misses the point. The BIG issue with Epiphany is that it can be used as a mesh-connected processor array!

Andreas is quite right, the future is Parallel, but not with GbE NIC's and switches implementing 'first' generation Beowulf clusters on ARM SBC's with very little RAM. The ARM CPU's are not the main resource: The Epiphany is - That's where the true potential of Parallella resides. I guess the Kickstarter project was a steep learning curve for everyone involved, but to create an Epiphany board that is crippled by unreliable USB and having to disable the UART to get reliable networking is not good.

I'm hanging on in there because I believe in Parallella - I've ordered four Porcupine boards and I'm going to try interconnecting the Epiphany processors in a N/S 4x16 mesh. However, it would be great if Adapteva could come up with a solution to the USB problems many of us are experiencing with Parallella. I'd love to use them as part of a Parallella University Program project at the University of Dundee (UK), but that would require replacing R-Pi's with Parallellas as 'Desktop' computers using the USB and HDMI as the primary interfaces for students to connect peripherals, not connecting via their NIC because USB doesn't work!

http://eggg.st-andrews.ac.uk/4273pi/

[Note: Under the UK Trades Descriptions Act, Parallella is not fit for purpose as a Desktop Computer without a working USB]

Bye,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 167
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby piotr5 » Wed Apr 15, 2015 8:20 am

I really know no electronics, but has anyone tried to monitor the power-consumption during start up? I mean, usb problems could be caused by too much power being used in a short time, so after an initial spike of power consumption, maybe there's another spike causing init-difficulties? in other words, according to my experience with computers the number one suspect for hardware-failure is power-input, number 2 would be heat, followed by humidity and/or dust. I saw each of these 3 make my devices malfunction already...
piotr5
 
Posts: 230
Joined: Sun Dec 23, 2012 2:48 pm

Re: USB Issue Troubleshooting

Postby ajtravis » Wed Apr 15, 2015 9:29 am

piotr5 wrote:I really know no electronics, but has anyone tried to monitor the power-consumption during start up? I mean, usb problems could be caused by too much power being used in a short time, so after an initial spike of power consumption, maybe there's another spike causing init-difficulties? in other words, according to my experience with computers the number one suspect for hardware-failure is power-input, number 2 would be heat, followed by humidity and/or dust. I saw each of these 3 make my devices malfunction already...

Hi, piotr5.

I think that's unlikely: I set a 5s delay in the pre-boot environment to allow for the possibility of high start-up currents that might cause initial fluctuations on the power rails but it made no difference to the (un)reliability of Parallella USB initialisation...

Bye,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 167
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

PreviousNext

Return to Troubleshooting

Who is online

Users browsing this forum: No registered users and 7 guests