USB Issue Troubleshooting

Hardware related problems and workarounds

USB Issue Troubleshooting

Postby aolofsson » Mon Aug 18, 2014 12:30 pm

Hi all,

There are quite a few posts related to USB port failures. The goal of this post is to summarize all of them and to set the record straight.

The issue:
The USB phy does not always get initialized properly coming out of reset on some boards. After successful initialization, the USB port works fine.

Statements:
*Since day one, we have tested the USB port on every board shipped
*We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
*We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
*We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.

Some data:
* The probability of successful USB initialization varies between boards.
* Keeping USB device plugged in during powerup increases probability of success
* A power cycle reset is more likely to work than a push button reset
* Probability of success increased by using longer USB cables
* Probability of success increased by using certain USB peripherals/hub setups
* Most boards work with all combination of hubs/peripherals (no restrictions)

How to report a USB problem:
* Failure mode (sometimes or always?)
* Headless/hdmi configuration
* FPGA/Linux/Ubuntu version being used
* Fan used?
* SKU#
* For SKU A101040 boards owners, an indication whether the U13 device is populated
* Power supply used
* Recommended: A picture of your USB setup (as shown here)

Sincerely,
Andreas
Attachments
IMG_20140818_072237111_HDR.jpg
IMG_20140818_072237111_HDR.jpg (353.69 KiB) Viewed 11070 times
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: USB Issue Troubleshooting

Postby ajtravis » Mon Aug 18, 2014 3:10 pm

aolofsson wrote:[...]
Statements:
*Since day one, we have tested the USB port on every board shipped
*We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
*We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
*We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.
[...]

Hi, Andreas.

I don't understand why you have invested so much time and effort debugging a problem that only affects a small number of customers when you could have resolved the matter by issuing RMA's for the correspondingly small number of faulty boards.

I requested an RMA for my 4@SKU101020 cards, but then decided to continue trying to identify where the USB problem was because other people seem to be having similar problems. Is it possible that these problems are confined to a particular SKU?

Bye,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 150
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby Morgaine » Mon Aug 18, 2014 3:53 pm

aolofsson wrote:The issue:
The USB phy does not always get initialized properly coming out of reset on some boards. After successful initialization, the USB port works fine.

Statements:
* Since day one, we have tested the USB port on every board shipped
* We have shipped 10,000 Parallella boards to 7,000 customers to date and have only received issue reports from a small number of customers
* We have spent significant time debugging this problem (internal to Adapteva and external) and unfortunately we have not found an acceptable bullet proof workaround yet
* We will keep working on resolving this issue, but given our limited resources and other high priority items that need to be dealt with, we can't give a time line for when this will be resolved.


It's good to see direct recognition that the problem exists on some boards, since that's the first step towards fixing it.

Some aspects of this situation are extremely puzzling though. For example, how can all 4 of ajtravis's boards suffer from the same problem? This can't be attributed to natural variability in individual units unless a whole production batch is faulty and ajtravis was unlucky enough to have received all boards from that batch.

Also, while the official Statements recognize that a USB initialization problem does manifest itself on some boards, they don't actually say whether the cause of this problem has been identified yet. While it's good to know that a workaround is being sought, the more important step is first identifying the cause, otherwise workarounds are just guesswork. Has the cause of unreliable USB initialization been determined yet?

Morgaine.
Morgaine
 
Posts: 42
Joined: Tue Jul 02, 2013 8:29 pm

Re: USB Issue Troubleshooting

Postby theover » Mon Aug 18, 2014 8:05 pm

Hi,

In my case of the 7010 board without mounted U13 chip, usb regularly fails at startup, maybe 50 % chance, i don't know.

If I reset with the button, USB always works, but of course this can upset the OS.

I've made a de-bounced startup circuit with the supplied wall wart, which seems to increase the chance of a startup with USB.

I don't think HDMI or not made a difference, I used the latest Linux and FPGA of a couple of months ago.

I've used various USB units, a long extension cable improved startup chance (but not near 100%), and conrad switchable hub.

T.
theover
 
Posts: 174
Joined: Mon Dec 17, 2012 4:50 pm

Re: USB Issue Troubleshooting

Postby ajtravis » Tue Aug 19, 2014 12:26 am

aolofsson wrote:[...]
How to report a USB problem:
* Failure mode (sometimes or always?)

Sometimes: 10 runs booting node1-4 of my 4@A101020 Parallella cluster (0 = USB phy failed to initialise, 1 = initialised)
Code: Select all
run   node1   node2   node3   node4   
1   0   0   0   0   
2   1   0   0   1   
3   1   0   1   0   
4   0   0   0   1   
5   0   0   0   0   
6   0   0   1   1   
7   0   0   1   1   
8   0   0   1   1   
9   0   0   1   1   
10   0   0   0   1   
               
total   2   0   5   7   35%

* Headless/hdmi configuration

Default HDMI config from http://www.parallella.org/create-sdcard/
* FPGA/Linux/Ubuntu version being used

kernel-hdmi-140603.tgz
parallella_e16_hdmi_gpiose_7010.bit.bin
* Fan used?

80mm 12V fan run at 5V
* SKU#

4@A101020 (node1-4)
* For SKU A101040 boards owners, an indication whether the U13 device is populated

N/A
* Power supply used

Adapteva supplied 5V/2.5A Jameco PSU with US->UK 240V adapter
* Recommended: A picture of your USB setup (as shown here)

IMG_2278.JPG
IMG_2278.JPG (211.72 KiB) Viewed 11003 times

IMG_2276.JPG
IMG_2276.JPG (240.87 KiB) Viewed 11003 times


Hi, Andreas.

In all cases, I could ssh into the nodes and run "lsusb" as well as test the wireless keyboard and mouse.
This is the "dmesg" output when the USB phy fails to initialise
USB-phy-missing.log
(12.26 KiB) Downloaded 283 times
. When the USB phy was detected I got this from "lsusb":
Code: Select all
Bus 001 Device 002: ID 046d:c52b Logitech, Inc. Unifying Receiver
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


Please let me know if you need any other information?

Bye,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 150
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby aolofsson » Tue Aug 19, 2014 12:42 am

Tony,

Thanks for the detailed feedback. I noticed in the picture that only one board had USB/HDMI connected. Under what conditions were the 10 tries done for board 1-4? Were all of them connected to USB dongles at power on? Your results fit reasonably well with our data if board 4 is the top one and you only had that USB cable connected. If this was not the case, then we are back to head scratching...

Andreas
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: USB Issue Troubleshooting

Postby ajtravis » Tue Aug 19, 2014 1:26 am

aolofsson wrote:Tony,

Thanks for the detailed feedback. I noticed in the picture that only one board had USB/HDMI connected. Under what conditions were the 10 tries done for board 1-4? Were all of them connected to USB dongles at power on? Your results fit reasonably well with our data if board 4 is the top one and you only had that USB cable connected. If this was not the case, then we are back to head scratching...

Hi, Andreas.

I tested one node at a time: I moved the uSD card, HDMI and USB cable to the next node after completing 10 runs on each node. The USB OTG cable and wireless dongle were attached before powering up the node. All the inactive nodes were powered off during my tests. The top board is node1 and the bottom board is node4.

Although these are the same nodes that I reported on in my previous tests http://forums.parallella.org/viewtopic.php?f=50&t=1483#p10180 I made a new default HDMI config uSD card and used it to re-test all the nodes. In my previous tests, I used a Maplin 5V/8.4A USB charger and the rear USB connector to power the boards, but this time I used the Adapteva-supplied PSU plugged into the barrel power connector and changed the PSU option link which had previously been set for USB power.

The boards work well, except for the problem initialising the USB. I've not had any NIC problems and I was able to demo "dsh" and MPI etc. without any problem. I've run the Epiphany demos on node1 OK, but I've not tested Epiphany on node2-4 yet.

Bye,

Tony.
Minke Informatics Limited, Registered in Scotland - Company No. SC419028
User avatar
ajtravis
 
Posts: 150
Joined: Fri Jul 18, 2014 12:54 pm
Location: Scotland (UK)

Re: USB Issue Troubleshooting

Postby RomanLudwig » Tue Aug 19, 2014 10:23 am

aolofsson wrote:* Failure mode (sometimes or always?)

Sometimes

aolofsson wrote:* Headless/hdmi configuration

hdmi configuration

aolofsson wrote:* FPGA/Linux/Ubuntu version being used

ubuntu-14.04-140611.img

aolofsson wrote:* Fan used?

5V 0.09A 60 mm Fan

aolofsson wrote:* SKU#

SKU P1602-DK02 SN 0011927

aolofsson wrote:* Power supply used

5V 2.0A FAIRWAY PSU

Hi Andreas,

i have more than 50% failures for the USB port on startup. As my wireless keyboard/mouse set has no numlock LED, it turned out to be helpful to plug in a Belkin Numeric Keypad at boot time. If the USB port starts up successfully the numlock LED of the keypad turns off after less than a second (and turns on again a couple of seconds later). On USB port failure the numlock LED shines constantly. So i can decide after less than a second whether to recycle power to try again. Once the numlock LED turned off, i can successfully change to my wireless dongle at any phase of the boot proccess.


Greetings

Roman
RomanLudwig
 
Posts: 1
Joined: Tue Aug 19, 2014 9:42 am

Re: USB Issue Troubleshooting

Postby Sean S » Tue Aug 19, 2014 4:02 pm

Hi Andreas

Can you please clarify "only received issue reports from a small number of customers"? Through what means are you referring to for receipt of issue reports and do you think this is a realistic reflection of the number of customers that are experiencing USB problems?

If you consider the number of view on the subject of "USB not working" which currently stands at "Views: 4882" then one of two things are possible

- either the same people have nothing better to do than view this thread over and over desperately hoping for a solution...
- or many people are having issues with USB.

You claim to have shipped 10 000 boards to 7 000 customers... Which would mean that roughly half of the people that have boards have viewed this thread - implying that they have issues with USB too. That in my head does not account for a "small number of customers".

It has been suggested that I RMA my board but being in the UK, I can't justify the cost of shipping at my expense. In addition as the board was a gift, I am being told that I cannot RMA the board but need to hassle someone else to go through the RMA process which I don't think it fair. So please don't make that suggestion to me.

I am happy to see that after all of noise, you have finally made a formal statement. However, this needs to be followed up with a full solution - as you said. And I think you need to offer people their money back as well as an option. Many people require USB to be stable.

Honestly, the Parallella concept is great. Just a pity that such a key feature has no resolution or understanding after such a long time.

Sean
Sean S
 
Posts: 30
Joined: Mon May 19, 2014 5:30 pm

Re: USB Issue Troubleshooting

Postby AndyC » Wed Aug 20, 2014 12:33 pm

* Failure mode (sometimes or always?)

75% failure


* Headless/hdmi configuration

hdmi


* FPGA/Linux/Ubuntu version being used

Standard ones from http://www.parallella.org/create-sdcard/


* Fan used?

5V60mm - avg temp 55


* SKU#

P1602-DK02


* Power supply used

4A 5V switched
AndyC
 
Posts: 184
Joined: Fri Jun 27, 2014 5:46 pm

Next

Return to Troubleshooting

Who is online

Users browsing this forum: No registered users and 4 guests