rev0; dropped/corrupt host writes under heavy core dma read

[notzed]

I'm starting to see some problems on my rev0 board which i haven't seen before - or just put down to bugs in my code. After finally getting my 'ezesdk' to a usable state i was playing with an fft routine.

Basically when multiple cores are busy with dma reading from external memory, writes from the host cpu to on-core memory are either going missing or getting corrupted (and corrupted 32-bit writes are being turned into 64-bit writes of junk too). This causes anything from the core just never receiving the go signal to hard system lockups.

Using memcpy instead of dma only makes it worse. More cores busy reading makes it worse.

I haven't really noticed issues with the inter-core stuff - so it seems it's probably something to do with the arm<>epiphany memory interface in the fpga code?

!stop press! If i remove the job processing and only do a dma write core->shared, then at least this "works" fine.

I was under the impression the new fpga images are incompatible with the rev0 board. Or is it just that the new kernels are? Is there anything I should or can try or am doing wrong?

update: tried another bit of code that tries dma + writing to a seq #, and that works. So maybe it's something else/just a bug.

[tnt]

Hi,

I noticed this issue like 1 year ago ...

viewtopic.php?f=23&t=223&start=20#p1786

That's a major reason why I stopped working on the epiphany for a while because I couldn't get data in/out the board without corrupting the control mailbox writes ...

I'm pretty disappointed that this hasn't been fixed yet.


Cheers,

Sylvain

[aolofsson]

Sylvain,
Please understand that you were both very early user of the gen0 board. We did fix an issue related to this board, but it did not make it into the bit stream shipped with the gen0 board. I believe this issue was fixed for the gen1 board. If not, then our testsuites and the 5,000 people who have received the gen1 board must have all missed it.(which would be shocking!) Could you try your code on a new gen1 board or should we just grab the tar ball from the forum post and try it?
Andreas

[tnt]

I do understand that this was early code, it was actually back on the zedboard at the time, not even the rev0. But this is the first I hear that you fixed the issue. If so, great ! I'll try to give it a shot on a production rev1 with the latest bitstream.

Is the AXI still at 40M btw ?

[psiegl]

I have a Parallella-I board and I wrote a small program which is not doing more than initializing all the 32 available DMA engines on my 16 core Epiphany with writes to the host DRAM. I start all of the DMAs in parallel after a barrier. What I see right now is that a bunch of (!!!!) stores will not make it to the DRAM. Even while using only 16 DMA engines it turns out to be the same problem (1 DMA engine per core).
I tried already several fancy techniques to synchronize the memory, but until now it looks like as if there is some hardware bug in the FPGA logic. Even writing directly from the cores via a memory (DRAM) reference doesn't always work. Especially not under heavy workload.

Until recently I was using the bitstream of rel.14.03.06-headless.tgz but somehow my board wasn't booting anymore (at least the ethernet connection wasn't showing up) so that I changed to rel.14.02.06.tgz where it is properly booting again, but I sadly detect the matmul-16 'BAD CORE' - bug there.

So right now it is pretty annoying because the Epiphany is quite fast, but sadly not useable because there is no way to read / write from / to the cores.

I've been cooling the Board with a 120x120mm FAN which is connected to the 5V power supply, but it looks like as if this is not sufficient. Some guys here are mentioning that the Epiphany needs at least a heatsink, otherwise it can't be used in a stable way. My intend is to buy one and I hope that this will solve the DMA problem, even if I somehow doubt that this is the issue right now.

Seriously: this is a very serious problem and needs to be adressed ... ASAP!

[aolofsson]

[aolofsson]

pseigl,

Thanks for the feedback!

-Can you help us out by publishing your exact source code for the test case. If this is indeed an issue, it's important that we an others on the forum can reproduce the error.

-Matmul-16 failing would suggest a bad chip, but you mentioned it working before?

If you found a real problem here, we could really use your help in getting some more details on your failing device (for everyone's sake)

Andreas

[psiegl]

Yes I'm interested in sharing my DMA code, but right now I try to bring up my parallella again.
First step will be to get it back running with the headless parallella.bit.bin so that I can evaluate if this makes a difference and afterwards I can shrink my code to a subset which could be evaluated by you guys.

EDIT: I managed to boot my Parallella-I Board with the headless bit file (rel.14.03.06-headless.tgz). I still see the problem "BAD CORE" occuring during the run of matmul-16, but at least my barriers as well as my DMA engines are now running fine. Let's see if I see any further problem ...

[notzed]

[tnt]

Nope, it can't work. Actually it could potentially damage it due to pin changes in the FPGA.