DMA crash ?

[ysapir]

OK, I finally was able to download and test the latest version of your program. I run on the E64 board and do see the "Write failure" messages, occurring in random frequency at each run. What's interesting that it always appear at the 1st iteration of the 500-iteration loop. I wonder what's so special about this 1st iteration. The only explanation I can think of is that some caching and buffering takes place in the process, and somehow the situation is "cleared" when further transactions are generated to the places previously visited.

I also do not think this is a "write error". I actually suspect it is "read errors". Write errors would destroy the program's image, and it looks like programs run perfectly.

Certainly, there's some more diagnostics work required here. I will have to look for ways to programmatically make sure that no caching or buffering is done on the way from the host to the device and back.

[tnt]

I don't see why you think the problem are read. The issue only shows up while DMA is active AFAIK and so the initial write of the software image wouldn't be affected.

I'll try to somehow write a test program that confirms that all reads are going through but not all writes.

[ysapir]

Right now it seems to be neither write or read, but rather a problem of transactions priorities, like I mention earlier. As you said, after gradually activating all the program "features" the point where I started seeing failures is when the "1" was written to "active". Otherwise, there are no failures.

Running in a loop (from the shell), I got to a point where the system crashed. This is a symptom of a host read transaction (like the one you do when reading-back the i*j values) that is stuck somewhere and never returns from the Epiphnay system. The system bus/CPU seems to wait for that transaction forever, and all system hang-up, requiring a cold-restart.

This makes sense considering that the DMA is being run in a tight back-to-back loop, as long as the "active" is asserted. Occasionally, an e_read() transaction manages to slip in, when there is an occasional gap between DMA transactions.

That said, It still does not explain why the failures occur only in the 1st "i" iterations. Still on the debug table.

[theover]

If this isn't a single board but general problem, isn't that rather serious, I mean are all chips faulty in the sense of certain Direct Memory Access behavior, and how important is that (I mean is this an extreme case, from the description it would seem off chip DMA somehow is suspect and should be replaced by memory r/w, or am I not following this right).

Hope it solves!

Theo V.

[tnt]

[ysapir]

[Post edited to better reflect the situation]

tnt is right, no need to panic

After some testing and debug we concluded that there may be a problem with the FPGA logic. We are still looking for the root cause.

In certain scenarios, e-write transactions from the host, targeted at the eCore's memory, get lost on the way, when there is an external DMA process running from the device.

For now, please use a work protocol a-la matmul demo, where communication from host to chip, when external DMA is involved, is done via mailboxes in the external memory. AFAICT, there is no problem working this way.

[tnt]

I see the update on KS mentionned some improvement in the FPGA logic. Anything that could have caused the issue here ?

[notzed]

I know this is an ancient thread (and i posted about it elsewhere on the rev0), but is this all fixed now? I seem to be seeing exactly the behaviour described with my software rasteriser.

I'm using the headful 7010 image, which I downloaded on 25th of july with the supplied plugpack, rev1.1 board i believe.

I seem to be seeing writes changed to writes of 0; well i'm definitely seeing that, i don't know if that's the only problem. It is much worse when the cores are busy accessing external ram (via dma or not) which would match the description above; i.e. if i add a printf in the loop which does the writing.

Changing the order of the writes around a bit and using fewer cores "fixes" it, but only because everything is running synchronous, but it's basically the same problem I hit before with the rev0 board.

[aolofsson]

notzed,
Thanks for reporting this! This is pretty bad news...thought this was solved in the latest FPGA stream. As soon as we get through the current SDK release, getting tests written and nailing down the root cause for this becomes 1st priority item.
Andreas