Inter board communication with Epiphany

[krmld]

[peteasa]

I am planning to try this out, once my Porcupine boards plus my second Parallella arrive!
Read the architecture reference http://www.adapteva.com/docs/epiphany_arch_ref.pdf and see that the maximum size grid is 32 x 32 cores or 8 x 8 Epiphany chips.
Now to get the inter board elink working well you are likely to want to get data on demand from the epiphany chip to the CPU. At the moment that is just a polled interface so the Arm cores can poll a message buffer over elink into the Epiphany chip. But soon, if not already, you should be able to use https://github.com/parallella/oh/tree/master/emailbox that will interrupt the CPU from the Epiphany side.
Also because the Parallella CPU uses one of the four elink connections and the Parallella board does not have a connection to the second elink connection (http://www.parallella.org/docs/parallella_schematic.pdf) you will be limited to 8 Parallella boards in a row... but wait there is an FPGA on the board with a lot of spare gpio pins... so it is not beyond the wit of man to connect the CPU elink connection to a home grown elink like connection (perhaps with for example 4 bit width rather than 8 bit width) from the FPGA so with a bit of HDL work you might be able to get 16 Parallella boards connected up...
You are also likely to need to do a bit of Linux kernel driver work to get the whole thing up and running.

[peteasa]

[aolofsson]

Sorry for the slow reply! I am very impressed by your bravery in tackling some of the FPGA and multiboard issues/

The registers touched by the sdk can be found here:
http://www.adapteva.com/docs/e16g301_datasheet.pdf

Notes:
-You will not be able to read a remote Epiphany from each ARM (ie ARM on P#1 can't access Epiphanu on P#2 in your picture). (the return transaction will get lost due to the preference for east/west in NOC)
-You should be able to do a write from ARM on P#1 to Epiphany on P#2 (but that is not very interesting?)
-You should reduce the frequency of the link (300MHz is too fast for most cables), this is done by writing to the IO register
-You will need to enable the North/South links by writing to the proper IO registers (see SDK example)
-run with the internal.ldf to simplify things further as a start, otherwise you would need to compile different programs for each board

Hope this helps?

[peteasa]

Excellent! Thanks for the tips.

I have been modifying / extending sdk (int e_reset_system(void) --> int e_reset_connected_system(int syscfgrow, int txclkmode, int cclkdivider)) to allow prevention of disabling the North / South elink on the appropriate board and modifying the C-clock and the Tx clock mode for the eMesh. No luck so far but now I have some pointers for other places to read.

One interesting thing was that I thought I would be able to read ESYSCOREID and get the configured Core ID row and column. However this always seemed to show row 32 column 8 when I read it from within e_reset_system().. The modified hello-world application correctly prints out the two different sets of core id's so I know that my wire links on the epiphany chip and the hdf files are correct.

[aolofsson]

The ESYSCOREID is in the FPGA link and is hard coded and not affected by the strap pins. (this is why the printf works)

Remind me which elink code you are using (pointer?)

[peteasa]

Ok I missed the ESYSCOREID hard coding bit.. thanks.

A second interesting thing is that the table in the referenced epiphany_arch_ref appears to have a typo. You can see there is a problem because according to the table when the Address-Row Tag Matches the Mesh-Node Column(!) and the Address-Column Tag matches the Mesh-Node Row(!) the routing direction is Into Mesh Node. Spot the typo where Row and Column appear to have been transposed in the table heading. The text in the paragraph 5.2 Routing Protocol has a similar typo because the text states that the transaction is routed East if the destination- address column tag is less than the column ID of the current router node. This implies that moving East will decrease the node column id. However the Figure 6 and Figure 8 shows moving East will increase the node column id. Assuming that these two observations are indeed typo's I do not understand why the return transaction will get lost because a message from one Parallella board will move to the required destination column then pass down the south connector to the north connector of the Parallella with the higher row numbers and the reply will pass back to the requires source column and then back up out of the north connector into the south connector of the originating Parallella board.

I am using elink - redesign fpga (https://github.com/parallella/parallell ... o/releases) and am using https://github.com/adapteva/epiphany-li ... af2ee5191c .. ie just before the fpga registers were changed.. It appears to work with the fpga that I have. I also plan to update soon to the latest of everything...

[aolofsson]

Thanks for highlighting the typo. Clearly a mistake, can't believe nobody noticed (or they did and they told me and I forgot to fix it...)

Attaching pictures that hopefully shows why it won't work. The arrows show the routing path followed.
The algorithm for routing is "go along row first, when you get a column match, move up/down a column. When row and column match the destination address you are home.
Same algorithm for read requests, writes, and read-return data.

drawing.png (23.72 KiB) Viewed 38090 times

[peteasa]

[aolofsson]

Fantastic! I really enjoying hearing about your progress. We have tested some of this stuff before and Bittware even had a 4 chip product on the market for a while, but still....it's always great to get more validation that this stuff actually works! btw. how are you hooking up the two boards together. Did you make your own cables?