Using Parallella in a Cluster is not clear

[kriegalex]

Hi,

Recently, I've been told that I would work with 2 Parallella boards that my school ordered. I'm working at 50% there aside of my studies. One of the first things my chief told me was that I should search how it works, how to run a simple program and then how to use the two of them to split even more a task (let's say a dotproduct).

The first part was not that hard, even if I had sometimes to modify your Makefiles because it didn't work (if I remember correctly for xtemp the makefile doesn't work). But for the rest, there are lot's of examples, we have access to the SDK documentation, it's nice. I already tested the Parallella with a matmult against a CPU and a GPU.

But for the second part, it's really (really) not clear how we are supposed to use multiple cards. Maybe I'm blind but right now the only things I found searching around is that I can order the parts "BSH-030-01-F-D-A" from samtec and then know by myself how to go from there (build the cable, connect and run a program). The only way I see is using the ethernet port and do something via the network, but it's not nice because the board has the 4 samtec connectors.

Maybe you should add something to the samples, in the documentation, here on the forum but at least somewhere :

- you can do that
- you cannot do that (never or only not right now)

The best thing would be a tutorial with all the things needed and a sample program with 2 cards and a basic operation like dotproduct or matmult.

I'm asking that because one of the major advantages of this board is that you can buy one or two at the beginning, and then keep adding them to improve your computing power while managing easily your budget and 40 boards only draw 80W of power, which is the TDP of a standard Xeon CPU. 40 boards would be nice to benchmark against a 80W Xeon CPU .

Best regards

P.S. : If it already exists, I'm very sorry that I didn't find it

[piotr5]

since you're told to use only 2 boards, I believe network connection is what you should use. I too would like to see instructions on how to build an e-link cable, and how to talk to another epiphany that way. still, through a network cable you get to talk to another host directly, so everything works quite symmetrically on the two. would you use eLink cable, the other host wouldn't even know another parallella is connected, you simply cannot start a program on host from within epiphany chip as far as I know. and if you need the additional memory, you need cooperation of that host-cpu. of course it is possible to change all that. but this is beyond the tasks you're told to perform. maybe some linux-kernel hacking to get host programs started from epiphany? maybe some fpga-hacking to get more direct access to main cpu or its memory?

as for xeon, the best I've seen is inel cpu with about 20 cores (40 threads?) for a few thousand dollars. also I think there are main-boards with multiple cpu-slots. all this technology seems quite old to me. pay the same money for parallellas and you have more cores, although then the bandwidth among them isn't as good. x86 mainboards have a few dozens GB/s connection to memory. even faster is connection between cores on the same cpu. epiphany is limited to only 2 GB/s per connection to memory, on parallella with eLink cables you could theoretically go up to 6GB/s. and epiphany is under-clocked (hopefully to a speed where heat/GHz is the lowest) which lowers that speed even more. therefore xeon has something parallella isn't capable of, no matter how many parallella you connect. a bit more recent technology is newest amd a-series, there you get 2-4 main cores and, as I understood, 12-32 64KB-cache vector-cores (simd-16) which now can communicate in the same way as the cores on xeon could. in terms of price this is quite similar to 2-3 parallella boards, except at a much higher speed per core and lots of energy-hunger in the main-board (the amd-cpu is only 45W TDP while the whole system usually will draw an additional 50W). I too dream of connecting 40 parallellas through elink and network, but apart from the work that's needed to accomplish that, wouldn't it be cheaper and quicker anyway to start working with amd A10Pro-7800B with its 32 x 16 x 3.5Ghz simd on 35W, if it ever will be available? of course, if there will be 1024-core parallella, at reasonable price and energy in a few years, that becomes a moot question...

[sebraa]

It's quite simple:

The Epiphany chip on the Parallella has 4 eLink interfaces. One of them is connected to the Zynq and the rest of the Parallella board. Two others are available on the connectors, if you want to interface them directly to another Epiphany chip. In this case, two 16-core Epiphany chips would appear as one 32-core Epiphany system.

Connecting two Parallella systems together is easiest done through the network interfaces. Then, you have two Linux systems containing an Epiphany chip each. This is the only solution which doesn't require special hardware and tinkering.

[9600]

[kriegalex]

Hi,

While being around on the forum, I found this , which is quite what looks like the temporary answer to my question until the official elink cables are out (or whatever will be used to connect and stack cards).

Best regards