Two-Parallella Cluster using eLink and Procupine

[claudio4parallella]

Hi all,

just to get in touch with everybody interested of the present topic.

I've realized a MPMD set of programs in order to manage on a single Parallella Board the 16 Cores, in a one by one way with input data and output data and synchronizing, starting and stopping all of them individually.
The same model of MPMD I've realized using two Parallella in Cluster using eLink and Porcupine boards via flat cable.
I can individually start, stop, get input and get output to and from each one of the 32 Cores.

The difference is :

- using a single Parallella Board, it's the Host program that may and must manage the 16 Cores directly, having direct access to all the RAMs, inside Epiphany and so within each Core range of memory address, too ;

- using a two Parallella Cluster (Master-Slave), it's one of the local Core's program of the Master Parallella that is in charge as Master-Core of driving all the 32 Cores (16 of them local and 16 of them on the eLinked board) having direct access to both the Epiphany Cores and RAM absolute address (0x908xxxxxxx and 0x808xxxxxxxxx).
The Host program on the Master Parallella could not have access to the Epiphany/Cores of the eLinked Parallella.
Only the Loader could have access to load the individual .elf (or .srec) executable program to the 32 Cores, even if on the Slave the set-up and running of the Epiphany must be ready to receive the Master commands. (I'm using the Interrupts-connected example of Parallel System Lab - Aviv Burshtein).

I'm available to show codes and examples of both my experiments, if necessary.

Thanks for any comment or different considerations from what I've understood.

My best to all of you

[jar]

A few people have struggled through what you have done over the past couple years. It would benefit the community to document the steps you took to configure the boards. My understanding is that you had to use the 2015.1 image?

One feature of COPRTHR-2 is having Epiphany core #0 initiating the work across all other epiphany cores. It actually launches programs in logarithmic time as the configured cores then configure and launch on the remaining cores.

[claudio4parallella]

[jar]

You can always replicate logarithmic time launcher in COPRTHR-2 if you want. It was documented in the paper "Advances in Run-Time Performance and Interoperability for the Adapteva Epiphany Coprocessor": https://arxiv.org/abs/1604.04207

COPRTHR-2 almost certainly won't do it automatically for your dual-Parallella setup. There are hard-coded base addresses as it's configured for the single E3 on a Parallella.

[claudio4parallella]

[jar]

This is a high quality writeup. Good job!

[claudio4parallella]

UPDATE

- I've closed my open issue about interrupt, now tests work properly and I can manage them

- About my 2Parallells eLink cluster, if I have well understood: the SLAVE Parallela must run its Host and its Device core program and without this running the MASTER Parallella could not load the tasks to the SLAVE Epiphany cores..... (hope my English is clear...). No way to upload the device tasks to the SLAVE Epiphany from the MASTER Host....

Thanks for comments

[claudio4parallella]

Here are my terminal screen of the eLink 2-Parallellas/Porcupine

SLAVE showing in a loop the local 16 Cores and Registers



MASTER showing in a loop the 32 Cores output, the total of output, the core-done counters



The 32 Cores are activated in sequence: core00 when reach 10000000 activate core01 and so on...... up to core73

[claudio4parallella]

Proof of concept results of two eLinked Parallellas in cluster (Master-Slave)

Purpose: realize synchronization between multicores in eLink cluster with individual tasks

Lesson learned:
- core 0 of MASTER parallella must be used to read/write/synchronize all the cores in Master Epiphany and Slave Epiphany
- internal.ldf works ok, not other choice

Attachments of resulting images:
1. example of a single Parallella: the 16 cores individually are drawing a 1024 step of Y=sin(alpha) [0..2Pi]
2. example of a dual eLinked Parallellas: the 31 cores individually are drawing a 1024 step of Y=sin(2*alpha) [0..2Pi]

Next step: 4 eLinked Parallellas and 64 cores

[claudio4parallella]

Wow! I found a Zynk7020 Parallella !
In the meantime the new board will arrive....
and my eLink cluster will become a full 64 cores machine
here attached is the 48 cores single kernel-by-core Proof of Concept