Broadcast and Multicast on Epiphany

[Farzaneh]

Dear Forum members

I am trying to implement broadcasting strategies between all cores on the epiphany. I had checked the forum topics regarding available library functions, but they mostly focus on small variable broadcasting (e.g. integer). My question is whether there is an available library to broadcast arrays between cores or not. If not how I can go through this implementation. My second goal is to implement multicast on the epiphany. I would be grateful if anybody could give me some pointers about multicasting as well.

Any help appreciated.
Thanks in advance
Farzaneh

[jar]

Yes there is! And it's new so you may have missed it.

The ARL OpenSHMEM for Epiphany implementation implements a vast number of common collective, atomic, reduction, and memory copying routines which have been optimized for the Epiphany architecture. You may download and use the source code here: https://github.com/USArmyResearchLab/openshmem-epiphany

OpenSHMEM is a standard interface for PGAS architectures and is ideal for Epiphany. I think people will like it and, in my opinion, should be the programming model of choice for the Epiphany architecture.

You may also want to refer to the publication of the work and the and the OpenSHMEM 1.3 standard documentation:

[jar]

You also asked about tips for multicast that I forgot to address. There is a pretty good example of it in ./parallella-examples/multicast, but you probably don't want to use it. Because of hardware errata, only the first four cores (first row) can perform the multicast (although all receive). Even if that's okay, you're only moving a single 32-bit value. Your code won't be portable. If you're fine with that, look at the example code.

[Farzaneh]

Thank you so much for your reply.

I had a quick look to OpenShMeM as well as multicast sample in Parallella, but what I am driving at is a bit different. I would like to fetch the same data from main memory to multiple thread's local SPM. Some of these data are brought in by all threads while others are brought in by few threads. If each thread brings in data independently, there can be NoC contention leading to higher memory access latency. However, with hardware support, this can be mitigated.

Please do advice if (bringing data by Core 0 using DMA + broadcast using OpenShMem library) would be better than (each core bringing data independently). Additionally, would it be possible to just send the data to a subset of cores? as I could not find any OpenSHMem examples.

Thanks in advance
Farzaneh

[jar]

If you want each core to have a copy of symmetric data from main memory, the fastest method is is to DMA into core 0 (or any core) and broadcast using the OpenSHMEM interface. The on-chip network is much faster at distributing data than the off-chip memory interface. Plus, you have the advantage of parallelism!...

Use the shmem_broadcast32 or shmem_broadcast64 routines for this operation. It uses a logarithmic scheme for speed, for example, if core 0 broadcasts to all 16 cores:

stage 0:
core 0 -> core 8

stage 1:
core 0 -> core 4
core 8 -> core 12

stage 2:
core 0 -> core 2
core 4 -> core 6
core 8 -> core 10
core 12 -> core 14

stage 3:
core 0 -> core 1
core 2 -> core 3
core 4 -> core 5
core 6 -> core 7
core 8 -> core 9
core 10 -> core 11
core 12 -> core 13
core 14 -> core 15

The operation is completed in the time it takes core 0 to send data to just four cores, not all 16.

[Farzaneh]

Thanks for your prompt reply. Apologize in advance for my long question:

I am trying to implement multicast using OpenSHMem. I wrote a simple program in which every two cores need the same data. As an example, we have below subset who share the same data. Therefore,
core0 fetches data from main memory and broadcast to core6. core1 fetches from main memory, broadcasting to core6 and so on.

core0 - core 6
core 1- core 7
core 2 - core 8
core 3- core 9
core 4- core 10
core 5 - core 11

Q1:
Is it possible to use shmem_broadcast64 to achieve multicast on epiphany?

Q2: If yes
What should be the sixth argument in the following code:
my_base = (id) %6;
shmem_barrier_all();
shmem_broadcast64(dest, source, NUM_ELEMS, 0, my_base, (???), 2, pSync);

Do we also need to modify PSync?

Thanks
Farzaneh

[jar]

[Farzaneh]

[jar]

Some things in the code changed recently, but that sounds like a bug. I'll look into it. It would help me if you could file an issue here with a simple test code to replicate the behavior: https://github.com/USArmyResearchLab/op ... any/issues

Thanks.

[Farzaneh]

Thank you so much.