Inter-process communication on the Parallella.

[Dr.BeauWebber]

OK, given a set of processor cores each running APL, how do we communicate between them ?

(This is a continuation of the discussion begun in : Why port array languages such as Apl to Parallella ? )

A conventional APL system will offer a range of communication processes, now usually including object-oriented methods of accessing .net, Java, R, etc. For AplX see http://www.microapl.co.uk/apl/aplxv5.html We need to find a set of communication methods that compactly gives us rapid communication between processes running on the cores of a Parallellla.

1) Well the methods that were built into aplc a number of years ago are based on piping :
a) using two pipes in for data and control, two pipes out for result and status : Basically based on std-in, std-out, std-err
b) piping data out of APL through a pipeline of shell commands and back into APL. A new mechanism []Pipe.
c) spawning a new long-lived process and connecting as many pipes to it as needed. A new mechanism []Spawn.
See the aplc.doc : http://home.earthlink.net/~swsirlin/aplc_doc.pdf
http://www.academia.edu/1166773/A_Pipe_has_two_ends._Using_APL_in_a_multiprocess_multiprocessor_environment._A_proposal_for_a_flexible_but_easy_to_use_syntax

2) Another possible mechanism is a global APL variable handling program, that local APLs can make read and write request to. We wish to preserve the rigorous APL way of knowing the full specification of the data : its size, shape and type, and this information is kept by the global program.

3) Alternatively it is possible to just share a variable such that a number of processes share the data but the data is kept by the process with the most recent data used to write the variable. Any particular read then accesses the data (and its type and structure information) from this, so a data read is then a local to local process transfer over the global interconnect. This still needs to be 'policed' by a process with a global overview.

Techniques 2 and 3 are similar to the []SVC Shared Variable Control in AplX.

Your thoughts on these and other possible mechanisms are most welcome.
If you have the knowledge / ability / time to implement any of these using the local/shared memory system on the Parallella, that would be most welcome.

[timpart]

I expect you have been reading the Architecture manual already but here are some thoughts.

Reads are much slower than writes. A read goes on a slow link to the destination core then the data are returned by a write back to the originator. If you want to transfer data quickly have one core write it to another's memory, not by a core reading another's memory.

The DMA engines are good at transferring data around memory while minimising the impact of the CPU to do other things. (Best if DMA access is to 4K block of RAM not currently being used by CPU if possible.) There is a small overhead in setting up the DMA so it is not suited to tiny amounts of data.

Pipes naturally lend themselves to writes by the originator which is convenient. I'd outline a pipe mechanism as follows.

Each core that can be a destination for a pipe should maintain a list of available RAM areas in its local memory. It should use a mutex variable present in its local memory while maintaining the data structures describing the list. The mutex will be quick to access as it's local.

When a core wants to pipe to another core it acquires a mutex on the destination core's free area list and acquires a suitable region of RAM updates list, releases mutex.

Sending core sets up DMA to read data from its local memory and write to destination core's memory. DMA is started.

Sending core does other things. Destination core carries on obliviously.

A while later sending core checks DMA and discovers it is complete. Sending core notifies destination core that the data are available (by another mutex and work to do list probably)

Destination core looks for more work and notices sending core's notification in its work list.

Tim

[Dr.BeauWebber]

Tim this is great.
Yes I had noticed that the DMA channels looked like important channels, but had got no further.
I will be brutally honest and say that I have hopes that someone or someones (as knowledgeable as yourself), will take it upon themselves to build pipe mechanisms. Whether tried and tested means, or more novel means particularly appropriate to these processors, will be the way forward, I do not know.
I will have quite a bit of my time taken in just getting the Aplc basically up and running to the best in such a shared memory context, where the size, shape and type of the data is preserved in the transfer. But pipe data can (initially anyway) be plain text streams of arbitrary length.
But certainly the first things that are needed are simple standard in and standard out pipe communication to the host processor.
I am trying to hack a simple fixed buffer substitute, but I am sure someone who reads this has better tools in their kit-box.
cheers,
Beau

[jeinstei]

I haven't had a chance to look real in depth yet on the Parallella, but are hardware mutexs available? A mutex per core would make any sort of inter-core communication much easier, as any internal or external resource could "lock" a core for use. I could imagine this leading to issues if a core got frozen, but have a watchdog per core that would auto reset could probably get around this problem.

It is also a question of how DMA or a process end on another core would grab the mutex if a different core already has the lock. If multiple cores are all vying for access to the same process, this could likely get very interesting, which shows the need for the high-speed FPGA to manage these things. Maybe support a table of mutexes in the FPGA?

I feel like I'm missing something that is probably in the hardware design already though.

[timpart]

Mutexes can be done by the TESTSET instruction on the Epiphany chip. The memory location must be on same chip as the core executing it. Can't be on external DRAM. Haven't seen anything to suggest the FPGA can do this to Epiphany internal RAM so would assume for time being not possible..

Tim

[ysapir]

Mutexes can be done by the TESTSET instruction on the Epiphany chip. The memory location must be on same chip as the core executing it. Can't be on external DRAM. Haven't seen anything to suggest the FPGA can do this to Epiphany internal RAM so would assume for time being not possible..

Tim

Note that the mutex operations are encapsulated in the mutex e-lib family of functions.

You are right in that the mutex location has to be on-chip, and is accessible (for now) only from an eCore.

[timpart]

Yes @ysapir should have mentioned the mutex functions, sorry was in a hurry and wanted to answer "is there hardware support" and convey that my answer was about Epiphany, not ARM which can also do hardware mutex but differently. (Can't remember how right now - it's changed since I first came across the chip back in the days when the A stood for Acorn.)

Would be good if one day we could get a mutex across both chips somehow. At the moment needs careful keeping out of each others way in the shared external DRAM unless I've missed something.

Tim

[Dr.BeauWebber]

OK,
I have been trying to create a very simple output communication process for Aplc using a buffer in external memory, shared with the host Arm processor. This is based on the esdk.4.13.04.24_linux_x86_64_armv7l.tgz hello_world.c example

Code: Select all

char outbuf[128] SECTION("shared_dram");
// char* pointer = outbuf ;
int counter = 0;
 

(I have tried it done various ways, sorry about the messy code, it is just for testing).

Now this works for a single message i.e. the hello world message, or

Code: Select all

char Word[] = "tEstAplctext";
  sprintf(outbuf,Word);
 counter = 13;

(or using a pointer).

However if I define a variable in Aplsc, the aplc workings tramp all over the buffer :

Code: Select all

 In aplc c code for source line 2
comment out
 aplc_format(&trs1, &V);
$ ./buildv5.sh
$ ./runv5.sh
  0: Message from eCore 0x8ca ( 3, 2): "tEstAplctext"
  1: Message from eCore 0x84b ( 1, 3): "tEstAplctext"
  2: Message from eCore 0x84b ( 1, 3): "tEstAplctext"
  3: Message from eCore 0x888 ( 2, 0): "tEstAplctext"
  4: Message from eCore 0x849 ( 1, 1): "tEstAplctext"
etc.

with
 aplc_format(&trs1, &V);
$ ./runv5.sh
  0: Message from eCore 0x8ca ( 3, 2): "tEst"
  1: Message from eCore 0x84b ( 1, 3): "tEst"
  2: Message from eCore 0x84b ( 1, 3): "tEst"
etc.

So I think I need a method to separate the buffer and the aplc workingspace,
so I can call sprintf repeatedly, and build up a page or so of text.
Can anyone tell me how to separate these ?

Currently I am calling the legacy.ldf and this also needs to be changed so the aplc program is in internal memory.
see http://forums.parallella.org/viewtopic.php?f=13&t=353

I can do no more until these are sorted.

[ysapir]

Please look at this post:

http://forums.parallella.org/viewtopic.php?f=13&t=353&p=2101#p2101

[Dr.BeauWebber]

Next stages :

I have been trying to create a very simple output communication process for Aplc using a buffer in external memory, shared with the host Arm processor]

The problems I was having, getting simple output this way, have, as you can see above, now been sorted.

What we now need to consider are more advanced methods (with proper handshake/synchronisation) of communicating between separate aplc processes on different cores, and between these and the host. processor.

Aplc has, if one wishes, the ability to declare the variable type.
While reviewing the documentation, I was reminded :
aplc_doc.pdf :

#global – a global variable; note that this is somewhat like the “extern” qualifier of c in that it implies
that the storage for this variable is given in some other place.

This offers a good way of having an overall process that specifies the correct sharing of global memory between the cores, and between cores and external memory, and the more localised memory for one particular core.