Connecting Multiple Parallella Boards

Any technical questions about the Epiphany chip and Parallella HW Platform.

Moderator: aolofsson

Connecting Multiple Parallella Boards

Postby aviv_ohad » Wed Apr 27, 2016 11:00 am

Hi All!

We have spent the last few months playing with the Parallella and trying to make several boards work together. We came across several problems that we could use some help solving.

At first we followed this post: https://parallella.org/forums/viewtopic ... r&start=10
with the old HDMI Build, and connected our boards this way: (link attached in case you cannot view the image)

Image
https://drive.google.com/file/d/0B5uBXg ... sp=sharing

Then we made our own new PCB connector:

Image
https://drive.google.com/file/d/0B5uBXg ... sp=sharing

The problems we came across are the following:

1. Hardwiring the core-id using a jumper does not seem to work for the headless builds it seems. When ever we try to run the Hello_World example after changing the hdf and xml file the board crashes and has to be rebooted.
Any clues as to why this happens or how to fix it?

2. We have been trying to use an interrupts based mailbox program to have communication between the boards up and running, and it only works with esdk.2015.1. When trying to use the new library with the HDMI build nothing seems to work at all. We read here : viewtopic.php?f=13&t=3073 that there are some missing features on the old FPGA build, but we would like to hear some more details about it.

3. changing core ID and Speed: As we plan to use our connector, we need to change these features without the comfortable pins that come with the Porcupine. We are able to write to the FPGA system Registers, but they don't seem to affect these two issues at the moment on both the headless and HDMI versions. Any ideas on how to tackle the problem,
or details on progress with the FPGA registers?

Thanks a lot!
aviv_ohad
 
Posts: 3
Joined: Sun Apr 03, 2016 7:52 am

Re: Connecting Multiple Parallella Boards

Postby aolofsson » Wed Apr 27, 2016 4:29 pm

aviv_ohad wrote:Then we made our own new PCB connector:
Image
https://drive.google.com/file/d/0B5uBXg ... sp=sharing


Very nice!!! Would love it if you could upload design files here to the repository of open source boards.
https://github.com/parallella/parallella-hw

aviv_ohad wrote:1. Hardwiring the core-id using a jumper does not seem to work for the headless builds it seems. When ever we try to run the Hello_World example after changing the hdf and xml file the board crashes and has to be rebooted. Any clues as to why this happens or how to fix it?


An example would help. What coordinates. What makes the board crash (read/write/ full app). Note that the FPGA bit stream will also likely have to change since the register and return address of the read transactions need to be correct to work. The FPGA registres should be "directly to the east" of the CHIP id coordiantes.

aviv_ohad wrote:2. We have been trying to use an interrupts based mailbox program to have communication between the boards up and running, and it only works with esdk.2015.1. When trying to use the new library with the HDMI build nothing seems to work at all. We read here : http://forums.parallella.org/viewtopic.php?f=13&t=3073 that there are some missing features on the old FPGA build, but we would like to hear some more details about it.

Note sure I understand? The old HDMI build is REALLY old and has a completely different driver, FPGA register map. It's impossible to mix and match. Safest best is to take the image as is.

aviv_ohad wrote:
3. changing core ID and Speed: As we plan to use our connector, we need to change these features without the comfortable pins that come with the Porcupine. We are able to write to the FPGA system Registers, but they don't seem to affect these two issues at the moment on both the headless and HDMI versions. Any ideas on how to tackle the problem, or details on progress with the FPGA registers?


What speed are your trying to change? RX/TX Epiphany clock?
We haven't tested implemented the software programmable chipid pins yet. Here are pointers showing how they would be hooked up (second file).

https://github.com/parallella/oh/blob/m ... lla_base.v
https://github.com/parallella/oh/blob/m ... tem_bd.tcl
User avatar
aolofsson
 
Posts: 1005
Joined: Tue Dec 11, 2012 6:59 pm
Location: Lexington, Massachusetts,USA

Re: Connecting Multiple Parallella Boards

Postby aviv_ohad » Sun May 01, 2016 4:18 pm

Thanks for the quick reply!

aolofsson wrote:Very nice!!! Would love it if you could upload design files here to the repository of open source boards.
https://github.com/parallella/parallella-hw


We will certainly upload our design files, with a decent explanation, once we manage to get our system up and running, and test it for errors.

aolofsson wrote:An example would help. What coordinates. What makes the board crash (read/write/ full app).


We have been using the ubuntu-14.04-headless-z7020-20150130.1 image file. First we reconfigured the hdf and xml files(to core id 908 for instance), then connect a jumper to match the core id on the hdf.
We then restart the system, compile and run the basic hello-world application. The board then crashes and never recovers- so we are forced to to format the sd card and write the image on to it again.
This prevented us from being able to debug the code properly and find out more details. Is there a known stability issue with the version we have been using? Is there a stable version you recommend using instead? if so, could you please point us to the vivado project files and the respective image file? We are now trying it with the newest available image file, and will let you know if there is any progress.

aolofsson wrote: Note that the FPGA bit stream will also likely have to change since the register and return address of the read transactions need to be correct to work. The FPGA registres should be "directly to the east" of the CHIP id coordiantes.


As we understand it, there should be no problem- the read reply should (according to Epiphany routing logic) be routed east first, and then intercepted by the zynq before it should have been routed north.
Here is a sketch to show what we mean:
https://www.dropbox.com/s/ddqelwyy4z9zg ... 0.jpg?dl=0

What are we missing? If we indeed must change FPGA registers address, could you point us to the files/modules that should be changed to do this?
Note that in the HDMI version this works like a charm...


aolofsson wrote: The old HDMI build is REALLY old and has a completely different driver, FPGA register map. It's impossible to mix and match. Safest best is to take the image as is.


Thanks for that, saved us a lot of precious time. Moved on to the headless version...

aolofsson wrote:What speed are your trying to change? RX/TX Epiphany clock?

Yes. Do you know how this is done?


aolofsson wrote:We haven't tested implemented the software programmable chipid pins yet. Here are pointers showing how they would be hooked up (second file).


Again, thanks for the precious time you saved us. Do you have any clue when this should be done?

some other thoughts and ideas:
1. Did you consider also implementing a module that would calculate the set the appropriate Zynq registers address in hardware? We think this might help users a lot...|
2. Are there plans to add sockets for eastbound and westbound off-chip traffic in future models?


Thanks again, hope to hear from you soon

Aviv and Ohad
aviv_ohad
 
Posts: 3
Joined: Sun Apr 03, 2016 7:52 am

Re: Connecting Multiple Parallella Boards

Postby peteasa » Sat May 07, 2016 6:31 pm

When I tried communicating between boards I had some strange effects but I never had to re-write the sd card! That seems unrelated to the epiphany connection.

I found that the highest clock speed for the 2015.1 fpga worked. If you want to try to change the speed look at https://github.com/adapteva/epiphany-li ... al.c#L1115 and https://github.com/adapteva/epiphany-li ... al.c#L1133 and https://github.com/adapteva/epiphany-li ... al.c#L1139. These are the places where the txcfg and rxcfg registers are setup for the full speed elink.

My advice would be to first get this working with the 2015.1 fpga and matching epiphany sdk before attempting with the latest fpga and software. In my testing viewtopic.php?f=23&t=3207&hilit=jumper#p15984 the North Parallella could load each of the 32 cores in the large workgroup. So I used this to prove that the South Parallella could write to the North Parallella. The epiphany code makes each of the 32 cores write its Id to all the other cores in the group. The North Parallella could then read the North cores and see Id's written from the South Parallella and the South Parallella could read the South cores and see the Id's written from the North Parallella.

The next bit of advice is to bring the system up gradually. For example first configure the hdf file for only one chip and run the hello world program for the 908 core (single chip with the appropriate hardware link) and then with the second board configure the hdf file and run the hello world program for the A08 core (single chip with different link). Check that this works as expected before trying to extend to multiple chips. Once working patch the epiphany-libs code to enable the North / South links and try again with single chip to confirm that your version of the epiphany-libs code still works. Then try enable the North / South links and see if you can create a group with 32 cores.

Notice that with the very latest fpga and matching kernel driver the disable of the elinks and the setting of the elink speed is all done in the kernel driver see https://github.com/parallella/parallell ... any.c#L482 so to replicate the 2015.1 fpga tests with the 2016.3 fpga https://www.parallella.org/2016/04/11/m ... se-2016-3/ will likely require a patch to the kernel driver as well as a patch to the epiphany-libs. Also because the 2016.3 fpga has higher elink speeds then it is likely that you will have to drop the speed of the elink to get it to work between boards. Thus it will not be easy to replicate the old inter board communications with the new fpga.

Peter.
User avatar
peteasa
 
Posts: 114
Joined: Fri Nov 21, 2014 7:04 pm

Re: Connecting Multiple Parallella Boards

Postby aviv_ohad » Thu Aug 25, 2016 6:39 pm

Hello everybody,
We are glad to tell you that we succeeded! Thanks to a lot of help from Peter and others, and a lot of time and effort,
We managed to build our very own Parallella Epiphany level cluster!
We wrote a user manual, some example code, and a technical report- all available in the link: https://github.com/Paralel-Systems-Lab/parallella-hw
We also added the design files to the same folder.
Hope this helps some of you! please let us know if you use it or have any issues!

CAM00434.jpg
CAM00434.jpg (717.43 KiB) Viewed 1968 times



Cheers ;)
aviv_ohad
 
Posts: 3
Joined: Sun Apr 03, 2016 7:52 am


Return to Epiphany and Parallella Q & A

Who is online

Users browsing this forum: No registered users and 5 guests