Core / Network Profiling (Cool video ^^)

[TomV]

Hey guys,

For a research project I am trying to compare a certain application running on the Parallella board, with a simulation model of that same application running on my PC. In order to do this, I need profiling data from the Epiphany chip. An example of what I have now can be found in this video

This turns out to be harder than I thought... Right now I have a free running timer (CTIMER0) counting down from MAX->0, and an event counter (CTRIMER1) running down from X->0, where X is the number of events that have to occur to fire an interrupt (sampling period if you will). At every interrupt I store the CTIMER0 value, knowing that for each sample I get, the event counter has caught X events. Not ideaal, but it seems to work best of all the methods I tried (a free running timer with a periodic interrupt would be much help in improving this btw...)

Anyway, it seems to work well for CPU events, but it is very hard to capture network traffic. The Architecture reference states:

Mesh traffic: Counts wait or access events on the local cMesh network node. Mesh event monitor event selection is programmed through the MESHCONFIG registers

All well and good, but since it apparently only captures cMesh events, I am having an extremely hard time figuring out what is happening on chip, leaving me with four questions:

Question 1: What exactly are Access and Wait events?
Seems self explanatory, but since every port (EAST, NORTH, SOUTH, WEST) has an up and a down link, it is crucial to know whether an access event is an outgoing packet, and ingoing packet or both in order to determine the direction of data I am capturing. Also, is an event always the same FLIT size (can I use it to measure network bandwidth?)

Question 2: When is the cMesh used?
Currently, I have an application where all the cores only talk with the Host via shared memory, and never exchange data on-chip. I would expect to get no cMesh traffic at all, but only traffic over the xMesh, maybe rMesh going east-west, or west-east. However, as you can see in the video, there is quite some on-chip cMesh traffic. Where does this come from? And is there a way to predict this?

Question 3: How does the E-Link connect to the cMesh?
In the video ( ) it becomes clear that data is coming off-chip on the east, travelling along the top row, and going down column by column to 'feed' the middle rows. I would expect data to only travel east-west or west-east since it originates from, or is destined for, the host. How does the E-Link decide on which row to feed in the data?

Question 4: How do I get a view of the entire network?
Since I can only see the cMesh, and my traffic is probably mainly going over the rMesh and xMesh, it seems I can never capture the data I need. Is there a cool trick anyone knows of that would allow me to monitor all traffic? The only thing I can think of atm is to alter the FPGA to capture all the destination and source addresses going over the E-Link, and using the deterministic routing to figure out where the data would have gone. This however, is a LOT of work

Thanks in advance for helping me out! (Btw, if anyone has any advice on better profiling techniques for the Epiphany in general, let me know!)

[aolofsson]

[TomV]

Hi Andreas,

thanks for the amazingly quick reply! It makes sense actually that data destined on-chip is using the cMesh (I think I even read that somewhere )..
Also the list of what is captured and how is very useful, thanks!

What I am trying to do at the moment is compare actual on-chip network traffic to a simulation model. The goal is to see if I can use the simulations to predict network usage/bottlenecks in an application, and to see if it can be used to aid software design. Judging from the list you made, there is no way to capture network activity for data going off-chip (is that correct? host-reads, and off-chip writes are not captured).

I suppose I could capture all the start/end times of my DMA transfers, and the sizes that go with the transfer to derive this (the routing is deterministic anyway). It would be nice to be able to see the total off-chip traffic, for instance to determine whether the transfer times are being limited by the eLink bandwidth, however I can imagine this would not be high on your priority list .

As for the unexpected NORTH / SOUTH traffic, unfortunately I can't put my code on Github as is. I will try to build an example to mimic my setup, this will take a day or two though I guess.

Thanks again for the help!

[joaofl]

Hi Tom.

Saw your post, and got interest about your work. Since this post if from April, I was wondering if you have results already, and maybe a publication with it.

I'm working on something similar, and it would be nice to learn some things from your experience.

Thanks.

Cheers,

João