- Code gets loaded from the host and started one at a time (but there is a big delay between the start time of the "main") If there is a barrier somewhere, the value of any timer started before the barrier will have very different values when they get there.
- If you want to synchronize the start of the programs, you will need to place a barrier at the beginning (before starting any timers)...I believe you did that?
- Depending on on chip communication patterns, if there is a lot of congestion on the on chip network, some cores may get non fair allocation
- If there is ANY off chip loads or program fetches, then this could account for serious differences ( I know you said there weren't..but this is the most common one)
- Beware of any ARM/epiphany communication with benchmarking/timing. Try to make the runtime big enough (seconds?), to make sure you don't hit some minimum thread switch cost or other fixed overhead (for example, each e_load/e_write call had fixed overhead that is fairly significant). 100,000 clock cycles is ~1ms, which is pretty fast.
Andreas
Statistics: Posted by aolofsson — Sat Apr 09, 2016 4:17 pm
]]>
]]>
I've found something interesting - I commented out the barriers in my code and now the timer values match up. This is the same behaviour that has been noticed by a number of people on this forum - for example and .
What could possibly be causing this discrepancy when barriers are involved? It seems to be a recurring issue.
Statistics: Posted by laideybug — Sat Apr 09, 2016 10:26 am
]]>
]]>
Following on from my previous post (), I decided to implement the algorithm using 2 workgroups that can communicate between each other with local memory writes, interrupts, and mutex calls. In order to do this, I had to write my own slightly modifed versions the eSDK functions for e_mutex_init(), e_mutex_lock(), e_mutex_unlock(), e_irq_set() and e_get_global_address() to allow them to access any core on the Epiphany, not just the cores in the caller core's workgroup.
I've deployed the ML algorithm on a 4 x 3 workgroup (which I'll call the network) with another 1 x 1 workgroup acting as an agent for the ARM. I have a dataset that needs to be processed by the network. The ARM reads in this dataset and writes it to shared memory. It then initialises and starts both workgroups.
After some basic initialisation, the cores in the network enter an IDLE state. The agent reads the first data sample from shared memory and copies it to the local memory of each core in the network. It then issues an E_SYNC interrupt call to each core to signal them to start processing the sample and then enters an IDLE state while waiting for the network to finish. During the process, the cores in the network sync using a number of e_barrier() calls. They also exchange information with each other using e_dma_copy(). I time how long the process takes on each core with the system timers (including the time spent in IDLE waiting to be started by the agent).
Once they have finished processing the sample, each core writes the timer value to the agent's local memory and increments a shared value located in the using a mutex (also located in the agent's local memory). The last core to increment this shared value (i.e. the last core to finish processing) issues an E_SYNC interrupt call to the agent to wake it up. The agent then collects and writes the totalled timer values of the network for each step to shared memory, and starts the whole process again for the next data sample.
The ARM polls the benchmark values for each sample in shared memory and then averages them and uses printf() to output them to Terminal.
My problem is this: the number of clock cycles that it takes for each core in the network to process each data sample is recorded as around ~100,000 cycles. However, the number of clock cycles recorded on the agent between the time it sends an E_SYNC interrupt call to each core to start processing and the time it is woken up again by an E_SYNC interrupt call from the last node to finish in the network is in the region of ~140,000,000 clock cycles. How can this possibly be? Even if each of the 12 cores in the network executed their processes one after each other (which is impossible, given that they sync with barriers), the total number of clock cycles for the agent to be woken up again should be nowhere near ~140,000,000. I've checked my code 100 times and I can't see any issues. The overall implementation works, just slowly. I'm using the internal.ldf linker script and all memory is allocated correctly as per e-objdump.
Am I going crazy or am I overlooking something glaringly obvious?
Statistics: Posted by laideybug — Thu Apr 07, 2016 6:25 pm
]]>