Accessing arbitrary host virtual memory from Epiphany

[dms1guy]

I have had a read of all the forum posts I could find on external memory (the 1 GB SDRAM) and its mapping from the host and Epiphany sides.

Could someone please check over my core assumptions and point out any misconceptions:

My assumptions are that:

1. The 1 GB SDRAM appears in the 1st Gigabyte of the host (ARM) address space (the 1st Megabyte not being accessible)

2. The 1GB SDRAM appears in the 3rd Gigabyte of the Epiphany address space
BUT out of every consecutive 64 MByte region, only the last 52MBytes are accessible because
With the default configuration of E16 Parallella, the Core-ID of Epiphany is set to (rows: 32-35, cols: 8-11),
and only the EAST e-link is connected to the FPGA, which is the glue to the SDRAM.
This means that Epiphany can only access its EAST addresses (rows 0-63, cols 12-63),
so only addresses with columns greater than 12 will get sent to the EAST link.
( address[31:26] = row, address[25:20 = column )
So addresses with column values (0..11) are not accessible.
For each row, each column addresses a 1 MByte region, and addresses with the first 12 column values
can't be routed to the EAST e-link, which means that the first 12 MBytes in each 64 MByte region are not accessible.
(as there are 64 columns per row, 1 MByte per column, hence each consecutive 64 MBytes)

This results in only the following addresses being routable to the EAST e-link

Address Range
----------------------
0x00c00000 - 0x03ffffff i.e. addresses for (row 0, col 12) .... (row 0, col 63)
0x04c00000 - 0x07ffffff i.e. addresses for (row 1, col 12) .... (row 1, col 63)
0x08c00000 - 0x0bffffff i.e. addresses for (row 2, col 12) .... (row 2, col 63)
...
0xfcc00000 - 0xffffffff i.e. addresses for (row 63, col 12) ... (row 63, col 63)

3. If a host (ARM) user-space memory region is to be accessed from the Epiphany device,
then the following conditions must be satisfied:

a) The physical address of the USER-SPACE memory must be known to the Epiphany side

b) The user-space memory must be committed (dirty i.e. backed with physical memory)

c) The user-space memory must be locked (i.e. cannot be swapped out)

d) The physical address of the user-space memory must not fall into an address range
that can not be routed out of the Epiphany EAST e-link (as per 2. above)

Can any one please confirm if these assumptions are correct.

-- dms1guy

[piotr5]

where did you get the idea arm-memory would be seen at 3G location from epiphany? I didn't notice that. AFAIK arm and epiphany see the same address-space except that in addition epiphany sees a mirror of the current core at a fixed location (so it can be addressed in 2 ways, with 2 addresses). that's to make program-distribution easier, compile once and run on any core you want to. and of course as you say from the point of view of epiphany any message sent to the first 4M of a 64M block will never reach the destination.

all that said, it should be pointed out that what happens after a message from epiphany left through the east eLink to fpga is up to fpga-programming, that behaviour could be changed. additionally it would be interesting to experiment with the router on epiphany-chips, so messages attempting to leave chip towards north or south will be redirected to east. therefore what you describe here is just current design of the boot-loader and default behaviour of epiphany. and you should also keep in mind the linux-kernel, it's being told only of a particular area for memory shared between epiphany and arm -- not really sure if dirtying and un-swapping some memory-area is all that's needed...

instead of reading forum-posts you'd better read the manual!

[dms1guy]

[dms1guy]

[piotr5]

I'm no linux-kernel guru. so I don't know how it could break. linux and its programs make certain assumptions about memory, ROM memory or this kind of shared memory would break those assumptions. do you know of other cases where what you suggest has been done? I suppose you could try but, unless you study the linux sourcecode, you cannot really foresee what would happen if your epiphany program had a bug. even less could you debug such a situation.

one particular case I could imagine is that your program writes into memory where it shouldn't, how do you notice that happened? it's a segmentation fault situation, but when an eCore did do the mistake, how could the ARM ever know which core to interrupt? so in your epiphany programs you must always check if the address you're about to write into is valid...

or, what if your mlock() in future versions of linux gets overruled by actually stopping the program which issued it and then writing something else into that memory for a while, before restoring the original data and continuing program-execution? I suspect linux is already doing that whenever you put your system into a hibernation state. so, what could stop the linux-developers to do the same in other situations too?

[dms1guy]

[piotr5]

thanks for the clarifications. it's true that epiphany could count as real-time. guess the architects of parallella's operating system didn't think of using that method. as always with computers: try it, make it public once it works, complain if other people's changes did break your programs. if in future there will be methods for swapping out memory without any time-loss at all, then and only then you'll have to think about splitting the notion of "real-time" into real-time processing and real-time I/O (where the latter could still suffer time-loss because you'd need to tell the hardware the new location for data, while the former receives that info from the virtual-memory hardware which directly works with the cache), as talking with epiphany counts as I/O...

so, how would you actually make sure the correct areas of physical memory are used?

[sebraa]

I have tested the simpler case of "provide a physical address" (taken from /proc/self/pagemap) "to the Epiphany and have it write there". It worked (unreliably) when I accessed (cleared) the page in question manually (I didn't use mlock). To me, this showed that the Epiphany is able to seriously corrupt ARM memory and one has to be extra careful when working with pointers.

Your assumptions seem sound to me. Additionally, I think caching for that specific page must be disabled as well, otherwise the ARM cores might read stale data.

However, I think that the Epiphany being able to access all (or most) physical memory is an oversight; for security reasons, the FPGA logic should ignore (or trap) accesses outside the shared memory area.

[dms1guy]

[piotr5]