Accessing arbitrary host virtual memory from Epiphany

Any technical questions about the Epiphany chip and Parallella HW Platform.

Moderator: aolofsson

Accessing arbitrary host virtual memory from Epiphany

Postby dms1guy » Fri Nov 27, 2015 5:55 pm

I have had a read of all the forum posts I could find on external memory (the 1 GB SDRAM) and its mapping from the host and Epiphany sides.

Could someone please check over my core assumptions and point out any misconceptions:

My assumptions are that:

1. The 1 GB SDRAM appears in the 1st Gigabyte of the host (ARM) address space (the 1st Megabyte not being accessible)

2. The 1GB SDRAM appears in the 3rd Gigabyte of the Epiphany address space
BUT out of every consecutive 64 MByte region, only the last 52MBytes are accessible because
With the default configuration of E16 Parallella, the Core-ID of Epiphany is set to (rows: 32-35, cols: 8-11),
and only the EAST e-link is connected to the FPGA, which is the glue to the SDRAM.
This means that Epiphany can only access its EAST addresses (rows 0-63, cols 12-63),
so only addresses with columns greater than 12 will get sent to the EAST link.
( address[31:26] = row, address[25:20 = column )
So addresses with column values (0..11) are not accessible.
For each row, each column addresses a 1 MByte region, and addresses with the first 12 column values
can't be routed to the EAST e-link, which means that the first 12 MBytes in each 64 MByte region are not accessible.
(as there are 64 columns per row, 1 MByte per column, hence each consecutive 64 MBytes)

This results in only the following addresses being routable to the EAST e-link

Address Range
----------------------
0x00c00000 - 0x03ffffff i.e. addresses for (row 0, col 12) .... (row 0, col 63)
0x04c00000 - 0x07ffffff i.e. addresses for (row 1, col 12) .... (row 1, col 63)
0x08c00000 - 0x0bffffff i.e. addresses for (row 2, col 12) .... (row 2, col 63)
...
0xfcc00000 - 0xffffffff i.e. addresses for (row 63, col 12) ... (row 63, col 63)

3. If a host (ARM) user-space memory region is to be accessed from the Epiphany device,
then the following conditions must be satisfied:

a) The physical address of the USER-SPACE memory must be known to the Epiphany side

b) The user-space memory must be committed (dirty i.e. backed with physical memory)

c) The user-space memory must be locked (i.e. cannot be swapped out)

d) The physical address of the user-space memory must not fall into an address range
that can not be routed out of the Epiphany EAST e-link (as per 2. above)

Can any one please confirm if these assumptions are correct.

-- dms1guy
Last edited by dms1guy on Sat Nov 28, 2015 8:39 am, edited 7 times in total.
User avatar
dms1guy
 
Posts: 21
Joined: Thu Sep 10, 2015 9:05 pm
Location: Isle of Man

Re: Accessing arbitrary host virtual memory from Epiphany

Postby piotr5 » Sat Nov 28, 2015 8:23 am

where did you get the idea arm-memory would be seen at 3G location from epiphany? I didn't notice that. AFAIK arm and epiphany see the same address-space except that in addition epiphany sees a mirror of the current core at a fixed location (so it can be addressed in 2 ways, with 2 addresses). that's to make program-distribution easier, compile once and run on any core you want to. and of course as you say from the point of view of epiphany any message sent to the first 4M of a 64M block will never reach the destination.

all that said, it should be pointed out that what happens after a message from epiphany left through the east eLink to fpga is up to fpga-programming, that behaviour could be changed. additionally it would be interesting to experiment with the router on epiphany-chips, so messages attempting to leave chip towards north or south will be redirected to east. therefore what you describe here is just current design of the boot-loader and default behaviour of epiphany. and you should also keep in mind the linux-kernel, it's being told only of a particular area for memory shared between epiphany and arm -- not really sure if dirtying and un-swapping some memory-area is all that's needed...

instead of reading forum-posts you'd better read the manual!
piotr5
 
Posts: 230
Joined: Sun Dec 23, 2012 2:48 pm

Re: Accessing arbitrary host virtual memory from Epiphany

Postby dms1guy » Sat Nov 28, 2015 1:00 pm

re: where did you get the idea arm-memory would be seen at 3G location from epiphany?


I actually got that from the Parallella Manual - section 4.1 - Zynq memory map

Code: Select all
Address     AddressSize Size    Function Note
Start       End
----------- ----------- -----   -------- --------------------------------------
0x0010_0000 0x3FFF_FFFF 1GB     DRAM     Accessible to all interconnect masters

0x4000_0000 0x7FFF_FFFF 1GB     PL       Custom logic address range

0x8000_0000 0xBFFF_FFFF 1GB     PL       Epiphany address range

0xFC00_0000 0xFCFF_FFFF 16MB    FLASH    Quad-SPI linear address for linear mode

0xFFFC_0000 0xFFFF_FFFF 252KB   OCM      OCM upper address range


It refers to the 3rd GB as Epiphany Address range, and I mistakenly assumed it was referring to where Epiphany would see the DRAM.

Thank you for correcting my mistaken assumption:
User avatar
dms1guy
 
Posts: 21
Joined: Thu Sep 10, 2015 9:05 pm
Location: Isle of Man

Re: Accessing arbitrary host virtual memory from Epiphany

Postby dms1guy » Sat Nov 28, 2015 1:19 pm

re: not really sure if dirtying and un-swapping some memory-area is all that's needed


It was never my intention to disable the swap system, as in my experience that can lead to system instability.

I was planning to use mlock() to fix virtual memory regions to a specific physical memory region.

Do you have any specific reservations of why mlock() might not work?
User avatar
dms1guy
 
Posts: 21
Joined: Thu Sep 10, 2015 9:05 pm
Location: Isle of Man

Re: Accessing arbitrary host virtual memory from Epiphany

Postby piotr5 » Sun Nov 29, 2015 9:12 am

I'm no linux-kernel guru. so I don't know how it could break. linux and its programs make certain assumptions about memory, ROM memory or this kind of shared memory would break those assumptions. do you know of other cases where what you suggest has been done? I suppose you could try but, unless you study the linux sourcecode, you cannot really foresee what would happen if your epiphany program had a bug. even less could you debug such a situation.

one particular case I could imagine is that your program writes into memory where it shouldn't, how do you notice that happened? it's a segmentation fault situation, but when an eCore did do the mistake, how could the ARM ever know which core to interrupt? so in your epiphany programs you must always check if the address you're about to write into is valid...

or, what if your mlock() in future versions of linux gets overruled by actually stopping the program which issued it and then writing something else into that memory for a while, before restoring the original data and continuing program-execution? I suspect linux is already doing that whenever you put your system into a hibernation state. so, what could stop the linux-developers to do the same in other situations too?
piotr5
 
Posts: 230
Joined: Sun Dec 23, 2012 2:48 pm

Re: Accessing arbitrary host virtual memory from Epiphany

Postby dms1guy » Sun Nov 29, 2015 12:22 pm

piotr5 wrote:linux and its programs make certain assumptions about memory, ROM memory or this kind of shared memory would break those assumptions.


I don't believe that is correct. The mlock documentation (see: http://linux.die.net/man/2/mlock) states:

Memory locking has two main applications: real-time algorithms and high-security data processing. Real-time applications require deterministic timing, and, like scheduling, paging is one major cause of unexpected program execution delays.

Cryptographic security software often handles critical bytes like passwords or secret keys as data structures. As a result of paging, these secrets could be transferred onto a persistent swap store medium, where they might be accessible to the enemy long after the security software has erased the secrets in RAM and terminated.


here the Linux assumptions regarding the locked memory are clearly stated. A stated goal is to service real-time applications and preserve deterministic timing. It even specifies any movement of the locked page through any kind of 'paging' behaviour as something that would disrupt the stated goal of preserving deterministic timing, so clearly, there can't be any arbitrary interruption of a locked memory page .. otherwise it defeats its own stated goals.

It further states a goal of security to prevent paging to backing store as a security hole .. so again, formal confirmation that the page data will not be copied to backing store ... to prevent security holes in Cryptographic applications


piotr5 wrote:what if your mlock() in future versions of linux gets overruled by actually stopping the program which issued it and then writing something else into that memory for a while, before restoring the original data and continuing program-execution? I suspect linux is already doing that whenever you put your system into a hibernation state. so, what could stop the linux-developers to do the same in other situations too?


Your suspicions on this count are justified as the mlock() documentation states:

(But be aware that the [b]suspend mode on laptops and some desktop computers will save a copy of the system's RAM to disk, regardless of memory locks[/b].)


But for my intended use-cases, I can live with the restriction that the program that issues the mlock() (and the Epiphany platform) must always be reset if it is ever placed into hibernation or equivalent.

piotr5 wrote:but, unless you study the linux sourcecode, you cannot really foresee what would happen if your epiphany program had a bug. even less could you debug such a situation.

one particular case I could imagine is that your program writes into memory where it shouldn't, how do you notice that happened? it's a segmentation fault situation, but when an eCore did do the mistake, how could the ARM ever know which core to interrupt? so in your epiphany programs you must always check if the address you're about to write into is valid...


I agree, writing software to manage any kind of critical code segments, especially in heterogeneous systems is never to be taken lightly, or by programmers without significant experience in writing (and debugging) concurrent software.
User avatar
dms1guy
 
Posts: 21
Joined: Thu Sep 10, 2015 9:05 pm
Location: Isle of Man

Re: Accessing arbitrary host virtual memory from Epiphany

Postby piotr5 » Mon Nov 30, 2015 1:15 pm

thanks for the clarifications. it's true that epiphany could count as real-time. guess the architects of parallella's operating system didn't think of using that method. as always with computers: try it, make it public once it works, complain if other people's changes did break your programs. if in future there will be methods for swapping out memory without any time-loss at all, then and only then you'll have to think about splitting the notion of "real-time" into real-time processing and real-time I/O (where the latter could still suffer time-loss because you'd need to tell the hardware the new location for data, while the former receives that info from the virtual-memory hardware which directly works with the cache), as talking with epiphany counts as I/O...

so, how would you actually make sure the correct areas of physical memory are used?
piotr5
 
Posts: 230
Joined: Sun Dec 23, 2012 2:48 pm

Re: Accessing arbitrary host virtual memory from Epiphany

Postby sebraa » Mon Nov 30, 2015 1:46 pm

I have tested the simpler case of "provide a physical address" (taken from /proc/self/pagemap) "to the Epiphany and have it write there". It worked (unreliably) when I accessed (cleared) the page in question manually (I didn't use mlock). To me, this showed that the Epiphany is able to seriously corrupt ARM memory and one has to be extra careful when working with pointers.

Your assumptions seem sound to me. Additionally, I think caching for that specific page must be disabled as well, otherwise the ARM cores might read stale data.

However, I think that the Epiphany being able to access all (or most) physical memory is an oversight; for security reasons, the FPGA logic should ignore (or trap) accesses outside the shared memory area.
sebraa
 
Posts: 495
Joined: Mon Jul 21, 2014 7:54 pm

Re: Accessing arbitrary host virtual memory from Epiphany

Postby dms1guy » Thu Dec 03, 2015 9:57 am

piotr5 wrote:thanks for the clarifications. it's true that epiphany could count as real-time. guess the architects of parallella's operating system didn't think of using that method. as always with computers: try it, make it public once it works


I'm working on it now. If my assumptions are validated, then of course I'll do a post to share my findings.

piotr5 wrote:so, how would you actually make sure the correct areas of physical memory are used?


As sebraa stated, I intend to convert the virtual address into a PFN (page frame number) by dividing it by the page size (which is 4096). Then use this to index into /proc/pid/pagemap (where pid = PID of the process executing on the host) to access a 64-bit entry, the lower 55 bits of which provide the PFN of the physical memory mapped to that virtual address. You then add the 12 least significant bits of the virtual address to the physical PFN to get the final physical address.

sebraa wrote:Your assumptions seem sound to me. Additionally, I think caching for that specific page must be disabled as well, otherwise the ARM cores might read stale data.


Excellent observation, I'll add that to my list of core assumptions.

sebraa wrote:However, I think that the Epiphany being able to access all (or most) physical memory is an oversight; for security reasons, the FPGA logic should ignore (or trap) accesses outside the shared memory area.


If the Epiphany is going to run 'arbitrary' code, then I agree that there is significant scope for introducing difficult to trace bugs through ARM memory corruption by the Epiphany side, and some kind of FPGA logic protection against this would be a good idea.

But, if the Epiphany is going to run 'managed' code, where memory accesses can be guaranteed to operate within acceptable parameters, then restricting access to non-shared memory via the FPGA could be a huge impediment, depending on how it was done..

For example, in my case,
I intend the Epiphany to run managed code and be the application engine, rather than the ARM side, which I intend to treat as an intelligent I/O controller. Accordingly the lions share of the 1 GB SDRAM should go to the application engine, which in this case would be the Epiphany (I was thinking ~700MB to the application engine (Epiphany) and ~300MB to the I/O Controller (Host/ARM).

If the FPGA logic restricted the Epiphany having direct access to non-shared 1GB SDRAM regions, then the only way to implement my desired scheme is to copy all transfers via the 32MB shared memory region, which would effectively double the number of memory transfers required for operations where Epiphany reads or writes system memory.
User avatar
dms1guy
 
Posts: 21
Joined: Thu Sep 10, 2015 9:05 pm
Location: Isle of Man

Re: Accessing arbitrary host virtual memory from Epiphany

Postby piotr5 » Thu Dec 03, 2015 10:22 am

sebraa wrote:However, I think that the Epiphany being able to access all (or most) physical memory is an oversight; for security reasons, the FPGA logic should ignore (or trap) accesses outside the shared memory area.

wont help much if you allow FPGA programming to users. so what you suggest is that for security reasons only certain apps and users are allowed to alter FPGA. just face it, either you allow experimenting with new parallell technologies or you have a secure system. any solution inbetween is a foul compromise.

keep in mind though: I also said in past that we need a kernel-driver which is aware that some memory can be accessed from epiphany and some can't, and let the kernel build up security around that. i.e. I suggested a path inbetween. now that I noticed how epiphany routers can be reprogrammed to overrule those limitations, I changed my mind...

btw, that epiphany flat memory model isn't secure isn't my own observation. I think someone commented on it during kickstarter. I definitely remember reading such a comment in the early days.
piotr5
 
Posts: 230
Joined: Sun Dec 23, 2012 2:48 pm

Next

Return to Epiphany and Parallella Q & A

Who is online

Users browsing this forum: MSN [Bot] and 6 guests

cron