Configuring parallela for a DB project

[exa]

When we're buying the parallela cluster kit, it comes with a 16 GB microsd. Can we change the microsd size, or can we upgrade it later on easily? It'd be nice to have it as large as possible for my project. Would it be possible to make options as this configurable in the shopping page? If I got 16 such kits, they would make 1024 cores and just 256GB disk, which is probably not enough for the data-intensive compute tasks I'm considering this hardware for. Well, it's some proprietary parallel data mining algorithms, basically.

The 1GB RAM limitation per board is also a bit challenging but that's not drastic.

[Gravis]

[hamster]

A NAS box doesn't scale too well. A single gigabit interface spreads thinly between between 16 nodes, and the 100 or so random I/Os of a spinning disk won't go far. How about using USB flash drives?

Maybe there is a software stack that would allow you to RAID and consolidate the capacity into a big distributed NAS? Or simpler still you could just nfs mount the storage off of the other systems to give yourself some file systems like

/cluster/node00/disk0 32GB
/cluster/node01/disk0 32GB
...
/cluster/node0E/disk0 32GB
/cluster/node0F/disk0 32GB

If the dataset is read-only you could most probably just use some form of hash to decide where to put the blocks of data, allowing easy access. As long as none of your flash keys break a multithreaded task should have no problem saturating a node's LAN bandwidth.

[Gravis]

[hamster]

Spurred on by your comments, I've done a quick bit of I/O testing on my Zedboard running Xillinux.

The simplistic testing was "dd if=/dev/(whatever) of=/dev/null bs=128k".

A low-end Lexar 32GB Class 4 SD card got 10.2MB/s for read.

A equally low end 32GB ADATA USB 2.0 stick got 24.3MB/s for read.

I've got a class 10 card around here somewhere.... but looking at the table at http://elinux.org/RPi_SD_cards it looks as though not all brands/models of SD cards are equal - I'm guessing that some class 4 cards are actually class 6 or class 10 .

There seems to be three sources of SD cards, regardless of class branding - those that read at about 4MB/s, those that read at about 11MB/s (like mine) and those that read at about 20MB/s.

[ticso]

Read bandwidth is not that bad for SD.
What really sucks for DB usage is read latency, which is often around 1ms.
Don't know why it is that slow, but I plan to directly attach NAND chips because of this.

[Gravis]

[hamster]

[ticso]

Fast Database servers work by having their data in fast random accessible storage.
Fastest is SRAM, next fastest DRAM, then disks.
SRAM is too expensive and disks too slow in most cases.
Therefor most databases use lots of DRAM - 100G or more is not uncommon depending on size of your workset.
Parallella has 1G only.
On the other hand 32G is cheap to get for standard PC hardware.
Easy to see why a PC is better choice, isn't it?

Next is disk.
Making Parallella a good database system requires fast "disk" - or external RAM.
1ms is typical for SD (card not reader) - some exceptional are doing 700-800us, but are low bandwidth.
Way to slow to concur with DRAM - especially if DRAM even has the size of your whole SD card.
You can get faster with better read time or more parallism.
SD can't do any of them.

Raw NAND chips have much better random access time.
One datasheet I have says 75us.
And they are multibanked allowing parallel access within chip - even more with multiple chips.
The reason why SD are slow is that they are cheap and small.
For example to transfer a logical block into a physical location you need to know where it is placed.
You can either cache this or read from flash.
Reading means additional 75us for each indirection.
SD cards are made for cameras running FAT and big picture files.
Linear read and write is what is required there.
Linear prereading adds 75us blocking time for next request.
If you compare measurement results you will notice that fast SD cards have higher read latency than slow cards.
A SSD is also prereading but has multiple ECC blocks allowing to read from other banks at the same time.

[hamster]

The database you are looking to set up doesn't sound like a normal database - what is the end use?

When you include delays in the networking stack to get data on and off the systems then such tiny latencies usually don't really matter.

And there is no way that a Parallella could compete with a tricked out PC (I got an 4GHz 8-core with 256SSD +32GB RAM for about $1000 wholesale), let alone server grade hardware (I've got blades at work that can take 24 32GB DIMMs... and you can get 1U SAN devices filled with TBs of flash if you have enough money).

I was more thinking along the lines of this....

Say you need to create a lookup catalogue for 100M book names, authors and ISBN numbers (maybe 200 bytes per book - for 20GB of data). If you were to try this in a database you would create a table with an index on it. When you feed a name it will navigate down through the index blocks, requiring a random read at each level, and then finally down to the data block with the name, author, ISBN and other stuff. To read a record might take 5 or 6 I/Os.

One other possibility would be to use an on-disk hash table. Hash the book name into a number between 0 and 32GB-1, then issue a single 128K read at the 1 byte prior to the hash value. If the records are stored as null terminated strings somewhere immediately on or as soon as possible after the hash value then you just start scanning sequentially in the block at that address for the name. When you see two null bytes in a row you know you are done (not found).

All the indexing work is done upfront, nearly any record can be found by book name with just one I/O (exceptions being for hashes where the reads wrap around the table, or an awfully common or lots of hash collisions (in which case you should move to a hash with a larger range, or a better hash)).

Even if you do fail to find it in the first block, it then becomes a streaming sequential read....