Perl Programming Questions

[JeminiZero]

Hello there. This is a question to anyone with who considers themselves a hardcore computer scientist. Because I'm not one and I am thinking of using perl with a project of insane magnitude.

Principally, I'm going to try and build a hash in perl. With about 4 billion entries. No you read it right. Its a 'b' not an 'm'. Like I said, its insane.

And so a few questions arise:
a) Is this even possible
b) How much RAM am I looking at to hold a hash of this size
c) Would it be more stable if I break this into multiple hashes (e.g. 4 seperate ones with a billion entries each).

[BugFix]

a) Is this even possible
b) How much RAM am I looking at to hold a hash of this size
c) Would it be more stable if I break this into multiple hashes (e.g. 4 seperate ones with a billion entries each).

It's possible on a 64 bit system. There's not enough address space in a 32 bit process to fit all that. Figure on a few dozen bytes per entry, minimum, plus whatever it is you're storing in the hash. So you're looking at a task for something like a 16 or 32GB box. That hardware is available, but not particularly cheap.

If you can't fit it all in memory, then a hash table is the wrong data structure to use. You'll have terrible VM pressure. Instead, look at tools designed to work on storage directly, like databases.

[JeminiZero]

It's possible on a 64 bit system. There's not enough address space in a 32 bit process to fit all that.

Will splitting into multiple hashes help?


Figure on a few dozen bytes per entry, minimum, plus whatever it is you're storing in the hash. So you're looking at a task for something like a 16 or 32GB box. That hardware is available, but not particularly cheap.

Ouch. And here I was thinking of running it on my desktop. How much RAM does a databse consume by comparison?

Thanks for the replies. :smallsmile: