zip tests
I usually keep this blog quite general purpose, but I thought I'd share this bit of tech knowledge.
If you can read any of this, it says that
my drive takes over a minute to read a 2.4 GB file (the current size of fableisland's raw db)
gzip with max compression (-9, -6 is default) is fast and good compression
bzip2 is great at compression but slow
pbzip2 really does scale (to at least two processors) wonderfully.
# ls -lah
-rw-r--r-- 1 root root 2.4G Jan 5 17:54 test.sql
# time cp test.sql /dev/null
real 1m10.412s
# time gzip -9 test.sql
real 5m15.720s
-rw-r--r-- 1 root root 237M Jan 5 16:47 test.sql.gz
# time bzip2 test.sql
real 42m19.576s
-rw-r--r-- 1 root root 175M Jan 5 16:59 test.sql.bz2
# time pbzip2 test-p.sql
real 23m25.958s
-rw-r--r-- 1 root root 175M Jan 5 17:54 test-p.sql.bz2