Home arrow Journal arrow Today Quick Tip: Compress your files faster with pbzip2
Today Quick Tip: Compress your files faster with pbzip2 PDF
Saturday, 14 May 2011
Everybody uses tgz (tar + gzip) to compress files. Some use bzip2 because it gives higher compress ratio per archived files.

A faster solution is pbzip2 which make use of every core of your CPU, and/or every CPU of your system.

If you benchmark a bzip2 archieving procees and in the mean time you use top -P you will see that your system load will not pass over 1 even if you have multiple cores. That means bzip2 does not scale. Try the same archieving step with pbzip2 and you'll see the difference in load and in time that will take to compress/decompress files.

  pbzip2 -z file.iso           #  or --compress, it will archieve file.iso file

  pbzip2 -d file.tbz             # or --decompress, it will unarchive file.tbz into file.iso

If you want to use pbzip2 with tar ( instead of bzip2) you have the following choices:

1. create an alias from tar to tar --use-compress-prog=pbzip2

  alias tar tar --use-compress-prog=pbzip2

move bzip2 binary and create a symlink to pbzip2

  mv /usr/bin/bzip2 /usr/bin/bzip2.old
  ln -s /usr/local/bin/pbzip2 /usr/bin/bzip2

3. use pipe

  tar -cf - . | pbzip2 > ../file.tbz

4. use --use-compress-prog parameter of tar (which will also pipe to pbzip2

  tar --use-compress-prog=pbzip2 -cf file.tbz .

There is also pigz - parallell gzip - multi thread gzip compression utility.

Last Updated ( Tuesday, 17 May 2011 )
< Prev   Next >

Other BSD Systems





Best BSD firewall?