Speed up bzip2/xz compression


Please note that this blog has been moved.

Now it has its own domain: mynixworld.info🙂

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

If you have a SMP system and are wondering if there exists such thing that can help you to use all CPUs to compress/decompress a file then the answer is YES, there is.

PBZIP2 is a parallel implementation of bzip2 block-sorting file compressor that uses pthreads and achieve near-linear speedup on SMP machines.

In fact there is even a version of BZIP2 that runs on cluster machines, it is called MPIBZIP2 which was designed for a distributed-memory message-passing architecture.

Edit: If you are going to use the Lempel-Ziv-Markov compression algorithm (LZMA) then even xz has a parallel implementation, called pxz.

About Eugen Mihailescu

Always looking to learn more about *nix world, about the fundamental concepts of arithmetic, algebra and geometry. I am also passionate about programming, database and systems administration.
This entry was posted in Uncategorized and tagged . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s