Parallel Gzip Tar
Old school Tar
Linux users get used to compress with tar command.
Pigz
For small file, don't bother. But when compressing takes time, here are some hint to speed up the process.
Most of us have multi core cpu. Use pigz. pigz is equivalent to gzip but it distributes the task to several threads.
Multiple archives
One can also use gnu parallel to generate faster multiple archives of all "*.data" files.
Build you own tar
One other option is to build tar from source with specific flag in the configure script. pigz use used instead of gzip.
Use 7zip
One other option to efficiently build archive is 7zip. It used LZMA or LZMA2 compression algorithm but handles others format also. On linux, it is provided as p7zip ( posix 7zip).
One may need to install : yum install p7zip. There is some limitation because 7zip do not handle correctly file linux permision. One may need to build a *.tar file and then *.tar.7z.
tar old school example
# Compress tar -czf file.tar.gz file1 file2 # List tar -tf file.tar.gz # Extract tar -xf file.tar.gz
Most of us have multi core cpu. Use pigz. pigz is equivalent to gzip but it distributes the task to several threads.
parallel tar example
# Compress tar -c file1 file2 | pigz -5 > out.tar.gz # Compress ( Use the tar flag "--use-compress-program=" to tell tar what compression program to use. ) tar -cf out.tar.gz --use-compress-program=pigz file1 file2 # List all file contained in an archive. This process is also multithreaded. tar -tf file.tar.gz --use-compress-program=pigz # Extract tar -xf out.tar.gz --use-compress-program=pigz
One can also use gnu parallel to generate faster multiple archives of all "*.data" files.
multiple archives generation
# install parallel : sudo apt-get install parallel or dnf install parallel # using gnu parallel : parallel creation of bz2 archive of all *.data file find . -name '*.data' 2>/dev/null | parallel tar -cjf {1}.tar.bz2 {1}
Build you own tar
compile tar
cd tar-1.28 ./configure --with-gzip=pigz make
Use 7zip
parallel tar example
# One may use to gzip a archive tar -c file1 file2 | gzip > out.tar.gz # One should use instead ( mx level of compression = 9 (Ultra) ) tar -c file1 file2 | 7z a -mx=3 -si out.tar.7z #restoring will work invoking 7z x -so out.tar.7z | tar xf -