Intel® Integrated Performance Primitives
Data Compression-based IPP_GZIP Interface for Linux*

License Definitions
Overview
Intel IPP Functions Used In The Interface
System Requirements
Package Contents
How To Build Source Code
Running the Interface
IPP_GZIP Command Syntax
IPP_GZIP In Multi-thread Environment
        Compression
        Decompression
        Compatibility
        Performance_Notes
Technical Support and Feedback
Disclaimer


License Definitions

By downloading and installing this sample, you hereby agree that the accompanying Materials are being provided to you under the terms and conditions of the End User License Agreement for the Intel® Integrated Performance Primitives product previously accepted by you. Please refer to the file ippEULA.rtf or ippEULA.txt located in the root directory of your Intel® IPP product installation for more information.

Overview

The Data Compression-based IPP_GZIP Interface (IPP_GZIP) illustrates the way of implementing effective lossless data compression solution by using Intel® Integrated Performance Primitives Data Compression domain API. Additionally, this interface shows the ways of parallelizing a user application using Intel® Virtual Machine to fully benefit from modern Intel® microprocessor architectures.

The interface uses dictionary-based IPP functions implementing Lempel-Ziv (LZ77) algorithm and original GZIP data formats according to RFC 1950, 1951 and 1952 specifications.

The result compressed data formats are fully compatible with the original GZIP formats, so the utilities are interchangeable. You can compress data with IPP_GZIP and decompress it with GZIP, or, vice versa, compress with GZIP and decompress using GZIP. However, to benefit from using multiprocessor/multicore architectures, use IPP_GZIP.

Additional information on this software as well as other Intel software performance products is available at http://www.intel.com/software/products/.

Intel IPP Functions Used In The Interface

IPP_GZIP interface uses the following functions from the Intel® Performance Primitives Data Compression library:

IPP_GZIP Function

IPP Function

Function Description

Compression

ippsDeflateLZ77_8u

Performs LZ77 encoding

ippsDeflateHuff_8u

Performs Huffman encoding

ippsCRC32_8u

Computes the CRC32 checksum for the source data buffer

Decompression/Test

ippsInflate_8u

Performs LZ77 decoding

ippsInflateBuildHuffTable

Performs Huffman decoding

System Requirements

Recommended hardware


Hardware requirements

Software requirements

Package Contents

The Intel® IPP Data Compression IPP_GZIP Interface for Linux* contains the following files:

./interfaces
ippEULA.rtf or ippEULA.txt End User License Agreement
support.txt Contains technical support information
./interfaces/data-compression/ipp_gzip
build_ia32.sh Shell script to build ipp_gzip interface on Linux systems based on the IA-32 architecture
build_intel64.sh Shell script to build ipp_gzip interface on Linux systems based on the Intel® 64 architecture
Makefile Makefile to build ipp_gzip interface
readme.htm This file
./interfaces/data-compression/ipp_gzip/src
ipp_gzip.c Interface's main function, upper-level dispatching functions and directory processing function
ipp_gzip_deflate.c Single-threaded and multi-threaded data compression function
ipp_gzip_deflate_mt.c Multi-threaded data compression function and service functions
ipp_gzip_inflate.c Single-threaded data decompression function
ipp_gzip_inflate_mt.c Multi-threaded data decompression function and service functions
ipp_gzip_io.c File input/output functions
ipp_gzip_utils.c Several common functions
./interfaces/data-compression/ipp_gzip/include
ipp_gzip.h Common header file for the interface
ipp_gzip_deflate.h Definitions and tables for deflate process
version.h Application version text
./interfaces/data-compression/ipp_gzip/vm/src
vm_affinity_common.c VM source file
vm_affinity_unix.c VM source file
vm_event_linux32.c VM source file
vm_file.c VM source file
vm_file_unix.c VM source file
vm_mmap_linux32.c VM source file
vm_mutex_linux32.c VM source file
vm_semaphore_linux32.c VM source file
vm_shared_object_linux32.c VM source file
vm_socket_linux32.c VM source file
vm_sys_info_linux32.c VM source file
vm_thread_linux32.c VM source file
vm_time_linux32.c VM source file
./interfaces/data-compression/ipp_gzip/vm/include
vm_affinity.h VM header file
vm_debug.h VM header file
vm_event.h VM header file
vm_file.h VM header file
vm_mmap.h VM header file
vm_mutex.h VM header file
vm_semaphore.h VM header file
vm_shared_object.h VM header file
vm_socket.h VM header file
vm_strings.h VM header file
vm_sys_info.h VM header file
vm_thread.h VM header file
vm_time.h VM header file
vm_types.h VM header file
./interfaces/data-compression/ipp_gzip/vm/include/sys
vm_file_unix.h VM header file
vm_types_linux32.h VM header file

How To Build Source Code

Setting Up Build Environment

Set up your build environment by creating an environment variable named IPPROOT that points to the root directory of your Intel® IPP installation.

The shared libraries are located in the sharedlib subdirectory of the root directory of the Intel® IPP installation (IPPROOT). This can be done by setting the LD_LIBRARY_PATH environment variable manually or by invoking the batch file "$IPPROOT/bin/ippvars.sh" from the Intel® IPP installation.

Building of a 32-bit Version

To build for a system based on the IA-32 architecture, run ./build_ia32.sh

 

The following files should be created as a result of correct building:

executable files:  ./bin/ia32_(compiler name)/ipp_gzip

Building of a 64-bit Version

To build for the Intel® Xeon® Processor with the Intel® 64 architecture based system, run ./build_intel64.sh
 

The following files should be created as a result of correct building:

executable files:  ./bin/intel64_(compiler name)/ipp_gzip

Note that with default settings, the interface will be built using the Intel C++ compiler. To change default settings, see Makefile for brief description of available variables and additional options. 

Running the Interface

To run this interface, the Intel® IPP shared libraries must be on the system's path. The libraries are located in the sharedlib subdirectory of the root directory of the Intel® IPP installation (IPPROOT). This can be done by setting the DYLD_LIBRARY_PATH environment variable manually or by invoking the batch file "$IPPROOT/tools/env/ippvars32.sh" from the Intel® IPP installation.

When built, the data-compression IPP_GZIP interface contains one executable file: ipp_gzip .

Type the following command at the command prompt : ipp_gzip -V

If IPP environment is set correctly, you must see the following version output:

Copyright © 2007-2009 Intel Corporation. All rights reserved.

IPP gzip . Version 2.0

 

Library: ippspxl.lib v7.0 beta build 91.13 (Jan 24 2009)

Try to get "-h" page using " ipp_gzip ?h":

Usage: ipp_gzip.exe [OPTION]... [FILE]... 
Compress or uncompress FILEs (by default, compress FILES in-place)  
Mandatory arguments to long options are mandatory for short options too.
    -c, --to-stdout  output to stdout 
    -c, --stdout     the same 
    -d, --decompress     decompress 
    -d, --uncompress     the same 
    -f, --force  force output file overwrite 
    -h, --help   print this text 
    -l, --list   list content of gzip archive 
    -n, --no-name    don't save/restore original filename/time 
    -N, --name   save/restore original filename/time 
    -r, --recursive  recurse into directories 
    -S, --suffix     make new suffix instead of .gz 
    -t, --test   test archive integrity 
    -T, --no-time    don't save/restore file date/timestamp 
    -v, --verbose    print some intermediate information 
    -V, --version    print current version number 
    -1, --fast   use faster compression method 
    -9, --best   use best compression method 
    -m, --num-threads    DEBUG: set number of threads to create 
    -j, --min-size       DEBUG: set minimum input file length to slice Input/Ouput options:
    -D, --DEBUG          DEBUG mode: prints some debug info 
    -w, --write-method   I/O: write method to use -w 0 (low/level) -w 1 (mmap) 
    -y, --read-method    I/O: read method to use -y 0 (low/level) -y 1(mmap)
    -u, --read-buffer    I/O: read buffer size in Kbytes 
    -i, --write-buffer   I/O: write buffer size in Kbytes 
    -s, --stat      display performance statistics
    -C, --clocks    display CPU clocks  

IPP_GZIP Command Syntax

IPP_GZIP command line syntax is compatible with the original GZIP utility syntax and has the following common format:

> ipp_gzip [option] [option] ? [file] [file] ?

Option

Description

Usage Example

-c
(or)
--to- stdout (or)
-- stdout

Uses " stdout " as application output file. " Stdout " is not always terminal. For example, when redirection is used "> file", " stdout " is specified file. Or, when pipe operator is used " | next_command ", " stdout " is pipe. Moreover, IPP_GZIP never puts compressed data to terminal-like stdout .

ipp_gzip ?c table.txt > mytable.gz
(compresses table.txt file and redirects " stdout " to mytable.gz file)
ipp_gzip ?c ?d mytable.gz
(decompresses mytable.gz file to screen)
ipp_gzip ?c table.txt | gzip ?t ?v
( compress table.txt with IPP_GZIP and tests the compressed data integrity with original GZIP)

-d
(or)
--decompress
(or)
--uncompress

Forces data decompression operation.

By default, when no "-d" option is specified, data processing is compression.

ipp_gzip ?d table.txt.gz
(decompresses table.txt.gz file into table.txt file)

-f
(or)
--force

Overwrites existing output file without any additional questions.
By default, IPP_GZIP asks for a user's approval to overwrite existing file with the same name

ipp_gzip ?d ?f table.txt.gz
(decompresses table.txt.gz into table.txt file and overwrites existing table.txt file if any)

-h
(or)
--help

Prints the list of options

ipp_gzip ?help

-l
(or)
--list

Prints the content of compressed file(s)

ipp_gzip ?l *. gz
(prints the content of all . gz files in current directory)

-n
(or)
--no-name

Does not save or restore original file name/date-time stamp.
By default, IPP_GZIP writes the name of the file it compresses into GZIP header, so, that the original file with its name could be restored even if the archive has been renamed. When "-n" option is used, IPP_GZIP creates decompressed file name from the name of the archive.

ipp_gzip ?n table.txt
(compresses table.txt file producing table.txt.gz file with no original name inside)

-N
(or)
--name

Saves original file name and date-time stamp (default case)

ipp_gzip ?N table.txt

-r
(or)
--recursive

When IPP_GZIP encounters a directory file, it processes all files inside this directory

ipp_gzip ?r archive
(if "archive" is a directory, compress all files ? other directories too ? inside "archive")

-S . suff
(or)
--suffix . suff

Uses specified ". suff " suffix instead of default ". gz ". Note that dot "." will NOT be added automatically

ipp_gzip ?S . gzip table.txt
(compresses table.txt into table.txt.gzip file)

-t
(or)
--test

Does not decompress the file, but tests its integrity. Notifies the user about all non-consistent problems inside the compressed file.

ipp_gzip ?t table.txt.gz

-v
(or)
--verbose

Prints some additional information after the following operations:
- compression (prints % % of compression)
- decompression (prints %% of compression)
- list (prints checksum, date stamp, compression method)

ipp_gzip ?d ?v table.txt.gz
(decompresses table.txt.gz file and prints the name of restored file and % value of compression)

-V
(or)
--version

Prints interface name, copyright information, target architecture, compiler optimization and library linkage. Does nothing extra.

 

-1
(or)
--fast

Uses the fastest compression method. Saves time during compression, but loses in compression ratio (and, produces larger files)

ipp_gzip - -fast table.txt

-9
(or)
--best

Uses the best compression method. Saves disk space, but loses in compression time (sometimes significantly)

ipp_gzip -9 table.txt

-m number
(or)
--num-threads number

Sets the number of active threads during multi-threaded operations.  See the next section for more details.

ipp_gzip ?m 8 huge_file.txt
(compresses huge_file.txt using 8 threads)

-j size
(or)
--min-size size

Sets the minimum file length that the parallel compression begins from. See the next section for more details.

ipp_gzip ?j 10000 small_file.dat
(if small_file.dat is less than 10000 bytes long, single-threaded compression is used)

-w number "-w 0" forces low-level i/o functions on file writes. "-w 1" forces memory mapping.  
-y number "-y 0" forces low-level i/o functions to be used during file reads
"-y 1" forces memory mapping functions
 
-u number
(or)
-i number
These options set the size of read (-u) or write (-i) buffers which will be used during i/o operarions. By default, number=64, i.e. 64KB buffers used. Sometimes, the increased sizes of i/o buffers improve the throughput of i/o operations. Use "-s" option to find the best combination of i/o methods and buffer size.  
-s Prints the number of CPU clocks used per input symbol (i.e. input byte). The lower values mean better performance.  
-C Prints the number of CPU clocks for testing purporse. The lower value mean better performance.  

IPP_GZIP In Multi-Thread Environment

Since the goal of IPP_GZIP is not only to use the IPP Data Compression functions for speeding-up the file processing, but also to fully utilize the benefits of the modern Intel® microprocessor architectures, it extensively uses multi-threading. Multi-threading has specific features depending on IPP_GZIP operation.

Compression

Compression is the most 'heavy' operation in terms of CPU resources; therefore, maximum benefit from multi-threading can be obtained, as it would be expected, during the compression. There are two ways of using multi-threading: multi-file threading and multi-chunk threading.

Multi-file threading is used when more than one file is specified on the command line. If we want to compress two files on a two-CPU computer (or on a single-unit Intel® Core™2 Duo processor computer), our natural decision will be to process each file in a separate thread and thus fully benefit from a dual-CPU computer. That is what IPP_GZIP does. For example:

        ??????? > ipp_gzip file1 file2

will compress file1 on one CPU and file2 on the other CPU. If our system has more than two CPUs, other CPUs will not be used. If number of files specified on the IPP_GZIP command line is more than number of available CPUs, all of them will be processed in parallel using existing CPUs. For example, file1 on CPU1, file 2 on CPU2, file3 on CPU1, etc.

Multi-chunk file processing is used when we process a single file on a multi-CPU computer. Thus, on a 4-CPU computer, the command line

        ??????? > ipp_gzip a-very-huge- file.dat

will split  the "a-very-huge- file.dat " file into 4 pieces (chunks) and will compress each chunk on separate CPUs combining processed data into a single output file " a-very-huge- file.dat.gz ". The compression ratio in this case will be, of course, a little bit worse than in the single-thread compression ? since LZ77 compression methods use statistical data (or pre-history) to compress better ? but this overhead (actually, 1-2%) is the cost of boosted compression performance ( 10x-20x times faster on 4/8-CPU computers vs. original GZIP compression speed).

The "-m" option can be used to control multi-thread operations. For example, by using "-m 2" on a 4-CPU computer we can limit IPP_GZIP to two threads. Or, vice versa, by using the "-m 4" option on a single-CPU computer we can produce archives as if they were compressed on a 4-core CPU. Of course, a forced multi-threading on a single-CPU computer will not speed-up the compression, but it will produce the archives which can be decompressed on a multi-CPU system and thus benefit from multi-CPU.

The "-j size" option controls multi-chunk compression. For example, if we are using a multiprocessor system, but the file to be processed is not big enough, we may not speed-up, but rather slow-down the compression because of thread creation/synchronization overhead. The default value of minimum file length is 256 KB and is defined by the "#define MIN_LENGTH_TO_SLICE ? "  value in the " ipp_gzip.h " file.

Decompression

Decompression is quite simple operation that does not require significant CPU resources, so the benefit from multi-thread decompression is not that big here. Decompression is almost a copy operation and is almost fully limited by input/output system performance. But, nevertheless, multi-thread decompression can be 2+ times faster than single-thread decompression.

Multi-file threading is the same as in the case of compression. If more than one file is specified on the IPP_GZIP command line, and the current system has more than one CPU, all files are decompressed in parallel.

Now about multi-chunk decompression. Of course, not every compressed file can be decompressed in parallel. The trick is that during multi-chunk compression, additional information about chunks ? chunk offsets and partial check sums ? is written to the comment field of the GZIP file header. This field is ignored by the original GZIP, so existence/absence of this field does not affect usual GZIP operations. This means that files compressed by IPP_GZIP using multiple chunks can be decompressed by the original GZIP tool.

As in the case of compression, the "-m <number>" option can control the number of processors used. For example, if we want to limit decompression to specified number of threads ? to save some CPU resources for other applications ? we can use "-m <less_thread_number> " option for that. If  the "-m" option is not specified, IPP_GZIP tries to use as much processors as possible. For example, the file previously compressed on a 4-CPU system using 4-thread processing is decompressed on a 2-CPU system using all two CPUs. If the number of chunks is less than the number of CPUs, other processors will not be used.

Compatibility

IPP_GZIP archive format is fully compatible with regular GZIP utility format (version 4.3 RFC 1952). Additional chunk/offset information is stored in comment field of GZIP archive header, which in general is not used by regular GZIP. This generally means, that archive files produced by GZIP may be processed by IPP_GZIP and vice versa. Starting from IPP 6.1 Gold Update 1 IPP_GZIP can work with files larger than 2 GB.  Along with this the format of service information in GZIP comment field was slightly changed. So, archive files produced by previous version of IPP_GZIP will not be decompressed by new version of IPP_GZIP in multi-threaded mode - you can force using of single-threaded decompression using "-m 1 -d" command line options or do nothing, because when IPP_GZIP meets improper comment field format it starts single-threaded mode decompression.

Performance Notes

IPP_GZIP interface is just a interface and not fully tuned for performance. Nevertheless, using of multi-threaded mode allows to significantly increase compression/decompression performance. Along with performance boost due to multi-core usage, additional performance benefit can be obtained from i/o and pure computing overlapping. Thus, overall performance boost can be even greater than single_thread_performance - number_of_cores. Nevertheless, using of memory mapping ("-y 1" and "-w 1") options must be done with additional care, because memory mapping brings some extra OS virtual memory functionality overload. When data file size is less or equal to physical memory size on your computer, memory mapping usage gives additional benefit, because IPP_GZIP writes target file in parallel with all available threads. But, when data file size is more/much more than your computer physical memory, using of memory mapping can significantly slow down or even hang your computer. So, do some performance experiments combining i/o methods and buffer sizes to choose best combination of options. In some cases you may want to limit the number of IPP_GZIP threads (with "-m" option) to lower computer's i/o subsystem load.

Technical Support and Feedback

To receive support or provide feedback refer to the "Technical Support and Feedback" section of the release notes (ReleaseNotes.htm) provided in the Intel® IPP product installation.

Your feedback on the Intel IPP samples is very important to us and your input will be considered for future releases. The Intel IPP sample code is intended only as an example of how to use the APIs to implement algorithms in different development environments. Please submit problems with installation, compiling, linking, runtime errors or incorrect output to Intel® Premier Support.

You can also share and discuss the experience of IPP sample usage with other developers at Intel Software Developer Forum.

Optimization Notice

The Intel® Integrated Performance Primitives (Intel® IPP) library contains functions that are more highly optimized for Intel microprocessors than for other microprocessors. While the functions in the Intel® IPP library offer optimizations for both Intel and Intel-compatible microprocessors, depending on your code and other factors, you will likely get extra performance on Intel microprocessors.

While the paragraph above describes the basic optimization approach for the Intel® IPP library as a whole, the library may or may not be optimized to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include Intel® Streaming SIMD Extensions 2 (Intel® SSE2), Intel® Streaming SIMD Extensions 3 (Intel® SSE3), and Supplemental Streaming SIMD Extensions 3 (Intel® SSSE3) instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors.

Intel recommends that you evaluate other library products to determine which best meets your requirements.

Disclaimer

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/#/en_US_01
Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. See http://www.intel.com/products/processor_number for details.

BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside, i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core, Intel Inside, Intel Inside logo, Intel NetBurst, Intel NetMerge, Intel NetStructure, Intel SingleDriver, Intel SpeedStep, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo, InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium, Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside, vPro Inside, VTune, Xeon, and Xeon Inside are trademarks of Intel Corporation in the U.S. and other countries.
* Other names and brands may be claimed as the property of others.

Microsoft, Windows, Visual Studio, Visual C++, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.