License Definitions
Overview
Intel IPP Functions
Used In The Interface
System Requirements
Package Contents
How To Build Source Code
Running the Interface
IPP_GZIP Command Syntax
IPP_GZIP In Multi-thread Environment
Compression
Decompression
Compatibility
Performance_Notes
Technical Support and Feedback
Disclaimer
By downloading and installing this sample, you hereby agree that the accompanying Materials are being provided to you under the terms and conditions of the End User License Agreement for the Intel® Integrated Performance Primitives product previously accepted by you. Please refer to the file ippEULA.rtf or ippEULA.txt located in the root directory of your Intel® IPP product installation for more information.
The Data Compression-based IPP_GZIP Interface (IPP_GZIP) illustrates the way of implementing effective lossless data compression solution by using Intel® Integrated Performance Primitives Data Compression domain API. Additionally, this interface shows the ways of parallelizing a user application using Intel® Virtual Machine to fully benefit from modern Intel® microprocessor architectures.
The interface uses dictionary-based IPP functions implementing Lempel-Ziv (LZ77) algorithm and original GZIP data formats according to RFC 1950, 1951 and 1952 specifications.
The result compressed data formats are fully compatible with the original GZIP formats, so the utilities are interchangeable. You can compress data with IPP_GZIP and decompress it with GZIP, or, vice versa, compress with GZIP and decompress using GZIP. However, to benefit from using multiprocessor/multicore architectures, use IPP_GZIP.
Additional information on this software as well as other Intel software performance products is available at http://www.intel.com/software/products/.
IPP_GZIP interface uses the following functions from the Intel® Performance Primitives Data Compression library:
IPP_GZIP Function |
IPP Function |
Function Description |
Compression |
ippsDeflateLZ77_8u |
Performs LZ77 encoding |
ippsDeflateHuff_8u |
Performs Huffman encoding |
|
ippsCRC32_8u |
Computes the CRC32 checksum for the source data buffer |
|
Decompression/Test |
ippsInflate_8u |
Performs LZ77 decoding |
ippsInflateBuildHuffTable |
Performs Huffman decoding |
The Intel® IPP Data Compression IPP_GZIP Interface for Linux* contains the following files:
./interfaces | |
---|---|
ippEULA.rtf or ippEULA.txt | End User License Agreement |
support.txt | Contains technical support information |
./interfaces/data-compression/ipp_gzip | |
build_ia32.sh | Shell script to build ipp_gzip interface on Linux systems based on the IA-32 architecture |
build_intel64.sh | Shell script to build ipp_gzip interface on Linux systems based on the Intel® 64 architecture |
Makefile | Makefile to build ipp_gzip interface |
readme.htm | This file |
./interfaces/data-compression/ipp_gzip/src | |
ipp_gzip.c | Interface's main function, upper-level dispatching functions and directory processing function |
ipp_gzip_deflate.c | Single-threaded and multi-threaded data compression function |
ipp_gzip_deflate_mt.c | Multi-threaded data compression function and service functions |
ipp_gzip_inflate.c | Single-threaded data decompression function |
ipp_gzip_inflate_mt.c | Multi-threaded data decompression function and service functions |
ipp_gzip_io.c | File input/output functions |
ipp_gzip_utils.c | Several common functions |
./interfaces/data-compression/ipp_gzip/include | |
ipp_gzip.h | Common header file for the interface |
ipp_gzip_deflate.h | Definitions and tables for deflate process |
version.h | Application version text |
./interfaces/data-compression/ipp_gzip/vm/src | |
vm_affinity_common.c | VM source file |
vm_affinity_unix.c | VM source file |
vm_event_linux32.c | VM source file |
vm_file.c | VM source file |
vm_file_unix.c | VM source file |
vm_mmap_linux32.c | VM source file |
vm_mutex_linux32.c | VM source file |
vm_semaphore_linux32.c | VM source file |
vm_shared_object_linux32.c | VM source file |
vm_socket_linux32.c | VM source file |
vm_sys_info_linux32.c | VM source file |
vm_thread_linux32.c | VM source file |
vm_time_linux32.c | VM source file |
./interfaces/data-compression/ipp_gzip/vm/include | |
vm_affinity.h | VM header file |
vm_debug.h | VM header file |
vm_event.h | VM header file |
vm_file.h | VM header file |
vm_mmap.h | VM header file |
vm_mutex.h | VM header file |
vm_semaphore.h | VM header file |
vm_shared_object.h | VM header file |
vm_socket.h | VM header file |
vm_strings.h | VM header file |
vm_sys_info.h | VM header file |
vm_thread.h | VM header file |
vm_time.h | VM header file |
vm_types.h | VM header file |
./interfaces/data-compression/ipp_gzip/vm/include/sys | |
vm_file_unix.h | VM header file |
vm_types_linux32.h | VM header file |
Set up your build environment by creating an environment variable named IPPROOT that points to the root directory of your Intel® IPP installation.
The shared libraries are located in the sharedlib subdirectory of the root directory of the Intel® IPP installation (IPPROOT). This can be done by setting the LD_LIBRARY_PATH environment variable manually or by invoking the batch file "$IPPROOT/bin/ippvars.sh" from the Intel® IPP installation.
To build for a system based on the IA-32 architecture, run ./build_ia32.sh
The following files should be created as a result of correct building:
executable files: ./bin/ia32_(compiler name)/ipp_gzip
To build for the Intel® Xeon® Processor with the Intel® 64 architecture based system, run ./build_intel64.sh
The following files should be created as a result of correct building:
executable files: ./bin/intel64_(compiler name)/ipp_gzip
Note that with default settings, the interface will be built using the Intel C++ compiler. To change default settings, see Makefile for brief description of available variables and additional options.
To run this interface, the Intel® IPP shared libraries must be on the system's path. The libraries are located in the sharedlib subdirectory of the root directory of the Intel® IPP installation (IPPROOT). This can be done by setting the DYLD_LIBRARY_PATH environment variable manually or by invoking the batch file "$IPPROOT/tools/env/ippvars32.sh" from the Intel® IPP installation.
When built, the data-compression IPP_GZIP interface contains one executable file: ipp_gzip .
Type the following command at the command prompt : ipp_gzip -V
If IPP environment is set correctly, you must see the following version output:
Copyright © 2007-2009 Intel Corporation. All rights reserved.
IPP gzip . Version 2.0
Library: ippspxl.lib v7.0 beta build 91.13 (Jan 24 2009)
Try to get "-h" page using " ipp_gzip ?h":
Usage: ipp_gzip.exe [OPTION]... [FILE]... Compress or uncompress FILEs (by default, compress FILES in-place)
Mandatory arguments to long options are mandatory for short options too. -c, --to-stdout output to stdout -c, --stdout the same -d, --decompress decompress -d, --uncompress the same -f, --force force output file overwrite -h, --help print this text -l, --list list content of gzip archive -n, --no-name don't save/restore original filename/time -N, --name save/restore original filename/time -r, --recursive recurse into directories -S, --suffix make new suffix instead of .gz -t, --test test archive integrity -T, --no-time don't save/restore file date/timestamp -v, --verbose print some intermediate information -V, --version print current version number -1, --fast use faster compression method -9, --best use best compression method -m, --num-threads DEBUG: set number of threads to create -j, --min-size DEBUG: set minimum input file length to slice Input/Ouput options: -D, --DEBUG DEBUG mode: prints some debug info -w, --write-method I/O: write method to use -w 0 (low/level) -w 1 (mmap) -y, --read-method I/O: read method to use -y 0 (low/level) -y 1(mmap) -u, --read-buffer I/O: read buffer size in Kbytes -i, --write-buffer I/O: write buffer size in Kbytes -s, --stat display performance statistics -C, --clocks display CPU clocks
IPP_GZIP command line syntax is compatible with the original GZIP utility syntax and has the following common format:
>
ipp_gzip
[option] [option] ? [file] [file] ?
Option |
Description |
Usage Example |
-c
|
Uses " stdout " as application output file. " Stdout " is not always terminal. For example, when redirection is used "> file", " stdout " is specified file. Or, when pipe operator is used " | next_command ", " stdout " is pipe. Moreover, IPP_GZIP never puts compressed data to terminal-like stdout . |
ipp_gzip
?c table.txt >
mytable.gz
|
-d
|
Forces data decompression operation.
|
ipp_gzip
?d
table.txt.gz
|
-f
|
Overwrites existing output file without any additional questions.
|
ipp_gzip
?d ?f
table.txt.gz
|
-h
|
Prints the list of options |
ipp_gzip ?help |
-l
|
Prints the content of compressed file(s) |
ipp_gzip
?l *.
gz
|
-n
|
Does not save or restore original file name/date-time stamp.
|
ipp_gzip
?n table.txt
|
-N
|
Saves original file name and date-time stamp (default case) |
ipp_gzip ?N table.txt |
-r
|
When IPP_GZIP encounters a directory file, it processes all files inside this directory |
ipp_gzip
?r archive
|
-S .
suff
|
Uses specified ". suff " suffix instead of default ". gz ". Note that dot "." will NOT be added automatically |
ipp_gzip
?S .
gzip
table.txt
|
-t
|
Does not decompress the file, but tests its integrity. Notifies the user about all non-consistent problems inside the compressed file. |
ipp_gzip
?t
table.txt.gz
|
-v
|
Prints some additional information after the following operations:
|
ipp_gzip
?d ?v
table.txt.gz
|
-V
|
Prints interface name, copyright information, target architecture, compiler optimization and library linkage. Does nothing extra. |
|
-1
|
Uses the fastest compression method. Saves time during compression, but loses in compression ratio (and, produces larger files) |
ipp_gzip - -fast table.txt |
-9
|
Uses the best compression method. Saves disk space, but loses in compression time (sometimes significantly) |
ipp_gzip -9 table.txt |
-m
number
|
Sets the number of active threads during multi-threaded operations. See the next section for more details. |
ipp_gzip
?m 8 huge_file.txt
|
-j
size
|
Sets the minimum file length that the parallel compression begins from. See the next section for more details. |
ipp_gzip
?j 10000
small_file.dat
|
-w number | "-w 0" forces low-level i/o functions on file writes. "-w 1" forces memory mapping. | |
-y number |
"-y 0" forces low-level i/o functions to be used during file reads "-y 1" forces memory mapping functions |
|
-u number
(or) -i number |
These options set the size of read (-u) or write (-i) buffers which will be used during i/o operarions. By default, number=64, i.e. 64KB buffers used. Sometimes, the increased sizes of i/o buffers improve the throughput of i/o operations. Use "-s" option to find the best combination of i/o methods and buffer size. | |
-s | Prints the number of CPU clocks used per input symbol (i.e. input byte). The lower values mean better performance. | |
-C | Prints the number of CPU clocks for testing purporse. The lower value mean better performance. |
Since the goal of IPP_GZIP is not only to use the IPP Data Compression functions for speeding-up the file processing, but also to fully utilize the benefits of the modern Intel® microprocessor architectures, it extensively uses multi-threading. Multi-threading has specific features depending on IPP_GZIP operation.
Compression is the most 'heavy' operation in terms of CPU resources; therefore, maximum benefit from multi-threading can be obtained, as it would be expected, during the compression. There are two ways of using multi-threading: multi-file threading and multi-chunk threading.
Multi-file threading is used when more than one file is specified on the command line. If we want to compress two files on a two-CPU computer (or on a single-unit Intel® Core™2 Duo processor computer), our natural decision will be to process each file in a separate thread and thus fully benefit from a dual-CPU computer. That is what IPP_GZIP does. For example:
??????? > ipp_gzip file1 file2
will compress file1 on one CPU and file2 on the other CPU. If our system has more than two CPUs, other CPUs will not be used. If number of files specified on the IPP_GZIP command line is more than number of available CPUs, all of them will be processed in parallel using existing CPUs. For example, file1 on CPU1, file 2 on CPU2, file3 on CPU1, etc.
Multi-chunk file processing is used when we process a single file on a multi-CPU computer. Thus, on a 4-CPU computer, the command line
??????? > ipp_gzip a-very-huge- file.dat
will split the "a-very-huge- file.dat " file into 4 pieces (chunks) and will compress each chunk on separate CPUs combining processed data into a single output file " a-very-huge- file.dat.gz ". The compression ratio in this case will be, of course, a little bit worse than in the single-thread compression ? since LZ77 compression methods use statistical data (or pre-history) to compress better ? but this overhead (actually, 1-2%) is the cost of boosted compression performance ( 10x-20x times faster on 4/8-CPU computers vs. original GZIP compression speed).
The "-m" option can be used to control multi-thread operations. For example, by using "-m 2" on a 4-CPU computer we can limit IPP_GZIP to two threads. Or, vice versa, by using the "-m 4" option on a single-CPU computer we can produce archives as if they were compressed on a 4-core CPU. Of course, a forced multi-threading on a single-CPU computer will not speed-up the compression, but it will produce the archives which can be decompressed on a multi-CPU system and thus benefit from multi-CPU.
The "-j size" option controls multi-chunk compression. For example, if we are using a multiprocessor system, but the file to be processed is not big enough, we may not speed-up, but rather slow-down the compression because of thread creation/synchronization overhead. The default value of minimum file length is 256 KB and is defined by the "#define MIN_LENGTH_TO_SLICE ? " value in the " ipp_gzip.h " file.
Decompression is quite simple operation that does not require significant CPU resources, so the benefit from multi-thread decompression is not that big here. Decompression is almost a copy operation and is almost fully limited by input/output system performance. But, nevertheless, multi-thread decompression can be 2+ times faster than single-thread decompression.
Multi-file threading is the same as in the case of compression. If more than one file is specified on the IPP_GZIP command line, and the current system has more than one CPU, all files are decompressed in parallel.
Now about multi-chunk decompression. Of course, not every compressed file can be decompressed in parallel. The trick is that during multi-chunk compression, additional information about chunks ? chunk offsets and partial check sums ? is written to the comment field of the GZIP file header. This field is ignored by the original GZIP, so existence/absence of this field does not affect usual GZIP operations. This means that files compressed by IPP_GZIP using multiple chunks can be decompressed by the original GZIP tool.
As in the case of compression, the "-m <number>" option can control the number of processors used. For example, if we want to limit decompression to specified number of threads ? to save some CPU resources for other applications ? we can use "-m <less_thread_number> " option for that. If the "-m" option is not specified, IPP_GZIP tries to use as much processors as possible. For example, the file previously compressed on a 4-CPU system using 4-thread processing is decompressed on a 2-CPU system using all two CPUs. If the number of chunks is less than the number of CPUs, other processors will not be used.
IPP_GZIP archive format is fully compatible with regular GZIP utility format (version 4.3 RFC 1952). Additional chunk/offset information is stored in comment field of GZIP archive header, which in general is not used by regular GZIP. This generally means, that archive files produced by GZIP may be processed by IPP_GZIP and vice versa. Starting from IPP 6.1 Gold Update 1 IPP_GZIP can work with files larger than 2 GB. Along with this the format of service information in GZIP comment field was slightly changed. So, archive files produced by previous version of IPP_GZIP will not be decompressed by new version of IPP_GZIP in multi-threaded mode - you can force using of single-threaded decompression using "-m 1 -d" command line options or do nothing, because when IPP_GZIP meets improper comment field format it starts single-threaded mode decompression.
IPP_GZIP interface is just a interface and not fully tuned for performance. Nevertheless, using of multi-threaded mode allows to significantly increase compression/decompression performance. Along with performance boost due to multi-core usage, additional performance benefit can be obtained from i/o and pure computing overlapping. Thus, overall performance boost can be even greater than single_thread_performance - number_of_cores. Nevertheless, using of memory mapping ("-y 1" and "-w 1") options must be done with additional care, because memory mapping brings some extra OS virtual memory functionality overload. When data file size is less or equal to physical memory size on your computer, memory mapping usage gives additional benefit, because IPP_GZIP writes target file in parallel with all available threads. But, when data file size is more/much more than your computer physical memory, using of memory mapping can significantly slow down or even hang your computer. So, do some performance experiments combining i/o methods and buffer sizes to choose best combination of options. In some cases you may want to limit the number of IPP_GZIP threads (with "-m" option) to lower computer's i/o subsystem load.
To receive support or provide feedback refer to the "Technical Support and Feedback" section of the release notes (ReleaseNotes.htm) provided in the Intel® IPP product installation.
Your feedback on the Intel IPP samples is very important to us and your input will be considered for future releases. The Intel IPP sample code is intended only as an example of how to use the APIs to implement algorithms in different development environments. Please submit problems with installation, compiling, linking, runtime errors or incorrect output to Intel® Premier Support.
You can also share and discuss the experience of IPP sample usage with other developers at Intel Software Developer Forum.
Optimization Notice |
---|
The Intel® Integrated Performance Primitives (Intel® IPP) library contains functions that are more highly optimized for Intel microprocessors than for other microprocessors. While the functions in the Intel® IPP library offer optimizations for both Intel and Intel-compatible microprocessors, depending on your code and other factors, you will likely get extra performance on Intel microprocessors. While the paragraph above describes the basic optimization approach for the Intel® IPP library as a whole, the library may or may not be optimized to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include Intel® Streaming SIMD Extensions 2 (Intel® SSE2), Intel® Streaming SIMD Extensions 3 (Intel® SSE3), and Supplemental Streaming SIMD Extensions 3 (Intel® SSSE3) instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Intel recommends that you evaluate other library products to determine which best meets your requirements. |
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS.
NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL
PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S
TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY
WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING
TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES
RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT
OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED
NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD
CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.
Intel may make changes to specifications and product descriptions at any time,
without notice. Designers must not rely on the absence or characteristics of any
features or instructions marked "reserved" or "undefined." Intel reserves these
for future definition and shall have no responsibility whatsoever for conflicts
or incompatibilities arising from future changes to them. The information here
is subject to change without notice. Do not finalize a design with this
information.
The products described in this document may contain design defects or errors
known as errata which may cause the product to deviate from published
specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest
specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this
document, or other Intel literature, may be obtained by calling 1-800-548-4725,
or go to:
http://www.intel.com/#/en_US_01
Intel processor numbers are not a measure of performance. Processor numbers
differentiate features within each processor family, not across different
processor families. See
http://www.intel.com/products/processor_number
for details.
BunnyPeople, Celeron, Celeron Inside, Centrino, Centrino Inside, Core Inside,
i960, Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core,
Intel Inside, Intel Inside logo, Intel NetBurst, Intel NetMerge,
Intel NetStructure, Intel SingleDriver, Intel SpeedStep,
Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo,
Intel StrataFlash, Intel Viiv, Intel vPro, Intel XScale, InTru, the InTru logo,
InTru soundmark, Itanium, Itanium Inside, MCS, MMX, Moblin, Pentium,
Pentium Inside, skoool, the skoool logo, Sound Mark, The Journey Inside,
vPro Inside, VTune, Xeon, and Xeon Inside
are trademarks of Intel Corporation in the U.S. and other countries.
* Other names and brands may be claimed as the property of others.
Microsoft, Windows, Visual Studio, Visual C++, and the Windows logo are trademarks, or registered trademarks of Microsoft Corporation in the United States and/or other countries.
Copyright (C) 2002-2010, Intel Corporation. All rights reserved.