Skip to content

libaec
libaec
Commit d6f47b1c authored by Mathis Rosenhauer's avatar Mathis Rosenhauer
Browse files
Options

Converted changelog to markdown

parent ba3a89a1
Hide whitespace changes
Inline Side-by-side
Showing with 84 additions and 345 deletions
CHANGELOG.md 0 → 100644
View file @ d6f47b1c
# libaec Changelog
All notable changes to libaec will be documented in this file.
## [Unreleased]
### Fixed
- Potential security vulnerabilities in decoder exposed by libFuzzer.
### Added
- Fuzz target for decoding and encoding.
### Changed
- Improved cmake support.
## [1.0.0] - 2016-11-16
### Added
- Include CCSDS test data with libaec. See THANKS.
### Changed
- Better compatibility with OSX for make check.
- Allow Cygwin to build DLLs.
## [0.3.4] - 2016-08-16
### Fixed
- Pad incomplete last line when in SZ compatibility mode.
## [0.3.3] - 2016-05-12
### Fixed
- Bug with zero blocks in the last RSI (reference sample interval)
when data size is not a multiple of RSIs or segments (64 blocks) and
the zero region reaches a segment boundary.
- More robust error handling.
### Changed
- Vectorization improvement for Intel compiler.
- Better compatibility with netcdf's build process.
## [0.3.2] - 2015-02-04
### Changed
- Allow nonconforming block sizes in SZ mode.
- Performance improvement for decoder.
## [0.3.1] - 2014-10-23
### Fixed
- Allow incomplete scanlines in SZ mode.
## [0.3] - 2014-08-06
### Changed
- Performance improvement for encoding pre-precessed data.
- More efficient coding of second extension if reference sample is
present.
- Port library to Windows (Visual Studio).
### Added
- Support building with CMake.
- Benchmarking target using ECHAM data (make bench).
## [0.2] - 2014-02-12
### Fixed
- Incorrect length calculation in assessment of Second Extension
coding.
- Unlimited encoding of fundamental sequences.
- Handle corrupted compressed data more gracefully.
### Added
- Additional testing with official CCSDS sample data.
- Support restricted coding options from latest standard.
### Changed
- Facilitate generation of SIMD instructions by compiler.
## [0.1] - 2013-05-21
### Added
- Initial release.
COPYING deleted 100644 → 0
View file @ ba3a89a1
Copyright 2012 - 2016
Mathis Rosenhauer, Moritz Hanke, Joerg Behrens
Deutsches Klimarechenzentrum GmbH
Bundesstr. 45a
20146 Hamburg
Germany
Luis Kornblueh
Max-Planck-Institut fuer Meteorologie
Bundesstr. 53
20146 Hamburg
Germany
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
ChangeLog deleted 100644 → 0
View file @ ba3a89a1
2016-11-16 Mathis Rosenhauer <rosenhauer@dkrz.de>
v1.0.0
Include CCSDS test data with libaec. See THANKS
Better compatibility with OSX for make check
Allow Cygwin to build DLLs
2016-08-16 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.3.4
Pad incomplete last line when in SZ compatibility mode
2016-05-12 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.3.3
Fixed bug with zero blocks in the last RSI (reference sample
interval) when data size is not a multiple of RSIs or segments (64
blocks) and the zero region reaches a segment boundary.
Vectorization improvement for Intel compiler
More robust error handling
Better compatibility with netcdf's build process
2015-02-04 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.3.2
Allow nonconforming block sizes in SZ mode.
Performance improvement for decoder.
2014-10-23 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.3.1
Allow incomplete scanlines in SZ mode.
2014-08-06 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.3
Performance improvement for encoding pre-precessed data.
More efficient coding of second extension if reference
sample is present.
Port library to Windows (Visual Studio).
Support building with CMake.
Benchmarking target using ECHAM data (make bench)
2014-02-12 Mathis Rosenhauer <rosenhauer@dkrz.de>
v0.2
Fixed incorrect length calculation in assessment of Second
Extension coding.
Unlimited encoding of fundamental sequences.
Additional testing with official CCSDS sample data.
Handle corrupted compressed data more gracefully.
Support restricted coding options from latest standard.
Facilitate generation of SIMD instructions by compiler.
2013-05-21 Mathis Rosenhauer <rosenhauer@dkrz.de>
Initial relaease: v0.1.
Makefile.am
View file @ d6f47b1c
ACLOCAL_AMFLAGS = -I m4
SUBDIRS = src tests
EXTRA_DIST = doc/patent.txt CMakeLists.txt cmake/config.h.in \
cmake/macros.cmake README.SZIP Copyright.txt data
cmake/macros.cmake README.md README.SZIP CHANGELOG.am Copyright.txt data
sampledata = 121B2TestData
sampledata_url = http://cwe.ccsds.org/sls/docs/SLS-DC/BB121B2TestData/$(sampledata).zip
......
README deleted 100644 → 0
View file @ ba3a89a1
**********************************************************************
libaec - Adaptive Entropy Coding library
**********************************************************************
Libaec provides fast lossless compression of 1 up to 32 bit wide
signed or unsigned integers (samples). The library achieves best
results for low entropy data as often encountered in space imaging
instrument data or numerical model output from weather or climate
simulations. While floating point representations are not directly
supported, they can also be efficiently coded by grouping exponents
and mantissa.
Libaec implements Golomb Rice coding as defined in the Space Data
System Standard documents 121.0-B-2 [1] and 120.0-G-2[2].
**********************************************************************
Patent
**********************************************************************
In doc/patent.txt a clarification on potentially applying
intellectual property rights is given.
**********************************************************************
Installation
**********************************************************************
See INSTALL for details.
**********************************************************************
Encoding
**********************************************************************
In this context efficiency refers to the size of the encoded
data. Performance refers to the time it takes to encode data.
Suppose you have an array of 32 bit signed integers you want to
compress. The pointer pointing to the data shall be called *source,
output goes into *dest.
#include <libaec.h>
...
struct aec_stream strm;
int32_t *source;
unsigned char *dest;
/* input data is 32 bits wide */
strm.bits_per_sample = 32;
/* define a block size of 16 */
strm.block_size = 16;
/* the reference sample interval is set to 128 blocks */
strm.rsi = 128;
/* input data is signed and needs to be preprocessed */
strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;
/* pointer to input */
strm.next_in = (unsigned char *)source;
/* length of input in bytes */
strm.avail_in = source_length * sizeof(int32_t);
/* pointer to output buffer */
strm.next_out = dest;
/* length of output buffer in bytes */
strm.avail_out = dest_length;
/* initialize encoding */
if (aec_encode_init(&strm) != AEC_OK)
return 1;
/* Perform encoding in one call and flush output. */
/* In this example you must be sure that the output */
/* buffer is large enough for all compressed output */
if (aec_encode(&strm, AEC_FLUSH) != AEC_OK)
return 1;
/* free all resources used by encoder */
aec_encode_end(&strm);
...
block_size can vary from 8 to 64 samples. Smaller blocks allow the
compression to adapt more rapidly to changing source
statistics. Larger blocks create less overhead but can be less
efficient if source statistics change across the block.
rsi sets the reference sample interval. A large RSI will improve
performance and efficiency. It will also increase memory requirements
since internal buffering is based on RSI size. A smaller RSI may be
desirable in situations where each RSI will be packetized and possible
error propagation has to be minimized.
Flags:
AEC_DATA_SIGNED: input data are signed integers. Specifying this
correctly increases compression efficiency. Default is unsigned.
AEC_DATA_PREPROCESS: preprocessing input will improve compression
efficiency if data samples are correlated. It will only cost
performance for no gain in efficiency if the data is already
uncorrelated.
AEC_DATA_MSB: input data is stored most significant byte first
i.e. big endian. You have to specify AEC_DATA_MSB even if your host
architecture is big endian. Default is little endian on all
architectures.
AEC_DATA_3BYTE: the 17 to 24 bit input data is stored in three
bytes. This flag has no effect for other sample sizes.
AEC_RESTRICTED: use a restricted set of code options. This option is
only valid for bits_per_sample <= 4.
AEC_PAD_RSI: assume that the encoded RSI is padded to the next byte
boundary while decoding. The preprocessor macro ENABLE_RSI_PADDING
needs to be defined while compiling for the encoder to honour this
flag.
Data size:
The following rules apply for deducing storage size from sample size
(bits_per_sample):
sample size storage size
1 - 8 bits 1 byte
9 - 16 bits 2 bytes
17 - 24 bits 3 bytes (only if AEC_DATA_3BYTE is set)
25 - 32 bits 4 bytes (if AEC_DATA_3BYTE is set)
17 - 32 bits 4 bytes (if AEC_DATA_3BYTE is not set)
If a sample requires less bits than the storage size provides, then
you have to make sure that unused bits are not set. Libaec does not
check this for performance reasons and will produce undefined output
if unused bits are set. All input data must be a multiple of the
storage size in bytes. Remaining bytes which do not form a complete
sample will be ignored.
Libaec accesses next_in and next_out buffers only bytewise. There are
no alignment requirements for these buffers.
Flushing:
aec_encode can be used in a streaming fashion by chunking input and
output and specifying AEC_NO_FLUSH. The function will return if either
the input runs empty or the output buffer is full. The calling
function can check avail_in and avail_out to see what occurred. The
last call to aec_encode() must set AEC_FLUSH to drain all
output. aec.c is an example of streaming usage of encoding and
decoding.
Output:
Encoded data will be written to the buffer submitted with
next_out. The length of the compressed data is total_out.
See libaec.h for a detailed description of all relevant structure
members and constants.
**********************************************************************
Decoding
**********************************************************************
Using decoding is very similar to encoding, only the meaning of input
and output is reversed.
#include <libaec.h>
...
struct aec_stream strm;
/* this is now the compressed data */
unsigned char *source;
/* here goes the uncompressed result */
int32_t *dest;
strm.bits_per_sample = 32;
strm.block_size = 16;
strm.rsi = 128;
strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;
strm.next_in = source;
strm.avail_in = source_length;
strm.next_out = (unsigned char *)dest;
strm.avail_out = dest_lenth * sizeof(int32_t);
if (aec_decode_init(&strm) != AEC_OK)
return 1;
if (aec_decode(&strm, AEC_FLUSH) != AEC_OK)
return 1;
aec_decode_end(&strm);
...
It is strongly recommended that the size of the output buffer
(next_out) is a multiple of the storage size in bytes. If the buffer
is not a multiple of the storage size and the buffer gets filled to
the last sample, the error code AEC_MEM_ERROR is returned.
It is essential for decoding that parameters like bits_per_sample,
block_size, rsi, and flags are exactly the same as they were for
encoding. Libaec does not store these parameters in the coded stream
so it is up to the calling program to keep the correct parameters
between encoding and decoding.
The actual values of coding parameters are in fact only relevant for
efficiency and performance. Data integrity only depends on consistency
of the parameters.
**********************************************************************
References
**********************************************************************
[1] Consultative Committee for Space Data Systems. Lossless Data
Compression. Recommendation for Space Data System Standards, CCSDS
121.0-B-2. Blue Book. Issue 2. Washington, D.C.: CCSDS, May 2012.
http://public.ccsds.org/publications/archive/121x0b2.pdf
[2] Consultative Committee for Space Data Systems. Lossless Data
Compression. Recommendation for Space Data System Standards, CCSDS
120.0-G-3. Green Book. Issue 3. Washington, D.C.: CCSDS, April
2013.
http://public.ccsds.org/publications/archive/120x0g3.pdf
configure.ac
View file @ d6f47b1c
......@@ -8,7 +8,7 @@ AC_CONFIG_HEADERS(config/config.h)
LT_INIT
gl_VISIBILITY
AM_INIT_AUTOMAKE
AM_INIT_AUTOMAKE([foreign])
# Checks for programs.
AC_PROG_CC
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment