README 7.79 KB
Newer Older
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
1 2 3
**********************************************************************
 libaec - Adaptive Entropy Coding library
**********************************************************************
4

5
Libaec provides fast lossless compression of 1 up to 32 bit wide
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
6 7 8 9 10 11
signed or unsigned integers (samples). The library achieves best
results for low entropy data as often encountered in space imaging
instrument data or numerical model output from weather or climate
simulations. While floating point representations are not directly
supported, they can also be efficiently coded by grouping exponents
and mantissa.
12

13
Libaec implements Golomb Rice coding as defined in the Space Data
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
14
System Standard documents 121.0-B-2 [1] and 120.0-G-2[2].
15

16

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
17
**********************************************************************
18
 Patent
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
19
**********************************************************************
20

21
In doc/patent.txt a clarification on potentially applying
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
22 23 24 25 26 27 28
intellectual property rights is given.


**********************************************************************
 Installation
**********************************************************************

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
29
See INSTALL for details.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49


**********************************************************************
 Encoding
**********************************************************************

In this context efficiency refers to the size of the encoded
data. Performance refers to the time it takes to encode data.

Suppose you have an array of 32 bit signed integers you want to
compress. The pointer pointing to the data shall be called *source,
output goes into *dest.

#include <libaec.h>

...
    struct aec_stream strm;
    int32_t *source;
    unsigned char *dest;

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
50
    /* input data is 32 bits wide */
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
    strm.bits_per_sample = 32;

    /* define a block size of 16 */
    strm.block_size = 16;

    /* the reference sample interval is set to 128 blocks */
    strm.rsi = 128;

    /* input data is signed and needs to be preprocessed */
    strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;

    /* pointer to input */
    strm.next_in = (unsigned char *)source;

    /* length of input in bytes */
66
    strm.avail_in = source_length * sizeof(int32_t);
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
67 68

    /* pointer to output buffer */
69
    strm.next_out = dest;
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
70 71

    /* length of output buffer in bytes */
72
    strm.avail_out = dest_length;
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87

    /* initialize encoding */
    if (aec_encode_init(&strm) != AEC_OK)
        return 1;

    /* Perform encoding in one call and flush output. */
    /* In this example you must be sure that the output */
    /* buffer is large enough for all compressed output */
    if (aec_encode(&strm, AEC_FLUSH) != AEC_OK)
        return 1;

    /* free all resources used by encoder */
    aec_encode_end(&strm);
...

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
88
block_size can vary from 8 to 64 samples. Smaller blocks allow the
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
89 90 91
compression to adapt more rapidly to changing source
statistics. Larger blocks create less overhead but can be less
efficient if source statistics change across the block.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
92 93

rsi sets the reference sample interval. A large RSI will improve
94 95
performance and efficiency. It will also increase memory requirements
since internal buffering is based on RSI size. A smaller RSI may be
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
96
desirable in situations where each RSI will be packetized and possible
97
error propagation has to be minimized.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
98 99 100 101 102 103

Flags:

AEC_DATA_SIGNED: input data are signed integers. Specifying this
correctly increases compression efficiency. Default is unsigned.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
104 105 106 107
AEC_DATA_PREPROCESS: preprocessing input will improve compression
efficiency if data samples are correlated. It will only cost
performance for no gain in efficiency if the data is already
uncorrelated.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
108 109 110 111 112 113

AEC_DATA_MSB: input data is stored most significant byte first
i.e. big endian. You have to specify AEC_DATA_MSB even if your host
architecture is big endian. Default is little endian on all
architectures.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
114 115
AEC_DATA_3BYTE: the 17 to 24 bit input data is stored in three
bytes. This flag has no effect for other sample sizes.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
116

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
117 118 119 120 121 122 123
AEC_RESTRICTED: use a restricted set of code options. This option is
only valid for bits_per_sample <= 4.

AEC_PAD_RSI: assume that the encoded RSI is padded to the next byte
boundary while decoding. The preprocessor macro ENABLE_RSI_PADDING
needs to be defined while compiling for the encoder to honour this
flag.
124

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
125 126
Data size:

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
127
The following rules apply for deducing storage size from sample size
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
128
(bits_per_sample):
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
129

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
130
sample size   storage size
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
131 132
 1 -  8 bits  1 byte
 9 - 16 bits  2 bytes
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
133 134 135
17 - 24 bits  3 bytes (only if AEC_DATA_3BYTE is set)
25 - 32 bits  4 bytes (if AEC_DATA_3BYTE is set)
17 - 32 bits  4 bytes (if AEC_DATA_3BYTE is not set)
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
136

137 138 139 140 141 142
If a sample requires less bits than the storage size provides, then
you have to make sure that unused bits are not set. Libaec does not
check this for performance reasons and will produce undefined output
if unused bits are set. All input data must be a multiple of the
storage size in bytes. Remaining bytes which do not form a complete
sample will be ignored.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
143

144 145 146
Libaec accesses next_in and next_out buffers only bytewise. There are
no alignment requirements for these buffers.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
147 148 149 150 151
Flushing:

aec_encode can be used in a streaming fashion by chunking input and
output and specifying AEC_NO_FLUSH. The function will return if either
the input runs empty or the output buffer is full. The calling
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
152
function can check avail_in and avail_out to see what occurred. The
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
153 154 155
last call to aec_encode() must set AEC_FLUSH to drain all
output. aec.c is an example of streaming usage of encoding and
decoding.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
156

157 158 159 160 161 162 163 164
Output:

Encoded data will be written to the buffer submitted with
next_out. The length of the compressed data is total_out.

See libaec.h for a detailed description of all relevant structure
members and constants.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
165 166 167 168 169

**********************************************************************
 Decoding
**********************************************************************

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
170 171
Using decoding is very similar to encoding, only the meaning of input
and output is reversed.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188

#include <libaec.h>

...
    struct aec_stream strm;
    /* this is now the compressed data */
    unsigned char *source;
    /* here goes the uncompressed result */
    int32_t *dest;

    strm.bits_per_sample = 32;
    strm.block_size = 16;
    strm.rsi = 128;
    strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;
    strm.next_in = source;
    strm.avail_in = source_length;
    strm.next_out = (unsigned char *)dest;
189
    strm.avail_out = dest_lenth * sizeof(int32_t);
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
190 191 192 193 194 195 196
    if (aec_decode_init(&strm) != AEC_OK)
        return 1;
    if (aec_decode(&strm, AEC_FLUSH) != AEC_OK)
        return 1;
    aec_decode_end(&strm);
...

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
197 198 199 200
It is strongly recommended that the size of the output buffer
(next_out) is a multiple of the storage size in bytes. If the buffer
is not a multiple of the storage size and the buffer gets filled to
the last sample, the error code AEC_MEM_ERROR is returned.
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
201

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
202 203
It is essential for decoding that parameters like bits_per_sample,
block_size, rsi, and flags are exactly the same as they were for
204
encoding. Libaec does not store these parameters in the coded stream
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
205 206 207
so it is up to the calling program to keep the correct parameters
between encoding and decoding.

208 209 210 211
The actual values of coding parameters are in fact only relevant for
efficiency and performance. Data integrity only depends on consistency
of the parameters.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
212 213 214 215 216 217

**********************************************************************
 References
**********************************************************************

[1] Consultative Committee for Space Data Systems. Lossless Data
218 219
Compression. Recommendation for Space Data System Standards, CCSDS
121.0-B-2. Blue Book. Issue 2. Washington, D.C.: CCSDS, May 2012.
220 221
http://public.ccsds.org/publications/archive/121x0b2.pdf

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
222
[2] Consultative Committee for Space Data Systems. Lossless Data
223
Compression.  Recommendation for Space Data System Standards, CCSDS
224 225 226
120.0-G-3. Green Book. Issue 3. Washington, D.C.: CCSDS, April
2013.
http://public.ccsds.org/publications/archive/120x0g3.pdf