README.md 7.4 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11
# libaec - Adaptive Entropy Coding library

Libaec provides fast lossless compression of 1 up to 32 bit wide
signed or unsigned integers (samples). The library achieves best
results for low entropy data as often encountered in space imaging
instrument data or numerical model output from weather or climate
simulations. While floating point representations are not directly
supported, they can also be efficiently coded by grouping exponents
and mantissa.

Libaec implements
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
12
[Golomb-Rice](http://en.wikipedia.org/wiki/Golomb_coding) coding as
13 14 15
defined in the Space Data System Standard documents [121.0-B-2][1] and
[120.0-G-2][2].

16 17
## Downloads

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
18
Source code and binary installer can be [downloaded here](https://gitlab.dkrz.de/k202009/libaec/tags).
19

20
## Patent
21

22
In [patent.txt](doc/patent.txt) a statement on potentially
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
23
applying intellectual property rights is given.
24 25 26

## Installation

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
27
See [INSTALL](INSTALL) for details.
28 29 30

## SZIP Compatibility

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
31
[Libaec can replace SZIP](README.SZIP).
32 33 34 35 36 37 38

## Encoding

In this context efficiency refers to the size of the encoded
data. Performance refers to the time it takes to encode data.

Suppose you have an array of 32 bit signed integers you want to
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
39 40
compress. The pointer pointing to the data shall be called `*source`,
output goes into `*dest`.
41

42
```c
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88
#include <libaec.h>

...
    struct aec_stream strm;
    int32_t *source;
    unsigned char *dest;

    /* input data is 32 bits wide */
    strm.bits_per_sample = 32;

    /* define a block size of 16 */
    strm.block_size = 16;

    /* the reference sample interval is set to 128 blocks */
    strm.rsi = 128;

    /* input data is signed and needs to be preprocessed */
    strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;

    /* pointer to input */
    strm.next_in = (unsigned char *)source;

    /* length of input in bytes */
    strm.avail_in = source_length * sizeof(int32_t);

    /* pointer to output buffer */
    strm.next_out = dest;

    /* length of output buffer in bytes */
    strm.avail_out = dest_length;

    /* initialize encoding */
    if (aec_encode_init(&strm) != AEC_OK)
        return 1;

    /* Perform encoding in one call and flush output. */
    /* In this example you must be sure that the output */
    /* buffer is large enough for all compressed output */
    if (aec_encode(&strm, AEC_FLUSH) != AEC_OK)
        return 1;

    /* free all resources used by encoder */
    aec_encode_end(&strm);
...
```

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
89
`block_size` can vary from 8 to 64 samples. Smaller blocks allow the
90 91 92 93
compression to adapt more rapidly to changing source
statistics. Larger blocks create less overhead but can be less
efficient if source statistics change across the block.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
94
`rsi` sets the reference sample interval. A large RSI will improve
95 96 97 98 99 100 101
performance and efficiency. It will also increase memory requirements
since internal buffering is based on RSI size. A smaller RSI may be
desirable in situations where each RSI will be packetized and possible
error propagation has to be minimized.

### Flags:

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
102
* `AEC_DATA_SIGNED`: input data are signed integers. Specifying this
103 104
  correctly increases compression efficiency. Default is unsigned.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
105
* `AEC_DATA_PREPROCESS`: preprocessing input will improve compression
106 107 108 109
  efficiency if data samples are correlated. It will only cost
  performance for no gain in efficiency if the data is already
  uncorrelated.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
110 111
* `AEC_DATA_MSB`: input data is stored most significant byte first
  i.e. big endian. You have to specify `AEC_DATA_MSB` even if your host
112 113 114
  architecture is big endian. Default is little endian on all
  architectures.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
115
* `AEC_DATA_3BYTE`: the 17 to 24 bit input data is stored in three
116 117
  bytes. This flag has no effect for other sample sizes.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
118 119
* `AEC_RESTRICTED`: use a restricted set of code options. This option is
  only valid for `bits_per_sample` <= 4.
120

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
121 122
* `AEC_PAD_RSI`: assume that the encoded RSI is padded to the next byte
  boundary while decoding. The preprocessor macro `ENABLE_RSI_PADDING`
123 124 125 126 127 128
  needs to be defined while compiling for the encoder to honour this
  flag.

### Data size:

The following rules apply for deducing storage size from sample size
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
129
(`bits_per_sample`):
130

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
131 132 133 134 135 136 137
 **sample size**  | **storage size**
--- | ---
 1 -  8 bits  | 1 byte
 9 - 16 bits  | 2 bytes
17 - 24 bits  | 3 bytes (only if `AEC_DATA_3BYTE` is set)
25 - 32 bits  | 4 bytes (if `AEC_DATA_3BYTE` is set)
17 - 32 bits  | 4 bytes (if `AEC_DATA_3BYTE` is not set)
138 139 140 141 142 143 144 145

If a sample requires less bits than the storage size provides, then
you have to make sure that unused bits are not set. Libaec does not
check this for performance reasons and will produce undefined output
if unused bits are set. All input data must be a multiple of the
storage size in bytes. Remaining bytes which do not form a complete
sample will be ignored.

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
146 147
Libaec accesses `next_in` and `next_out` buffers only bytewise. There
are no alignment requirements for these buffers.
148 149 150

### Flushing:

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
151 152
`aec_encode` can be used in a streaming fashion by chunking input and
output and specifying `AEC_NO_FLUSH`. The function will return if either
153
the input runs empty or the output buffer is full. The calling
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
154 155
function can check `avail_in` and `avail_out` to see what occurred. The
last call to `aec_encode()` must set `AEC_FLUSH` to drain all
156
output. [aec.c](src/aec.c) is an example of streaming usage of encoding and
157 158 159 160 161
decoding.

### Output:

Encoded data will be written to the buffer submitted with
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
162
`next_out`. The length of the compressed data is `total_out`.
163 164 165 166 167 168 169 170 171 172

See libaec.h for a detailed description of all relevant structure
members and constants.


## Decoding

Using decoding is very similar to encoding, only the meaning of input
and output is reversed.

173
```c
174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199
#include <libaec.h>

...
    struct aec_stream strm;
    /* this is now the compressed data */
    unsigned char *source;
    /* here goes the uncompressed result */
    int32_t *dest;

    strm.bits_per_sample = 32;
    strm.block_size = 16;
    strm.rsi = 128;
    strm.flags = AEC_DATA_SIGNED | AEC_DATA_PREPROCESS;
    strm.next_in = source;
    strm.avail_in = source_length;
    strm.next_out = (unsigned char *)dest;
    strm.avail_out = dest_lenth * sizeof(int32_t);
    if (aec_decode_init(&strm) != AEC_OK)
        return 1;
    if (aec_decode(&strm, AEC_FLUSH) != AEC_OK)
        return 1;
    aec_decode_end(&strm);
...
```

It is strongly recommended that the size of the output buffer
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
200
(`next_out`) is a multiple of the storage size in bytes. If the buffer
201
is not a multiple of the storage size and the buffer gets filled to
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
202
the last sample, the error code `AEC_MEM_ERROR` is returned.
203

Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
204 205
It is essential for decoding that parameters like `bits_per_sample`,
`block_size`, `rsi`, and `flags` are exactly the same as they were for
206 207 208 209 210 211 212 213 214 215 216 217 218 219
encoding. Libaec does not store these parameters in the coded stream
so it is up to the calling program to keep the correct parameters
between encoding and decoding.

The actual values of coding parameters are in fact only relevant for
efficiency and performance. Data integrity only depends on consistency
of the parameters.


## References

[Consultative Committee for Space Data Systems. Lossless Data
Compression. Recommendation for Space Data System Standards, CCSDS
121.0-B-2. Blue Book. Issue 2. Washington, D.C.: CCSDS, May 2012.][1]
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
220
[1]: http://public.ccsds.org/publications/archive/121x0b2.pdf
221 222 223 224

[Consultative Committee for Space Data Systems. Lossless Data
Compression.  Recommendation for Space Data System Standards, CCSDS
120.0-G-3. Green Book. Issue 3. Washington, D.C.: CCSDS, April 2013.][2]
Mathis Rosenhauer's avatar
Mathis Rosenhauer committed
225
[2]: http://public.ccsds.org/publications/archive/120x0g3.pdf