-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathmanual.texi
2243 lines (1894 loc) · 89.2 KB
/
manual.texi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\input texinfo @c -*- Texinfo -*-
@setfilename bzip2.info
@ignore
This file documents bzip2 version 1.0.2, and associated library
libbzip2, written by Julian Seward ([email protected]).
Copyright (C) 1996-2002 Julian R Seward
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for verbatim copies.
@end ignore
@ifinfo
@format
START-INFO-DIR-ENTRY
* Bzip2: (bzip2). A program and library for data compression.
END-INFO-DIR-ENTRY
@end format
@end ifinfo
@iftex
@c @finalout
@settitle bzip2 and libbzip2
@titlepage
@title bzip2 and libbzip2
@subtitle a program and library for data compression
@subtitle copyright (C) 1996-2002 Julian Seward
@subtitle version 1.0.2 of 30 December 2001
@author Julian Seward
@end titlepage
@parindent 0mm
@parskip 2mm
@end iftex
@node Top,,, (dir)
The following text is the License for this software. You should
find it identical to that contained in the file LICENSE in the
source distribution.
@bf{------------------ START OF THE LICENSE ------------------}
This program, @code{bzip2},
and associated library @code{libbzip2}, are
Copyright (C) 1996-2002 Julian R Seward. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
@itemize @bullet
@item
Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
@item
The origin of this software must not be misrepresented; you must
not claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
@item
Altered source versions must be plainly marked as such, and must
not be misrepresented as being the original software.
@item
The name of the author may not be used to endorse or promote
products derived from this software without specific prior written
permission.
@end itemize
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Julian Seward, Cambridge, UK.
@code{jseward@@acm.org}
@code{bzip2}/@code{libbzip2} version 1.0.2 of 30 December 2001.
@bf{------------------ END OF THE LICENSE ------------------}
Web sites:
@code{http://sources.redhat.com/bzip2}
@code{http://www.cacheprof.org}
PATENTS: To the best of my knowledge, @code{bzip2} does not use any patented
algorithms. However, I do not have the resources available to carry out
a full patent search. Therefore I cannot give any guarantee of the
above statement.
@chapter Introduction
@code{bzip2} compresses files using the Burrows-Wheeler
block-sorting text compression algorithm, and Huffman coding.
Compression is generally considerably better than that
achieved by more conventional LZ77/LZ78-based compressors,
and approaches the performance of the PPM family of statistical compressors.
@code{bzip2} is built on top of @code{libbzip2}, a flexible library
for handling compressed data in the @code{bzip2} format. This manual
describes both how to use the program and
how to work with the library interface. Most of the
manual is devoted to this library, not the program,
which is good news if your interest is only in the program.
Chapter 2 describes how to use @code{bzip2}; this is the only part
you need to read if you just want to know how to operate the program.
Chapter 3 describes the programming interfaces in detail, and
Chapter 4 records some miscellaneous notes which I thought
ought to be recorded somewhere.
@chapter How to use @code{bzip2}
This chapter contains a copy of the @code{bzip2} man page,
and nothing else.
@quotation
@unnumberedsubsubsec NAME
@itemize
@item @code{bzip2}, @code{bunzip2}
- a block-sorting file compressor, v1.0.2
@item @code{bzcat}
- decompresses files to stdout
@item @code{bzip2recover}
- recovers data from damaged bzip2 files
@end itemize
@unnumberedsubsubsec SYNOPSIS
@itemize
@item @code{bzip2} [ -cdfkqstvzVL123456789 ] [ filenames ... ]
@item @code{bunzip2} [ -fkvsVL ] [ filenames ... ]
@item @code{bzcat} [ -s ] [ filenames ... ]
@item @code{bzip2recover} filename
@end itemize
@unnumberedsubsubsec DESCRIPTION
@code{bzip2} compresses files using the Burrows-Wheeler block sorting
text compression algorithm, and Huffman coding. Compression is
generally considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of the PPM
family of statistical compressors.
The command-line options are deliberately very similar to those of GNU
@code{gzip}, but they are not identical.
@code{bzip2} expects a list of file names to accompany the command-line
flags. Each file is replaced by a compressed version of itself, with
the name @code{original_name.bz2}. Each compressed file has the same
modification date, permissions, and, when possible, ownership as the
corresponding original, so that these properties can be correctly
restored at decompression time. File name handling is naive in the
sense that there is no mechanism for preserving original file names,
permissions, ownerships or dates in filesystems which lack these
concepts, or have serious file name length restrictions, such as MS-DOS.
@code{bzip2} and @code{bunzip2} will by default not overwrite existing
files. If you want this to happen, specify the @code{-f} flag.
If no file names are specified, @code{bzip2} compresses from standard
input to standard output. In this case, @code{bzip2} will decline to
write compressed output to a terminal, as this would be entirely
incomprehensible and therefore pointless.
@code{bunzip2} (or @code{bzip2 -d}) decompresses all
specified files. Files which were not created by @code{bzip2}
will be detected and ignored, and a warning issued.
@code{bzip2} attempts to guess the filename for the decompressed file
from that of the compressed file as follows:
@itemize
@item @code{filename.bz2 } becomes @code{filename}
@item @code{filename.bz } becomes @code{filename}
@item @code{filename.tbz2} becomes @code{filename.tar}
@item @code{filename.tbz } becomes @code{filename.tar}
@item @code{anyothername } becomes @code{anyothername.out}
@end itemize
If the file does not end in one of the recognised endings,
@code{.bz2}, @code{.bz},
@code{.tbz2} or @code{.tbz}, @code{bzip2} complains that it cannot
guess the name of the original file, and uses the original name
with @code{.out} appended.
As with compression, supplying no
filenames causes decompression from standard input to standard output.
@code{bunzip2} will correctly decompress a file which is the
concatenation of two or more compressed files. The result is the
concatenation of the corresponding uncompressed files. Integrity
testing (@code{-t}) of concatenated compressed files is also supported.
You can also compress or decompress files to the standard output by
giving the @code{-c} flag. Multiple files may be compressed and
decompressed like this. The resulting outputs are fed sequentially to
stdout. Compression of multiple files in this manner generates a stream
containing multiple compressed file representations. Such a stream
can be decompressed correctly only by @code{bzip2} version 0.9.0 or
later. Earlier versions of @code{bzip2} will stop after decompressing
the first file in the stream.
@code{bzcat} (or @code{bzip2 -dc}) decompresses all specified files to
the standard output.
@code{bzip2} will read arguments from the environment variables
@code{BZIP2} and @code{BZIP}, in that order, and will process them
before any arguments read from the command line. This gives a
convenient way to supply default arguments.
Compression is always performed, even if the compressed file is slightly
larger than the original. Files of less than about one hundred bytes
tend to get larger, since the compression mechanism has a constant
overhead in the region of 50 bytes. Random data (including the output
of most file compressors) is coded at about 8.05 bits per byte, giving
an expansion of around 0.5%.
As a self-check for your protection, @code{bzip2} uses 32-bit CRCs to
make sure that the decompressed version of a file is identical to the
original. This guards against corruption of the compressed data, and
against undetected bugs in @code{bzip2} (hopefully very unlikely). The
chances of data corruption going undetected is microscopic, about one
chance in four billion for each file processed. Be aware, though, that
the check occurs upon decompression, so it can only tell you that
something is wrong. It can't help you recover the original uncompressed
data. You can use @code{bzip2recover} to try to recover data from
damaged files.
Return values: 0 for a normal exit, 1 for environmental problems (file
not found, invalid flags, I/O errors, &c), 2 to indicate a corrupt
compressed file, 3 for an internal consistency error (eg, bug) which
caused @code{bzip2} to panic.
@unnumberedsubsubsec OPTIONS
@table @code
@item -c --stdout
Compress or decompress to standard output.
@item -d --decompress
Force decompression. @code{bzip2}, @code{bunzip2} and @code{bzcat} are
really the same program, and the decision about what actions to take is
done on the basis of which name is used. This flag overrides that
mechanism, and forces bzip2 to decompress.
@item -z --compress
The complement to @code{-d}: forces compression, regardless of the
invokation name.
@item -t --test
Check integrity of the specified file(s), but don't decompress them.
This really performs a trial decompression and throws away the result.
@item -f --force
Force overwrite of output files. Normally, @code{bzip2} will not overwrite
existing output files. Also forces @code{bzip2} to break hard links
to files, which it otherwise wouldn't do.
@code{bzip2} normally declines to decompress files which don't have the
correct magic header bytes. If forced (@code{-f}), however, it will
pass such files through unmodified. This is how GNU @code{gzip}
behaves.
@item -k --keep
Keep (don't delete) input files during compression
or decompression.
@item -s --small
Reduce memory usage, for compression, decompression and testing. Files
are decompressed and tested using a modified algorithm which only
requires 2.5 bytes per block byte. This means any file can be
decompressed in 2300k of memory, albeit at about half the normal speed.
During compression, @code{-s} selects a block size of 200k, which limits
memory use to around the same figure, at the expense of your compression
ratio. In short, if your machine is low on memory (8 megabytes or
less), use -s for everything. See MEMORY MANAGEMENT below.
@item -q --quiet
Suppress non-essential warning messages. Messages pertaining to
I/O errors and other critical events will not be suppressed.
@item -v --verbose
Verbose mode -- show the compression ratio for each file processed.
Further @code{-v}'s increase the verbosity level, spewing out lots of
information which is primarily of interest for diagnostic purposes.
@item -L --license -V --version
Display the software version, license terms and conditions.
@item -1 (or --fast) to -9 (or --best)
Set the block size to 100 k, 200 k .. 900 k when compressing. Has no
effect when decompressing. See MEMORY MANAGEMENT below.
The @code{--fast} and @code{--best} aliases are primarily for GNU
@code{gzip} compatibility. In particular, @code{--fast} doesn't make
things significantly faster. And @code{--best} merely selects the
default behaviour.
@item --
Treats all subsequent arguments as file names, even if they start
with a dash. This is so you can handle files with names beginning
with a dash, for example: @code{bzip2 -- -myfilename}.
@item --repetitive-fast
@item --repetitive-best
These flags are redundant in versions 0.9.5 and above. They provided
some coarse control over the behaviour of the sorting algorithm in
earlier versions, which was sometimes useful. 0.9.5 and above have an
improved algorithm which renders these flags irrelevant.
@end table
@unnumberedsubsubsec MEMORY MANAGEMENT
@code{bzip2} compresses large files in blocks. The block size affects
both the compression ratio achieved, and the amount of memory needed for
compression and decompression. The flags @code{-1} through @code{-9}
specify the block size to be 100,000 bytes through 900,000 bytes (the
default) respectively. At decompression time, the block size used for
compression is read from the header of the compressed file, and
@code{bunzip2} then allocates itself just enough memory to decompress
the file. Since block sizes are stored in compressed files, it follows
that the flags @code{-1} to @code{-9} are irrelevant to and so ignored
during decompression.
Compression and decompression requirements, in bytes, can be estimated
as:
@example
Compression: 400k + ( 8 x block size )
Decompression: 100k + ( 4 x block size ), or
100k + ( 2.5 x block size )
@end example
Larger block sizes give rapidly diminishing marginal returns. Most of
the compression comes from the first two or three hundred k of block
size, a fact worth bearing in mind when using @code{bzip2} on small machines.
It is also important to appreciate that the decompression memory
requirement is set at compression time by the choice of block size.
For files compressed with the default 900k block size, @code{bunzip2}
will require about 3700 kbytes to decompress. To support decompression
of any file on a 4 megabyte machine, @code{bunzip2} has an option to
decompress using approximately half this amount of memory, about 2300
kbytes. Decompression speed is also halved, so you should use this
option only where necessary. The relevant flag is @code{-s}.
In general, try and use the largest block size memory constraints allow,
since that maximises the compression achieved. Compression and
decompression speed are virtually unaffected by block size.
Another significant point applies to files which fit in a single block
-- that means most files you'd encounter using a large block size. The
amount of real memory touched is proportional to the size of the file,
since the file is smaller than a block. For example, compressing a file
20,000 bytes long with the flag @code{-9} will cause the compressor to
allocate around 7600k of memory, but only touch 400k + 20000 * 8 = 560
kbytes of it. Similarly, the decompressor will allocate 3700k but only
touch 100k + 20000 * 4 = 180 kbytes.
Here is a table which summarises the maximum memory usage for different
block sizes. Also recorded is the total compressed size for 14 files of
the Calgary Text Compression Corpus totalling 3,141,622 bytes. This
column gives some feel for how compression varies with block size.
These figures tend to understate the advantage of larger block sizes for
larger files, since the Corpus is dominated by smaller files.
@example
Compress Decompress Decompress Corpus
Flag usage usage -s usage Size
-1 1200k 500k 350k 914704
-2 2000k 900k 600k 877703
-3 2800k 1300k 850k 860338
-4 3600k 1700k 1100k 846899
-5 4400k 2100k 1350k 845160
-6 5200k 2500k 1600k 838626
-7 6100k 2900k 1850k 834096
-8 6800k 3300k 2100k 828642
-9 7600k 3700k 2350k 828642
@end example
@unnumberedsubsubsec RECOVERING DATA FROM DAMAGED FILES
@code{bzip2} compresses files in blocks, usually 900kbytes long. Each
block is handled independently. If a media or transmission error causes
a multi-block @code{.bz2} file to become damaged, it may be possible to
recover data from the undamaged blocks in the file.
The compressed representation of each block is delimited by a 48-bit
pattern, which makes it possible to find the block boundaries with
reasonable certainty. Each block also carries its own 32-bit CRC, so
damaged blocks can be distinguished from undamaged ones.
@code{bzip2recover} is a simple program whose purpose is to search for
blocks in @code{.bz2} files, and write each block out into its own
@code{.bz2} file. You can then use @code{bzip2 -t} to test the
integrity of the resulting files, and decompress those which are
undamaged.
@code{bzip2recover}
takes a single argument, the name of the damaged file, and writes a
number of files @code{rec00001file.bz2}, @code{rec00002file.bz2}, etc,
containing the extracted blocks. The output filenames are designed so
that the use of wildcards in subsequent processing -- for example,
@code{bzip2 -dc rec*file.bz2 > recovered_data} -- processes the files in
the correct order.
@code{bzip2recover} should be of most use dealing with large @code{.bz2}
files, as these will contain many blocks. It is clearly futile to use
it on damaged single-block files, since a damaged block cannot be
recovered. If you wish to minimise any potential data loss through
media or transmission errors, you might consider compressing with a
smaller block size.
@unnumberedsubsubsec PERFORMANCE NOTES
The sorting phase of compression gathers together similar strings in the
file. Because of this, files containing very long runs of repeated
symbols, like "aabaabaabaab ..." (repeated several hundred times) may
compress more slowly than normal. Versions 0.9.5 and above fare much
better than previous versions in this respect. The ratio between
worst-case and average-case compression time is in the region of 10:1.
For previous versions, this figure was more like 100:1. You can use the
@code{-vvvv} option to monitor progress in great detail, if you want.
Decompression speed is unaffected by these phenomena.
@code{bzip2} usually allocates several megabytes of memory to operate
in, and then charges all over it in a fairly random fashion. This means
that performance, both for compressing and decompressing, is largely
determined by the speed at which your machine can service cache misses.
Because of this, small changes to the code to reduce the miss rate have
been observed to give disproportionately large performance improvements.
I imagine @code{bzip2} will perform best on machines with very large
caches.
@unnumberedsubsubsec CAVEATS
I/O error messages are not as helpful as they could be. @code{bzip2}
tries hard to detect I/O errors and exit cleanly, but the details of
what the problem is sometimes seem rather misleading.
This manual page pertains to version 1.0.2 of @code{bzip2}. Compressed
data created by this version is entirely forwards and backwards
compatible with the previous public releases, versions 0.1pl2, 0.9.0,
0.9.5, 1.0.0 and 1.0.1, but with the following exception: 0.9.0 and
above can correctly decompress multiple concatenated compressed files.
0.1pl2 cannot do this; it will stop after decompressing just the first
file in the stream.
@code{bzip2recover} versions prior to this one, 1.0.2, used 32-bit
integers to represent bit positions in compressed files, so it could not
handle compressed files more than 512 megabytes long. Version 1.0.2 and
above uses 64-bit ints on some platforms which support them (GNU
supported targets, and Windows). To establish whether or not
@code{bzip2recover} was built with such a limitation, run it without
arguments. In any event you can build yourself an unlimited version if
you can recompile it with @code{MaybeUInt64} set to be an unsigned
64-bit integer.
@unnumberedsubsubsec AUTHOR
Julian Seward, @code{jseward@@acm.org}.
@code{http://sources.redhat.com/bzip2}
The ideas embodied in @code{bzip2} are due to (at least) the following
people: Michael Burrows and David Wheeler (for the block sorting
transformation), David Wheeler (again, for the Huffman coder), Peter
Fenwick (for the structured coding model in the original @code{bzip},
and many refinements), and Alistair Moffat, Radford Neal and Ian Witten
(for the arithmetic coder in the original @code{bzip}). I am much
indebted for their help, support and advice. See the manual in the
source distribution for pointers to sources of documentation. Christian
von Roques encouraged me to look for faster sorting algorithms, so as to
speed up compression. Bela Lubkin encouraged me to improve the
worst-case compression performance. The @code{bz*} scripts are derived
from those of GNU @code{gzip}. Many people sent patches, helped with
portability problems, lent machines, gave advice and were generally
helpful.
@end quotation
@chapter Programming with @code{libbzip2}
This chapter describes the programming interface to @code{libbzip2}.
For general background information, particularly about memory
use and performance aspects, you'd be well advised to read Chapter 2
as well.
@section Top-level structure
@code{libbzip2} is a flexible library for compressing and decompressing
data in the @code{bzip2} data format. Although packaged as a single
entity, it helps to regard the library as three separate parts: the low
level interface, and the high level interface, and some utility
functions.
The structure of @code{libbzip2}'s interfaces is similar to
that of Jean-loup Gailly's and Mark Adler's excellent @code{zlib}
library.
All externally visible symbols have names beginning @code{BZ2_}.
This is new in version 1.0. The intention is to minimise pollution
of the namespaces of library clients.
@subsection Low-level summary
This interface provides services for compressing and decompressing
data in memory. There's no provision for dealing with files, streams
or any other I/O mechanisms, just straight memory-to-memory work.
In fact, this part of the library can be compiled without inclusion
of @code{stdio.h}, which may be helpful for embedded applications.
The low-level part of the library has no global variables and
is therefore thread-safe.
Six routines make up the low level interface:
@code{BZ2_bzCompressInit}, @code{BZ2_bzCompress}, and @* @code{BZ2_bzCompressEnd}
for compression,
and a corresponding trio @code{BZ2_bzDecompressInit}, @* @code{BZ2_bzDecompress}
and @code{BZ2_bzDecompressEnd} for decompression.
The @code{*Init} functions allocate
memory for compression/decompression and do other
initialisations, whilst the @code{*End} functions close down operations
and release memory.
The real work is done by @code{BZ2_bzCompress} and @code{BZ2_bzDecompress}.
These compress and decompress data from a user-supplied input buffer
to a user-supplied output buffer. These buffers can be any size;
arbitrary quantities of data are handled by making repeated calls
to these functions. This is a flexible mechanism allowing a
consumer-pull style of activity, or producer-push, or a mixture of
both.
@subsection High-level summary
This interface provides some handy wrappers around the low-level
interface to facilitate reading and writing @code{bzip2} format
files (@code{.bz2} files). The routines provide hooks to facilitate
reading files in which the @code{bzip2} data stream is embedded
within some larger-scale file structure, or where there are
multiple @code{bzip2} data streams concatenated end-to-end.
For reading files, @code{BZ2_bzReadOpen}, @code{BZ2_bzRead},
@code{BZ2_bzReadClose} and @* @code{BZ2_bzReadGetUnused} are supplied. For
writing files, @code{BZ2_bzWriteOpen}, @code{BZ2_bzWrite} and
@code{BZ2_bzWriteFinish} are available.
As with the low-level library, no global variables are used
so the library is per se thread-safe. However, if I/O errors
occur whilst reading or writing the underlying compressed files,
you may have to consult @code{errno} to determine the cause of
the error. In that case, you'd need a C library which correctly
supports @code{errno} in a multithreaded environment.
To make the library a little simpler and more portable,
@code{BZ2_bzReadOpen} and @code{BZ2_bzWriteOpen} require you to pass them file
handles (@code{FILE*}s) which have previously been opened for reading or
writing respectively. That avoids portability problems associated with
file operations and file attributes, whilst not being much of an
imposition on the programmer.
@subsection Utility functions summary
For very simple needs, @code{BZ2_bzBuffToBuffCompress} and
@code{BZ2_bzBuffToBuffDecompress} are provided. These compress
data in memory from one buffer to another buffer in a single
function call. You should assess whether these functions
fulfill your memory-to-memory compression/decompression
requirements before investing effort in understanding the more
general but more complex low-level interface.
Yoshioka Tsuneo (@code{QWF00133@@niftyserve.or.jp} /
@code{tsuneo-y@@is.aist-nara.ac.jp}) has contributed some functions to
give better @code{zlib} compatibility. These functions are
@code{BZ2_bzopen}, @code{BZ2_bzread}, @code{BZ2_bzwrite}, @code{BZ2_bzflush},
@code{BZ2_bzclose},
@code{BZ2_bzerror} and @code{BZ2_bzlibVersion}. You may find these functions
more convenient for simple file reading and writing, than those in the
high-level interface. These functions are not (yet) officially part of
the library, and are minimally documented here. If they break, you
get to keep all the pieces. I hope to document them properly when time
permits.
Yoshioka also contributed modifications to allow the library to be
built as a Windows DLL.
@section Error handling
The library is designed to recover cleanly in all situations, including
the worst-case situation of decompressing random data. I'm not
100% sure that it can always do this, so you might want to add
a signal handler to catch segmentation violations during decompression
if you are feeling especially paranoid. I would be interested in
hearing more about the robustness of the library to corrupted
compressed data.
Version 1.0 is much more robust in this respect than
0.9.0 or 0.9.5. Investigations with Checker (a tool for
detecting problems with memory management, similar to Purify)
indicate that, at least for the few files I tested, all single-bit
errors in the decompressed data are caught properly, with no
segmentation faults, no reads of uninitialised data and no
out of range reads or writes. So it's certainly much improved,
although I wouldn't claim it to be totally bombproof.
The file @code{bzlib.h} contains all definitions needed to use
the library. In particular, you should definitely not include
@code{bzlib_private.h}.
In @code{bzlib.h}, the various return values are defined. The following
list is not intended as an exhaustive description of the circumstances
in which a given value may be returned -- those descriptions are given
later. Rather, it is intended to convey the rough meaning of each
return value. The first five actions are normal and not intended to
denote an error situation.
@table @code
@item BZ_OK
The requested action was completed successfully.
@item BZ_RUN_OK
@itemx BZ_FLUSH_OK
@itemx BZ_FINISH_OK
In @code{BZ2_bzCompress}, the requested flush/finish/nothing-special action
was completed successfully.
@item BZ_STREAM_END
Compression of data was completed, or the logical stream end was
detected during decompression.
@end table
The following return values indicate an error of some kind.
@table @code
@item BZ_CONFIG_ERROR
Indicates that the library has been improperly compiled on your
platform -- a major configuration error. Specifically, it means
that @code{sizeof(char)}, @code{sizeof(short)} and @code{sizeof(int)}
are not 1, 2 and 4 respectively, as they should be. Note that the
library should still work properly on 64-bit platforms which follow
the LP64 programming model -- that is, where @code{sizeof(long)}
and @code{sizeof(void*)} are 8. Under LP64, @code{sizeof(int)} is
still 4, so @code{libbzip2}, which doesn't use the @code{long} type,
is OK.
@item BZ_SEQUENCE_ERROR
When using the library, it is important to call the functions in the
correct sequence and with data structures (buffers etc) in the correct
states. @code{libbzip2} checks as much as it can to ensure this is
happening, and returns @code{BZ_SEQUENCE_ERROR} if not. Code which
complies precisely with the function semantics, as detailed below,
should never receive this value; such an event denotes buggy code
which you should investigate.
@item BZ_PARAM_ERROR
Returned when a parameter to a function call is out of range
or otherwise manifestly incorrect. As with @code{BZ_SEQUENCE_ERROR},
this denotes a bug in the client code. The distinction between
@code{BZ_PARAM_ERROR} and @code{BZ_SEQUENCE_ERROR} is a bit hazy, but still worth
making.
@item BZ_MEM_ERROR
Returned when a request to allocate memory failed. Note that the
quantity of memory needed to decompress a stream cannot be determined
until the stream's header has been read. So @code{BZ2_bzDecompress} and
@code{BZ2_bzRead} may return @code{BZ_MEM_ERROR} even though some of
the compressed data has been read. The same is not true for
compression; once @code{BZ2_bzCompressInit} or @code{BZ2_bzWriteOpen} have
successfully completed, @code{BZ_MEM_ERROR} cannot occur.
@item BZ_DATA_ERROR
Returned when a data integrity error is detected during decompression.
Most importantly, this means when stored and computed CRCs for the
data do not match. This value is also returned upon detection of any
other anomaly in the compressed data.
@item BZ_DATA_ERROR_MAGIC
As a special case of @code{BZ_DATA_ERROR}, it is sometimes useful to
know when the compressed stream does not start with the correct
magic bytes (@code{'B' 'Z' 'h'}).
@item BZ_IO_ERROR
Returned by @code{BZ2_bzRead} and @code{BZ2_bzWrite} when there is an error
reading or writing in the compressed file, and by @code{BZ2_bzReadOpen}
and @code{BZ2_bzWriteOpen} for attempts to use a file for which the
error indicator (viz, @code{ferror(f)}) is set.
On receipt of @code{BZ_IO_ERROR}, the caller should consult
@code{errno} and/or @code{perror} to acquire operating-system
specific information about the problem.
@item BZ_UNEXPECTED_EOF
Returned by @code{BZ2_bzRead} when the compressed file finishes
before the logical end of stream is detected.
@item BZ_OUTBUFF_FULL
Returned by @code{BZ2_bzBuffToBuffCompress} and
@code{BZ2_bzBuffToBuffDecompress} to indicate that the output data
will not fit into the output buffer provided.
@end table
@section Low-level interface
@subsection @code{BZ2_bzCompressInit}
@example
typedef
struct @{
char *next_in;
unsigned int avail_in;
unsigned int total_in_lo32;
unsigned int total_in_hi32;
char *next_out;
unsigned int avail_out;
unsigned int total_out_lo32;
unsigned int total_out_hi32;
void *state;
void *(*bzalloc)(void *,int,int);
void (*bzfree)(void *,void *);
void *opaque;
@}
bz_stream;
int BZ2_bzCompressInit ( bz_stream *strm,
int blockSize100k,
int verbosity,
int workFactor );
@end example
Prepares for compression. The @code{bz_stream} structure
holds all data pertaining to the compression activity.
A @code{bz_stream} structure should be allocated and initialised
prior to the call.
The fields of @code{bz_stream}
comprise the entirety of the user-visible data. @code{state}
is a pointer to the private data structures required for compression.
Custom memory allocators are supported, via fields @code{bzalloc},
@code{bzfree},
and @code{opaque}. The value
@code{opaque} is passed to as the first argument to
all calls to @code{bzalloc} and @code{bzfree}, but is
otherwise ignored by the library.
The call @code{bzalloc ( opaque, n, m )} is expected to return a
pointer @code{p} to
@code{n * m} bytes of memory, and @code{bzfree ( opaque, p )}
should free
that memory.
If you don't want to use a custom memory allocator, set @code{bzalloc},
@code{bzfree} and
@code{opaque} to @code{NULL},
and the library will then use the standard @code{malloc}/@code{free}
routines.
Before calling @code{BZ2_bzCompressInit}, fields @code{bzalloc},
@code{bzfree} and @code{opaque} should
be filled appropriately, as just described. Upon return, the internal
state will have been allocated and initialised, and @code{total_in_lo32},
@code{total_in_hi32}, @code{total_out_lo32} and
@code{total_out_hi32} will have been set to zero.
These four fields are used by the library
to inform the caller of the total amount of data passed into and out of
the library, respectively. You should not try to change them.
As of version 1.0, 64-bit counts are maintained, even on 32-bit
platforms, using the @code{_hi32} fields to store the upper 32 bits
of the count. So, for example, the total amount of data in
is @code{(total_in_hi32 << 32) + total_in_lo32}.
Parameter @code{blockSize100k} specifies the block size to be used for
compression. It should be a value between 1 and 9 inclusive, and the
actual block size used is 100000 x this figure. 9 gives the best
compression but takes most memory.
Parameter @code{verbosity} should be set to a number between 0 and 4
inclusive. 0 is silent, and greater numbers give increasingly verbose
monitoring/debugging output. If the library has been compiled with
@code{-DBZ_NO_STDIO}, no such output will appear for any verbosity
setting.
Parameter @code{workFactor} controls how the compression phase behaves
when presented with worst case, highly repetitive, input data. If
compression runs into difficulties caused by repetitive data, the
library switches from the standard sorting algorithm to a fallback
algorithm. The fallback is slower than the standard algorithm by
perhaps a factor of three, but always behaves reasonably, no matter how
bad the input.
Lower values of @code{workFactor} reduce the amount of effort the
standard algorithm will expend before resorting to the fallback. You
should set this parameter carefully; too low, and many inputs will be
handled by the fallback algorithm and so compress rather slowly, too
high, and your average-to-worst case compression times can become very
large. The default value of 30 gives reasonable behaviour over a wide
range of circumstances.
Allowable values range from 0 to 250 inclusive. 0 is a special case,
equivalent to using the default value of 30.
Note that the compressed output generated is the same regardless of
whether or not the fallback algorithm is used.
Be aware also that this parameter may disappear entirely in future
versions of the library. In principle it should be possible to devise a
good way to automatically choose which algorithm to use. Such a
mechanism would render the parameter obsolete.
Possible return values:
@display
@code{BZ_CONFIG_ERROR}
if the library has been mis-compiled
@code{BZ_PARAM_ERROR}
if @code{strm} is @code{NULL}
or @code{blockSize} < 1 or @code{blockSize} > 9
or @code{verbosity} < 0 or @code{verbosity} > 4
or @code{workFactor} < 0 or @code{workFactor} > 250
@code{BZ_MEM_ERROR}
if not enough memory is available
@code{BZ_OK}
otherwise
@end display
Allowable next actions:
@display
@code{BZ2_bzCompress}
if @code{BZ_OK} is returned
no specific action needed in case of error
@end display
@subsection @code{BZ2_bzCompress}
@example
int BZ2_bzCompress ( bz_stream *strm, int action );
@end example
Provides more input and/or output buffer space for the library. The
caller maintains input and output buffers, and calls @code{BZ2_bzCompress} to
transfer data between them.
Before each call to @code{BZ2_bzCompress}, @code{next_in} should point at
the data to be compressed, and @code{avail_in} should indicate how many
bytes the library may read. @code{BZ2_bzCompress} updates @code{next_in},
@code{avail_in} and @code{total_in} to reflect the number of bytes it
has read.
Similarly, @code{next_out} should point to a buffer in which the
compressed data is to be placed, with @code{avail_out} indicating how
much output space is available. @code{BZ2_bzCompress} updates
@code{next_out}, @code{avail_out} and @code{total_out} to reflect the
number of bytes output.
You may provide and remove as little or as much data as you like on each
call of @code{BZ2_bzCompress}. In the limit, it is acceptable to supply and
remove data one byte at a time, although this would be terribly
inefficient. You should always ensure that at least one byte of output
space is available at each call.
A second purpose of @code{BZ2_bzCompress} is to request a change of mode of the
compressed stream.
Conceptually, a compressed stream can be in one of four states: IDLE,
RUNNING, FLUSHING and FINISHING. Before initialisation
(@code{BZ2_bzCompressInit}) and after termination (@code{BZ2_bzCompressEnd}), a
stream is regarded as IDLE.
Upon initialisation (@code{BZ2_bzCompressInit}), the stream is placed in the
RUNNING state. Subsequent calls to @code{BZ2_bzCompress} should pass
@code{BZ_RUN} as the requested action; other actions are illegal and
will result in @code{BZ_SEQUENCE_ERROR}.
At some point, the calling program will have provided all the input data
it wants to. It will then want to finish up -- in effect, asking the
library to process any data it might have buffered internally. In this
state, @code{BZ2_bzCompress} will no longer attempt to read data from
@code{next_in}, but it will want to write data to @code{next_out}.
Because the output buffer supplied by the user can be arbitrarily small,
the finishing-up operation cannot necessarily be done with a single call
of @code{BZ2_bzCompress}.
Instead, the calling program passes @code{BZ_FINISH} as an action to
@code{BZ2_bzCompress}. This changes the stream's state to FINISHING. Any
remaining input (ie, @code{next_in[0 .. avail_in-1]}) is compressed and
transferred to the output buffer. To do this, @code{BZ2_bzCompress} must be
called repeatedly until all the output has been consumed. At that
point, @code{BZ2_bzCompress} returns @code{BZ_STREAM_END}, and the stream's
state is set back to IDLE. @code{BZ2_bzCompressEnd} should then be
called.
Just to make sure the calling program does not cheat, the library makes
a note of @code{avail_in} at the time of the first call to
@code{BZ2_bzCompress} which has @code{BZ_FINISH} as an action (ie, at the
time the program has announced its intention to not supply any more
input). By comparing this value with that of @code{avail_in} over
subsequent calls to @code{BZ2_bzCompress}, the library can detect any
attempts to slip in more data to compress. Any calls for which this is
detected will return @code{BZ_SEQUENCE_ERROR}. This indicates a
programming mistake which should be corrected.
Instead of asking to finish, the calling program may ask
@code{BZ2_bzCompress} to take all the remaining input, compress it and
terminate the current (Burrows-Wheeler) compression block. This could
be useful for error control purposes. The mechanism is analogous to
that for finishing: call @code{BZ2_bzCompress} with an action of
@code{BZ_FLUSH}, remove output data, and persist with the
@code{BZ_FLUSH} action until the value @code{BZ_RUN} is returned. As
with finishing, @code{BZ2_bzCompress} detects any attempt to provide more
input data once the flush has begun.
Once the flush is complete, the stream returns to the normal RUNNING
state.
This all sounds pretty complex, but isn't really. Here's a table
which shows which actions are allowable in each state, what action
will be taken, what the next state is, and what the non-error return
values are. Note that you can't explicitly ask what state the
stream is in, but nor do you need to -- it can be inferred from the
values returned by @code{BZ2_bzCompress}.
@display
IDLE/@code{any}
Illegal. IDLE state only exists after @code{BZ2_bzCompressEnd} or
before @code{BZ2_bzCompressInit}.
Return value = @code{BZ_SEQUENCE_ERROR}
RUNNING/@code{BZ_RUN}
Compress from @code{next_in} to @code{next_out} as much as possible.
Next state = RUNNING
Return value = @code{BZ_RUN_OK}
RUNNING/@code{BZ_FLUSH}
Remember current value of @code{next_in}. Compress from @code{next_in}
to @code{next_out} as much as possible, but do not accept any more input.
Next state = FLUSHING
Return value = @code{BZ_FLUSH_OK}
RUNNING/@code{BZ_FINISH}
Remember current value of @code{next_in}. Compress from @code{next_in}
to @code{next_out} as much as possible, but do not accept any more input.
Next state = FINISHING
Return value = @code{BZ_FINISH_OK}
FLUSHING/@code{BZ_FLUSH}
Compress from @code{next_in} to @code{next_out} as much as possible,
but do not accept any more input.
If all the existing input has been used up and all compressed
output has been removed
Next state = RUNNING; Return value = @code{BZ_RUN_OK}
else
Next state = FLUSHING; Return value = @code{BZ_FLUSH_OK}
FLUSHING/other
Illegal.
Return value = @code{BZ_SEQUENCE_ERROR}
FINISHING/@code{BZ_FINISH}
Compress from @code{next_in} to @code{next_out} as much as possible,
but to not accept any more input.
If all the existing input has been used up and all compressed
output has been removed
Next state = IDLE; Return value = @code{BZ_STREAM_END}
else
Next state = FINISHING; Return value = @code{BZ_FINISHING}
FINISHING/other
Illegal.
Return value = @code{BZ_SEQUENCE_ERROR}
@end display
That still looks complicated? Well, fair enough. The usual sequence
of calls for compressing a load of data is:
@itemize @bullet
@item Get started with @code{BZ2_bzCompressInit}.
@item Shovel data in and shlurp out its compressed form using zero or more
calls of @code{BZ2_bzCompress} with action = @code{BZ_RUN}.
@item Finish up.
Repeatedly call @code{BZ2_bzCompress} with action = @code{BZ_FINISH},
copying out the compressed output, until @code{BZ_STREAM_END} is returned.
@item Close up and go home. Call @code{BZ2_bzCompressEnd}.
@end itemize
If the data you want to compress fits into your input buffer all
at once, you can skip the calls of @code{BZ2_bzCompress ( ..., BZ_RUN )} and
just do the @code{BZ2_bzCompress ( ..., BZ_FINISH )} calls.
All required memory is allocated by @code{BZ2_bzCompressInit}. The
compression library can accept any data at all (obviously). So you
shouldn't get any error return values from the @code{BZ2_bzCompress} calls.
If you do, they will be @code{BZ_SEQUENCE_ERROR}, and indicate a bug in
your programming.
Trivial other possible return values:
@display
@code{BZ_PARAM_ERROR}