compress
Compresses a stream of bytes.
The compress
operator is deprecated. You should use the
bespoke operators instead.
These operators offer more options for some of the formats.
Description
The compress
operator compresses bytes in a pipeline incrementally with a
known codec.
The operator uses Apache Arrow's compression utilities under the hood, and transparently supports all options that Apache Arrow supports for streaming compression.
Besides the supported brotli
, bz2
, gzip
, lz4
, and zstd
, Apache Arrow
also ships with codecs for lzo
, lz4_raw
, lz4_hadoop
and snappy
, which
only support oneshot compression. Support for them is not currently implemented.
codec: string
An identifier of the codec to use. Currently supported are brotli
, bz2
,
gzip
, lz4
, and zstd
.
level = int (optional)
The compression level to use. The supported values depend on the codec used. If omitted, the default level for the codec is used.