Web lists-archives.com

Re: RFC: Support for zstd in .deb packages?

On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:
> (ZSTD)
> On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> > Recently Julian mentioned it again on IRC, and we each started
> > implementing support in dpkg and apt respectively, to allow easier
> > evaluation. I stopped when I realized the code was getting too messy,
> > but after few weeks Julian and Bálint Réczey ended up implementing
> > the support for apt and dpkg [D], so that they could merge it in
> > Ubuntu for their upcoming release, which they did.
> > 
> > Their main driving reason (AFAICT) has been (de)compression speed.
> With default level:
> 	comp	decomp	size
> xz	8.038s	0.356s	4320292
> bz2	2.265s	0.730s	5234516
> zst	0.274s	0.102s	5657626
> gz	0.880s	0.152s	6515505
> Z	0.499s	0.133s	8932459
> lzo	0.100s	0.095s	9198874
> _Compression_ speed is insane; decompression is merely nice.  Obviously,
> tuneable level turns this single point into an envelope, but even xz at its
> lowest levels can be multiple times faster than gzip while delivering better
> compression.

We only care about zstd at level 19 for dpkg and apt.

> Don't.  For .debs, that is.
> A .deb that is getting processed goes through dpkg, whose status files
> writes and all those fsyncs are so slow that it's pointless to optimize
> every last bit of decompression.  If someone already goes as far as
> recompressing packages, we got two fine choices: "xz -0" and "none"; the
> former being faster than gzip while compressing better, and the latter being
> as fast as cat.  You don't get any further than cat on the size-vs-speed
> slider...

Our major use case is cloud initial setup, image building, CI, buildds, all
of which do not require any syncs, and can safely use eatmydata, for example;
hence the enormous speed up.

There's still a speed up with syncs, but it's less than 10% IIRC. This will
get faster over time as SSDs increase their IOPS.

> On the other hand, adding .tar.zst [2] would be a much welcome addition for
> files referenced from .dsc.  A machine that installs dpkg-dev has enough
> storage that a half-MB library is beneath any notice, and zstd is a fine
> replacement for gzip and the likes.

this was backed out at guillem's request.

debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en