Web lists-archives.com

Re: .deb format: let's use 0.939, zstd, drop bzip2




Le 08/05/2019 à 19:38, Adam Borowski a écrit :
> Hi!
> I've recently did some research on how can we improve the speed of unpacking
> packages.  There's a lot of other stages that can be improved, but let's
> talk about the .deb format.
> 
> First, the 0.939 format, as described in "man deb-old".  While still being
> accepted by dpkg, it had been superseded before even the very first stable
> release.  Why?  It has at least two upsides over 2.0:
> 
> * there's no 10¹⁰ bytes (~9.31GB) limit
>   While no package this big is in the archive _yet_ (max being 1⎖652⎖244⎖360
>   bytes), both storage sizes and software bloat grow pretty fast, thus we'll
>   break this barrier in a few years.  And there's a world outside the
>   official archive -- I bet someone already has been burned by this limit.
> * it's faster by a small but non-negligible factor.  A task "unpack all
>   packages in default XFCE GUI install" gets done by stock dpkg (after
>   repacking everything as gzip) 3% faster.
> 
> Obviously, 3% is not worth fighting for, but as the size limit needs fixing
> anyway...
> 
> Alas, while current dpkg handles 0.939 archives well, it supports only two
> compressors: gzip and cat.  Neither of them is adequate these days.  Thus,
> we'd need to enable others -- which means not being able to unpack new .debs
> with old dpkg.  Barring ugly versioned pre-depends on dpkg, that'd require
> waiting two release cycles.
> 
> So let's pick compressors to enable.  For compression ratio, xz still wins
> (at least among popular compressors).  But there's a thing to say about
> zstd: firefox.deb zstd -19 takes to unpack:
> * 2.644s .xz, stock dpkg
> * 2.532s .xz, my tool (libarchive based)
> * 0.290s .zst, my tool
> * 0.738s .gz, stock dpkg
> * 0.729s .gz 0.939, stock dpkg
> File sizes being 60628216 gz, 47959544 zstd, 44506304 xz.
> 
> XFCE install total: 723M xz, 773M zstd, 963M gzip.
> 
> Thus, even though we'd want to stick with xz for the official archive, speed
> gains from zstd are so massive that it's tempting to add support for it,
> at least for non-official uses, possibly also for common Build-Depends.
> The usual objection, "we don't want to bloat the Essential set" doesn't hold
> water because 1. libzstd is already a part of the Required set in Buster,
> 2. a non-default compressor can be dlopened.
> 
> Thoughts?
> 
> But, the dlopen idea shows a potential victim: bzip2.  Let's kill it.
> 
> Stats for Buster's packages:
> 
> .deb format:
> 2.0:    100%
> 
> control:
> gz      11671
> xz      45210
> 
> data:
> gz      966
> xz      55915
> 
> With not a single package in the archive still using bz2, removing support
> would be reasonable.  It'd be okay to give a clear error message telling the
> user to install libbz2-1.0 (dlopen) or bzip2 (pipe) -- so folks can still
> unpack historic .debs if need be.
> 
> Meow!

Hi,

devscripts MR!122[1] proposes to add Zstandard support to uscan.

Cheers,
Xavier

https://salsa.debian.org/debian/devscripts/merge_requests/122