Project

General

Profile

Bug #12347

Investigate why the reproducible ISO is larger than 3.0~beta2

Added by intrigeri over 2 years ago. Updated over 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Build system
Target version:
-
Start date:
03/15/2017
Due date:
% Done:

30%

Feature Branch:
feature/5630-deterministic-builds
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

3.0~beta2 was 1099M, my current builds from feature/5630-deterministic-builds are 1191M. I've verified that the package list is almost the same. The size difference comes from live/filesystem.squashfs only. The "funny" thing is that in theory, we include less data in this ISO: we exclude/delete lots of stuff that would harm build reproducibility.

So, either we somehow include more data in the SquashFS, or the newer mksquashfs compresses less well than the one we used before.

History

#1 Updated by u over 2 years ago

  • Assignee set to u

#2 Updated by u over 2 years ago

give me the shaaa!

#3 Updated by lamby over 2 years ago

.oO ( Or maybe sorting a bunch of stuff via deterministic modes means that's being less efficient with inodes or something… )

#4 Updated by intrigeri over 2 years ago

give me the shaaa!

Compression details found in the build logs (both with 4 CPUs):

feature/5630-deterministic-builds:

Creating 4.0 filesystem on filesystem.squashfs, block size 1048576.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
    compressed data, compressed metadata, compressed fragments, compressed xattrs
    duplicates are removed
Filesystem size 1188906.36 Kbytes (1161.04 Mbytes)
    30.30% of uncompressed filesystem size (3923494.69 Kbytes)
Inode table size 1108626 bytes (1082.64 Kbytes)
    21.20% of uncompressed inode table size (5230220 bytes)
Directory table size 1389566 bytes (1357.00 Kbytes)
    34.73% of uncompressed directory table size (4001042 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
    97.50% of uncompressed xattr table size (80 bytes)
Number of duplicate files found 6878
Number of inodes 150113
Number of files 119805
Number of fragments 2228
Number of symbolic links  18054
Number of device nodes 6
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 12248

feature/stretch:

Creating 4.0 filesystem on filesystem.squashfs, block size 1048576.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
    compressed data, compressed metadata, compressed fragments, compressed xattrs
    duplicates are removed
Filesystem size 1089689.83 Kbytes (1064.15 Mbytes)
    27.55% of uncompressed filesystem size (3955809.15 Kbytes)
Inode table size 1136786 bytes (1110.14 Kbytes)
    21.47% of uncompressed inode table size (5294329 bytes)
Directory table size 1406442 bytes (1373.48 Kbytes)
    34.58% of uncompressed directory table size (4067027 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
    97.50% of uncompressed xattr table size (80 bytes)
Number of duplicate files found 6878
Number of inodes 152111
Number of files 121782
Number of fragments 2244
Number of symbolic links  18052
Number of device nodes 6
Number of fifo nodes 0
Number of socket nodes 12
Number of directories 12259

#5 Updated by intrigeri over 2 years ago

  • 00. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K (baseline)
    • size: 1191M
    • build time (intri's Jenkins): 29 minutes
    • build time (Tails' Jenkins): 47 minutes
  • 01. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xe
    • size: same as baseline => fail
  • 02. -comp xz -Xbcj x86 -Xpreset 9
    • bugfix/12347-test-02
    • size: 1215M => fail
    • build time (intri's Jenkins): 25 minutes
  • if -Xpreset 9 is too slow, try 7 or 8 and adjust the next steps
  • 03. -comp xz -Xbcj x86 -Xpreset 9 -Xe
  • 04. -comp xz -Xbcj x86 -Xe
    • size: 1215M => fail
    • build time (intri's Jenkins): 25 minutes
  • 05. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xpreset 9
    • bugfix/12347-test-05
    • size: same as baseline
    • build time (intri's Jenkins): 27 minutes
    • build time (Tails' Jenkins): 47 minutes
    • no size improvement, not much faster than baseline => fail
  • 06. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xpreset 9 -Xe
    • bugfix/12347-test-06
    • size: same as baseline
    • build time (intri's Jenkins): 27 minutes
    • build time (Tails' Jenkins): 48 minutes
    • no size improvement, not faster than baseline => fail
  • 07. -comp xz -Xbcj x86 -b 8096K -Xdict-size 8096K
    • "-b block size not power of two or not between 4096 and 1Mbyte" => fail
  • 08. -comp lzma -Xe -Xpreset 9
    • bugfix/12347-test-08
    • build time (intri's Jenkins): 22 minutes
    • build time (Tails' Jenkins): 39 minutes
    • size: 1208M
    • fails to boot => fail
    • a tiny bit bigger, but substantially faster to compress
  • 09. -comp lzo -Xcompression-level 9
    • segmentation fault (kernel: traps: mksquashfs[27815] general protection ip:409091 sp:7f3a55923e30 error:0 in mksquashfs[400000+2c000]) => fail
  • 10. -comp lz4 -Xhc
    • size: much bigger than baseline => fail
    • not supported in Debian's Linux kernel => fail
  • 11. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -no-exports
    • size: 1190M => small improvement
    • build time (intri's Jenkins): 30 minutes
  • 12. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -Xlc 0 -Xlp 2 -Xpb 2
    • size: 1201M => fail
    • build time (intri's Jenkins): 31 minutes
  • 13. -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -read-queue 16M -write-queue 16M
    • invalid values => fail

#6 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 30

So at this point, it looks like the only (tiny) improvement we can make to our current settings is adding -no-exports. I'll do it right away.

Keeping in mind that Tails 2.11 is 1154M, going up to 1190M for 3.0 feels OK (and it'll get about 60M smaller once I2P is removed), so I personally don't find it worth investigating any further.

Ulrike, what do you think?

One other thing we could do is 1. update our SquashFS sort file (in case it causes the regression; note that we will do this anyway for the 3.0 release, so it may be worth waiting a bit); 2. build an ISO; 3. if the regression is still there, report it to the new squashfs-tools upstream (and likely they'll ask us to bisect the 103 patches we added on top of the version that didn't expose this problem).

#7 Updated by u over 2 years ago

  • Assignee changed from u to intrigeri

intrigeri wrote:

Keeping in mind that Tails 2.11 is 1154M, going up to 1190M for 3.0 feels OK (and it'll get about 60M smaller once I2P is removed), so I personally don't find it worth investigating any further.

Ulrike, what do you think?

I agree.

One other thing we could do is 1. update our SquashFS sort file (in case it causes the regression; note that we will do this anyway for the 3.0 release, so it may be worth waiting a bit); 2. build an ISO; 3. if the regression is still there, report it to the new squashfs-tools upstream (and likely they'll ask us to bisect the 103 patches we added on top of the version that didn't expose this problem).

Where is the sorting of the squashfs file 1.) tracked? Would you mind tracking 2.)? And then for 3.) I have the feeling I can report this upstream but I won't be able to dissect all of those patches one by one.

#8 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to u

Where is the sorting of the squashfs file 1.) tracked?

./config/binary_rootfs/squashfs.sort

Would you mind tracking 2.)?

I can do 2 builds + compare whenever you ask me.

And then for 3.) I have the feeling I can report this upstream but I won't be able to dissect all of those patches one by one.

You're aware of git bisect, right? If this feels to complicated anyway, let's forget about it and just close this ticket.

#9 Updated by u over 2 years ago

  • Assignee changed from u to intrigeri

And then for 3.) I have the feeling I can report this upstream but I won't be able to dissect all of those patches one by one.

You're aware of git bisect, right? If this feels to complicated anyway, let's forget about it and just close this ticket.

Haha, no i was not aware of git bisect! Feels complicated to me, and I would prefer closing this ticket, as the ISO size is not change significantly. However, reassigning to you in case you prefer to look into it anyway.

#10 Updated by intrigeri over 2 years ago

  • Assignee changed from intrigeri to anonym

My time budget for this project is basically over, reassigning to someone who has more left :)

#11 Updated by intrigeri about 2 years ago

Either we think it's part of the contract (and then it needs target version + deliverable for), or not (and then it needs neither).

#12 Updated by bertagaz about 2 years ago

intrigeri wrote:

Either we think it's part of the contract (and then it needs target version + deliverable for), or not (and then it needs neither).

I don't think that's really an issue now that most of RB are in stable.

There may have been a bump in size with reproducible builds, but so far it doesn't seem to have had impact on anything, even our infra. If it had, it's probably not that much (like a few douzens MB). So maybe let's consider this as a fair cost?

#13 Updated by intrigeri about 2 years ago

There may have been a bump in size with reproducible builds, but so far it doesn't seem to have had impact on anything, even our infra.
If it had, it's probably not that much (like a few douzens MB).

Thankfully we have actual data so we don't have to guess. When this ticket was raised the difference was 92MB, i.e. +8.3% on the ISO size.

The impact I'm worried about is the one on users: how many incremental updates one can apply in a row (soon obsolete thanks to #12707, so it's too late to bother now), download time for end-users (for the ISO, and possibly for incremental upgrades if the size of IUKs is affected as well).

Now, I think it would be OK to merely report this to the author of the squashfs-tools patch and close this ticket.

#14 Updated by anonym about 2 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check set to Info Needed

I'm not sure how to proceed. Do you have any quick pointers? If I understand it correctly I would have to install an old squashfs-tools, e.g. Jessie's 1:4.2+20130409-2, to be able to get back the "baseline", right?

#15 Updated by intrigeri about 2 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check deleted (Info Needed)

Yes. Keep in mind that this ticket is not a sponsor deliverable and has no target version.

#16 Updated by intrigeri over 1 year ago

  • Parent task deleted (#5630)

(We now have reproducible builds so let's try to close #5630 at some point.)

Also available in: Atom PDF