Project

General

Profile

Bug #12032

Feature #5630: Reproducible builds

The SquashFS creation is not deterministic

Added by intrigeri almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
12/14/2016
Due date:
% Done:

100%

Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

This applies both to the SquashFS in the ISO and the one in IUKs. I'm told that -no-fragments should fix it, but:

  • this might make the compression rate worse;
  • one might need to pass a fixed value to -processors too.

Contact info of the people investigating this: and .


Related issues

Related to Tails - Feature #11974: Reproducible IUK builds Resolved 11/20/2016
Blocked by Tails - Feature #11966: Reproducible website build Resolved 11/19/2016

Associated revisions

Revision b006d507 (diff)
Added by intrigeri almost 3 years ago

Pass -no-fragments to mksquashfs (refs: #12032).

This may render the SquashFS creation deterministic, but this may make
the resulting SquashFS substantially larger.

Revision da42d59a (diff)
Added by intrigeri over 2 years ago

Revert "Pass -no-fragments to mksquashfs (refs: #12032)."

This reverts commit b006d507b348ac5b53d793a438096c8e64930eb6.

History

#1 Updated by intrigeri almost 3 years ago

  • Description updated (diff)

#2 Updated by intrigeri almost 3 years ago

#4 Updated by intrigeri over 2 years ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10

intrigeri wrote:

Next step: test with latest patchset applied by lynxyis

uploaded squashfs-tools 4.3-3.0tails4 that has this patchset + our own not-merged-yet patches on top.

#5 Updated by intrigeri over 2 years ago

#6 Updated by intrigeri over 2 years ago

  • Type of work changed from Test to Research

According to diff -r --brief --no-dereference, there is no remaining difference in the content of the files stored in the SquashFS! :) Which means we are now in a much better position to work on this ticket.

Sadly, even with squashfs-tools 4.3-3.0tails4 the SquashFS is not reproducible yet, regardless of whether we pass -no-fragments. FTR the other options passed to mksquashfs are -comp xz -Xbcj x86 -b 1024K -Xdict-size 1024K -no-progress -sort squashfs.sort.

Next steps:

  • report this to Alexander Couzens, asking if there's another set of options we should try
  • ensure deterministic sorting of files in the SquashFS, by generating a squashfs.sort file in a deterministic manner and appending it to squashfs.sort (I have WIP about this locally)

#7 Updated by intrigeri over 2 years ago

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 1189902.38 Kbytes (1162.01 Mbytes)
        30.32% of uncompressed filesystem size (3924527.33 Kbytes)
Inode table size 1108786 bytes (1082.80 Kbytes)
        21.20% of uncompressed inode table size (5230321 bytes)
Directory table size 1390142 bytes (1357.56 Kbytes)
        34.74% of uncompressed directory table size (4001427 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
        97.50% of uncompressed xattr table size (80 bytes)

vs.

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 1048576
        compressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 1189902.42 Kbytes (1162.01 Mbytes)
        30.32% of uncompressed filesystem size (3924527.33 Kbytes)
Inode table size 1109362 bytes (1083.36 Kbytes)
        21.21% of uncompressed inode table size (5230321 bytes)
Directory table size 1390062 bytes (1357.48 Kbytes)
        34.74% of uncompressed directory table size (4001427 bytes)
Xattr table size 78 bytes (0.08 Kbytes)
        97.50% of uncompressed xattr table size (80 bytes)

#8 Updated by intrigeri over 2 years ago

intrigeri wrote:

According to diff -r --brief --no-dereference, there is no remaining difference in the content of the files stored in the SquashFS! :)

I've looked more closely and while this is true, the inode data differs: in particular, the mtime of files that come from config/chroot_local-includes/ varies; as a consequence, of course the mtime of their parent directory varies too. Also, the uid/gid of /etc/resolv.conf vary. I'll file tickets about those.

#9 Updated by intrigeri over 2 years ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (intrigeri)
  • % Done changed from 10 to 100

I'll report success back to the new squashfs-tools upstream.

Also available in: Atom PDF