Project

General

Profile

Bug #16294

SquashFS metadata is not reproducible in buster

Added by lamby 10 days ago. Updated 8 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Build system
Target version:
Start date:
01/05/2019
Due date:
% Done:

90%

QA Check:
Pass
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

├── live/filesystem.squashfs
│ ├── unsquashfs -s {}
│ │ @@ -1,10 +1,10 @@
│ │  Found a valid SQUASHFS 4:0 superblock
│ │ -Creation or last append time Sat Jan  5 12:15:29 2019
│ │ -Filesystem size 1710322.73 Kbytes (1670.24 Mbytes)
│ │ +Creation or last append time Sat Jan  5 14:54:01 2019
│ │ +Filesystem size 1710322.79 Kbytes (1670.24 Mbytes)

Timing should be fixed; filesystem size difference very likely symptom of other issues.


Related issues

Related to Tails - Bug #16310: Buster IUKs are not built reproducibly Resolved 01/06/2019
Duplicated by Tails - Bug #16295: Various base directories/files have varying mtimes Duplicate 01/05/2019
Blocks Tails - Feature #16285: feature/buster branch is not reproducible Resolved 01/05/2019
Blocks Tails - Feature #15507: Core work 2019Q1: Foundations Team Confirmed 04/08/2018

History

#1 Updated by lamby 10 days ago

  • Blocks Feature #16285: feature/buster branch is not reproducible added

#2 Updated by lamby 10 days ago

  • QA Check set to Dev Needed

#3 Updated by intrigeri 10 days ago

│ │ -Creation or last append time Sat Jan 5 12:15:29 2019
│ │ -Filesystem size 1710322.73 Kbytes (1670.24 Mbytes)
│ │ +Creation or last append time Sat Jan 5 14:54:01 2019
│ │ +Filesystem size 1710322.79 Kbytes (1670.24 Mbytes)

I suspect our patched squashfs-tools is not used because it's older than the version in Buster. We should probably build and upload 1:4.3-7.0tails1 to our feature-buster APT suite. Happy to do that if I'm given a debdiff against the testing/sid source package with our patches forward-ported :)

#4 Updated by lamby 10 days ago

  • Duplicated by Bug #16295: Various base directories/files have varying mtimes added

#5 Updated by lamby 10 days ago

  • Status changed from New to In Progress

#6 Updated by intrigeri 10 days ago

#7 Updated by lamby 10 days ago

  • Assignee changed from lamby to intrigeri

#8 Updated by intrigeri 10 days ago

  • Target version set to Tails_4.0

#9 Updated by intrigeri 10 days ago

  • Related to Bug #16310: Buster IUKs are not built reproducibly added

#10 Updated by intrigeri 10 days ago

  • Assignee changed from intrigeri to lamby
  • QA Check changed from Dev Needed to Info Needed

It FTBFS for me, am I doing anything wrong?

make[1]: Entering directory '/<<PKGBUILDDIR>>'
dh_auto_configure -- CFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" CXXFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security" FCFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" FFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" GCJFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong" LDFLAGS="-Wl,-z,relro" OBJCFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security" OBJCXXFLAGS="-g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security" 
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
   debian/rules override_dh_auto_build
make[1]: Entering directory '/<<PKGBUILDDIR>>'
LZMA_XZ_SUPPORT=1 LZ4_SUPPORT=1 LZO_SUPPORT=1 XZ_SUPPORT=1 dh_auto_build
    cd squashfs-tools && make -j9 "INSTALL=install --strip-program=true" 
make[2]: Entering directory '/<<PKGBUILDDIR>>/squashfs-tools'
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o mksquashfs.o mksquashfs.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o read_fs.o read_fs.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o action.o action.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o swap.o swap.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o pseudo.o pseudo.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o compressor.o compressor.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o sort.o sort.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o progressbar.o progressbar.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o read_file.o read_file.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o info.o info.c
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o restore.o restore.c
mksquashfs.c: In function 'reader_read_file':
mksquashfs.c:2271:5: error: 'content_clamp_time' undeclared (first use in this function)
  if(content_clamp_time != -1 && buf2.st_mtime >= content_clamp_time)
     ^~~~~~~~~~~~~~~~~~
mksquashfs.c:2271:5: note: each undeclared identifier is reported only once for each function it appears in
cc -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -fgnu89-inline  -I. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DLZMA_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT -DLZ4_SUPPORT -DXATTR_SUPPORT -DXATTR_DEFAULT -Wdate-time -D_FORTIFY_SOURCE=2  -c -o process_fragments.o process_fragments.c
mksquashfs.c: In function 'dir_scan':
mksquashfs.c:3147:18: error: 'content_clamp_time' undeclared (first use in this function)
   buf.st_mtime = content_clamp_time != -1 ? content_clamp_time : time(NULL);
                  ^~~~~~~~~~~~~~~~~~
mksquashfs.c: In function 'dir_scan1':
mksquashfs.c:3380:6: error: 'content_clamp_time' undeclared (first use in this function)
   if(content_clamp_time != -1 && buf.st_mtime >= content_clamp_time)
      ^~~~~~~~~~~~~~~~~~
mksquashfs.c: In function 'dir_scan2':
mksquashfs.c:3538:18: error: 'content_clamp_time' undeclared (first use in this function)
   buf.st_mtime = content_clamp_time != -1 ? content_clamp_time : time(NULL);
                  ^~~~~~~~~~~~~~~~~~
mksquashfs.c: In function 'main':
mksquashfs.c:5524:17: error: 'content_clamp_time' undeclared (first use in this function)
   forced_time = content_clamp_time = (time_t)epoch;
                 ^~~~~~~~~~~~~~~~~~
make[2]: *** [<builtin>: mksquashfs.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/<<PKGBUILDDIR>>/squashfs-tools'
dh_auto_build: cd squashfs-tools && make -j9 "INSTALL=install --strip-program=true" returned exit code 2
make[1]: *** [debian/rules:13: override_dh_auto_build] Error 2
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:19: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

#11 Updated by intrigeri 10 days ago

  • Related to deleted (Bug #16310: Buster IUKs are not built reproducibly)

#12 Updated by intrigeri 10 days ago

  • Blocks Bug #16310: Buster IUKs are not built reproducibly added

#13 Updated by lamby 10 days ago

  • Assignee changed from lamby to intrigeri
  • QA Check changed from Info Needed to Dev Needed

Sorry about that; force-pushed the wrong local (backup) tree.

+ afabac8...0e18d71 debian -> debian (forced update)

Note that this one also makes the following additional change during the rebase:

https://salsa.debian.org/lamby/pkg-tails-squashfs-tools/commit/4b3e0811af6944e39a75398934dfe7e6ee1ff6cd#fa505f5dcaca9b1e38cec0c19f204a6b49d7ed9f_0_9

Can you quickly check that bit too.

#14 Updated by intrigeri 10 days ago

  • Assignee changed from intrigeri to lamby
  • QA Check changed from Dev Needed to Info Needed

Sorry about that; force-pushed the wrong local (backup) tree.

NP! :)

May you please also force-push an updated signed tag?

#15 Updated by intrigeri 10 days ago

Note that this one also makes the following additional change during the rebase:

https://salsa.debian.org/lamby/pkg-tails-squashfs-tools/commit/4b3e0811af6944e39a75398934dfe7e6ee1ff6cd#fa505f5dcaca9b1e38cec0c19f204a6b49d7ed9f_0_9

Can you quickly check that bit too.

Had a quick look, doesn't look crazy. I'm not qualified for a proper code review so let's hope upstream considers it some day.

#16 Updated by intrigeri 9 days ago

  • Assignee changed from lamby to intrigeri
  • QA Check changed from Info Needed to Ready for QA

#17 Updated by intrigeri 9 days ago

  • % Done changed from 0 to 50

squashfs-tools | 1:4.3-7.0tails1 | feature-buster | amd64, source

Triggered a build+rebuild+diffoscope, let's see how it goes!

#19 Updated by intrigeri 9 days ago

  • Assignee changed from intrigeri to lamby
  • % Done changed from 50 to 90

intrigeri wrote:

Triggered a build+rebuild+diffoscope, let's see how it goes!

Only fontconfig issues on https://jenkins.tails.boum.org/view/Tails_ISO/job/reproducibly_build_Tails_ISO_feature-buster/69/artifact/build-artifacts/diffoscope.iso.html AFAICT. Please double-check and close as resolved if you agree :)

#20 Updated by lamby 9 days ago

  • Status changed from In Progress to Resolved
  • QA Check changed from Ready for QA to Pass

#21 Updated by lamby 8 days ago

  • Status changed from Resolved to Confirmed
  • QA Check deleted (Pass)
Mon 07 19:26 <@lamby> intrigeri: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-buster/74/ Hm…
Mon 07 19:48 < intrigeri> lamby: was going to paste exactly that link. I have good hopes that getting back lynxis' patches will fix 
                          that (iirc that did the trick last time, when fixing timestamps etc. was not enough).

Re-opening.

#22 Updated by lamby 8 days ago

  • Assignee changed from lamby to intrigeri

intrigeri, what was the "cheat word" you mentioned over dinner? :)

#23 Updated by intrigeri 8 days ago

  • Assignee changed from intrigeri to lamby

intrigeri, what was the "cheat word" you mentioned over dinner? :)

"fragments" :)

#24 Updated by lamby 8 days ago

  • Status changed from Confirmed to Fix committed
  • Assignee changed from lamby to intrigeri
  • QA Check set to Ready for QA

Please see https://salsa.debian.org/lamby/pkg-tails-fontconfig for a:

squashfs-tools (1:4.3-8.0tails1) feature-buster; urgency=medium

  * Rebase/re-apply 0098-remove-frag_deflator_thread.patch dropped in
    1:4.3-3.0tails4 in an attempt to fix the filesystem metadata.
    (Closes:Tails:#16294)

 -- Chris Lamb <lamby@debian.org>  Tue, 08 Jan 2019 11:11:49 +0100

(ps. I don't think you pushed to https://git-tails.immerda.ch for the previous upload? Or at least my remotes have not updated AFAICT...?)

#25 Updated by intrigeri 8 days ago

  • Status changed from Fix committed to In Progress

("Fix committed" means something else here: https://tails.boum.org/contribute/working_together/Redmine/ :)

#26 Updated by intrigeri 8 days ago

Please see https://salsa.debian.org/lamby/pkg-tails-fontconfig for a:

I'll check your squashfs-tools repo instead :)

> squashfs-tools (1:4.3-8.0tails1) feature-buster; urgency=medium

>   * Rebase/re-apply 0098-remove-frag_deflator_thread.patch dropped in
>     1:4.3-3.0tails4 in an attempt to fix the filesystem metadata.
>     (Closes:Tails:#16294)

>  -- Chris Lamb <lamby@debian.org>  Tue, 08 Jan 2019 11:11:49 +0100

> 

Will try to do this today!

(ps. I don't think you pushed to https://git-tails.immerda.ch for the previous upload? Or at least my remotes have not updated AFAICT...?)

I have pushed there (https://git.tails.boum.org/squashfs-tools/commit/?h=debian) but Gitolite rejects tags with a % so I had to rename it to debian/4.3-7.0tails1.

#27 Updated by intrigeri 8 days ago

Uploaded, triggered a build on Jenkins, and retrying my (manual) test procedure for #16310.

#28 Updated by intrigeri 8 days ago

  • Blocks deleted (Bug #16310: Buster IUKs are not built reproducibly)

#29 Updated by intrigeri 8 days ago

  • Related to Bug #16310: Buster IUKs are not built reproducibly added

#30 Updated by lamby 8 days ago

  • Status changed from In Progress to Resolved
  • QA Check changed from Ready for QA to Pass

Also available in: Atom PDF