Project

General

Profile

Bug #15187

fontconfig cache is not reproducible in Buster

Added by intrigeri over 1 year ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
01/17/2018
Due date:
09/30/2018
% Done:

100%

Estimated time:
0.00 h
QA Check:
Pass
Feature Branch:
Type of work:
Code
Blueprint:
Starter:
Affected tool:

Description

See attached diffoscope output.

I don't remember if we expected Buster to have everything we need, or if there's still some upstream or Debian bug report open about that. I guess that's https://bugs.debian.org/864082, which is not marked as forwarded upstream (strange, probably a mere oversight). We could of course build a patched fontconfig package as we do for Stretch, but it would be nice to seize this opportunity to try harder to upstream this.

feature-buster_diffoscope.html View (442 KB) intrigeri, 01/17/2018 05:19 PM

0001-Ensure-cache-checksums-are-determinstic.patch View (3.79 KB) lamby, 05/03/2018 03:46 AM


Related issues

Related to Tails - Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 Resolved 05/19/2017
Related to Tails - Feature #16285: feature/buster branch is not reproducible Resolved 01/05/2019
Blocks Tails - Feature #15944: Port Tails to Buster In Progress 09/12/2018
Blocked by Tails - Bug #15857: Make feature/buster build Resolved 08/29/2018
Blocks Tails - Feature #16209: Core work: Foundations Team Confirmed 03/22/2019

History

#1 Updated by intrigeri over 1 year ago

  • Related to Bug #12567: fontconfig cache is not generated reproducibly even with patch from Debian#857892 added

#2 Updated by intrigeri over 1 year ago

  • Status changed from Confirmed to In Progress
  • % Done changed from 0 to 10
  • Type of work changed from Communicate to Code

So there were two issues:

  • https://bugs.debian.org/857892, fixed in the version of fontconfig that's in Buster (we apply a similar patch to upstream's on Stretch) => case closed
  • https://bugs.debian.org/864082 (i.e. the solution to #12567), that has two parts:
    • a patch against the upstream source, that I've just marked as forwarded upstream; upstream has replied with a different design proposal that totally makes sense to me; no follow-up from our part so far
    • a Debian packaging change that depends on the new feature proposed upstream

Next step: implement what upstream suggested for the second problem. It looks easy: I suspect that even with my non-existing C skills I could get something worth sending upstream in a few hours (including integration with the upstream build system + Debian packaging and testing in an ISO build). I would start from this code (the Python option is more tempting but I don't think it's acceptable to add this dependency to fontconfig in Debian).

#3 Updated by lamby over 1 year ago

Thanks for marking it as forwarded upstream. Usually when this happens (and "upstream" is a mailing list) it's because posts get marked for moderation and I can only really keep polling the HTML archives for so so many days until I get distracted by something else :)

Let me know if I can help here - very welcome to jump in and resolve this, just let me know and assign over etc. etc.

#4 Updated by intrigeri about 1 year ago

  • Assignee set to lamby
  • Estimated time set to 3.00 h

#5 Updated by intrigeri about 1 year ago

  • Due date set to 09/30/2018

#6 Updated by intrigeri about 1 year ago

#9 Updated by lamby about 1 year ago

I've reworked the patch on upstream's advice. I've:

- Sent it here to https://bugs.debian.org/864082
- Forwarded it to the upstream mailing list (although I think it is stuck in their spam queue, alas, but it should eventually be available https://lists.freedesktop.org/archives/fontconfig/2018-May/thread.html)
- Attached it here

#10 Updated by lamby about 1 year ago

#11 Updated by intrigeri about 1 year ago

  • Assignee changed from intrigeri to lamby

I've reworked the patch on upstream's advice. I've:

- Sent it here to https://bugs.debian.org/864082
- Forwarded it to the upstream mailing list
[…]

Great! Can you handle the next steps i.e. ensure this is reviewed and merged upstream (or worst case, in Debian), pinging the relevant folks as needed and following-up on whatever comment they may have?

#12 Updated by lamby about 1 year ago

Sure. Already on it :)

#18 Updated by lamby about 1 year ago

https://bugs.debian.org/864082#49 - Pinged Debian maintainer to cut a new release and updated forwarded URI

#19 Updated by lamby about 1 year ago

#21 Updated by intrigeri about 1 year ago

Will we need changes in the Debian packaging, aside of importing the changes from upstream?

#22 Updated by lamby about 1 year ago

intrigeri wrote:

Will we need changes in the Debian packaging, aside of importing the changes from upstream?

No.

#23 Updated by lamby 10 months ago

Pinged bug and Keith IRL…

#24 Updated by lamby 9 months ago

Pinged upstream bug

#25 Updated by intrigeri 9 months ago

  • Target version changed from Tails_4.0 to Tails_3.11

#26 Updated by intrigeri 9 months ago

#27 Updated by lamby 8 months ago

Finally merged in debian as 2.13.1-1

#28 Updated by intrigeri 8 months ago

#29 Updated by intrigeri 8 months ago

#30 Updated by lamby 8 months ago

  • Assignee changed from lamby to CyrilBrulebois
  • QA Check set to Ready for QA

#31 Updated by lamby 8 months ago

  • Blocked by Bug #15857: Make feature/buster build added

#32 Updated by lamby 7 months ago

  • Assignee changed from CyrilBrulebois to intrigeri

Hm, according to https://bugs.debian.org/864082#101:

Unfortunately, fontconfig still installs unreproducibly. Try […]

Hmpf!

I think I would need some hour(s) to even confirm/investigate this, hence assigning over to you, intri.

#33 Updated by intrigeri 7 months ago

  • Assignee changed from intrigeri to lamby
  • Estimated time changed from 3.00 h to 7.00 h
  • QA Check changed from Ready for QA to Dev Needed

Sure, adding 4 hours that should hopefully be enough to debug this, get a fix upstream and in Buster :)

#34 Updated by lamby 7 months ago

Can reproduce:


$ diffoscope --markdown=- debian-fontconfig*/var/cache/fontconfig

# Comparing debian-fontconfig1/var/cache/fontconfig & debian-fontconfig2/var/cache/fontconfig

## file list

    @@ -1,5 +1,5 @@
    +7fd806a4-197a-4989-8a34-2c49019d041b-le64.cache-7
    +95c367ca-9c9b-47d7-9625-c03688da4239-le64.cache-7
     CACHEDIR.TAG
    -a4fcff53-9cdb-4103-baea-3115d0f9e21e-le64.cache-7
    -afd762ff-b72e-4c3f-98f5-19b3b7cf7f95-le64.cache-7
    -c8c796f6-9945-4521-bb11-2ad6a193bcf2-le64.cache-7
    -ee218622-3364-4921-aaae-6e7d011e7c5e-le64.cache-7
    +cb3a236e-83c1-49d4-92f9-a44aa67ef71c-le64.cache-7
    +dc12f21a-6ea1-4373-b9e4-cfc7bd8165f7-le64.cache-7

## stat {}

    @@ -1,8 +1,8 @@

       Size: 4096          Blocks: 8          IO Block: 4096   directory
     Links: 2
     Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

    -Modify: 2018-10-29 15:00:44.915312284 +0000
    +Modify: 2018-10-29 15:02:00.475850377 +0000

      Birth: -

## Comparing debian-fontconfig1/var/cache/fontconfig/CACHEDIR.TAG & debian-fontconfig2/var/cache/fontconfig/CACHEDIR.TAG

### stat {}

    @@ -1,8 +1,8 @@

       Size: 200           Blocks: 8          IO Block: 4096   regular file
     Links: 1
     Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

    -Modify: 2018-10-29 15:00:44.915312284 +0000
    +Modify: 2018-10-29 15:02:00.475850377 +0000

      Birth: -

#36 Updated by lamby 6 months ago

Upstream were interested in my patch but it sparked a dicussion around removing this mechanism entirely (yay?) but that has some deeper issues with Flatpak/bind-mounting, etc. I've sent a gentle ping of sorts offering my assistance: https://lists.freedesktop.org/archives/fontconfig/2018-November/006405.html

#37 Updated by lamby 6 months ago

I've pinged upstream here. I'm not really sure what else I can do at the moment. https://lists.freedesktop.org/archives/fontconfig/2018-November/006416.html

#38 Updated by intrigeri 6 months ago

  • Estimated time changed from 7.00 h to 4.00 h

(Removing what's already been accounted for in Q2.)

#39 Updated by intrigeri 6 months ago

  • Target version changed from Tails_3.11 to Tails_3.12

#40 Updated by lamby 5 months ago

From: Chris Lamb <chris@chris-lamb.co.uk>
To: fontconfig@lists.freedesktop.org
Cc: 864082@bugs.debian.org
Subject: Next steps for a reproducible Fontconfig?
Date: Friday, 4 January 2019 1:29 PM

[Adding 864082@bugs.debian.org to CC]

Dear fontconfig maintainers,

I've just spent a coffee-or-two unpicking this to get the latest status
and to load the history back into my brain.

As a bit of background, I'm working on the Reproducible Builds
effort and fontconfig — in its usual usage, or at least in Debian
at the time — generated unreproducible cache files.

This was due to it using the timestamps of each directory in the
`checksum` member of the `_FcCache` struct. This is so that it can
identify which cache files remain valid and/or require regeneration
(or similar logic).

So therefore in June 2017 I sent an initial patch:

  https://lists.freedesktop.org/archives/fontconfig/2017-June/005948.html

… which, after some (private?) discussion regarding the implementation, resulted in:

  https://lists.freedesktop.org/archives/fontconfig/2018-May/006285.html

… and that was merged after some further round-trips in f098adac54:

   https://lists.freedesktop.org/archives/fontconfig/2018-May/006289.html

… which was released as part of:

   $ git tag --contains f098adac54 | head -n1
   2.13.1

So far, so good. However, Johannes Schauer then reported that
fontconfig "still" installs unreproduciby:

  https://bugs.debian.org/864082#101

… so I prepared a new patch:

  https://lists.freedesktop.org/archives/fontconfig/2018-October/006374.html

… and that was "soft NACK'd" in the sense that Keith mentions:

I've dug into this a bit more and I think an architectural change in the
cache files made last year is probably not what we want.

       — https://lists.freedesktop.org/archives/fontconfig/2018-October/006376.html

(I am now inferring that it was this "architectural change" 
resulted in the regression Johannes reported, rather than the bug
being incomplete from the beginning.)

Anyway, the upshot from my proposal was that some larger/different
changes are/were "requested" instead.

Behdad Esfahbod also chimed in with:

I don't like the new mechanism either, but I think it was added to resolve
bind-mounted font dirs

       — https://lists.freedesktop.org/archives/fontconfig/2018-October/006381.html

… in the context of Flatpak apps. Keith then addressed all this
with a branch which he published here:

  https://gitlab.freedesktop.org/keithp/fontconfig

… the most salient commit being (I think?):

  https://gitlab.freedesktop.org/keithp/fontconfig/commit/a04751b2e624d034becec7588159ef2f9a8dfc1b

Since then, I don't believe there has been any review of this
branch both in the sense of the code itself but also in terms of
the architectural changes that it implies. I might be able to help
on the former front but without knowing the "lore" of Fontconfig I
simply cannot comment on the latter parts.

Anyway, I'd love to get this resolved once and for all ideally get
it into Debian buster which is about to start "freezing" very
soon.

What would be the best way for me to help here? Can I entreat Keith
to merge his branch? I can put some cycles onto this issue if that is
of some assistance.

Best wishes,

-- 
Chris Lamb
chris-lamb.co.uk / @lolamby

https://lists.freedesktop.org/archives/fontconfig/2019-January/006420.html / https://bugs.debian.org/864082#120

#41 Updated by intrigeri 5 months ago

#42 Updated by intrigeri 5 months ago

#43 Updated by lamby 5 months ago

  • Subject changed from fontconfig cache is not reproducible on Buster to fontconfig cache is not reproducible in Buster

#44 Updated by lamby 5 months ago

  • Blocks Feature #16285: feature/buster branch is not reproducible added

#45 Updated by lamby 5 months ago

  • Assignee changed from lamby to intrigeri

Please check the master branch at https://salsa.debian.org/lamby/pkg-tails-fontconfig.

ps. Should the Target Version be 4.0 here?

#46 Updated by lamby 5 months ago

  • Assignee changed from intrigeri to lamby

Taking ticket back ; upstream's test suite fails at the last second!!

#47 Updated by lamby 5 months ago

  • Assignee changed from lamby to intrigeri
  • Target version changed from Tails_3.12 to Tails_4.0

Apologies for that - it had got 90% of the way through the testsuite and I sent the message prematurely instead of waiting before it had definitely passed before hitting Submit... :D

Try 8abd008bc6fcecbdf45a45f2fd7a504054ddfe19 on master at git@salsa.debian.org:lamby/pkg-tails-fontconfig.git

#48 Updated by intrigeri 5 months ago

  • Assignee changed from intrigeri to lamby
  • % Done changed from 10 to 20
  • QA Check changed from Dev Needed to Ready for QA

Thanks! Built and uploaded to feature-buster.

The URL for the 1st CI job that will benefit from this will probably be: https://jenkins.tails.boum.org/job/reproducibly_build_Tails_ISO_feature-buster/71/. Please keep an eye on it and check whether this fixes the fontconfig cache reproducibility issues :)

#49 Updated by lamby 5 months ago

  • Status changed from In Progress to Fix committed
  • % Done changed from 20 to 90
  • QA Check changed from Ready for QA to Pass

This is fixed in Tails, but keeping this open for the time being as a) the build is not entirely reproducible yet and it b) it would be nice to get a fix upstream and/or in Debian.

Marking as "passed QA", "fix committed" & "90% done" however.

#50 Updated by lamby 5 months ago

  • Blocks deleted (Feature #16285: feature/buster branch is not reproducible)

#51 Updated by lamby 5 months ago

  • Related to Feature #16285: feature/buster branch is not reproducible added

#52 Updated by intrigeri 4 months ago

  • Estimated time changed from 4.00 h to 0.00 h

Removing what's been accounted for in 2018Q4. Shout if you need more :)

#54 Updated by intrigeri 4 months ago

Some upstream movement https://lists.freedesktop.org/archives/fontconfig/2019-January/006464.html

Yeah, I'm super happy the Flatpak folks and fontconfig crew have been collaborating towards a solution that satisfies Flatpak's needs, without breaking reproducibility! :)

#55 Updated by intrigeri 2 months ago

#56 Updated by intrigeri 2 months ago

#57 Updated by intrigeri about 2 months ago

  • Status changed from Fix committed to In Progress

Let's mark this fix committed once we have a reproducible feature/buster build.

#58 Updated by intrigeri about 2 months ago

  • Status changed from In Progress to Resolved
  • Assignee deleted (lamby)
  • % Done changed from 90 to 100

Also available in: Atom PDF