Project

General

Profile

Bug #10976

persistence.conf lost, recoverable by reconfiguring

Added by emmapeel almost 3 years ago. Updated 9 days ago.

Status:
In Progress
Priority:
Elevated
Assignee:
Category:
Persistence
Target version:
Start date:
05/09/2018
Due date:
% Done:

100%

QA Check:
Dev Needed
Feature Branch:
Type of work:
Research
Blueprint:
Starter:
Affected tool:

Description

Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.

But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)

The Persistence is easily recovered by reconfiguring it again, with same passphrase.

I guess I should add a note on known issues, and that is how I will resolve this ticket. But I wanted the devs to know that (I think) something on 1.8.2 made the persistence.conf a little bit more fragile.

Not sure what will happen with Tails 2.0, maybe this dissapears.


Related issues

Related to Tails - Feature #14544: Spend software developer time on smallish UX improvements In Progress 08/31/2018
Related to Tails - Bug #15598: Explain loss of persistence.conf in known issues Resolved 05/09/2018

Associated revisions

Revision 0a533354 (diff)
Added by intrigeri 5 months ago

Drop Icedove → Thunderbird migration code.

We've done the switch a year ago, in Tails 3.0, and #15693 shows
that this migration code causes issues, so let's get rid of it.

Besides, I have a hunch that this code is the only possible rationale
explanation for the dreaded "empty persistence.conf file" problem, putting aside
user intervention: it's the only piece of code we have, apart of the Tails
Persistence Setup wizard that affected users tell they did not use, that
modifies persistence.conf (refs: #10976).

History

#1 Updated by intrigeri almost 3 years ago

  • Assignee set to emmapeel
  • QA Check set to Info Needed

Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.

But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)

The Persistence is easily recovered by reconfiguring it again, with same passphrase.

I'm not sure what you mean with "persistence lost". The fact it can be recovered leads me to think that the persistent data is not lost, simply not set up / mounted properly, probably due to a corrupted live-persistence.conf file, and:

  • the user is still proposed to unlock their persistent volume in the Greeter;
  • the data is still available in /live/persistence/TailsData_unlocked once unlocked.

This would be consistent with the fact that re-creating a proper live-persistence.conf by re-configuring the persistence makes it possible to have the persistent data accessible via usual paths again.

Right?

If I got it right, two things:

  • I've heard similar reports regularly for years, so I don't think this problem is new; now, I totally believe you that it apparently happens much more often recently.
  • what we need is the debugging info from a session in which the persistence seems to be lost, and the content of live-persistence.conf in that same session. It's just too sad to always hear about this problem, but never have seen the info that would help us understand what's happening.

Cheers!

#2 Updated by emmapeel almost 3 years ago

intrigeri wrote:

Once in a while this happens since many Tails, maybe the persistence.conf file gets broken.

But since Tails 1.8.2 many of such reports have been made, most of the times only rebooting normally (no emergency shutdown, nothing)

The Persistence is easily recovered by reconfiguring it again, with same passphrase.

I'm not sure what you mean with "persistence lost". The fact it can be recovered leads me to think that the persistent data is not lost, simply not set up / mounted properly, probably due to a corrupted live-persistence.conf file

Exactly!

  • the user is still proposed to unlock their persistent volume in the Greeter;

yes

  • the data is still available in /live/persistence/TailsData_unlocked once unlocked.

yes

This would be consistent with the fact that re-creating a proper live-persistence.conf by re-configuring the persistence makes it possible to have the persistent data accessible via usual paths again.

Right?

If I got it right, two things:

  • I've heard similar reports regularly for years, so I don't think this problem is new; now, I totally believe you that it apparently happens much more often recently.

Yes, I think this last month there were more.

  • what we need is the debugging info from a session in which the persistence seems to be lost, and the content of live-persistence.conf in that same session. It's just too sad to always hear about this problem, but never have seen the info that would help us understand what's happening.

Ok, will try to get this information.

#3 Updated by sajolida over 2 years ago

I experienced this myself in c7425c8012ba2f65e0b9876d11a573f7.

#4 Updated by intrigeri over 2 years ago

  • Category set to Persistence
  • Assignee changed from emmapeel to sajolida
  • QA Check deleted (Info Needed)

sajolida wrote:

I experienced this myself in c7425c8012ba2f65e0b9876d11a573f7.

And in there, searching for "ext4" I find:

COMMAND=/usr/local/sbin/live-persist --read-write --log-file=/var/log/live-persist
activate /dev/mapper/TailsData_unlocked
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): warning: mounting fs with errors, running e2fsck is recommended
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Feb 23 10:31:54 amnesia kernel: EXT4-fs (dm-0): re-mounted. Opts: data=ordered,acl

So, for some reason that filesystem was corrupted (possibly filling the FS as you say you did didn't help ext4 being robust, I dunno, that might be a place where we can improve UX), which may have various interesting user-visible consequences such as files disappearing. I doubt that anything Tails -specific leads to such FS corruption, but one never knows. I'm not quite sure what we should do at this point.

#5 Updated by sajolida over 2 years ago

  • Assignee deleted (sajolida)

Me neither.

#6 Updated by adamantium about 2 years ago

intrigeri wrote:

sajolida wrote:

So, for some reason that filesystem was corrupted (possibly filling the FS as you say you did didn't help ext4 being robust, I dunno, that might be a place where we can improve UX), which may have various interesting user-visible consequences such as files disappearing. I doubt that anything Tails -specific leads to such FS corruption, but one never knows. I'm not quite sure what we should do at this point.

I just experienced this problem after being forced to do a manual upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system partion for automatic upgrade). Data is still there at /live/persistence/TailsData_unlocked, but no Persistent folder under the Places menu.

I am going to backup all the data present there, but it would be helpful if some documentation on how to fix the "Missing Persistent Folder" were present in places other users with the same problem might look. The documentation should address the following questions a user might have:

1. When one opens "configure persistent volume" and sees only Personal Data highlighted with the green checkmark, I experience the question: "If I select (or don't select) these options I think I originally selected when I originally configured Tails umpteen versions ago, will the selection/nonselection cause this wizard to reset/delete my existing files, saved passwords, xmpp user accounts, gpg keys/rings, wifi passwords, etc to a default empty unconfigured state?

2. Even if I copy the contents of /live/persistence/TailsData_unlocked to a burned DVD or external usb hard drive, and the Tails persistence wizard does something I don't like, will I be able to truly restore my data/configs in a meaningful way that allows me to just start using Tails like I always have?

3. Is it possible for me to completely clone this Tails USB drive (perfect bit for bit backup of both the Tails system AND TailsData encrypted partition), perhaps with clonezilla or a clonezilla image?

Troubleshooting documentation needs to be written for this missing (but not deleted) persistence files/directories issue. Perhaps even a tutorial that explains EXACTLY what to do and what NOT to do in the Tails persistence config wizard. We've all experienced some kind of data loss at some time. Not so fun. I'd really like to spare fellow Tails users from the frustration if not nightmare. A good addition to Tails documentation or other destination someone might look at when having the same problem would be a considerate action to Tails users.

#7 Updated by sajolida about 2 years ago

I just experienced this problem after being forced to do a manual
upgrade from Tails 2.5 to 2.6 (insufficient space on Tails system
partion for automatic upgrade).

Did this happen on the first boot after the manual upgrade? or after
some time?

I am going to backup all the data present there

Do you still have the content of persistence.conf in your backups? It's
a file at the root of the persistent volume (also available from
/live/persistence/TailsData_unlocked/persistence.conf).

but it would be
helpful if some documentation on how to fix the "Missing Persistent
Folder" were present in places other users with the same problem
might look. The documentation should address the following questions
a user might have:

Yeap, and thanks for all your very relevant suggestions. This is a very
painful situation and we need to fix that.

Documentation could be one way but we're still gathering information to
see if we could actually prevent this from happening.

If we can't fix this for real, I'm also interested in thinking a bit
more about the user scenario: how would someone goes from the problem
happening to them, to reading and applying this troubleshooting
documentation? Because if we don't work on that, only few people might
apply the doc and likely after already being very confused, scared,
angry, and all.

For example, if the problem is a missing persistence.conf, could this be
detected and some user interface triggered when unlocking the persistence?

#8 Updated by adamantium about 2 years ago

First boot after the manual upgrade.

I checked the persistence.conf, and the file existed, but was completely blank.

Good news is that my backup was unnecessary, as running the persistence configuration wizard recreated a successfully fleshed out persistence.conf. Though it was tedious to create a new tails usb and manually copy the persistence related folders and apply the reassign ownership command, it was successful, and built my confidence in the manual persistence backup process. We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails. Once Tails has more backup/restore functionality plus robust torified VOIP, I see increased adoption as a primary OS.

I like the idea of detecting a blank or corrupted persistence.conf file with a "repair" button for a user to click or possibly "backup, then repair", right there in the Tails Greeter, then enter the persistence password, then check the existing persistence related folders, and recreate a proper persistence.conf.

#9 Updated by elouann about 2 years ago

I checked the persistence.conf, and the file existed, but was completely blank.

I had the same experience, but persistence.conf simply disappeared in my case

We might want to create an automatic persistence backup tool to add to the useability and robustness of Tails.

I agree. Some requirements and tests was published, do you want to have a look?
See https://tails.boum.org/blueprint/backups/

#10 Updated by sajolida about 2 years ago

  • Assignee set to intrigeri

First boot after the manual upgrade.

I checked the persistence.conf, and the file existed, but was
completely blank.

That's interesting... I'm not a file system expert but it looks like:

  • A weird coincidence to have it happen on the very first boot after the
    manual upgrade.
  • A weird file system corruption to empty a file instead of removing it.

I'll let intrigeri add more knowledge here...

We might want to create an automatic persistence backup tool to add
to the useability and robustness of Tails. Once Tails has more
backup/restore functionality plus robust torified VOIP, I see
increased adoption as a primary OS.

That's in our roadmap, hopefully for next year. I will most probably
conduct some user research to understand better what people need and
what would work for them before we pick up amongst the different option
that we have. If you're interested in giving your input on this, and are
fine sharing an email with us, could you send me your address at
.

I like the idea of detecting a blank or corrupted persistence.conf
file with a "repair" button for a user to click or possibly "backup,
then repair", right there in the Tails Greeter, then enter the
persistence password, then check the existing persistence related
folders, and recreate a proper persistence.conf.

Yeap. I thought about something like this as well. We could maybe even
always replace an empty or inexistent persistence.conf with its backup
without prompting the user.

#11 Updated by intrigeri about 2 years ago

  • Target version set to Tails_2.9.1

(Looks like I should have a look at it in a somewhat timely manner.)

#12 Updated by adamantium about 2 years ago

sajolida wrote:

I checked the persistence.conf, and the file existed, but was
completely blank.

That's interesting... I'm not a file system expert but it looks like:

  • A weird coincidence to have it happen on the very first boot after the
    manual upgrade.
  • A weird file system corruption to empty a file instead of removing it.

I agree that it is strange for a file as critical as persistence.conf to be either blanked or deleted. I didn't think this file was touchable without root access (setting password during Tails greeter, etc.). Any speculation as to what could cause this suspicious result?

That's in our roadmap, hopefully for next year. I will most probably
conduct some user research to understand better what people need and
what would work for them before we pick up amongst the different option
that we have. If you're interested in giving your input on this, and are
fine sharing an email with us, could you send me your address at
.

Look for an email from me soon.

Yeap. I thought about something like this as well. We could maybe even
always replace an empty or inexistent persistence.conf with its backup
without prompting the user.

My only concerns here would be:

  • Does the "self-healing" function we might add create a possible vulnerability in Tails? Ex: would it be possible for a malicious agent to copy a malicious code/script/executable/startup_executable to a folder in /live/Persistence/TailsData_unlocked, then find a way to delete/blank/corrupt persistence.conf, and then upon next boot have our self-heal function recreate a persistence.conf that then executes the rogue code every startup?
  • What occasionally produces the blanked/deleted persistence.conf file in the first place? Have we been hacked? [My intent is NOT to be alarmist or offensive. My understanding of things is far above the average user, but not remotely to the level of sajolida and intrigeri. Truly I have extreme respect for the team that makes Tails happen.]

#13 Updated by Mango about 2 years ago

It seems like I currently have this problem after a couple of boot times since I upgraded to latest update. The persistence storage disppaears if you type in passphrase during login.

I thought I had lost the data but I login into a different TailsOS Usb and mount the the USB with the "missing" persistence sotrage. Typed in password and I was able to mount and access those data files.

I checked the persistence.conf file and it shows 0kb. Is their a temporary fix to this problem where it would restore the missing persistence link under 'places'?

#14 Updated by sajolida about 2 years ago

I'm sorry about that :( Please backup your data and try to reconfigure
the persistent volume from Applications → Tails → Configure persistent
volume.

#15 Updated by intrigeri almost 2 years ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#16 Updated by intrigeri almost 2 years ago

  • Subject changed from Persistence lost, recoverable by reconfiguring to persistence.conf lost, recoverable by reconfiguring
  • Assignee changed from intrigeri to emmapeel
  • Target version deleted (Tails 2.10)
  • QA Check set to Info Needed

I've just re-read this entire ticket and sadly, "what we need is the debugging info from a session in which the persistence seems to be lost" still holds true.

Regarding when this problem happens immediately after a manual upgrade: I really don't understand. The manual upgrade process does not unlock the persistent filesystem, and it doesn't even touch that partition, so I don't see how it could empty persistence.conf in there. In this case even more than in the general case, I'll need debugging info to try and understand what's happening.

So I'm adding help desk people as watchers, and hopefully, one of these days, a user will send a bug report during the session that's affected by the problem, and our amazing help desk will forward the debugging info to me. And when you get such a bug report, please ask the OP:

  • the output of df -h;
  • the output of ls -l /live/persistence/*_unlocked/*.conf* (I'm interested in the mtime, not just in the size);
  • the output of ls -l /live/persistence/;
  • the output of getfacl /live/persistence/*_unlocked /live/persistence/*_unlocked/*.conf*;
  • whether they have reconfigured their persistence, in any way, during the previous Tails session (goal: pinpoint a bug in the persistence setup software);
  • whether they have done a clean shutdown to end the previous Tails session (i.e. via the GNOME top-right menu).

#17 Updated by sajolida almost 2 years ago

https://twitter.com/seancc317/status/822966170278051841

Jan 22
sean collins-cruz
@seancc317

@Tails_live what can we do if persistent folder is gone after update? Am I totally screwed?

Seems to be the same and also happened after an upgrade.

#18 Updated by emmapeel 9 months ago

  • Assignee changed from emmapeel to sajolida

I found out about a group of users that were resorting to a very painful workaround:

WHen this happened to them, they will start another Tails from scratch and copy the contents, instead of just recofinguring the persistent.conf file. Ouch!

Couldn't we at least add some documentation somewhere while we wait for this information?

#19 Updated by sajolida 9 months ago

  • Blocks Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing added

#20 Updated by adamantium 9 months ago

I recently had the "blanked 0k persistence.conf" phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time. There might have been some lack of proper shutdown (just pull the usb out of the computer and wait for the auto "security" shutdown). I can't remember the details, and I don't have a debug log (sorry sajolida and intrigeri). I also don't know how to create or find such debug logs.

Do we have any more ideas on the possibility of detecting whether persistent features are present and self-healing by automatically recreating a new persistence.conf? Any further speculation on what could cause a blanked persistence.conf? This is truly mystifying.

#21 Updated by sajolida 8 months ago

  • Blocks deleted (Feature #14758: Core work 2017Q4 → 2018Q1: Technical writing)

#22 Updated by sajolida 8 months ago

  • Blocks Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing added

#23 Updated by sajolida 8 months ago

I'm out of budget for this quarter.

#24 Updated by sajolida 7 months ago

  • Assignee changed from sajolida to cbrownstein

Cody: I'm assiging this one to you as part of the big problem space "Persistent storage vs Backups".
Please check which parts of it are the best to tackle first.

#25 Updated by intrigeri 7 months ago

adamantium wrote:

I recently had the "blanked 0k persistence.conf" phenomena happen for me again. This was maybe a few boots after I manually upgraded to Tails 3.5. I successfully backed up and ran the configure persistence wizard and got it back, same as last time.

See #10976#note-16 :)

#26 Updated by intrigeri 7 months ago

I got one report from help desk, sent on Apr 2, about this with some useful debugging info. tl;dr:

  • the persistent FS has plenty of free space
  • there's no *.conf.insecure_disabled config file
  • all permissions and ACLs are good
  • /live/persistence/TailsData_unlocked was last modified on Apr 1 at 06:20; same for persistence.conf; so something definitely happened at that time
  • persistence.conf is empty
  • live-additional-software.conf was last modified on Apr 1 at 08:06 and is empty too
  • The user says they didn't reconfigure their persistence nor use the emergency shutdown during the last session the persistent volume was working normally.

I've not found anything in our code, nor in live-boot, that touches persistence.conf except Tails Persistence Setup and live-persist (and the latter can only rename it to persistence.conf.insecure_disabled if permissions are wrong). So I fail to understand how this can happen without user intervention :/

#27 Updated by sajolida 5 months ago

  • Assignee changed from cbrownstein to intrigeri
  • QA Check deleted (Info Needed)

How hard would it be possible to have a backup of persistence.conf, detect when the bug happens, and restore automatically from that?

  • persistence.conf.bak would get updated every time persistence.conf is modified by the user interface.
  • When unlocking the persistence, if persistence.conf is empty or unexisting we could compare the two files and copy back persistence.conf.bak onto persistence.conf.

This would automate what we are asking users to do when the bug happens...

#28 Updated by sajolida 5 months ago

  • Blocks deleted (Feature #15411: Core work 2018Q2 → 2018Q3: Technical writing)

#29 Updated by sajolida 5 months ago

I rephrased #10976#note-27 after posting it in order to be more neutral. Sorry!

#30 Updated by intrigeri 5 months ago

  • Target version set to Tails_3.10.1
  • QA Check set to Info Needed

Thanks for pushing me to think about workarounds to what looks more and more like an ext4 bug, possibly triggered by user actions that users (understandably) don't expect will trigger any issue and don't think of as potentially related.

I'm happy look into this at some point but it would not be realistic to expect this to happen before September. Meanwhile, whoever sees this happen (in particular Tails contributors) should send me a WhisperBack report + the debug info I've requested 1 year ago on this ticket, both from the very first session during which the problem is detected. Ideally, attach a timeline of what happened in the days before the bug, i.e. when this Tails was booted with persistence enabled, when it was upgraded, etc. (and technical users can look at that debugging info and from the files mtimes, they'll guess what part of the timeline I'm mostly interested in: what happened around these mtimes :) This might help identify and fix the root cause before I spend time looking for a workaround.

And BTW I was wrong: we do have a piece of code that can, if buggy, trigger that: migrate_icedove_to_thunderbird() modifies persistence.conf. It's been there for a year and it would take me a while to understand if the sed-based code can cause any issue, so perhaps a first step would be to drop that code.

#31 Updated by emmapeel 5 months ago

It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note, as I think this is very scary users and potentially leads to lost information, see for example:

https://twitter.com/Vyshakcool95/status/1010193002923376641

#32 Updated by intrigeri 5 months ago

It is great to try to fix this issue for good, but I humbly ask again for a mention in the known issues page or some note,

Agreed: see the #15598 sub-task.

#33 Updated by emmapeel 5 months ago

Sorry, I had overlooked that issue. Great!

#34 Updated by sajolida 5 months ago

emmapeel: Don't hesitate to ping cbrownstein on behalf of front desk on #15598.

#35 Updated by intrigeri 5 months ago

  • Status changed from Confirmed to In Progress

Applied in changeset commit:e0e1b7b5f4b8256f988c3eda58b048041d6a8a5f.

#36 Updated by intrigeri 3 months ago

Dear help desk, 3.9~rc1 removes the code I (vaguely hopefully) suspect was the culprit. But it also adds code that modifies the permissions on live-additional-software.conf on upgrades from 3.8 or older. So please keep an eye on this and tell me whether such issues happen again when running 3.9~rc1 or newer. The timing of the problem happening vs. upgrade to 3.9~rc1 or newer matters so please try to collect this info.

#37 Updated by sajolida 3 months ago

  • Related to Feature #14544: Spend software developer time on smallish UX improvements added

#38 Updated by intrigeri 2 months ago

  • Assignee changed from intrigeri to goupille

intrigeri wrote:

Dear help desk, 3.9~rc1 removes the code I (vaguely hopefully) suspect was the culprit. But it also adds code that modifies the permissions on live-additional-software.conf on upgrades from 3.8 or older. So please keep an eye on this and tell me whether such issues happen again when running 3.9~rc1 or newer. The timing of the problem happening vs. upgrade to 3.9~rc1 or newer matters so please try to collect this info.

Same question for upgrades to 3.9.

#39 Updated by goupille 2 months ago

  • Assignee changed from goupille to intrigeri

at least one user is affected with tails 3.9 (I'm forwarding the bug report to you)

#40 Updated by goupille 2 months ago

goupille wrote:

at least two different users are affected with tails 3.9 (I'm forwarding a bug report to you)

#41 Updated by intrigeri about 2 months ago

I've analyzed report 2abf0655f6cc39a34db3b78ab6ba6a6c:

  • The persistent filesystem had not been unmounted cleanly. So data corruption is to be expected.
  • persistence.conf is not empty: it has /home/amnesia/Persistent. I don't know if the user fixed it already themselves or not.
  • Apparently, timing is unrelated to upgrading Tails (or at least the user does not say so).
  • I don't know if help desk tried to gather the info I've requested on 10976#note-16. I'll try to do it myself.

#42 Updated by intrigeri 25 days ago

  • Target version changed from Tails_3.10.1 to Tails_3.11

#43 Updated by goupille 19 days ago

intrigeri wrote:

  • I don't know if help desk tried to gather the info I've requested on 10976#note-16. I'll try to do it myself.

I just resent you a report from another user with those informations (ticket number in the subject)

#44 Updated by pablonatalino 18 days ago

I experienced this myself now.

#45 Updated by intrigeri 15 days ago

Analyzed 4dc80ae26f8e3397fbe1437e1e1b232f: empty config file, mtime is ~20 minutes before the user sent their bug report. Permissions look OK, no disabled config file, no disk space issue.

Same for "Upgrade problems PLEASE HELP" except mtime is 2 days before help desk forwarded me the report (I was not told when the report was sent).

#46 Updated by sajolida 13 days ago

Do I understand correctly and the fix proposed in #10976#note-36 was not enough to solve the issue?

If so, should we consider building resiliency mechanisms even though we don't know what is the root cause of the problem, like proposed in #10976#note-27?

#47 Updated by intrigeri 9 days ago

  • QA Check changed from Info Needed to Dev Needed

Do I understand correctly and the fix proposed in #10976#note-36 was not enough to solve the issue?

Yes.

If so, should we consider building resiliency mechanisms even though we don't know what is the root cause of the problem, like proposed in #10976#note-27?

Yes, that's the plan I agreed with and why this ticket is assigned to me with this target version. I'm not happy I did not manage to get to it since mid-September but it's not because I think it's unimportant :/

#48 Updated by sajolida 9 days ago

Thanks! And sorry if my request for clarification sounded pushy.

#49 Updated by intrigeri 9 days ago

  • Related to Bug #15598: Explain loss of persistence.conf in known issues added

#50 Updated by intrigeri 9 days ago

  • Priority changed from Normal to Elevated

Also available in: Atom PDF