Project

General

Profile

Bug #5571

Test suite: filesystem shares vs. snapshots

Added by Tails over 6 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Test suite
Target version:
Start date:
08/08/2013
Due date:
07/15/2015
% Done:

100%

Feature Branch:
test/5571-no-more-filesystem-shares
Type of work:
Code
Blueprint:
Starter:
No
Affected tool:

Description

Filesystem shares cannot (due to QEMU limitations) be added to an active VM, and cannot (due to QEMU limitations) be active (i.e. mounted) during a snapshot save. Hence unmounting before save and remounting after restore them seems like a good idea.

However, the 9p* modules seem to get into a broken state during a save + restore (the "tags" used as mount source cannot be found), so unloading before save and reload after restore comes to mind. But loading 9pnet_virtio fails after a restore with probe of virtio2 failed with error -2 (in dmesg) and the shares remain unmountable.

We should research this further, and determine whether we're just doing something wrong, or if this is an upstream bug.

<blockquote>

Given we don't use 9p shares at all, do we care more than what's needed to just quickly report this to Debian and be done with it?

Actually we do: git grep "I setup a filesystem share". This bug is what forces us to "drop" the background snapshot in the MAT can clean a PDF file scenario. But it's perhaps not a big issue. I'm fine with closing this bug and living with this as a limitation, I just want someone else's oppinion. After all, we have more important stuff to do.

</blockquote>

qemu_2.1+dfsg-12+deb8u2-with-p9-live-migration.patch View (8.79 KB) anonym, 09/13/2015 11:36 AM


Subtasks

Feature #6224: Reproduce the 9p reload bug with a recent kernelResolved

Feature #6225: Report the 9p kernel bugRejected

Feature #6226: Ask about FS shares vs. snapshots on libvirt listResolved

Bug #10214: Get 9p live migration support into QEMURejected

Associated revisions

Revision 914f0271 (diff)
Added by anonym about 3 years ago

Add a helper to easily share files from the host to the guest.

It's an alternative to filesystem shares, which cannot be used together
with snapshots.

Will-fix: #5571

Revision a11c39b6 (diff)
Added by anonym about 3 years ago

Share files with the guest without using filesystem shares.

So now we can use snapshots in the affected scenarios, which this commit
also implements.

Will-fix: #5571

Revision 8bc7ed16 (diff)
Added by anonym about 3 years ago

Remove support for filesystem shares.

We don't use them any more due to their incompatibility with snapshots,
which we rely heavily on.

Will-fix: #5571

Revision aecf387e (diff)
Added by intrigeri almost 3 years ago

Drop forgotten debugging statement (refs: #5571).

Revision 05e8bebc
Added by intrigeri almost 3 years ago

Merge branch 'test/5571-no-more-filesystem-shares' into stable (Fix-committed: #5571)

Revision e081742a (diff)
Added by intrigeri almost 3 years ago

Drop documentation about filesystem shares: we don't use them anymore (refs: #5571).

History

#1 Updated by intrigeri over 6 years ago

  • Starter set to No

The August dev meeting said that this is no practical issue right now, since we only use FS shares for things that don't care about these bugs:

  • MAT: should be moved to a dedicated feature anyway
  • usb_install: doesn't use background snapshots

... but a limitation in what we can do in the future, when we want to
have libvirt fs shares work even when scenarios are using
a background snapshot.

So, we want to report this bug where it should be.

#2 Updated by intrigeri over 6 years ago

  • Type of work changed from Discuss to Code

#3 Updated by intrigeri almost 6 years ago

  • Category set to Test suite

#4 Updated by BitingBird over 5 years ago

  • Subject changed from test suite: fs shares vs snapshots to Test suite: fs shares vs snapshots

#5 Updated by intrigeri over 5 years ago

  • Subject changed from Test suite: fs shares vs snapshots to Test suite: filesystem shares vs. background snapshots

#6 Updated by intrigeri over 5 years ago

  • Tracker changed from Feature to Bug

#7 Updated by anonym almost 5 years ago

  • Target version set to Tails_1.4.1

#9 Updated by anonym almost 5 years ago

  • Assignee set to anonym

#10 Updated by kytv over 4 years ago

Edit: wrong ticket.

#11 Updated by intrigeri over 4 years ago

  • Target version changed from Tails_1.4.1 to Tails_1.5

#13 Updated by anonym over 4 years ago

  • Target version changed from Tails_1.5 to Tails_1.6

#15 Updated by anonym about 4 years ago

I have some pretty good news! Live migration of (p9) shared filesystems can be fixed with this patch series ([Qemu-devel] [PATCH 0/2] Virtio-9p live migration patchset). So, yeah, this is a QEMU problem, not a libvirt one. It is pretty obvious to me now, years later when I have a much better understanding of our virtualization stack, but I am still apparently brain-dead enough to not reconsider what I have been reading in these tickets since then. Oh well...

So, the patches apply cleanly on Jessie's QEMU, 2.1+dfsg-12+deb8u2, and I have attached a Debian source patch that applies against that version. With the patched QEMU, the filesystem shares work after restoring a snapshot without any extra work (e.g. no unmounting + remounting). Woo! For us to use this we'd need to rethink how we use the shares completely, but that's a later question.

So how do we proceed? It seems to me that the current children tickets are irrelevant. New ones should be created about communicating our need for this on qemu-devel@, and possibly try to refresh the patches for QEMU's current master (on patch doesn't apply cleanly any more). What do you think?

#18 Updated by intrigeri about 4 years ago

So how do we proceed? It seems to me that the current children tickets are irrelevant. New ones should be created about communicating our need for this on qemu-devel@, and possibly try to refresh the patches for QEMU's current master (on patch doesn't apply cleanly any more). What do you think?

I say close the obsolete tickets and create new ones.

Any estimate wrt. how close to be applied upstream these patches are?

#19 Updated by anonym about 4 years ago

  • Type of work changed from Code to Wait

intrigeri wrote:

So how do we proceed? It seems to me that the current children tickets are irrelevant. New ones should be created about communicating our need for this on qemu-devel@, and possibly try to refresh the patches for QEMU's current master (on patch doesn't apply cleanly any more). What do you think?

I say close the obsolete tickets and create new ones.

Done.

Any estimate wrt. how close to be applied upstream these patches are?

Nope. The last incarnation of the patch series was sent almost a year ago, and apparently it dropped one of the original patches that fixed a "theoretical potential bug". I've emailed both the original and second submitter to ask if any of them are interested in pushing for it again. => Wait

#20 Updated by intrigeri about 4 years ago

Type of work changed from Code to Wait

I think one subtask should have "Type of work = Wait", but not this parent ticket.

#21 Updated by anonym about 4 years ago

  • Type of work changed from Wait to Code

intrigeri wrote:

Type of work changed from Code to Wait

I think one subtask should have "Type of work = Wait", but not this parent ticket.

Done.

#22 Updated by anonym about 4 years ago

  • Target version changed from Tails_1.6 to Tails_1.7

#23 Updated by sajolida about 4 years ago

Note that Tails 1.7 is passed the milestone for SponsorS (October 15).

#24 Updated by sajolida about 4 years ago

Sorry, actually this was due on July 15 :)

#25 Updated by intrigeri about 4 years ago

This is about an internal commitment. What was promised internally has been completed already. The new subtasks are new, bonus stuff. I don't know how to translate that into Redmine and have no time to look into it further right now, just to try and lower a bit the level of stress in passing.

#26 Updated by anonym about 4 years ago

Implementation detail: We might want to use some variation of the procedure we use for #10288 (see its description).

#27 Updated by anonym about 4 years ago

Hm. I've just seen that one can share folders via SPICE but it requires spice-webdavd which is not packaged in Debian. I couldn't get it to compile inside Tails (neither on Wheezy (too old libsoup) or Jessie (some weird linking error)). Any way, it's an alternative I guess...

#28 Updated by anonym about 4 years ago

  • Target version changed from Tails_1.7 to Tails_1.8

#29 Updated by anonym about 4 years ago

  • Type of work changed from Code to Wait

#31 Updated by intrigeri about 4 years ago

(Encoding the fact that what was promised has been done already.)

#32 Updated by intrigeri about 4 years ago

  • Type of work changed from Wait to Code

Type of work changed from Code to Wait

Reverting again, see #5571#note-20.

#33 Updated by intrigeri about 4 years ago

  • Target version changed from Tails_1.8 to Tails_2.2

Postponing to after January, since times will be a bit crazy until then.

#34 Updated by anonym almost 4 years ago

  • Target version changed from Tails_2.2 to Tails_2.3

I'll give this one some more attention in March.

#35 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.3 to Tails_2.4

#36 Updated by anonym over 3 years ago

  • Target version changed from Tails_2.4 to Tails_2.5

#37 Updated by intrigeri over 3 years ago

  • Target version changed from Tails_2.5 to Tails_2.7

#38 Updated by anonym about 3 years ago

  • Subject changed from Test suite: filesystem shares vs. background snapshots to Test suite: filesystem shares vs. snapshots

Since the hallowed #6094 was merged, we have no "background" snapshots any more.

#39 Updated by anonym about 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check set to Info Needed

Given the complexity of making filesystem shares working with snapshots, I think we should just stop using filesystem shares. We just need a fast channel to transfer data from the host to the guest filesystem (e.g. SPICE, or the remote shell if we solve #11887 + #11888) and then I think we can replace steps like this:

Given I setup a filesystem share containing the Tails ISO

with
Given I plug a USB drive containing the Tails ISO

The only drawbacks (compared to a filesystem share) I can see is that this will need additional space and time for these transferred files. Whatever. At the moment we reboot when we need this, which at least takes a lot more time, and I think we can leave the peak disk usage as-is by just reordering features depending on this (for large files, e.g. the ISO) to run early.

What do you think?

#40 Updated by intrigeri about 3 years ago

  • Assignee changed from intrigeri to anonym
  • QA Check changed from Info Needed to Dev Needed

What do you think?

Sounds good.

#41 Updated by anonym about 3 years ago

I just realized that we don't need the SPICE/remote shell, but can use guestfs to copy in the files to the virtual disk. So this is not blocked at all by anything. Woo!

#42 Updated by anonym about 3 years ago

  • Status changed from Confirmed to In Progress
  • Feature Branch set to test/5571-no-more-filesystem-shares

anonym wrote:

I just realized that we don't need the SPICE/remote shell, but can use guestfs to copy in the files to the virtual disk. So this is not blocked at all by anything. Woo!

Implemented! Let's see what Jenkins thinks about it.

#43 Updated by bertagaz about 3 years ago

  • Target version changed from Tails_2.7 to Tails_2.9.1

#44 Updated by anonym about 3 years ago

Had to push some fixes. What say you, Jenkins?

#45 Updated by anonym almost 3 years ago

Had to push another fix. Jenkins?

#46 Updated by anonym almost 3 years ago

  • Target version changed from Tails_2.9.1 to Tails 2.10

#47 Updated by anonym almost 3 years ago

And again. Jenkins?

#48 Updated by anonym almost 3 years ago

  • Assignee changed from anonym to intrigeri
  • QA Check changed from Dev Needed to Ready for QA

Jenkins finally seems to have accepted this branch. Please review and merge!

PS: For this branch I tried out a workflow where I only relied on Jenkins' test suite runs in order to get an idea of how easy it would be to contribute test suite work without being able to run it locally. As I expected, anything beside trivial image bumps ends up with many back-and-fourths making this an awkward experience, and the long feedback loop makes this useless for learning how to write tests.

#49 Updated by intrigeri almost 3 years ago

  • Status changed from In Progress to 11

#50 Updated by intrigeri almost 3 years ago

  • Assignee deleted (intrigeri)
  • QA Check changed from Ready for QA to Pass

I've pushed a few fixes on top of the branch and merged into stable and devel. Then I've merged into feature/stretch, and resolved conflicts (including one caused by having dogtail'ified the same code both on feature/stretch and on this branch, but in different ways; meh) the best I could. Cool!

#51 Updated by anonym almost 3 years ago

  • Status changed from 11 to Resolved

Also available in: Atom PDF