Project

General

Profile

Feature #5356

Erase video memory on shutdown

Added by Tails over 5 years ago. Updated about 2 years ago.

Status:
Confirmed
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

QA Check:
Feature Branch:
Type of work:
Research
Blueprint:
Starter:
No
Affected tool:

Description

Ways to recover video memory from a shutdown system do exist: https://hsmr.cc/palinopsia/. Hence, Tails should erase that memory on shutdown.

Talking with a friend of mine who used to work on video card drivers for X.org (slightly paraphrased, didn't need to take you through the entire conversation):

<brutalchaos> MostAwesomeDude: is it possible to erase video ram for all types of video cards during shutdown?
<MostAwesomeDude> You can't clear the VRAM until the computer's off, because the BIOS will keep writing to it up until the last second.
<brutalchaos> Is there a way to flush the vram then?
<MostAwesomeDude> Basically, become root, find the PCI BAR where VRAM's mapped, and walk over it. There's a couple things called mmapw and mmapr that can help you; Google for them.

Another idea, maybe related to the previous (this isn't really my strong point), is mapping all RAM, including video memory as virtual RAM and with the use of kfree in the kernel, writing all 0's to it.

When this is fixed, the entry in the FAQ should be removed.


Related issues

Related to Tails - Bug #9099: Document that Tails may leave traces in video memory Resolved 03/23/2015

History

#1 Updated by BitingBird over 4 years ago

  • Subject changed from erase video memory on shutdown to Erase video memory on shutdown
  • Description updated (diff)
  • Starter set to No

#2 Updated by goupille almost 4 years ago

not sure if this is related, but a user experienced that :

just before the greeter, he saw "a part of his previous session on the screen while the rest was scrambled", then it disappeared and the greeter replaced it.

we can't recontact that user.

#3 Updated by intrigeri almost 4 years ago

just before the greeter, he saw "a part of his previous session on the screen while
the rest was scrambled", then it disappeared and the greeter replaced it.

Never seen that, but indeed it seems to be related.
Do we have details about the hardware and graphics drivers?

#4 Updated by BitingBird over 3 years ago

About this topic, see https://hsmr.cc/palinopsia/

#5 Updated by intrigeri over 3 years ago

  • Description updated (diff)

#6 Updated by intrigeri over 3 years ago

  • Description updated (diff)

#7 Updated by intrigeri over 3 years ago

  • Description updated (diff)

#8 Updated by intrigeri over 3 years ago

The code by the palinopsia PoC to read from VRAM might be a useful starting point to instead write into it at shutdown time.

#9 Updated by intrigeri over 3 years ago

  • Related to Bug #9099: Document that Tails may leave traces in video memory added

#10 Updated by BitingBird over 3 years ago

  • Description updated (diff)

desc: when fixed, remove FAQ entry.

#11 Updated by DanOver over 3 years ago

The OpenGL API can be used to write random data in VRAM, or not?

#12 Updated by cypherpunks almost 3 years ago

intrigeri wrote:

The code by the palinopsia PoC to read from VRAM might be a useful starting point to instead write into it at shutdown time.

That wouldn't work. According to the PoC page: It allocates a number of texture buffers in VRAM without initializing them, then directly renders them onto the screen, thereby accessing previously used buffers.

As for using the OpenGPL API to write data to VRAM, I don't think that's reliable though, because you cannot write to arbitrary addresses. How do you know it would not just overwrite the same 32 MiB over and over? I guess you could try rendering something massive, but even then it would likely do some weird optimizations that leave some memory untouched.

However, there is a way to access video memory directly, using the slram driver. An (untested) example:

# lspci -vvv VGA
0000:02:00.0 VGA compatible controller: ATI Technologies Inc R300 AD [Radeon 9500 Pro] (prog-if 00 [VGA])
        Subsystem: PC Partner Limited: Unknown device 7c07
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2000ns min), cache line size 08
        Interrupt: pin A routed to IRQ 255
        Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M]
        Region 1: I/O ports at b000 [size=256]
        Region 2: Memory at e9000000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at e8000000 [disabled] [size=128K]
        Capabilities: [58] AGP version 3.0
                Status: RQ=256 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW+ AGP3+ Rate=x4,x8
                Command: RQ=1 ArqSz=0 Cal=0 SBA+ AGP- GART64- 64bit- FW- Rate=<none>
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
# modprobe slram map=VRAM,0xd8000000,+0x7c00000
# modprobe mtdblock
# dd if=/dev/urandom of=/dev/mtdblock0 bs=4M

#13 Updated by Dr_Whax over 2 years ago

I got this from the folks at #nouveau.

05:45 #nouveau: < DrWhax> mupuf: Hi! I'm a contributor to the Tails team, were looking into ways to effectively clear the VRAM to counter the following attack: https://hsmr.cc/palinopsia/. We looked into slram to directly write to the video memory, I wonder if this is a correct approach or if we could use the nouveau driver for any of this.
05:46 #nouveau: < mupuf> DrWhax: hey, great. You contact me because of the presentation I gave at XDC2012?
05:47 #nouveau: < DrWhax> mupuf: I got refered to you by RSpliet 
05:47 #nouveau: < mupuf> ok
05:47 #nouveau: < mupuf> well, this is definitely solvable for all open source drivers
05:48 #nouveau: < mupuf> it is going to come at a cost, but there are mitigation techniques
05:48 #nouveau: < karolherbst> mupuf: there could be a drm boot options maybe?
05:48 #nouveau: < mupuf> karolherbst: security should not be an option
05:48 #nouveau: < mupuf> it should be a default
05:48 #nouveau: < karolherbst> yeah, depends on the costs though
05:48 #nouveau: < mupuf> and if we are smart enough, we may reduce the cost to be negligeable
05:48 #nouveau: < karolherbst> right
05:49 #nouveau: < karolherbst> but then timing might be important too
05:49 #nouveau: < karolherbst> the most secure way is to clear buffers when they get unused
05:49 #nouveau: < mupuf> in any case, the presentation Tim and I gave at
XDC2012 contained some information and discussions
05:49 #nouveau: < mupuf> let me find you the link
05:50 #nouveau: < mupuf> karolherbst: not really, you just need to clear them when the pool of cleared buffers is empty
05:50 #nouveau: < karolherbst> mupuf: mhh and in the meantime?
05:50 #nouveau: < mupuf> and you need to allocate buffers from the application's pool as much as possible
05:51 #nouveau: < karolherbst> yeah, but I think they want a rock solid method
05:51 #nouveau: < mupuf> then you need to have a background task that clears buffers
05:51 #nouveau: < karolherbst> where you can't read out memory at any given point in time
05:51 #nouveau: < karolherbst> *used buffers
05:51 #nouveau: < mupuf> a simple method is to clear buffers on demand
05:52 #nouveau: < mupuf> karolherbst: that is not really what the attack is showing
05:52 #nouveau: < mupuf> it shows two different GL apps getting buffers with the content of another app
05:52 #nouveau: < karolherbst> no
05:52 #nouveau: < karolherbst> two different apps
05:52 #nouveau: < mupuf> on unloading the driver: clear the RAM --> this one should be easy to implement
05:53 #nouveau: < mupuf> DrWhax:
http://www.x.org/wiki/Events/XDC2012/XDC2012AbstractMartinPeres-Timoth%C3%A9eRavier/
05:55 #nouveau: < mupuf> DrWhax: We will need this feature in the kernel drivers at some point or another
05:55 #nouveau: < mupuf> so you should not get too much resistance
05:56 #nouveau: < mupuf> but I would suggest you implement the concept of pool of pages per graphic context
05:56 #nouveau: < mupuf> and having a pool of cleared buffers
05:57 #nouveau: < mupuf> whe the pool of cleared buffers is empty, it should reclaim pages from the apps' unused pages
05:58 #nouveau: < mupuf> that should be an effective way of reducing the memory bw consumption needed for clearing buffers
06:00 #nouveau: < mupuf> DrWhax: I am not going to lie to you, this is not a trivial task
06:01 #nouveau: < mupuf> but if you do it in TTM, you get both nouveau and radeon in one go
06:01 #nouveau: < mupuf> if you can do it at the GEM level, you do it for all the open source drivers
06:01 #nouveau: < mupuf> so it has a high reward 
06:10 #nouveau: < karolherbst> pmoreau: funny though, the temperature with nvidia isn't really lower
06:31 #nouveau: < DrWhax> mupuf: thanks for the explanation!
06:32 #nouveau: < DrWhax> mupuf: and that's great that we'd be able to get both in one go!

I'm convinced we'd probably need to find someone in order to tackle this ticket. The good thing is, the free and open source software community will benefit from it!

Shall I close the ticket and create another one for waiting till somebody solves it?

#14 Updated by cypherpunks over 2 years ago

I don't think it's necessary to close it. There are still potential mitigations, such as telling the driver to limit the amount of VRAM to use (which can be done from userspace, usually as a modprobe argument) to a small enough value that it can be overwritten over the AGP bus. Since it would be smaller, the 8 MiB/s speed limit would be less problematic. Having less VRAM would not impact Tails users badly because very little is needed anyway. GPU hardware is packed with memory for video games and the like, so even 128 MiB, for example, may be more than enough.

If that is too dependent on the specific hardware in use, then I'm sure a simple VRAM benchmark program would work to erase a large amount of memory. They certainly exist, such as http://www.mikelab.kiev.ua/index_en.php?page=PROGRAMS/vmt_en (though that is for Windows), which acts to be a version of memtest86 for video memory.

Unfortunately, I think that IRC log shows they are misunderstanding the problem. It looks like they are trying to mitigate the issue of reading from uninitialized buffers, which the palinopsia page talks about the most. Of course that's still a security issue and should be worked on (I've been skimming through the amdgpu source myself to mitigate the issue on my own system), it would not let us wipe VRAM on shutdown. What we need is the ability to either write to arbitrary video memory addresses, which likely can only be done completely from kernelmode and which they are almost certainly not going to expose to userspace, or to allocate large amounts of memory to benchmark the VRAM, which is actually within our grasp.

#15 Updated by intrigeri over 2 years ago

  • Priority changed from Normal to Low

Thanks for the insight! It's good to see that this might be solvable in the end. I agree we should keep this ticket open then. Still, I'm downgrading it to low priority, which means here that it's unlikely that someone on our team fixes it any time soon (due to lack of time and skills), but help and patches would be warmly welcome :)

#16 Updated by cypherpunks about 2 years ago

I'm still thinking about this, but I'm pretty sure I found a way to very rapidly and effectively erase all video memory. It'll require having /dev/mem fully writable, at least by root, but that shouldn't be a problem because the memory wipe occurs after kexec into a new kernel anyway, so the new kernel could contain a writable /dev/mem or a kernel module to create an equivalent character device without compromising Tails' security model. In the lspci example in an above post, writing 128 MiB directly to d8000000 would overwrite some memory on the graphics card, since that region of system memory is mapped to VRAM. In order to overwrite the rest of the memory, the PCI configuration space can be modified to point the mapped region 128 MiB forward, and it could be written to again. Repeat this process until VRAM is full. Both writing to VRAM through memory-mapped addresses and writing to PCI configuration space is very fast (since the former is stored in high-bandwidth VRAM over 2.5-8 GHz channels, and the latter is just kept on registers in the PCI device itself).

Unfortunately, the format and methods of changing the sliding window of mapped memory differs for each major type of graphics card, so we'd have to keep a text file which uses the VID (vendor identification), which is thankfully stored in the same place in all PCI configurations, in order to know how to write to the rest of the PCI configuration space. That'll be necessary for us to reliably move the sliding window so it can be written to again. In the case that the VID is not detected, perhaps it can either be guessed based on how similar it is to an existing model, or the routine can bail and warn the user that VRAM could not be wiped, and tell them to report to the Tails devs the model of their graphics card and its configuration space (lspci -xxxx) so it can be added to the list.

Does that seem like a solution which would be acceptable in Tails? All it takes is a writable /dev/mem once in the memory-wiping initramfs, writing to PCI configuration space, and a text file which matches known VIDs to the routine needed to move the sliding window.

Obviously it would be better for our GPU drivers to just initialize freed memory automatically, but we know that's not going to happen soon for any sizeable number of COTS devices!

#17 Updated by cypherpunks about 2 years ago

Heh, I just realized that what I described was basically what was already posted in the OP. I'm kind of surprised it took so long for it to be actually understood. Slide the PCI bar and write over it each time. Should be very fast.

<MostAwesomeDude> Basically, become root, find the PCI BAR where VRAM's mapped, and walk over it. There's a couple things called mmapw and mmapr that can help you; Google for them.

Another possibility might be to shut down the PCI device, which might be possible either through sysfs, or by modifying the PCI configuration space directly over memory-mapped I/O (since iopl() access is allowed with CAP_SYS_RAWIO, at least until we have grsecurity). I'm not sure if shutting down the PCI device would cut power to the VRAM, but it might. Since wiping system RAM takes a little time, it would give the VRAM a head start to fade if it starts right before the main memory wipes (and right after the wipe message is displayed, of course).

But I think sliding and wiping the memory-mapped BAR is better.

Also available in: Atom PDF