Erase video memory on shutdown
Ways to recover video memory from a shutdown system do exist: https://hsmr.cc/palinopsia/. Hence, Tails should erase that memory on shutdown.
Talking with a friend of mine who used to work on video card drivers for X.org (slightly paraphrased, didn't need to take you through the entire conversation):
<brutalchaos> MostAwesomeDude: is it possible to erase video ram for all types of video cards during shutdown? <MostAwesomeDude> You can't clear the VRAM until the computer's off, because the BIOS will keep writing to it up until the last second. <brutalchaos> Is there a way to flush the vram then? <MostAwesomeDude> Basically, become root, find the PCI BAR where VRAM's mapped, and walk over it. There's a couple things called mmapw and mmapr that can help you; Google for them.
Another idea, maybe related to the previous (this isn't really my strong point), is mapping all RAM, including video memory as virtual RAM and with the use of kfree in the kernel, writing all 0's to it.
When this is fixed, the entry in the FAQ should be removed.
#3 Updated by intrigeri about 5 years ago
just before the greeter, he saw "a part of his previous session on the screen while
the rest was scrambled", then it disappeared and the greeter replaced it.
Never seen that, but indeed it seems to be related.
Do we have details about the hardware and graphics drivers?
#12 Updated by cypherpunks about 4 years ago
The code by the palinopsia PoC to read from VRAM might be a useful starting point to instead write into it at shutdown time.
That wouldn't work. According to the PoC page: It allocates a number of texture buffers in VRAM without initializing them, then directly renders them onto the screen, thereby accessing previously used buffers.
As for using the OpenGPL API to write data to VRAM, I don't think that's reliable though, because you cannot write to arbitrary addresses. How do you know it would not just overwrite the same 32 MiB over and over? I guess you could try rendering something massive, but even then it would likely do some weird optimizations that leave some memory untouched.
However, there is a way to access video memory directly, using the slram driver. An (untested) example:
# lspci -vvv VGA 0000:02:00.0 VGA compatible controller: ATI Technologies Inc R300 AD [Radeon 9500 Pro] (prog-if 00 [VGA]) Subsystem: PC Partner Limited: Unknown device 7c07 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B- Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 32 (2000ns min), cache line size 08 Interrupt: pin A routed to IRQ 255 Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M] Region 1: I/O ports at b000 [size=256] Region 2: Memory at e9000000 (32-bit, non-prefetchable) [size=64K] Expansion ROM at e8000000 [disabled] [size=128K] Capabilities:  AGP version 3.0 Status: RQ=256 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW+ AGP3+ Rate=x4,x8 Command: RQ=1 ArqSz=0 Cal=0 SBA+ AGP- GART64- 64bit- FW- Rate=<none> Capabilities:  Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- # modprobe slram map=VRAM,0xd8000000,+0x7c00000 # modprobe mtdblock # dd if=/dev/urandom of=/dev/mtdblock0 bs=4M
#13 Updated by Dr_Whax about 4 years ago
I got this from the folks at #nouveau.
05:45 #nouveau: < DrWhax> mupuf: Hi! I'm a contributor to the Tails team, were looking into ways to effectively clear the VRAM to counter the following attack: https://hsmr.cc/palinopsia/. We looked into slram to directly write to the video memory, I wonder if this is a correct approach or if we could use the nouveau driver for any of this. 05:46 #nouveau: < mupuf> DrWhax: hey, great. You contact me because of the presentation I gave at XDC2012? 05:47 #nouveau: < DrWhax> mupuf: I got refered to you by RSpliet 05:47 #nouveau: < mupuf> ok 05:47 #nouveau: < mupuf> well, this is definitely solvable for all open source drivers 05:48 #nouveau: < mupuf> it is going to come at a cost, but there are mitigation techniques 05:48 #nouveau: < karolherbst> mupuf: there could be a drm boot options maybe? 05:48 #nouveau: < mupuf> karolherbst: security should not be an option 05:48 #nouveau: < mupuf> it should be a default 05:48 #nouveau: < karolherbst> yeah, depends on the costs though 05:48 #nouveau: < mupuf> and if we are smart enough, we may reduce the cost to be negligeable 05:48 #nouveau: < karolherbst> right 05:49 #nouveau: < karolherbst> but then timing might be important too 05:49 #nouveau: < karolherbst> the most secure way is to clear buffers when they get unused 05:49 #nouveau: < mupuf> in any case, the presentation Tim and I gave at XDC2012 contained some information and discussions 05:49 #nouveau: < mupuf> let me find you the link 05:50 #nouveau: < mupuf> karolherbst: not really, you just need to clear them when the pool of cleared buffers is empty 05:50 #nouveau: < karolherbst> mupuf: mhh and in the meantime? 05:50 #nouveau: < mupuf> and you need to allocate buffers from the application's pool as much as possible 05:51 #nouveau: < karolherbst> yeah, but I think they want a rock solid method 05:51 #nouveau: < mupuf> then you need to have a background task that clears buffers 05:51 #nouveau: < karolherbst> where you can't read out memory at any given point in time 05:51 #nouveau: < karolherbst> *used buffers 05:51 #nouveau: < mupuf> a simple method is to clear buffers on demand 05:52 #nouveau: < mupuf> karolherbst: that is not really what the attack is showing 05:52 #nouveau: < mupuf> it shows two different GL apps getting buffers with the content of another app 05:52 #nouveau: < karolherbst> no 05:52 #nouveau: < karolherbst> two different apps 05:52 #nouveau: < mupuf> on unloading the driver: clear the RAM --> this one should be easy to implement 05:53 #nouveau: < mupuf> DrWhax: http://www.x.org/wiki/Events/XDC2012/XDC2012AbstractMartinPeres-Timoth%C3%A9eRavier/ 05:55 #nouveau: < mupuf> DrWhax: We will need this feature in the kernel drivers at some point or another 05:55 #nouveau: < mupuf> so you should not get too much resistance 05:56 #nouveau: < mupuf> but I would suggest you implement the concept of pool of pages per graphic context 05:56 #nouveau: < mupuf> and having a pool of cleared buffers 05:57 #nouveau: < mupuf> whe the pool of cleared buffers is empty, it should reclaim pages from the apps' unused pages 05:58 #nouveau: < mupuf> that should be an effective way of reducing the memory bw consumption needed for clearing buffers 06:00 #nouveau: < mupuf> DrWhax: I am not going to lie to you, this is not a trivial task 06:01 #nouveau: < mupuf> but if you do it in TTM, you get both nouveau and radeon in one go 06:01 #nouveau: < mupuf> if you can do it at the GEM level, you do it for all the open source drivers 06:01 #nouveau: < mupuf> so it has a high reward 06:10 #nouveau: < karolherbst> pmoreau: funny though, the temperature with nvidia isn't really lower 06:31 #nouveau: < DrWhax> mupuf: thanks for the explanation! 06:32 #nouveau: < DrWhax> mupuf: and that's great that we'd be able to get both in one go!
I'm convinced we'd probably need to find someone in order to tackle this ticket. The good thing is, the free and open source software community will benefit from it!
Shall I close the ticket and create another one for waiting till somebody solves it?
#14 Updated by cypherpunks almost 4 years ago
I don't think it's necessary to close it. There are still potential mitigations, such as telling the driver to limit the amount of VRAM to use (which can be done from userspace, usually as a modprobe argument) to a small enough value that it can be overwritten over the AGP bus. Since it would be smaller, the 8 MiB/s speed limit would be less problematic. Having less VRAM would not impact Tails users badly because very little is needed anyway. GPU hardware is packed with memory for video games and the like, so even 128 MiB, for example, may be more than enough.
If that is too dependent on the specific hardware in use, then I'm sure a simple VRAM benchmark program would work to erase a large amount of memory. They certainly exist, such as http://www.mikelab.kiev.ua/index_en.php?page=PROGRAMS/vmt_en (though that is for Windows), which acts to be a version of memtest86 for video memory.
Unfortunately, I think that IRC log shows they are misunderstanding the problem. It looks like they are trying to mitigate the issue of reading from uninitialized buffers, which the palinopsia page talks about the most. Of course that's still a security issue and should be worked on (I've been skimming through the amdgpu source myself to mitigate the issue on my own system), it would not let us wipe VRAM on shutdown. What we need is the ability to either write to arbitrary video memory addresses, which likely can only be done completely from kernelmode and which they are almost certainly not going to expose to userspace, or to allocate large amounts of memory to benchmark the VRAM, which is actually within our grasp.
#15 Updated by intrigeri almost 4 years ago
- Priority changed from Normal to Low
Thanks for the insight! It's good to see that this might be solvable in the end. I agree we should keep this ticket open then. Still, I'm downgrading it to low priority, which means here that it's unlikely that someone on our team fixes it any time soon (due to lack of time and skills), but help and patches would be warmly welcome :)
#16 Updated by cypherpunks over 3 years ago
I'm still thinking about this, but I'm pretty sure I found a way to very rapidly and effectively erase all video memory. It'll require having /dev/mem fully writable, at least by root, but that shouldn't be a problem because the memory wipe occurs after kexec into a new kernel anyway, so the new kernel could contain a writable /dev/mem or a kernel module to create an equivalent character device without compromising Tails' security model. In the lspci example in an above post, writing 128 MiB directly to d8000000 would overwrite some memory on the graphics card, since that region of system memory is mapped to VRAM. In order to overwrite the rest of the memory, the PCI configuration space can be modified to point the mapped region 128 MiB forward, and it could be written to again. Repeat this process until VRAM is full. Both writing to VRAM through memory-mapped addresses and writing to PCI configuration space is very fast (since the former is stored in high-bandwidth VRAM over 2.5-8 GHz channels, and the latter is just kept on registers in the PCI device itself).
Unfortunately, the format and methods of changing the sliding window of mapped memory differs for each major type of graphics card, so we'd have to keep a text file which uses the VID (vendor identification), which is thankfully stored in the same place in all PCI configurations, in order to know how to write to the rest of the PCI configuration space. That'll be necessary for us to reliably move the sliding window so it can be written to again. In the case that the VID is not detected, perhaps it can either be guessed based on how similar it is to an existing model, or the routine can bail and warn the user that VRAM could not be wiped, and tell them to report to the Tails devs the model of their graphics card and its configuration space (lspci -xxxx) so it can be added to the list.
Does that seem like a solution which would be acceptable in Tails? All it takes is a writable /dev/mem once in the memory-wiping initramfs, writing to PCI configuration space, and a text file which matches known VIDs to the routine needed to move the sliding window.
Obviously it would be better for our GPU drivers to just initialize freed memory automatically, but we know that's not going to happen soon for any sizeable number of COTS devices!
#17 Updated by cypherpunks over 3 years ago
Heh, I just realized that what I described was basically what was already posted in the OP. I'm kind of surprised it took so long for it to be actually understood. Slide the PCI bar and write over it each time. Should be very fast.
<MostAwesomeDude> Basically, become root, find the PCI BAR where VRAM's mapped, and walk over it. There's a couple things called mmapw and mmapr that can help you; Google for them.
Another possibility might be to shut down the PCI device, which might be possible either through sysfs, or by modifying the PCI configuration space directly over memory-mapped I/O (since iopl() access is allowed with CAP_SYS_RAWIO, at least until we have grsecurity). I'm not sure if shutting down the PCI device would cut power to the VRAM, but it might. Since wiping system RAM takes a little time, it would give the VRAM a head start to fade if it starts right before the main memory wipes (and right after the wipe message is displayed, of course).
But I think sliding and wiping the memory-mapped BAR is better.