[regression since 12.04] ntfs-3g refuses to mount Windows8 not using hibernation

It sounds like the behavior change since 11.10 is because ntfs-3g itself has started behaving more safely. I think the correct fix for us is to document this in the release notes.

(Preferably with a workaround telling users how to disable kernel hibernate in Windows 8; see related bug #1042159 in wubi)

In the above situation (full shutdown, with hibernation configured, but not enabled), what are the outputs of :
ls -l /windows/hiberfil.sys
head -c 8 /windows/hiberfil.sys | od -t x1
(/windows to be replaced by the mount point of the Windows8 system partition)
This obviously requires mounting in read-only mode (mount option "ro")

Note : refusing to mount is probably not a regression, but the correct decision in the presence of data concealed by Windows8 and not synced to disk, and there probably are cases where ntfs-3g should refuse to mount but does not.

According to https://bugs.launchpad.net/wubi/+bug/1042159/comments/12 , the problem seems to appear when using Win8 "Restart", not "ShutDown". Updating bug description.

For those reading French, please see some test results here: http://forum.ubuntu-fr.org/viewtopic.php?pid=10585481#p10585481

************ Here is my SUMMARY:

- Malbo's post#34 Steps 1, 4 & 7: after using Win8RP "Hibernation" (long sleep), the hiberfil.sys md5 has always changed (Steps 4 and 7), and it always left the "48 49 42 52 09 00 00 00" signature:
head -c 8 /media/Acer/hiberfil.sys | od -t x1
0000000 48 49 42 52 09 00 00 00
0000010

- Steps 2, 5 et 6: after using Win8RP "Restart", Malbo observed 2 cases: once the signature changed (Step 5), twice it didn't change (Step 2 and 6). So once the signature remained "48 49 42 52 09 00 00 00" (Step2), but twice it became or remained "57 41 4b 45 09 00 00 00" (Step 5 and 6).

- Step 3 and post#37: after using Win8RP "ShutDown", the signature did not change: at Step3 it remained "48 49 42 52 09 00 00 00", and in #37 it remained "57 41 4b 45 09 00 00 00".

************ My current CONCLUSIONS:

1) if signature is NOT "48 49 42 52 09 00 00 00", then Win8RP is NOT HIBERNATED. It is either restarted or ShutDown.
2) if signature is "48 49 42 52 09 00 00 00", then Win8RP is either hibernated, or restarted, or shutdown.

--> checking the result of "head -c 8 /media/Acer/hiberfil.sys | od -t x1" is NOT ALWAYS enough to determine if Win8RP is hibernated or not.

Now the question is if these signatures are static, or depend on a particular Kernel / Patch level / Update applied.

and another question is to find another clue (when "48 49 42 52 09 00 00 00") to make the difference between hibernation and other states. Maybe the answer is located inside $LogFile or the rest of hiberfil.sys . See Jean-Pierre's test call : http://tuxera.com/forum/viewtopic.php?f=2&t=29601&sid=09167e1a650c27499c66ca74b41118b1

Part of the trouble seems to be generated by the "hybrid sleep" mode of Windows 8. This is evident by comparing the comments # 16 (hybrid sleep enabled) and # 18 (hybrid sleep disabled) for the bug #1042159 : https://bugs.launchpad.net/wubi/+bug/1042159
With hybrid sleep turned off, troubles due to Sleep mode are resolved but there is still the problem of "Turn on Fast Startup" in Windows 8 Power Options" placing the Win8 partition in hibernation state.

Status changed to 'Confirmed' because the bug affects multiple users.

When the "fast restarting" of Windows 8 is enabled, the ntfs partitions of internal disk (this means including non-Windows system partition) may be left in an inconsistent state, and Windows 8 restarts using a saved copy of the metadata (instead of what is actually on disk), leading to potential data loss. The checks based on the hiberfil.sys cannot be relied on, as this file only exists in the Windows system partition. Also note that the "Shut down" button of Windows 8 does not show the kind of restarting (fast or standard) which is triggered.

The only known way to avoid inconsistencies and data losses is to disable the fast restart and hibernation. See :
http://jp-andre.pagesperso-orange.fr/advanced-ntfs-3g.html#windows8

Following Jean-Pierre's comment #10, i created Bug #1053383
("Wubi on Windows8 leads to potential data loss")

For wubi, that means it fails to boot.

I haven't noticed any problems using Wubi or mounting NTFS from normal dual boot (12.04.1) with Windows 8 - fast start is enabled.

The following is from a fresh Quantal install using Wubi:
bcbc@ubuntu:~$ ls -la /host/hiberfil.sys
-rwxrwxrwx 1 root root 5065179136 Sep 19 22:49 /host/hiberfil.sys
bcbc@ubuntu:~$ head -c 8 /host/hiberfil.sys | od -t x1
0000000 57 41 4b 45 09 00 00 00
0000010
bcbc@ubuntu:~$

My previous post was after "Restarting" the computer from Windows 8 and booting into Wubi. The next test was to "Shut down" from Windows 8 and boot Wubi. That also worked, but I've observed a different behaviour when booting Wubi from Windows 8 manager as compared to Win 7 and that is:
1. Computer starts, BIOS screen
2. Windows boot manager, select Ubuntu(Wubi)
3. Computer restarts, BIOS reappears, boots directly into Ubuntu(Wubi)

So it appears that Windows is taking care of the 'fast-start' hibernated image from the Windows Boot Manager when required. (Of course when it's really hibernated it just boots straight into Windows)

It's different when I shut down Windows 8 and then boot the non-Wubi install (12.04.1). Then when I try to mount the windows partition I get:
bcbc@neptune:~$ sudo mount /dev/sda1 /mnt
Windows is hibernated, refused to mount.
Failed to mount '/dev/sda1': Operation not permitted
bcbc@neptune:~$ sudo mount -r /dev/sda1 /mnt
bcbc@neptune:~$ head -c 8 /mnt/hiberfil.sys | od -t x1
0000000 48 49 42 52 09 00 00 00
0000010
bcbc@neptune:~$

AFAIK, leaving Windows 8 by selecting "restart" and rebooting to Linux is safe. Also using the Windows dual-boot manager is safe, because it syncs the cached data to disk before starting another OS. The unsafe situation is booting directly into another OS through another boot manager such as grub.

The test based on hiberfil.sys is only relevant for the Windows system partition. Testing the logfile is apparently better (mind the single quotes, and replace win8 by your mount point) :
head -c 30 '/win8/$LogFile' | tail -c 4 | od -t x1
If you get "01 00 01 00" you are in a safe situation, if you get "00 00 02 00" you are in an unsafe situation. This is to be confirmed, please report what you get.

You would guess that no ntfs-3g version if able to play the new rules, so beware.

For a Windows 8 shutdown and booting (non-wubi) Ubuntu, I get:
bcbc@neptune:~$ head -c 30 '/mnt/$LogFile' | tail -c 4 | od -t x1
0000000 00 00 02 00
0000004

But it won't mount it, except read-only so, in that sense, it's safe. But it would be a pain for a user trying to boot from an Ubuntu USB and trying to partition and install, or someone using Grub.

bcbc@neptune:~$ sudo umount /mnt
bcbc@neptune:~$ sudo mount /dev/sda1 /mnt
Windows is hibernated, refused to mount.
Failed to mount '/dev/sda1': Operation not permitted
The NTFS partition is hibernated. Please resume and shutdown Windows
properly, or mount the volume read-only with the 'ro' mount option, or
mount the volume read-write with the 'remove_hiberfile' mount option.
For example type on the command line:

            mount -t ntfs-3g -o remove_hiberfile /dev/sda1 /mnt

@bcbc : You only looked at the case of "Shut down" but for me, the worst case with Windows 8 RTM, it is not "Shut down", it is the "hybrid sleep" as I explain in this comment : https://bugs.launchpad.net/wubi/+bug/1042159/comments/18

Okay, I hadn't noticed (or heard about) the bit about 'hybrid sleep'.

I switched it on, used the sleep, and then woke up from the sweep.

Then I hit "Restart" and tried to boot into Wubi, but it didn't work. I can't mount the windows partition except in read-only mode.
bcbc@neptune:~$ head -c 30 '/mnt/$LogFile' | tail -c 4 | od -t x1
0000000 01 00 01 00
0000004

After that I performed a "Shutdown" from Windows, Wubi boots fine, including on subsequent "Restarts" from Windows.
bcbc@neptune:~$ head -c 30 '/host/$LogFile' | tail -c 4 | od -t x1
0000000 01 00 01 00
0000004

After going in to Win8 and doing another "Sleep", followed by another "Restart", then Wubi won't boot again. So it seems easily reproducible.
PS in case it matters, this is what hiberfil.sys looks like:
bcbc@neptune:~$ head -c 8 /mnt/hiberfil.sys | od -t x1
0000000 48 49 42 52 09 00 00 00
0000010

I'll guess I'll be turning off 'hybrid sleep' now.

@bcbc
 I concluded that you see the same thing that I detailed in comments # 16 and # 17 of bug report 1042159: https://bugs.launchpad.net/wubi/+bug/1042159/comments/ 16
Your tests also show that "hybrid sleep" is bad for Wubi

I still fail to see why this is a bug in wubi or ntfs-3g. The way I see it is that Windows8 chooses to hibernate by default on shutdown (unless fast-start is disabled), without the knowledge of the user. The fact that the user is unaware of it, doesn't change the fact that it's hibernated. So it's correct for ntfs-3g to refuse to mount it.
Also, the fact that Microsoft consider "Restart" a true restart and "Shut down" a hidden, system hibernation is not intuitive or logical (to me).

The problem with Wubi and hybrid sleep, is in my mind a windows 8 bug. Since in this case, it's Shutdown (not restart) that cleans up the mess left behind by the hybrid sleep. Now suddenly Restart is the inferior option that leaves a quasi-hibernated state.

Why would it be up to ntfs-3g to try to interpret this mess and what should it do with a system-hibernated state versus a user-hibernated state?

For Wubi, it's probably easy enough to check and switch off 'hybrid sleep' to get around the issue, but I think it's unlikely that Microsoft intended this and maybe they'll fix it.

References:
1. this shows that Windows 8 shut down is intentionally hibernating http://blogs.msdn.com/b/b8/archive/2011/09/08/delivering-fast-boot-times-in-windows-8.aspx

"Now here’s the key difference for Windows 8: as in Windows 7, we close the user sessions, but instead of closing the kernel session, we hibernate it. Compared to a full hibernate, which includes a lot of memory pages in use by apps, session 0 hibernation data is much smaller, which takes substantially less time to write to disk. If you’re not familiar with hibernation, we’re effectively saving the system state and memory contents to a file on disk (hiberfil.sys) and then reading that back in on resume and restoring contents back to memory. Using this technique with boot gives us a significant advantage for boot times, since reading the hiberfile in and reinitializing drivers is much faster on most systems (30-70% faster on most systems we’ve tested)."

2. That same link talks about "Restart" being the full reset (which is not the case after hybrid sleep):
" Also, choosing Restart from the UI will do a full shutdown, followed by a cold boot."

Regardless of your decision whether to decide automatically what to do or not, I think you should make sure the "remove_hiberfile" option is interpreted correctly, because it isn't. You should at least be able to decide whether to force the deletion or not.