www.cendio.com
Bug 5560 - new files on unfsd side aren't properly shown to nfs client (Windows)
: new files on unfsd side aren't properly shown to nfs client (Windows)
Status: CLOSED FIXED
: ThinLinc
Local drive redirection
: 4.3.0
: PC Unknown
: P2 Normal
: 4.10.1
Assigned To:
:
:
:
:
  Show dependency treegraph
 
Reported: 2015-06-02 13:23 by
Modified: 2019-06-18 12:34 (History)
Acceptance Criteria:
* Files added or removed on the client should show up right away on the server * atime and mtime should be shown correctly on the server * ctime should be shown correctly on the server with a Linux or macOS client * ctime should be shown the same as mtime on the server with a Windows client * open() with O_EXCL should work fine with any client, even when exporting crappy fs like FAT


Attachments


Note

You need to log in before you can comment on or make changes to this bug.


Description From cendio 2015-06-02 13:23:40
A file created on the server side (nfs) is not shown when listing a directory
on client side.

This bug does not affect the linux build.
------- Comment #1 From cendio 2015-06-02 14:57:54 -------
This is not a regression, same issue is reproducible using 4.2.0 and 4.3.0
client.

Steps to reproduce:

 1. Configure local drive redirection and connect to a ThinLinc session

 2. Start a terminal window and go into ~/thindrives/<share>/

 3. Using the Explorer on the windows client, browse to exported directory and 
    create a file.

 4. List the files within the mounted export and the new file created in step 3 
    is not visible.

However, you can run 'stat filename.txt' so some data is sent over nfs
protocol.
------- Comment #2 From cendio 2016-08-29 16:31:16 -------
Support issue 21040 had the same problem using ThinLinc client 4.6.0 on Win10
platform. I couldn't reproduce at first. There is a corner case to this issue.
You need to export a directory directly under C: eg. `C:\thinlincshare` to hit
this bug.
------- Comment #3 From cendio 2016-09-01 12:06:40 -------
(In reply to comment #2)
> Support issue 21040 had the same problem using ThinLinc client 4.6.0 on Win10
> platform. I couldn't reproduce at first. There is a corner case to this issue.
> You need to export a directory directly under C: eg. `C:\thinlincshare` to hit
> this bug.

This information is invalid. I am now in a state that i can not have a working
case at all. I have tested with different ThinLinc servers (NFS client
version). With different configurations for caching. Non producing a working
environment.

I do see that a 'ls' within a directory does not send a readdir nfs operation.
This means that the client seems to see that they already have the correct
information in the cache.

'ls' does an stat nfs operation on current directory. I have verified that the
statbuf contains proper information such as st_mtime before and after a change
within a directory.

The problem is not reproducible using a linux client. So the suspected parts
are the win32 backend.

We should look into on what premises the kernel nfs client actual invalidates
cached data and performes a readdir nfs operation.
------- Comment #5 From cendio 2019-05-21 16:06:44 -------
Adding or removing files on Windows changes mtime, but not ctime. But on a
Linux system both mtime and ctime are changed. This might explain why this only
happens on Windows.

Note that this behaviour is not fully supported by the documentation for stat()
and friends. It doesn't say anything specific about ctime and directories, but
it does state that it should be updated when the inode is updated (as opposed
to the contents). However no fields reported by stat change when a file is
added or removed.

The Linux nfs man page also states that only mtime is examined:

>       To detect when directory entries have been added or removed on the
>       server, the Linux NFS client watches a directory's mtime.  If the
>       client detects a change in a directory's mtime, the client drops all
>       cached LOOKUP results for that directory.  Since the directory's
>       mtime is a cached attribute, it may take some time before a client
>       notices it has changed.  See the descriptions of the acdirmin,
>       acdirmax, and noac mount options for more information about how long
>       a directory's mtime is cached.
------- Comment #6 From cendio 2019-05-21 16:09:36 -------
There is a mount option called "lookupcache" which has this as part of the
description:

>                      If none is specified, the client revalidates both
>                      types of directory cache entries before an application
>                      can use them.  This permits quick detection of files
>                      that were created or removed by other clients, but can
>                      impact application and server performance.

However I'm still seeing the bug even with lookupcache=none specified.

Not even "noac" works, which is documented as turning off all metadata caching.
------- Comment #7 From cendio 2019-05-21 18:57:03 -------
Updating ctime seems to work, and I'm seeing some suspicious parts of the Linux
kernel to confirm this.

I filed a bug with them:

https://bugzilla.kernel.org/show_bug.cgi?id=203665

But it's not realistic to wait for a fix to roll out to every system out there.
We'll have to work around this.
------- Comment #8 From cendio 2019-05-23 12:22:38 -------
Windows doesn't have "change time", so the plan is to go ahead and report
modification time in the ctime field. That should be close enough.

Related to this is the handling of creation verifiers. If a file is created
exclusively (O_EXCL), then there is some extra magic involved to handle
possible retransmissions of the create command.

The client provides a random "verifier" to the server, and the server is
supposed to store this in persistent storage. The recommended (and de-facto)
method is to store this information in atime and mtime. The client will update
these to sane values once the create has succeeded.

However FAT does not have enough space for the verifier in those fields. So
instead we started using the creation time. We also chopped off some bits as
there still wasn't enough room.

This happened to work smoothly because Windows reports creation time in the
ctime field. If we start putting modification time in there then we'll need to
change things. We probably want to rethink using creation time at all, because
we don't have a good way of setting it to something sane once we're done.
------- Comment #9 From cendio 2019-05-23 12:25:41 -------
So the idea for the verifier is to reduce it from 64 bits down to something
that fits in atime and mtime even on FAT. This adds some risk for collision,
but we should not be under such a load that it's really something to worry
about*.

Fortunately we already have a good hash function in unfs with low collision
risk, namely FNV1a. It gives us 32 bits out, which is small enough that it fits
in atime and mtime.

* As a side note, not even the Linux kernel perfectly preserves the verifier.
It chops off the highest bit in atime and mtime as having it set confuses some
systems.
------- Comment #10 From cendio 2019-05-23 12:32:57 -------
Whilst testing this, I discovered that _stat64i() doesn't work correctly on
Windows. It will report times that are off by one hour in some cases. Not
entirely sure of the conditions, but right now it is misreporting every time
that is during winter time. GetFileTime() does not suffer from this
fortunately, so we should probably use that instead.

As a fun note, "dir" is also reporting things incorrectly so you have to use
Explorer to see proper file timestamps.
------- Comment #11 From cendio 2019-05-23 15:29:00 -------
Argh! More fun obstacles!

Windows likes to store things in local time rather than UTC on FAT disks. That
means it has to convert things back and forth as the API is using UTC. For the
access time field that means things might shift around, as it has a resolution
of whole days and cropping to just a date can then cause it to shift.

So it means we need to have some margin for these conversions when we store
things. Fortunately we still have bits to spare...
------- Comment #12 From cendio 2019-05-23 16:49:32 -------
There is some magic in fh_decomp() where it fakes mtime in some cases so that
the NFS client will notice changes to a directory. I would assume that this
also needs to consider ctime to work with current Linux kernels.

It was added because FAT doesn't track mtime for the root directory, so any
files added or removed there won't get noticed.

But during testing everything works just fine. I guess modern Linux kernels
have realised that an mtime of 0 means "unknown" and will always recheck the
directory.

That code also seems incomplete as it will only trigger if the FAT root is also
the exported directory. If I export the parent directory of where the FAT drive
is mounted then the workaround never triggers.


Since things seem to work in practice I'm not touching this code and focusing
on the specific Windows bug.
------- Comment #13 From cendio 2019-05-23 16:50:50 -------
A thing to note is that the verifier problem mentioned earlier is a FAT
problem, not a Windows problem. And FAT drives are common on Linux as well, so
that fix should probably be made generic.
------- Comment #17 From cendio 2019-05-24 12:03:00 -------
Should work well now. Check the acceptance criteria for things to test.
------- Comment #18 From cendio 2019-05-28 13:25:07 -------
Tested with build 6138:

* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* atime and mtime should be shown correctly on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown correctly on the server with a Linux or macOS client
Windows NTFS: -
Windows FAT32: -
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown the same as mtime on the server with a Windows client
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: -
Linux ext4: -
macOS: -

* open() with O_EXCL should work fine with any client, even when exporting
crappy fs like FAT
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK
------- Comment #19 From cendio 2019-06-10 14:17:59 -------
Something is broken. It is no longer possible to export root directories using
NTFS. It gives you a "access denied" on the server when trying to access the
directory.
------- Comment #20 From cendio 2019-06-10 15:07:47 -------
The problem doesn't seem to be the root directory itself, but rather that there
is a file in there that I don't have access to. So when we do READDIR, we also
stat() that file to get the inode. But a stat() call now involves getting the
FILETIMEs, which are failing with ERROR_ACCESS_DENIED.

Not sure why though since explorer and dir can both access this just fine.
------- Comment #22 From cendio 2019-06-10 16:21:53 -------
Works better now.

This completely changes the calls used on Windows, so the Windows client needs
to be re-tested. Test exporting C:\ to check for the latest found bug.
------- Comment #23 From cendio 2019-06-11 12:21:51 -------
* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: Not OK

Apparently, the time of a folder is not updated when files in that folder are
added or removed.
------- Comment #24 From cendio 2019-06-11 12:58:23 -------
Apparently this is per design:

"Note: The design and behavior of the FAT file system is different with regards
to the modified time stamp. On a FAT file system, the modified date of a folder
does not change if the contents of the folder change."

https://web.archive.org/web/20080219020154/http://support.microsoft.com/kb/299648


Looks like we need to fix a general workaround on Windows...
------- Comment #25 From cendio 2019-06-12 09:43:22 -------
Note that FAT is also partially broken on Linux, even though I stated this
before:

(In reply to comment #12)
> 
> But during testing everything works just fine. I guess modern Linux kernels
> have realised that an mtime of 0 means "unknown" and will always recheck the
> directory.
> 

Apparently I didn't test carefully enough. It seems like adding or removing
directory works, probably because the link count of the root directory changes.
But regular files are not noticed.

So we may have to make this workaround a bit more general than just Windows.
------- Comment #27 From cendio 2019-06-17 14:11:17 -------
Tested with build 6157:

* Files added or removed on the client should show up right away on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* atime and mtime should be shown correctly on the server
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown correctly on the server with a Linux or macOS client
Windows NTFS: -
Windows FAT32: -
Linux FAT32: OK
Linux ext4: OK
macOS: OK

* ctime should be shown the same as mtime on the server with a Windows client
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: -
Linux ext4: -
macOS: -

* open() with O_EXCL should work fine with any client, even when exporting
crappy fs like FAT
Windows NTFS: OK
Windows FAT32: OK
Linux FAT32: OK
Linux ext4: OK
macOS: OK