百度智能云

All Product Document

          Cloud File Storage

          NFS Protocol Problems

          NFS Protocol Versions Supported

          Currently, CFS supports the NFS 4.1 protocol.

          Why are other versions of NFS not supported?

          Starting with the 2.6 series, the Linux kernel has supported the NFS 4.0 protocol, so the only reason to support NFS 3 and below may be the Windows operating system. But the Windows operating system does not support the NFS protocol well. On Windows, the preferred shared file system is the SMB/CIFS protocol, which itself is also supported by newer Linux (kernel version > 3.4). Therefore, CFS chooses not to support NFS 3 and below.

          Among the NFS 4.X series, the 4.0 and 4.1 versions are more widely used. 4.1 Some design flaws in Version 4.0 were fixed and features such as session, pNFS and directory delegation were introduced. Although only small version numbers have changed, the actual differences are quite large. CFS chooses to support the better-performing NFS 4.1 protocol. At the same time, CFS has development plans for the NFS 4.0 protocol.

          Support for the SMB/CIFS protocol is currently under development and is expected to be open for beta at the end of 2019.

          The Linux kernel's official development of NFS 4.1 started with Version 2.6.31, released experimental support to Version 2.6.36, and finally provided formal support to Version 3.7.For details, please see Link. Therefore, users should try to choose kernels of 3.7 and above. In general, newer stable versions of the kernel can provide better performance and experience.

          The kernel version displayed by some operating systems does not have the same code as the official kernel of the same version in Linux. For example, CentOS/RHEL will make some bugfixes, modifications or transplants. Their codes are usually ahead of the official kernel of the same version. Therefore, the reality of each operating system is subject to the official documentation and source code.

          For common operating systems, we recommend the following versions:

          • CentOS/RHEL: 6.6 and above are recommended. See Link for more detailed bug information of the NFS protocol on the operating system;
          • Ubuntu: 14.04 and above are recommended;
          • Debian: 8 and above are recommended;
          • OpenSUSE: 42.3 and above are recommended.

          Attributes Unsupported

          The attributes not supported by CFS include: FATTR4_ACL, FATTR4_ACLSUPPORT, FATTR4_ARCHIVE, FATTR4_HIDDEN, FATTR4_MIMETYPE, FATTR4_QUOTA_AVAIL_HARD, FATTR4_QUOTA_AVAIL_SOFT, FATTR4_QUOTA_USED, FATTR4_SYSTEM, FATTR4_TIME_BACKUP, FATTR4_TIME_CREATE, FATTR4_DIR_NOTIF_DELAY, FATTR4_DIRENT_NOTIF_DELAY, FATTR4_DACL, FATTR4_SACL, FATTR4_CHANGE_POLICY, FATTR4_FS_STATUS, FATTR4_LAYOUT_HINT, FATTR4_LAYOUT_ALIGNMENT, FATTR4_FS_LOCATIONS_INFO, FATTR4_MDSTHRESHOLD, FATTR4_RETENTION_GET, FATTR4_RETENTION_SET, FATTR4_RETENTEVT_GET, FATTR4_RETENTEVT_SET, FATTR4_RETENTION_HOLD, FATTR4_MODE_SET_MASKED, FATTR4_FS_CHARSET_CAP.

          The file locks supported by CFS are nonblocking. It is recommended to lock the entire file or part of the byte range. Other types of locks are not currently supported, which means that you need to pay attention to the following when using 3 common locking interfaces:

          • flock must also specify the LOCK_NB parameter;
          • lockf does not support the F_LOCK option;
          • fcntl does not support the F_SETLKW option.

          CFS does not support attributes such as Delegation, pNFS, Access Control List (ACL) and Kerberos security.

          For performance reasons, CFS does not update the time_access attribute of the file read during read-only operations.

          CFS does not have persistent Session or Replay Cache. When a failover occurs, the client will re-establish the Session. The behavior complies with RFC 5661 and is usually handled automatically by the client.

          "Mount: wrong fs type, bad option, bad superblock" is reported during mounting

          Most operating systems do not install the NFS client by default and require manual installation.

          • If you use the CentOS/RHEL operating system, run the following command:
          yum install nfs-utils
          • If you use the Ubuntu/Debian operating system, run the following command:
          apt-get install nfs-common

          The above commands require root authority.

          The list obtained when traversing the folder has missing items, duplicates, update lag, etc.

          When a client traverses a folder through the the programming interface of readdir/getdents/getdents64 or through the ls command, the list seen by the user may have missing items and duplicates if another client has concurrent operations such as create, delete and rename under the folder. This is due to the interference of concurrent operations. The root cause is that a complete traversal consists of several consecutive acquisition operations. Only a part of the file items can be acquired at a time, and the directory has changed during the consecutive operations.

          Under normal circumstances, the client will use the directory entry and attribute cache to improve performance, but the cache also means that one client's changes may not be immediately seen by other clients. This leads to an update lag. The user can use the noac option to disable caching when mounting. This method will cause a large increase in the number of OP_GETATTR remote calls, and greatly damage performance. We do not recommend it.

          Client requests with the same host name and intranet IP interfere with each other

          For two clients of the same file system, if their host names and intranet IP addresses are the same, CFS cannot distinguish them and the requests will interfere with each other. The problem usually occurs in the container scenario because the intranet IP is the same on the same machine. If the host name is the same, it will be considered as the same client.

          Some older Linux kernels will truncate the host name when encoding the client owner character of the NFS protocol and take the previous part of the characters. Therefore, the difference part of the host name should be encoded as forward as possible. Based on our experience, it is safer to control the difference characters within the first 32 characters.

          In short, you should avoid situations where the host name and the intranet IP are the same.

          The client cannot connect to the mount target, and the kernel log says "nfs: server XXX not responding, still trying"

          In the kernel log, XXX is the mount target address, and the following is represented by . Follow the steps below to troubleshoot the problem:

          • Ping to confirm that the address is reachable. If it is not reachable, check if the corresponding mount target address still exists on the webpage;
          • If there is no problem in the above step, telnet 2049, and confirm that the port number of the backend NFS protocol is connected. If it is not connected, check whether the relevant port number is disabled by the security group policy.
          • If the above step is OK, check whether the mount parameter is set to noresvport. If not, try to mount again with the new options.

          If the problem persists, you can send a work ticket describing the problem and contact the engineer for troubleshooting.

          Some files with names similar to ".nfsXXXX" were found in the directory

          These files are produced by the so-called "Silly Rename" mechanism.

          In UNIX-like operating systems, a file which is deleted after opened becomes invisible, but the process can continue to read and write it through the file descriptor. The file is actually deleted when no more processes open it. In earlier versions of NFS (NFS 3 and earlier), there is no guarantee that the server will continue to retain the file. Therefore, the kernel uses a small trick to rename the file to a hidden file such as ".nfsXXXX" when it is deleted. It will be deleted if it is not in use anymore. This is the so-called "silly rename" mechanism. If the client is down or the network is interrupted, the client may not successfully delete the ".nfsXXXX" files, and they will remain.

          The NFS 4.X protocol has been able to deal with the problems, but the kernel still uses the "Silly Rename" mechanism for backward compatibility.

          You can filter out such files when listing the items in the folder. You can safely delete the files when you are sure that no processes are using them.

          Low Client Throughput or QPS

          The actual throughput of the client is low, which may be caused by the sunrpc slot setting. You can confirm it with the following command:

          cat /proc/sys/sunrpc/tcp_slot_table_entries

          If the value is low, you can increase it to 128:

          echo "options sunrpc tcp_slot_table_entries=128" >> /etc/modprobe.d/sunrpc.conf
          echo "options sunrpc tcp_max_slot_table_entries=128" >> /etc/modprobe.d/sunrpc.conf

          The above settings will take effect after restarting. You can use the following command to modify the current system values:

          sysctl -w sunrpc.tcp_slot_table_entries=128

          If you click cat/proc/sys/sunrpc/tcp_slot_table_entries again, you can see that the settings have been updated.

          Modifying the setting value has no effect on the mounted file system, so you need to remount the file system.

          Multiple clients concurrently append write the same file data

          The NFS protocol does not support atomic append writes. To achieve the effect of append write in the local file system, you need to lock the file first, seek to the correct location, and then write the data. The sample code is as follows:

          int lock_file(int fd) {
              for (;;) {
                  int rc = flock(fd, LOCK_EX | LOCK_NB);
                  if (rc == 0) {
                      return 0;
                  }
                  if (errno == EINTR) {
                      continue; // interupt by a signal, retry
                  } else if (errno == EWOULDBLOCK) {
                      usleep((rand() % 10) * 100); // sleep a while to retry
                      contunue;
                  } else {
                      break;
                  }
              }
              return -1; // lock fail
          }
          
          int unlock_file(int fd) {
              for (;;) {
                  int rc = flock(fd, LOCK_UN);
                  if (rc == 0) {
                      return 0;
                  }
                  if (errno == EINTR) {
                      continue; // interupt by a signal, retry
                  } else {
                      break;
                  }
              }
              return -1; // unlock fail
          }
          
          int append_write(int fd, const char* data, ssize_t size) {
              int rc = lock_file(fd);
              if (rc != 0) {
                  return rc;
              }
              off_t offset = lseek(fd, 0, SEEK_END);
              if (offset < 0) {
                  unlock_file(fd);
                  return -1;
              }
              while (size > 0) {
                  ssize_t nwriten = pwrite(fd, buf, size, offset);
                  if (nwriten >= 0) {
                      size -= nwriten;
                      offset += nwriten;
                  } else if (errno == EINTR) {
                      continue;
                  } else {
                      unlock_file(fd);
                      return -1;
                  }
              }
              return unlock_file(fd);
          }
          Previous
          Billing Problems
          Next
          SMB Protocol Issues