Atomic copies using rename()

The Unix rename(2) system call—rename in Perl, Ruby, and other languages—caMelCased renameTo in Java—offers atomic file renaming under a single filesystem. Atomic here means the file is either renamed, or it is not, even if the system should crash. Workflows that move files can benefit from atomic copies: a truncated /etc/passwd file could render a system unusable; an incomplete index.html may confuse customers or bring ridicule to a company; an incomplete report.csv may silently omit crucial data.

Technorati Tags: coding, Unix

Implementation

The sample Ruby code below first writes the source file to a temporary file in the same directory as the permanent filename, then uses the rename call once the copy completes. rsync, cfengine, and other programs use this method to ensure complete all-or-nothing file copies.

#!/usr/bin/ruby # # Usage: $0 source_file destination_file # # TODO error, exception handling and cleanup as # necessary for the implementation. require "ftools"; require "tempfile"; program = File.basename($0); src_file = ARGV[0]; src_stat = File.stat(src_file); dst_file = ARGV[1]; # If directory the target, copy file into # that directory if FileTest::directory?(dst_file) dst_dir = dst_file dst_file = File.join(dst_dir,File.basename(src_file)) else dst_dir = File.dirname(dst_file) end File.mkpath(dst_dir); tf = Tempfile.open("#{program}-tmp", tmpdir=dst_dir) File.copy(src_file, tf.path) File.chmod(src_stat.mode, tf.path) # TODO apply checksum verification here if paranoid # that the filesystem or underlying storage might have # corrupted the file, or if calculating the MDN as part # of the AS2 protocol. File.rename(tf.path, dst_file)

Concerns with rename()

The rename documentation often includes dire warnings: that rename does not work across filesystem boundaries, that it is not portable, or that it fails should the destination file exist. The first concern is not a problem for files being renamed under the same filesystem, much less the same parent directory. The common mv(1) command uses rename(2) should the source and destination reside on the same filesystem:

$ ktrace mv source/cat dest $ kdump -f ktrace.out| tail -22 5228 mv NAMI "dest" 5228 mv RET stat 0 5228 mv CALL lstat(0xbffff41f,0xbffff220) 5228 mv NAMI "source/cat" 5228 mv RET lstat 0 5228 mv CALL lstat(0xbffff42a,0xbffff1c0) 5228 mv NAMI "dest" 5228 mv RET lstat 0 5228 mv CALL access(0xbfffedc0,0) 5228 mv NAMI "dest/cat" 5228 mv RET access 0 5228 mv CALL lstat(0xbffff41f,0xbfffece0) 5228 mv NAMI "source/cat" 5228 mv RET lstat 0 5228 mv CALL access(0xbfffedc0,0x2) 5228 mv NAMI "dest/cat" 5228 mv RET access 0 5228 mv CALL rename(0xbffff41f,0xbfffedc0) 5228 mv NAMI "source/cat" 5228 mv NAMI "dest/cat" 5228 mv RET rename 0 5228 mv CALL exit(0)

Portability can be a concern, as detailed in perlport:

Some platforms can't delete or rename files held open by the system, this limitation may also apply to changing filesystem metainformation like file permissions or owners. Remember to "close" files when you are done with them. Don't "unlink" or "rename" an open file. Don't "tie" or "open" a file already tied or opened; "untie" or "close" it first.

Portability is best addressed by unit tests: when building the software, perform various test rename calls, and use these results to determine if the software will work on the new system. Unit tests should also consider various failure or degraded performance conditions, for example when data arrives slowly—copies over a wide area network, or large amounts of data—a condition unlikely in a test environment unless forced somehow.

Behavior when the target file exists depends on the implementation; mv(1) offers a -f option, or fails. Other software may need to clobber the target, fail with an alert, or first perform metadata comparisons, such as file modification time, file size, or a checksum on the file contents. Checking whether the target exists prior to the rename call is perhaps a needless race condition.

Network Copies

A similar method can also be used with protocols such as SFTP for file transfers: first upload the data to a temporary file, then use the rename command to move the complete file to the final location. Note that uploading and renaming—while a good practice over directly copying the file—does not guarantee that the data was received, uncorrupted, by the target system. In contrast, the AS2 protocol returns success only if the file contents are cryptographically verified by the target system, making it suitable for workflows where the data must not be truncated, corrupted, or otherwise altered.

Jeremy Mates’s Blog

Atomic copies using rename()

Implementation

Concerns with rename()

Network Copies