PHP file_put_contents File Locking

The Senario:

You have a file with a string (average sentence worth) on each line. For arguments sake lets say this file is 1Mb in size (thousands of lines).

You have a script that reads the file, changes some of the strings within the document (not just appending but also removing and modifying some lines) and then overwrites all the data with the new data.

The Questions:

  1. Does ‘the server’ PHP, OS or httpd etc. already have systems in place to stop issues like this (reading/writing half way through a write)?

  2. If it does, please explain how it works and give examples or links to relevant documentation.

  3. If not, are there things I can enable or set-up, such as locking a file until a write is completed and making all other reads and/or writes fail until the previous script has finished writing?

My Assumptions and Other Information:

  1. The server in question is running PHP and Apache or Lighttpd.

  2. If the script is called by one user and is halfway through writing to the file and another user reads the file at that exact moment. The user who reads it will not get the full document, as it hasn’t been written yet. (If this assumption is wrong please correct me)

  3. I’m only concerned with PHP writing and reading to a text file, and in particular, the functions “fopen”/”fwrite” and mainly “file_put_contents”. I have looked at the “file_put_contents” documentation but have not found the level of detail or a good explanation of what the “LOCK_EX” flag is or does.

  4. The scenario is an example of a worst case scenario where I would assume these issues are more likely to occur, due to the large size of the file and the way the data is edited. I want to learn more about these issues and don’t want or need answers or comments such as “use mysql” or “why are you doing that” because I’m not doing that, I just want to learn about file read/writing with PHP and don’t seem to be looking in the right places/documentation and yes I understand PHP is not the perfect language for working with files in this way.

9

I know this is ages old, but in case someone runs into this. IMHO the way to go about it is like this:

1) Open the original file (e.g. original.txt) using file_get_contents(‘original.txt’).

2) Make your changes/edits.

3) Use file_put_contents(‘original.txt.tmp’) and write it to a temp file original.txt.tmp.

4) Then move the tmp file to the original file, replacing the original file. For this you use rename(‘original.txt.tmp’, ‘original.txt’).

Advantages: While the file is being processed and written to the file is not locked and others can still read the old content. At least on Linux/Unix boxes rename is an atomic operation. Any interruptions during the file writing don’t touch the original file. Only once the file has been fully written to disk is it moved. More interesting read on this in the comments to http://php.net/manual/en/function.rename.php

Edit to address commments(too for comment):

https://stackoverflow.com/questions/7054844/is-rename-atomic has further references to what you might need to do if you are operating across filesystems.

On the shared lock for the reading I am not sure why that would be needed as in this implementation there is no writing to the file directly. PHP’s flock (which is used to get the lock) is a little but unreliable and can be ignored by other processes. Thats why I am suggesting using the rename.

The rename file should ideally be named uniquely to the process doing the renaming so as to make sure not 2 processes do the same thing. But this of course does not prevent editing of the same file by more than one person at the same time. But at least the file will be left intact (last edit wins).

Step 3) & 4) would then become this:

$tempfile = uniqid(microtime(true)); // make sure we have a unique name
file_put_contents($tempFile); // write temp file
rename($tempfile, 'original.txt'); // ideally on the same filesystem

4

1) No
3) No

There are several issues with the original suggested approach:

Firstly, some UNIX-like systems such as Linux may not have locking support implemented. The OS does not lock files by default. I have seen the syscalls being NOP (no-operation), but that’s a few years back, so you need to verify whether a lock set by your instance of the application is respected by another instance. (i.e. 2 concurrent visitors). If the locking is still unimplemented [very likely it is], the OS lets you overwrite that file.

Reading large files line-by-line is not feasible for performance reasons. I suggest using file_get_contents() to load the whole file into memory and then explode() it to get the lines. Alternatively, use fread() to read the file in blocks. The aim is to minimize the number of read calls.

In regards to file locking:

LOCK_EX means an exclusive lock (typically for writing). Only one process may hold an exclusive lock for a given file at a given time.
LOCK_SH is a shared lock (typically for reading), More than one process may hold a shared lock for a given file at a given time.
LOCK_UN unlocks the file. Unlocking is done automatically in case you use file_get_contents()
http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems

Elegant solution

PHP supports data stream filters which are intended for processing data in files or from other inputs. You may want to create one such a filter properly using the standard API.
http://php.net/manual/en/function.stream-filter-register.php
http://php.net/manual/en/filters.php

Alternative solution (in 3 steps):

  1. Create a queue. Instead of processing one filename, use the database or other mechanism to store unique filenames somewhere in pending/ and processed in /processed. This way nothing gets overwritten. The database will be also useful for storing additional information, such as metadata, reliable timestamps, processing results, and other.

  2. For files up to a few MB, read the whole file into memory and then process it (file_get_contents() + explode() + foreach())

  3. For larger files read the file in blocks (i.e. 1024 Bytes) and process + write in real-time each block as reading (careful about the last line which doesn’t end with n. It needs to be processed in the next batch)

3

In PHP documentation for file_put_contents() you can find in example#2 the usage for LOCK_EX, puting simply:

file_put_contents('somefile.txt', 'some text', LOCK_EX);

The LOCK_EX is a constant with an integer value than can be used on some functions in a bitwise.

There are also a specific function in order to control locking for files: flock() manner.

1

An issue you didn’t mention that you also need to be careful of is race conditions where two instances of your script are running at the nearly the same time, for example this order of occurrences:

  1. Script instance 1: Reads file
  2. Script instance 2: Reads file
  3. Script instance 1: Writes changes to file
  4. Script instance 2: Overwrites first script instance’s changes to file with its own changes (since at this point its read has become stale).

So when updating a large file, you need to LOCK_EX that file before you read it and not release the lock until the writes have been made. In this example I believe that will cause the second script instance to hang for a little bit while it waits its turn to access the file, but this is better than lost data.

Trang chủ Giới thiệu Sinh nhật bé trai Sinh nhật bé gái Tổ chức sự kiện Biểu diễn giải trí Dịch vụ khác Trang trí tiệc cưới Tổ chức khai trương Tư vấn dịch vụ Thư viện ảnh Tin tức - sự kiện Liên hệ Chú hề sinh nhật Trang trí YEAR END PARTY công ty Trang trí tất niên cuối năm Trang trí tất niên xu hướng mới nhất Trang trí sinh nhật bé trai Hải Đăng Trang trí sinh nhật bé Khánh Vân Trang trí sinh nhật Bích Ngân Trang trí sinh nhật bé Thanh Trang Thuê ông già Noel phát quà Biểu diễn xiếc khỉ Xiếc quay đĩa Dịch vụ tổ chức sự kiện 5 sao Thông tin về chúng tôi Dịch vụ sinh nhật bé trai Dịch vụ sinh nhật bé gái Sự kiện trọn gói Các tiết mục giải trí Dịch vụ bổ trợ Tiệc cưới sang trọng Dịch vụ khai trương Tư vấn tổ chức sự kiện Hình ảnh sự kiện Cập nhật tin tức Liên hệ ngay Thuê chú hề chuyên nghiệp Tiệc tất niên cho công ty Trang trí tiệc cuối năm Tiệc tất niên độc đáo Sinh nhật bé Hải Đăng Sinh nhật đáng yêu bé Khánh Vân Sinh nhật sang trọng Bích Ngân Tiệc sinh nhật bé Thanh Trang Dịch vụ ông già Noel Xiếc thú vui nhộn Biểu diễn xiếc quay đĩa Dịch vụ tổ chức tiệc uy tín Khám phá dịch vụ của chúng tôi Tiệc sinh nhật cho bé trai Trang trí tiệc cho bé gái Gói sự kiện chuyên nghiệp Chương trình giải trí hấp dẫn Dịch vụ hỗ trợ sự kiện Trang trí tiệc cưới đẹp Khởi đầu thành công với khai trương Chuyên gia tư vấn sự kiện Xem ảnh các sự kiện đẹp Tin mới về sự kiện Kết nối với đội ngũ chuyên gia Chú hề vui nhộn cho tiệc sinh nhật Ý tưởng tiệc cuối năm Tất niên độc đáo Trang trí tiệc hiện đại Tổ chức sinh nhật cho Hải Đăng Sinh nhật độc quyền Khánh Vân Phong cách tiệc Bích Ngân Trang trí tiệc bé Thanh Trang Thuê dịch vụ ông già Noel chuyên nghiệp Xem xiếc khỉ đặc sắc Xiếc quay đĩa thú vị
Trang chủ Giới thiệu Sinh nhật bé trai Sinh nhật bé gái Tổ chức sự kiện Biểu diễn giải trí Dịch vụ khác Trang trí tiệc cưới Tổ chức khai trương Tư vấn dịch vụ Thư viện ảnh Tin tức - sự kiện Liên hệ Chú hề sinh nhật Trang trí YEAR END PARTY công ty Trang trí tất niên cuối năm Trang trí tất niên xu hướng mới nhất Trang trí sinh nhật bé trai Hải Đăng Trang trí sinh nhật bé Khánh Vân Trang trí sinh nhật Bích Ngân Trang trí sinh nhật bé Thanh Trang Thuê ông già Noel phát quà Biểu diễn xiếc khỉ Xiếc quay đĩa
Thiết kế website Thiết kế website Thiết kế website Cách kháng tài khoản quảng cáo Mua bán Fanpage Facebook Dịch vụ SEO Tổ chức sinh nhật