I’m currently working on a project that will enable file searching based on metadata found in the file.
It’ll be comprised of 2 parts:
- a filesystem crawler that passively scans for changes and indexes the results in a database
- a search front-end that queries the database
The database will use a pseudo-hierarchical (ie flattened) model to create a map of the directory/file structure.
The fields stored in the index will include:
- file/folder name
- parent directory
- modified time
- metadata
- …
What I’m wondering is, how does modified time on files/folders typically work and how can it be used to reduce the area that the crawler will have to index?
Note: assume the crawler will regularly run on a fixed interval to keep the index up-to-date.
I assume you speak about unix-like OS. From Wikipedia (stat()):
stat() is a Unix system call that returns file attributes about an inode. The semantics of stat() vary between operating systems. As an example, the Unix command ls uses it to retrieve information on (among many others):
- atime: time of last access (
ls -lu),- mtime: time of last modification (
ls -l), and- ctime: time of last status change (
ls -lc).
From man 2 stat:
The field
st_mtimeis changed by file modifications, for example, bymknod(2),truncate(2),utime(2)andwrite(2)(of more than zero bytes). Moreover,st_mtimeof a directory is changed by the creation or deletion of files in that directory. Thest_mtimefield is not changed for changes in owner, group, hard link count, or mode.
That means that last modification time is set on a file when:
- file is created (
mknod) - file is truncated (
truncate) - modification time is changed explicitly (
utime) - one or more bytes of data was written into a file (
write) – that includes cases when same data was written twice and nothing actually changed
I am not sure if it is a complete list.
As for mtime on a directory:
- It changes when you create, move or delete a file or a directory which is immediate child of this directory. Simple heuristic:
mtimeof a directory changes when the output ofls(just plain command, no recursion or any additional parameters) changes. - It does not change when you only modify the contents of any files inside (no matter how deeply nested), or do anything inside its’ subdirectories. No propagation.
I’m wondering if storing the data you’ve mentioned in a DB would be of any help – it is already stored in a file system, which is a database by itself, and all the tools for querying it are already there.
3