← UNIX File I/O & Descriptors (APUE)

Select a question from the sidebar

15 questions in this topic

Question 1 / 15

How does the open() system call work, and what are the commonly used flags?

open() is the fundamental UNIX system call for accessing files. It returns a file descriptor (a small non-negative integer) that serves as an index into the process's file descriptor table.

Prototype:

#include <fcntl.h>
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);

Access mode flags (exactly one required):

O_RDONLY -- open for reading only
O_WRONLY -- open for writing only
O_RDWR -- open for reading and writing

Common modifier flags (OR-ed with access mode):

O_CREAT -- create file if it does not exist (requires mode argument)
O_TRUNC -- truncate to zero length if file exists
O_APPEND -- append on each write (atomic seek-to-end + write)
O_EXCL -- with O_CREAT, fail if file exists (atomic create-or-fail)
O_NONBLOCK -- non-blocking mode for FIFOs and devices
O_CLOEXEC -- close fd on exec (avoids fd leak to child processes)
O_SYNC -- synchronous writes (data + metadata flushed to disk)

The kernel always assigns the lowest available file descriptor number. The mode argument (e.g., 0644) is masked by the process's umask when creating files.

Pseudo Code

1. Call open(pathname, flags, mode)
2. Kernel checks permissions against effective UID/GID
3. Kernel allocates an entry in the open file table
4. Kernel adds an entry in the process fd table pointing to the open file entry
5. Returns lowest available fd number, or -1 on error (sets errno)
6. Common pattern: fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0644)

Code

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

int main(void) {
    /* Open for reading only */
    int fd1 = open("/etc/passwd", O_RDONLY);
    if (fd1 == -1) {
        perror("open /etc/passwd");
        return 1;
    }
    printf("fd1 = %d\n", fd1);  /* likely 3 */

    /* Create or truncate a file for writing */
    int fd2 = open("output.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd2 == -1) {
        perror("open output.txt");
        close(fd1);
        return 1;
    }
    printf("fd2 = %d\n", fd2);  /* likely 4 */

    /* Atomic create -- fail if file already exists */
    int fd3 = open("lockfile", O_WRONLY | O_CREAT | O_EXCL, 0600);
    if (fd3 == -1) {
        if (errno == EEXIST)
            fprintf(stderr, "lockfile already exists\n");
        else
            perror("open lockfile");
    } else {
        printf("fd3 = %d\n", fd3);
        close(fd3);
    }

    close(fd1);
    close(fd2);
    return 0;
}

fd1 = 3
fd2 = 4
fd3 = 5

Key Points

•open() returns the lowest available file descriptor number
•Exactly one of O_RDONLY, O_WRONLY, or O_RDWR must be specified
•O_CREAT requires a third mode argument; actual permissions = mode & ~umask
•O_CREAT | O_EXCL is atomic: create-or-fail with no race condition
•O_APPEND guarantees atomic seek-to-end + write (even across processes)
•Always check the return value; -1 means error and errno is set

Gotchas

•Forgetting the mode argument with O_CREAT causes undefined permissions (garbage from the stack)
•O_RDONLY is typically 0, so you cannot test for it with bitwise AND
•O_TRUNC on a file opened O_RDONLY is undefined behavior on some systems
•O_CLOEXEC should be used by default to prevent fd leaks across exec()
•On Linux, open() can fail with EMFILE (per-process limit) or ENFILE (system-wide limit)

open() opens a file and returns a file descriptor (small int). Use flags like O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC, O_APPEND, O_EXCL. Always check return value for -1.

•Returns lowest available fd or -1 on error
•O_CREAT needs a mode argument

#include <fcntl.h>
#include <unistd.h>

int main(void) {
    int fd = open("file.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) { perror("open"); return 1; }
    write(fd, "hello\n", 6);
    close(fd);
    return 0;
}

Omitting mode with O_CREAT gives garbage permissions

Question 2 / 15

What are file descriptors 0, 1, and 2, and how does the fd lifecycle work with close()?

Every UNIX process starts with three open file descriptors by convention:

FD	Symbolic Constant	stdio Stream	Purpose
0	`STDIN_FILENO`	`stdin`	Standard input
1	`STDOUT_FILENO`	`stdout`	Standard output
2	`STDERR_FILENO`	`stderr`	Standard error

These constants are defined in <unistd.h>. The shell sets these up before the program runs; they typically point to the terminal device.

fd lifecycle:

open() / creat() / socket() / pipe() -- allocates a new fd (lowest available)
Use fd with read(), write(), lseek(), fcntl(), etc.
close(fd) -- releases the fd, decrements the reference count in the open file table entry. When the reference count reaches 0, the kernel releases the open file table entry.

Important: close() returns 0 on success, -1 on error. On NFS, a delayed write error can be reported on close(), so you should check its return value for critical files.

Pseudo Code

1. Process starts with fd 0 (stdin), 1 (stdout), 2 (stderr) open
2. open() assigns the lowest available fd number
3. Use fd with read/write/lseek/fcntl
4. close(fd) releases the descriptor
5. If fd was last reference to open file entry, kernel frees that entry
6. Closing an already-closed fd is an error (EBADF)

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

int main(void) {
    /* Standard file descriptors */
    printf("STDIN_FILENO  = %d\n", STDIN_FILENO);   /* 0 */
    printf("STDOUT_FILENO = %d\n", STDOUT_FILENO);  /* 1 */
    printf("STDERR_FILENO = %d\n", STDERR_FILENO);  /* 2 */

    /* Close stdin, then open a file -- it gets fd 0 */
    close(STDIN_FILENO);
    int fd = open("/dev/null", O_RDONLY);
    printf("After closing stdin, new fd = %d\n", fd);  /* 0 */

    /* Always check close() return on critical files */
    int wfd = open("important.dat", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (wfd == -1) { perror("open"); return 1; }
    write(wfd, "data\n", 5);
    if (close(wfd) == -1) {
        perror("close");  /* NFS can report write errors here */
        return 1;
    }

    close(fd);
    return 0;
}

STDIN_FILENO  = 0
STDOUT_FILENO = 1
STDERR_FILENO = 2
After closing stdin, new fd = 0

Key Points

•fd 0 = stdin, fd 1 = stdout, fd 2 = stderr by POSIX convention
•Use STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO from <unistd.h>
•open() always assigns the lowest available fd number
•close() decrements the open file table reference count
•A process has a per-process limit on open fds (RLIMIT_NOFILE)

Gotchas

•Closing fd 0 then opening a file gives it fd 0 -- can cause subtle bugs if code assumes fd 0 is a terminal
•Double-close of an fd is an error (EBADF) and can close a different file if the fd was reused
•On NFS, close() can return an error for a failed deferred write -- always check return value
•Forgetting to close fds leads to fd exhaustion (EMFILE error)

fd 0 = stdin, 1 = stdout, 2 = stderr. open() returns the lowest unused fd. close() releases the fd. Always check close() return value for important files.

•Standard fds: 0=stdin, 1=stdout, 2=stderr
•open() assigns lowest available fd

#include <unistd.h>
#include <fcntl.h>

int main(void) {
    /* fd 0, 1, 2 are stdin, stdout, stderr */
    close(STDIN_FILENO);  /* free fd 0 */
    int fd = open("/dev/null", O_RDONLY);  /* gets fd 0 */
    close(fd);
    return 0;
}

Double-close is a bug -- EBADF and possible race with fd reuse

Question 3 / 15

How do read() and write() handle partial I/O, and why must you use loops?

read() and write() are not guaranteed to transfer the full requested number of bytes in a single call. They return the number of bytes actually transferred, which can be less than requested.

Reasons for short reads:

Reading from a pipe/FIFO/socket when fewer bytes are available
Reading from a terminal (typically returns one line)
Interrupted by a signal (returns bytes transferred so far, or -1 with EINTR)
Reading near end-of-file
Kernel buffer limits

Reasons for short writes:

Disk full condition
Writing to a pipe/socket with limited buffer space
Signal interruption
Non-blocking I/O on a fd that cannot accept all bytes

Return values:

read(): number of bytes read (0 = EOF), or -1 on error
write(): number of bytes written, or -1 on error

You must always loop until all bytes are transferred or an error/EOF occurs. This is so common that many projects define readn() and writen() helper functions (as shown in APUE by Stevens).

Pseudo Code

1. Call read(fd, buf, nbytes)
2. If return == 0, EOF reached
3. If return == -1 and errno == EINTR, retry
4. If return == -1 (other error), handle error
5. If return < nbytes, advance buffer pointer, reduce count, loop
6. Same logic for write() but 0 return is unusual (not EOF)

Code

#include <unistd.h>
#include <errno.h>
#include <stddef.h>

/* Read exactly n bytes (APUE-style readn) */
ssize_t readn(int fd, void *buf, size_t n) {
    size_t nleft = n;
    ssize_t nread;
    char *ptr = (char *)buf;

    while (nleft > 0) {
        nread = read(fd, ptr, nleft);
        if (nread == -1) {
            if (errno == EINTR)
                continue;   /* interrupted, retry */
            return -1;      /* real error */
        }
        if (nread == 0)
            break;          /* EOF */
        nleft -= (size_t)nread;
        ptr += nread;
    }
    return (ssize_t)(n - nleft);
}

/* Write exactly n bytes (APUE-style writen) */
ssize_t writen(int fd, const void *buf, size_t n) {
    size_t nleft = n;
    ssize_t nwritten;
    const char *ptr = (const char *)buf;

    while (nleft > 0) {
        nwritten = write(fd, ptr, nleft);
        if (nwritten == -1) {
            if (errno == EINTR)
                continue;
            return -1;
        }
        nleft -= (size_t)nwritten;
        ptr += nwritten;
    }
    return (ssize_t)n;
}

Key Points

•read() and write() can return fewer bytes than requested (partial I/O)
•read() returns 0 for EOF, -1 for error
•EINTR means interrupted by signal -- usually retry
•Always loop to handle partial I/O on pipes, sockets, and terminals
•readn()/writen() helper functions are the standard APUE pattern

Gotchas

•Assuming read() fills the entire buffer is the most common UNIX programming bug
•write() returning 0 is technically possible but unusual; do not treat it as EOF
•On signals: if SA_RESTART is set, the kernel auto-restarts; otherwise you get EINTR
•Reading from /dev/urandom can also short-read under extreme load

read()/write() can transfer fewer bytes than requested. Always loop. read() returns 0 for EOF, -1 for error. Handle EINTR (signal interruption) by retrying.

•Never assume read/write transfers all bytes
•Loop until done, EOF, or error

#include <unistd.h>
#include <errno.h>

ssize_t readn(int fd, void *buf, size_t n) {
    size_t left = n;
    char *p = (char *)buf;
    while (left > 0) {
        ssize_t nr = read(fd, p, left);
        if (nr == -1) { if (errno == EINTR) continue; return -1; }
        if (nr == 0) break;
        left -= nr; p += nr;
    }
    return (ssize_t)(n - left);
}

Short reads are normal on pipes, sockets, and terminals

Question 4 / 15

How do dup() and dup2() work for file descriptor duplication?

dup() and dup2() create a copy of a file descriptor. Both the old and new fd share the same open file table entry (same file offset, status flags, etc.).

Prototypes:

#include <unistd.h>
int dup(int oldfd);           /* returns lowest available fd */
int dup2(int oldfd, int newfd); /* makes newfd a copy of oldfd */

dup() returns the lowest available fd that is a copy of oldfd.

dup2() atomically closes newfd (if open) and makes it a copy of oldfd. If oldfd == newfd, it returns newfd without closing it.

The classic use case is I/O redirection -- redirecting stdout to a file:

Open the file
dup2(fd, STDOUT_FILENO) -- now fd 1 points to the file
Close the original fd
Any printf() or write(1, ...) goes to the file

dup2 vs close+dup: dup2() is atomic. If you do close(newfd); dup(oldfd); there is a race window where a signal handler could open a file and grab the fd number you wanted.

Pseudo Code

1. dup(oldfd): kernel finds lowest free fd, copies oldfd's pointer to it
2. dup2(oldfd, newfd): if newfd is open, close it; copy oldfd to newfd slot
3. Both old and new fd share the same open file table entry
4. They share file offset and status flags
5. They have independent FD_CLOEXEC flags

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

int main(void) {
    /* Redirect stdout to a file */
    int fd = open("output.log", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) { perror("open"); return 1; }

    /* Save original stdout */
    int saved_stdout = dup(STDOUT_FILENO);
    if (saved_stdout == -1) { perror("dup"); return 1; }

    /* Redirect: fd 1 now points to output.log */
    if (dup2(fd, STDOUT_FILENO) == -1) { perror("dup2"); return 1; }
    close(fd);  /* no longer need the original fd */

    printf("This goes to output.log\n");  /* writes to file */

    /* Restore stdout */
    dup2(saved_stdout, STDOUT_FILENO);
    close(saved_stdout);

    printf("This goes to terminal\n");  /* back to terminal */
    return 0;
}

This goes to terminal

Key Points

•dup() returns lowest available fd; dup2() targets a specific fd
•Both old and new fd share the same open file table entry (offset, flags)
•dup2() is atomic: close + copy in one step, no race condition
•Classic use: I/O redirection (dup2(fd, STDOUT_FILENO))
•FD_CLOEXEC flag is NOT inherited by the duplicate

Gotchas

•dup2 does NOT set FD_CLOEXEC on the new fd -- use dup3() with O_CLOEXEC on Linux
•If oldfd is invalid, dup2 returns -1 and does NOT close newfd
•dup2(fd, fd) is a no-op that returns fd (does not close it)
•After dup2, both fds share the same offset -- writing via one advances the other

dup(fd) copies fd to lowest available number. dup2(old, new) atomically closes new and copies old to it. Both share the same file offset and flags.

•dup2() is atomic close+copy
•Old and new fd share file offset

#include <unistd.h>
#include <fcntl.h>

int main(void) {
    int fd = open("out.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    dup2(fd, STDOUT_FILENO);  /* stdout -> out.txt */
    close(fd);
    write(STDOUT_FILENO, "redirected\n", 11);
    return 0;
}

dup2 does not set FD_CLOEXEC on the new descriptor

Question 5 / 15

What does fcntl() do, and what are its most important operations?

fcntl() (file control) is a multipurpose system call for manipulating file descriptor properties.

Prototype:

#include <fcntl.h>
int fcntl(int fd, int cmd, ... /* arg */);

Key operations:

F_DUPFD -- duplicate fd (like dup but with a minimum fd number)
F_GETFD / F_SETFD -- get/set fd flags (currently only FD_CLOEXEC)
F_GETFL / F_SETFL -- get/set file status flags (O_APPEND, O_NONBLOCK, O_SYNC, etc.)
F_GETLK / F_SETLK / F_SETLKW -- advisory record locking
F_GETOWN / F_SETOWN -- get/set process ID for SIGIO/SIGURG signals

Important distinction:

fd flags (F_GETFD/F_SETFD) are per-fd (not shared across dup'd fds)
file status flags (F_GETFL/F_SETFL) are per-open-file-entry (shared across dup'd fds)

The most common use of fcntl() is setting O_NONBLOCK on an existing fd, since you cannot always control the flags at open() time (e.g., for pipe fds or inherited fds).

Pseudo Code

1. Get current flags: flags = fcntl(fd, F_GETFL)
2. Modify flags: flags |= O_NONBLOCK  (or flags &= ~O_NONBLOCK)
3. Set new flags: fcntl(fd, F_SETFL, flags)
4. For FD_CLOEXEC: fcntl(fd, F_SETFD, FD_CLOEXEC)
5. For locking: fill struct flock, call fcntl(fd, F_SETLK, &lock)

Code

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

/* Set a file descriptor to non-blocking mode */
int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL);
    if (flags == -1) { perror("fcntl F_GETFL"); return -1; }
    flags |= O_NONBLOCK;
    if (fcntl(fd, F_SETFL, flags) == -1) { perror("fcntl F_SETFL"); return -1; }
    return 0;
}

/* Set close-on-exec flag */
int set_cloexec(int fd) {
    int flags = fcntl(fd, F_GETFD);
    if (flags == -1) { perror("fcntl F_GETFD"); return -1; }
    flags |= FD_CLOEXEC;
    if (fcntl(fd, F_SETFD, flags) == -1) { perror("fcntl F_SETFD"); return -1; }
    return 0;
}

int main(void) {
    int fd = open("test.txt", O_RDWR | O_CREAT, 0644);
    if (fd == -1) { perror("open"); return 1; }

    /* Print current file status flags */
    int flags = fcntl(fd, F_GETFL);
    printf("Access mode: ");
    switch (flags & O_ACCMODE) {
        case O_RDONLY: printf("O_RDONLY\n"); break;
        case O_WRONLY: printf("O_WRONLY\n"); break;
        case O_RDWR:   printf("O_RDWR\n");   break;
    }

    set_nonblocking(fd);
    set_cloexec(fd);

    close(fd);
    return 0;
}

Access mode: O_RDWR

Key Points

•fcntl() is a multipurpose fd control system call
•F_GETFL/F_SETFL operate on file status flags (shared via dup)
•F_GETFD/F_SETFD operate on fd flags (per-descriptor, not shared)
•Most common use: setting O_NONBLOCK on an existing fd
•Access mode (O_RDONLY etc.) cannot be changed via F_SETFL

Gotchas

•You must read-modify-write flags (GET, OR/AND, SET) -- never just SET without GET first
•F_GETFL returns -1 on error, but -1 is also a valid (unlikely) flag combination -- always check errno
•O_ACCMODE mask is needed to extract the access mode from F_GETFL result
•F_SETFL cannot change the access mode (O_RDONLY/O_WRONLY/O_RDWR)

fcntl() manipulates fd properties: duplicate fds, get/set fd flags (FD_CLOEXEC), get/set file status flags (O_NONBLOCK, O_APPEND), and advisory locking.

•Always read-modify-write: GET, modify, SET
•fd flags are per-descriptor; file status flags are per-open-file

#include <fcntl.h>

int set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL);
    if (flags == -1) return -1;
    return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

Never SET flags without GET first -- you will clobber existing flags

Question 6 / 15

How does lseek() work, and what are file holes?

lseek() repositions the file offset of an open file descriptor.

Prototype:

#include <unistd.h>
off_t lseek(int fd, off_t offset, int whence);

whence values:

SEEK_SET -- offset from beginning of file
SEEK_CUR -- offset from current position
SEEK_END -- offset from end of file

File holes (sparse files): If you seek past the end of a file and write, the gap is called a "hole." The filesystem does not allocate disk blocks for holes -- they read as zero bytes. This creates a sparse file where the logical size is larger than the physical disk usage.

Example: A file with 1 byte at offset 0 and 1 byte at offset 1GB uses only 2 blocks on disk, not 1GB of space.

lseek() returns the new offset on success, or (off_t)-1 on error. You can use lseek(fd, 0, SEEK_CUR) to get the current offset without moving it.

Pseudo Code

1. lseek(fd, 0, SEEK_SET) -- go to beginning
2. lseek(fd, 0, SEEK_END) -- go to end (returns file size)
3. lseek(fd, 0, SEEK_CUR) -- get current offset
4. lseek(fd, 1000000, SEEK_SET); write(fd, "x", 1) -- creates a hole
5. Holes read as '\0' but use no disk blocks

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>

int main(void) {
    int fd = open("sparse.dat", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) { perror("open"); return 1; }

    /* Write at beginning */
    write(fd, "A", 1);

    /* Seek far ahead -- creates a hole */
    off_t pos = lseek(fd, 1000000, SEEK_SET);
    printf("Seeked to offset: %lld\n", (long long)pos);

    /* Write at the new position */
    write(fd, "B", 1);

    close(fd);

    /* Check file size vs disk usage */
    struct stat st;
    stat("sparse.dat", &st);
    printf("Logical size:  %lld bytes\n", (long long)st.st_size);    /* 1000001 */
    printf("Disk blocks:   %lld (512-byte blocks)\n", (long long)st.st_blocks);
    /* st_blocks will be much less than 1000001/512 */

    return 0;
}

Seeked to offset: 1000000
Logical size:  1000001 bytes
Disk blocks:   8 (512-byte blocks)

Key Points

•lseek() repositions the file offset; returns new offset or -1
•SEEK_SET = absolute, SEEK_CUR = relative, SEEK_END = from end
•Seeking past EOF and writing creates a hole (sparse file)
•Holes read as zero bytes but consume no disk blocks
•lseek(fd, 0, SEEK_CUR) returns current offset without moving

Gotchas

•lseek() does not work on pipes, FIFOs, or sockets (returns ESPIPE)
•Comparing lseek return with -1 requires casting to off_t since some offsets could be negative-looking on 32-bit
•Copying a sparse file with cp or cat fills in the holes (use cp --sparse=always)
•off_t may be 32-bit by default; compile with -D_FILE_OFFSET_BITS=64 for large file support

lseek(fd, offset, whence) sets the file offset. SEEK_SET/SEEK_CUR/SEEK_END. Seeking past EOF and writing creates a sparse file with holes that read as zeros but use no disk space.

•Holes read as zero, use no disk blocks
•Does not work on pipes or sockets

#include <unistd.h>
#include <fcntl.h>

int main(void) {
    int fd = open("sparse.dat", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    write(fd, "A", 1);
    lseek(fd, 1000000, SEEK_SET);  /* hole */
    write(fd, "B", 1);
    close(fd);  /* file is ~1MB logically, tiny on disk */
    return 0;
}

off_t is 32-bit by default on some systems -- use _FILE_OFFSET_BITS=64

Question 7 / 15

What is the three-level file descriptor architecture (fd table, open file table, v-node table)?

UNIX uses a three-level data structure to manage open files:

1. Per-process file descriptor table (fd table):

One per process, indexed by fd number (0, 1, 2, ...)
Each entry contains: fd flags (just FD_CLOEXEC) and a pointer to an open file table entry

2. System-wide open file table (file table):

One for the entire kernel
Each entry contains: file status flags (O_APPEND, O_NONBLOCK, etc.), current file offset, reference count, pointer to v-node entry
Created by each open() call (even for the same file)

3. V-node table (or inode table):

One per open file (unique per file, not per open)
Contains: file type, pointers to filesystem-specific functions (read/write ops), inode information (size, permissions, timestamps, disk block locations)

Key implications:

dup()/dup2() create a new fd table entry pointing to the same open file table entry (shared offset)
Two separate open() calls on the same file create different open file table entries (independent offsets)
fork() duplicates the fd table -- child inherits pointers to the same open file table entries (shared offsets between parent and child)
O_APPEND works because the kernel sets offset to file size atomically in the open file table entry on each write

Pseudo Code

Level 1: Per-process FD table
  fd 0 -> [FD_CLOEXEC=0, ptr -> OFT entry A]
  fd 1 -> [FD_CLOEXEC=0, ptr -> OFT entry B]
  fd 3 -> [FD_CLOEXEC=1, ptr -> OFT entry A]  (via dup)

Level 2: System-wide Open File Table
  Entry A -> [flags=O_RDWR, offset=100, refcount=2, ptr -> vnode X]
  Entry B -> [flags=O_WRONLY|O_APPEND, offset=500, refcount=1, ptr -> vnode X]

Level 3: V-node (inode) table
  Vnode X -> [type=REG, size=1024, inode=12345, blocks=[...]]

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

int main(void) {
    /* Two open() calls on the same file: independent offsets */
    int fd1 = open("data.txt", O_RDWR | O_CREAT | O_TRUNC, 0644);
    int fd2 = open("data.txt", O_RDWR);
    /* fd1 and fd2 have DIFFERENT open file table entries */
    /* They have independent offsets */

    write(fd1, "AAAA", 4);  /* fd1 offset = 4 */
    write(fd2, "BB", 2);    /* fd2 offset = 2 (overwrites first 2 bytes) */
    /* File contents: "BBAA" */

    /* dup creates shared offset */
    int fd3 = dup(fd1);
    /* fd3 shares open file table entry with fd1 */
    off_t off1 = lseek(fd1, 0, SEEK_CUR);  /* offset = 4 */
    off_t off3 = lseek(fd3, 0, SEEK_CUR);  /* offset = 4 (same!) */
    printf("fd1 offset=%lld, fd3 offset=%lld\n",
           (long long)off1, (long long)off3);

    write(fd1, "CC", 2);  /* offset advances to 6 for BOTH fd1 and fd3 */
    off3 = lseek(fd3, 0, SEEK_CUR);
    printf("After write via fd1, fd3 offset=%lld\n", (long long)off3);  /* 6 */

    close(fd1);
    close(fd2);
    close(fd3);
    return 0;
}

fd1 offset=4, fd3 offset=4
After write via fd1, fd3 offset=6

Key Points

•Three levels: per-process fd table, system open file table, v-node table
•dup/fork share the open file table entry (shared offset and flags)
•Separate open() calls create separate open file table entries (independent offsets)
•FD_CLOEXEC is per-fd; file status flags and offset are per-open-file-entry
•fork() causes parent and child to share offsets (both point to same open file entry)

Gotchas

•After fork(), parent and child share offsets -- writing in one moves the offset for the other
•Two processes opening the same file independently have separate offsets and can overwrite each other
•Closing a dup'd fd does not affect the other fd or the offset, but decrements the refcount
•The v-node is shared even across independent opens -- that is how the kernel knows it is the same file

Three levels: (1) per-process fd table (fd flags + pointer), (2) system-wide open file table (offset, status flags, refcount), (3) v-node/inode table (file metadata). dup()/fork() share level 2; separate open() calls create new level 2 entries.

•dup() shares offset; separate open() does not
•fork() inherits shared open file table entries

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

int main(void) {
    int fd1 = open("f.txt", O_RDWR | O_CREAT, 0644);
    int fd2 = dup(fd1);  /* shares offset with fd1 */
    int fd3 = open("f.txt", O_RDWR);  /* independent offset */
    write(fd1, "AB", 2);
    printf("fd2 offset=%lld\n", (long long)lseek(fd2, 0, SEEK_CUR)); /* 2 */
    printf("fd3 offset=%lld\n", (long long)lseek(fd3, 0, SEEK_CUR)); /* 0 */
    close(fd1); close(fd2); close(fd3);
    return 0;
}

fd2 offset=2
fd3 offset=0

After fork, parent and child share file offsets -- coordinate writes carefully

Question 8 / 15

How do atomic operations with O_APPEND and O_CREAT|O_EXCL prevent race conditions?

UNIX provides two important atomic file operations to avoid race conditions:

O_APPEND -- Atomic append: When a file is opened with O_APPEND, every write() atomically seeks to the end of the file and writes. This is done as a single kernel operation, so multiple processes appending to the same file (e.g., a log file) will not overwrite each other's data.

Without O_APPEND, the sequence lseek(fd, 0, SEEK_END); write(fd, data, len); is NOT atomic -- another process could write between the seek and write, causing data loss.

O_CREAT | O_EXCL -- Atomic create-or-fail: When both flags are specified, open() will fail with EEXIST if the file already exists. The check-and-create is atomic. This is the correct way to implement lock files.

Without O_EXCL, checking stat() then open() has a TOCTOU (time-of-check-to-time-of-use) race: another process could create the file between the check and the open.

Note: O_CREAT | O_EXCL does NOT follow symbolic links. If the path is a symlink, it fails with EEXIST regardless. This is a security feature to prevent symlink attacks.

Pseudo Code

RACE (broken):
  Process A: lseek(fd, 0, SEEK_END)  -> offset 1000
  Process B: lseek(fd, 0, SEEK_END)  -> offset 1000
  Process A: write(fd, dataA, 100)   -> writes at 1000
  Process B: write(fd, dataB, 50)    -> writes at 1000 (OVERWRITES A!)

ATOMIC (correct with O_APPEND):
  Process A: write(fd, dataA, 100)   -> kernel atomically seeks to end, writes at 1000
  Process B: write(fd, dataB, 50)    -> kernel atomically seeks to end, writes at 1100

Code

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

/* Safe concurrent log writing */
void append_log(const char *msg) {
    int fd = open("app.log", O_WRONLY | O_APPEND | O_CREAT, 0644);
    if (fd == -1) { perror("open log"); return; }
    /* write() is atomic to EOF thanks to O_APPEND */
    write(fd, msg, strlen(msg));
    close(fd);
}

/* Safe lock file creation */
int acquire_lock(const char *lockpath) {
    int fd = open(lockpath, O_WRONLY | O_CREAT | O_EXCL, 0600);
    if (fd == -1) {
        if (errno == EEXIST) {
            fprintf(stderr, "Lock already held\n");
            return -1;
        }
        perror("open lockfile");
        return -1;
    }
    /* Write PID to lock file for debugging */
    char buf[32];
    int len = snprintf(buf, sizeof(buf), "%d\n", getpid());
    write(fd, buf, (size_t)len);
    close(fd);
    return 0;
}

void release_lock(const char *lockpath) {
    unlink(lockpath);
}

int main(void) {
    append_log("Server started\n");

    if (acquire_lock("/tmp/myapp.lock") == 0) {
        printf("Lock acquired\n");
        /* do work ... */
        release_lock("/tmp/myapp.lock");
    }
    return 0;
}

Lock acquired

Key Points

•O_APPEND makes write() atomic: seek-to-end + write in one kernel operation
•O_CREAT | O_EXCL is atomic create-or-fail -- no TOCTOU race
•O_EXCL does not follow symlinks (security feature)
•Multiple processes can safely append to the same log file with O_APPEND
•Lock files should use O_CREAT | O_EXCL, not stat() + open()

Gotchas

•O_APPEND guarantees atomicity of the seek+write but NOT of multiple writes appearing contiguous
•NFS does not properly support O_APPEND atomicity -- use fcntl locking on NFS
•O_EXCL lock files are not cleaned up on crash -- need a stale lock detection strategy
•POSIX guarantees O_APPEND atomic writes only up to PIPE_BUF bytes for pipes; for regular files it is always atomic

O_APPEND makes each write() atomically seek to EOF then write (safe for concurrent log appending). O_CREAT | O_EXCL atomically creates a file or fails if it exists (safe lock file pattern).

•O_APPEND: atomic seek-to-end + write
•O_CREAT|O_EXCL: atomic create-or-fail

#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

/* Atomic append */
int fd = open("log", O_WRONLY | O_APPEND | O_CREAT, 0644);
write(fd, "line\n", 5);  /* always at EOF */
close(fd);

/* Atomic lock file */
int lk = open("/tmp/app.lock", O_WRONLY | O_CREAT | O_EXCL, 0600);
if (lk == -1 && errno == EEXIST) { /* already locked */ }

NFS does not support O_APPEND atomicity properly

Question 9 / 15

How does O_NONBLOCK affect I/O operations?

The O_NONBLOCK flag changes the behavior of I/O operations so they never block the calling process.

Without O_NONBLOCK (default blocking behavior):

read() on a pipe with no data blocks until data arrives or all writers close
write() on a full pipe blocks until space is available
open() on a FIFO blocks until the other end opens
accept() on a socket blocks until a connection arrives

With O_NONBLOCK:

read() with no data returns -1 with errno = EAGAIN (or EWOULDBLOCK)
write() with no buffer space returns -1 with errno = EAGAIN
open() on a read-only FIFO returns immediately
accept() with no pending connections returns -1 with EAGAIN

Non-blocking I/O is essential for event-driven programming (used with select(), poll(), or epoll()) where a single thread handles multiple I/O sources.

You can set O_NONBLOCK at open() time or later with fcntl(fd, F_SETFL, flags | O_NONBLOCK).

Pseudo Code

1. Set O_NONBLOCK via open() or fcntl()
2. Attempt read/write
3. If returns -1 and errno == EAGAIN:
   a. No data available (read) or buffer full (write)
   b. Register fd with poll/epoll and wait
   c. Retry when poll says fd is ready
4. If returns > 0: partial or complete I/O succeeded
5. If read returns 0: EOF (peer closed)

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>

int main(void) {
    int pipefd[2];
    if (pipe(pipefd) == -1) { perror("pipe"); return 1; }

    /* Set read end to non-blocking */
    int flags = fcntl(pipefd[0], F_GETFL);
    fcntl(pipefd[0], F_SETFL, flags | O_NONBLOCK);

    /* Try to read from empty pipe -- would block normally */
    char buf[128];
    ssize_t n = read(pipefd[0], buf, sizeof(buf));
    if (n == -1) {
        if (errno == EAGAIN || errno == EWOULDBLOCK)
            printf("No data available (non-blocking)\n");
        else
            perror("read");
    }

    /* Write some data, then read succeeds */
    write(pipefd[1], "hello", 5);
    n = read(pipefd[0], buf, sizeof(buf));
    if (n > 0) {
        buf[n] = '\0';
        printf("Read: %s\n", buf);  /* "hello" */
    }

    close(pipefd[0]);
    close(pipefd[1]);
    return 0;
}

No data available (non-blocking)
Read: hello

Key Points

•O_NONBLOCK prevents read/write/open/accept from blocking
•Returns -1 with EAGAIN/EWOULDBLOCK when operation would block
•Essential for event-driven I/O with select/poll/epoll
•Can be set at open() or later with fcntl(F_SETFL)
•Non-blocking open() on a FIFO returns immediately for O_RDONLY

Gotchas

•EAGAIN and EWOULDBLOCK may be the same value on Linux but are different constants on some systems
•O_NONBLOCK on regular files has no effect -- regular file reads never block (they wait for disk I/O)
•Non-blocking write can return a short write (less than requested) -- must handle partial writes
•Opening a FIFO with O_WRONLY | O_NONBLOCK fails with ENXIO if no reader

O_NONBLOCK makes I/O operations return immediately instead of blocking. If the operation would block, it returns -1 with errno = EAGAIN. Used with select()/poll()/epoll() for event-driven I/O.

•Returns EAGAIN instead of blocking
•Used with event loops (select/poll/epoll)

#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(void) {
    int pfd[2];
    pipe(pfd);
    fcntl(pfd[0], F_SETFL, fcntl(pfd[0], F_GETFL) | O_NONBLOCK);
    char buf[64];
    if (read(pfd[0], buf, 64) == -1 && errno == EAGAIN)
        printf("Would block -- no data\n");
    close(pfd[0]); close(pfd[1]);
    return 0;
}

Would block -- no data

O_NONBLOCK has no effect on regular file I/O

Question 10 / 15

What are pread() and pwrite(), and why are they useful for multithreaded programs?

pread() and pwrite() perform I/O at a specified offset without changing the file's current offset. This makes them atomic with respect to the file position.

Prototypes:

#include <unistd.h>
ssize_t pread(int fd, void *buf, size_t count, off_t offset);
ssize_t pwrite(int fd, const void *buf, size_t count, off_t offset);

Why they matter for threads: All threads in a process share the same file descriptors (and therefore the same file offsets in the open file table). If two threads do lseek() + read() on the same fd, they race on the shared offset:

Thread A: lseek(fd, 100, SEEK_SET)
Thread B: lseek(fd, 200, SEEK_SET) -- overwrites A's seek
Thread A: read(fd, buf, 50) -- reads from 200, not 100!

With pread(fd, buf, 50, 100), the offset is specified in the call and the file's current offset is untouched. No locking needed.

Other uses:

Database systems that read/write at specific page offsets
Any scenario where multiple I/O streams target different parts of the same file

Pseudo Code

Without pread (racy):
  Thread A: lseek(fd, offsetA, SEEK_SET)
  Thread B: lseek(fd, offsetB, SEEK_SET)  <-- overwrites A's position
  Thread A: read(fd, bufA, n)  <-- reads from offsetB!

With pread (safe):
  Thread A: pread(fd, bufA, n, offsetA)  <-- atomic, does not change fd offset
  Thread B: pread(fd, bufB, n, offsetB)  <-- atomic, does not change fd offset

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <pthread.h>
#include <string.h>

#define PAGE_SIZE 4096

static int shared_fd;

void *reader_thread(void *arg) {
    int page_num = *(int *)arg;
    char buf[PAGE_SIZE];
    /* pread: read at offset without affecting shared file offset */
    ssize_t n = pread(shared_fd, buf, PAGE_SIZE,
                      (off_t)page_num * PAGE_SIZE);
    if (n > 0) {
        printf("Thread read page %d: first byte = 0x%02x\n",
               page_num, (unsigned char)buf[0]);
    }
    return NULL;
}

int main(void) {
    shared_fd = open("database.dat", O_RDWR | O_CREAT, 0644);
    if (shared_fd == -1) { perror("open"); return 1; }

    /* Write some pages using pwrite */
    char page0[PAGE_SIZE], page1[PAGE_SIZE];
    memset(page0, 0xAA, PAGE_SIZE);
    memset(page1, 0xBB, PAGE_SIZE);
    pwrite(shared_fd, page0, PAGE_SIZE, 0);
    pwrite(shared_fd, page1, PAGE_SIZE, PAGE_SIZE);

    /* Read different pages concurrently -- no locking needed */
    pthread_t t1, t2;
    int p0 = 0, p1 = 1;
    pthread_create(&t1, NULL, reader_thread, &p0);
    pthread_create(&t2, NULL, reader_thread, &p1);
    pthread_join(t1, NULL);
    pthread_join(t2, NULL);

    close(shared_fd);
    return 0;
}

Thread read page 0: first byte = 0xaa
Thread read page 1: first byte = 0xbb

Key Points

•pread/pwrite perform I/O at a specified offset without changing the file offset
•They are atomic with respect to the file position (no lseek + read/write race)
•Essential for multithreaded programs sharing a file descriptor
•Used heavily in database engines for page-level I/O
•Equivalent to lseek + read/write but without the race condition

Gotchas

•pread/pwrite can still return short reads/writes -- you must loop
•pwrite with O_APPEND ignores the offset argument (always appends)
•They share the same file lock state as read/write
•Not available on very old UNIX systems (but POSIX since SUSv2)

pread(fd, buf, n, offset) and pwrite(fd, buf, n, offset) do I/O at a specific offset without changing the fd's current position. Thread-safe alternative to lseek() + read()/write().

•Atomic seek+read/write in one call
•Does not modify the file's current offset

#include <unistd.h>
#include <fcntl.h>

int main(void) {
    int fd = open("data.bin", O_RDWR | O_CREAT, 0644);
    char buf[] = "page0";
    pwrite(fd, buf, 5, 0);       /* write at offset 0 */
    pwrite(fd, "page1", 5, 4096); /* write at offset 4096 */
    char out[5];
    pread(fd, out, 5, 0);        /* read from offset 0 */
    close(fd);
    return 0;
}

pwrite with O_APPEND ignores the offset

Question 11 / 15

What is scatter-gather I/O with readv() and writev()?

Scatter-gather I/O (readv()/writev()) allows reading into or writing from multiple non-contiguous buffers in a single system call.

Prototypes:

#include <sys/uio.h>
ssize_t readv(int fd, const struct iovec *iov, int iovcnt);
ssize_t writev(int fd, const struct iovec *iov, int iovcnt);

struct iovec:

struct iovec {
    void  *iov_base;  /* starting address of buffer */
    size_t iov_len;   /* number of bytes */
};

Advantages:

Atomicity -- writev writes all buffers in one kernel operation (important for protocols with headers + data)
Performance -- one syscall instead of multiple read/write calls (fewer context switches)
Convenience -- no need to concatenate buffers before writing

Use cases:

Writing an HTTP response: header + body in one writev()
Reading structured data into separate fields
Network protocols with fixed headers and variable payloads

Pseudo Code

1. Define array of struct iovec, each with base pointer and length
2. Call writev(fd, iov, count) to write all buffers atomically
3. Call readv(fd, iov, count) to scatter input into multiple buffers
4. Returns total bytes transferred or -1 on error
5. May return short count (partial I/O) -- handle like read/write

Code

#include <sys/uio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(void) {
    int fd = open("output.dat", O_WRONLY | O_CREAT | O_TRUNC, 0644);
    if (fd == -1) { perror("open"); return 1; }

    /* Write header + body in one atomic operation */
    char header[] = "HTTP/1.1 200 OK\r\nContent-Length: 13\r\n\r\n";
    char body[] = "Hello, World!";

    struct iovec iov[2];
    iov[0].iov_base = header;
    iov[0].iov_len  = strlen(header);
    iov[1].iov_base = body;
    iov[1].iov_len  = strlen(body);

    ssize_t nwritten = writev(fd, iov, 2);
    printf("writev wrote %zd bytes\n", nwritten);
    close(fd);

    /* Read back into separate buffers */
    fd = open("output.dat", O_RDONLY);
    char hdr_buf[64];
    char body_buf[32];
    struct iovec riov[2];
    riov[0].iov_base = hdr_buf;
    riov[0].iov_len  = strlen(header);  /* read exactly header size */
    riov[1].iov_base = body_buf;
    riov[1].iov_len  = strlen(body);

    ssize_t nread = readv(fd, riov, 2);
    body_buf[strlen(body)] = '\0';
    printf("readv read %zd bytes, body: %s\n", nread, body_buf);

    close(fd);
    return 0;
}

writev wrote 52 bytes
readv read 52 bytes, body: Hello, World!

Key Points

•readv/writev transfer data to/from multiple buffers in one syscall
•writev is atomic for regular files (no interleaving from other writes)
•Reduces number of system calls compared to multiple write() calls
•struct iovec has iov_base (pointer) and iov_len (size)
•IOV_MAX defines the maximum number of iovec entries (at least 16, often 1024)

Gotchas

•writev can still do a short write -- you must handle partial I/O
•The iovec array is read by the kernel, not modified -- readv fills the buffers
•Exceeding IOV_MAX entries causes EINVAL
•writev atomicity is not guaranteed for sockets -- data may be interleaved with other writes

readv()/writev() do scatter-gather I/O: read into or write from multiple buffers in one syscall. Uses struct iovec array. One syscall = fewer context switches + atomic writes.

•One syscall for multiple buffers
•Atomic for regular files

#include <sys/uio.h>
#include <unistd.h>
#include <string.h>

int main(void) {
    char hdr[] = "HDR:";
    char data[] = "payload";
    struct iovec iov[2] = {
        { .iov_base = hdr,  .iov_len = strlen(hdr) },
        { .iov_base = data, .iov_len = strlen(data) }
    };
    writev(STDOUT_FILENO, iov, 2);  /* prints: HDR:payload */
    return 0;
}

IOV_MAX limits the number of iovec entries

Question 12 / 15

What are /dev/null and /dev/zero, and how are they used in systems programming?

UNIX provides special device files that are useful for I/O operations:

/dev/null (the bit bucket):

Reading from /dev/null returns EOF immediately (0 bytes)
Writing to /dev/null always succeeds and discards all data
Used to discard unwanted output: ./program > /dev/null 2>&1
In C, redirect stderr: dup2(open("/dev/null", O_WRONLY), STDERR_FILENO)
Daemons redirect all three standard fds to /dev/null

/dev/zero:

Reading from /dev/zero returns an infinite stream of zero bytes (\0)
Used to create zero-filled buffers or files
Used with mmap() for anonymous memory mapping on some systems
dd if=/dev/zero of=file bs=1M count=100 creates a 100MB zero-filled file

/dev/random and /dev/urandom:

Provide random bytes from the kernel's entropy pool
/dev/random may block if entropy is low; /dev/urandom never blocks
Used for cryptographic key generation and secure randomness

Pseudo Code

1. /dev/null: open, dup2 to fd 1 or 2 to silence output
2. /dev/zero: read from it to get zero-filled buffers
3. Common daemon pattern:
   open /dev/null
   dup2(devnull_fd, STDIN_FILENO)
   dup2(devnull_fd, STDOUT_FILENO)
   dup2(devnull_fd, STDERR_FILENO)
   close(devnull_fd)

Code

#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>

int main(void) {
    /* Redirect stderr to /dev/null (silence errors) */
    int devnull = open("/dev/null", O_RDWR);
    if (devnull == -1) { perror("open /dev/null"); return 1; }
    dup2(devnull, STDERR_FILENO);
    fprintf(stderr, "This is silenced\n");  /* goes nowhere */

    /* Read zero bytes from /dev/zero */
    int devzero = open("/dev/zero", O_RDONLY);
    if (devzero == -1) { perror("open /dev/zero"); return 1; }
    char buf[1024];
    ssize_t n = read(devzero, buf, sizeof(buf));
    printf("Read %zd bytes from /dev/zero\n", n);  /* 1024 */

    /* Verify all bytes are zero */
    int all_zero = 1;
    for (int i = 0; i < (int)n; i++) {
        if (buf[i] != 0) { all_zero = 0; break; }
    }
    printf("All zeros: %s\n", all_zero ? "yes" : "no");  /* yes */

    close(devzero);
    close(devnull);
    return 0;
}

Read 1024 bytes from /dev/zero
All zeros: yes

Key Points

•/dev/null discards all writes, returns EOF on read
•/dev/zero returns infinite zero bytes on read
•Daemons redirect stdin/stdout/stderr to /dev/null
•/dev/urandom provides non-blocking random bytes
•These are character device files, not regular files

Gotchas

•/dev/null opened O_RDONLY cannot be used for dup2 to stdout -- use O_RDWR
•/dev/random may block on Linux if entropy pool is empty -- prefer /dev/urandom for most uses
•Reading from /dev/zero in a loop without a limit will consume CPU forever
•On some embedded systems, these device files may not exist

/dev/null discards all writes and gives EOF on read. /dev/zero provides infinite zero bytes. Daemons redirect stdin/stdout/stderr to /dev/null.

•/dev/null = bit bucket; /dev/zero = zero source
•Used in daemon initialization

#include <unistd.h>
#include <fcntl.h>

int main(void) {
    /* Silence stdout */
    int fd = open("/dev/null", O_WRONLY);
    dup2(fd, STDOUT_FILENO);
    close(fd);
    write(STDOUT_FILENO, "gone\n", 5);  /* discarded */
    return 0;
}

Open /dev/null with O_RDWR if you dup2 to both stdin and stdout

Question 13 / 15

How do file descriptor limits work, and how can you check or change them?

Every process has limits on the number of file descriptors it can have open simultaneously.

Two types of limits:

Soft limit (RLIMIT_NOFILE cur): The actual enforced limit. A process can raise this up to the hard limit.
Hard limit (RLIMIT_NOFILE max): The ceiling. Only root (or CAP_SYS_RESOURCE) can raise this. A process can irreversibly lower it.

System-wide limit:

/proc/sys/fs/file-max on Linux -- total open files across all processes
ENFILE error when this is exceeded

Per-process limit:

EMFILE error when a process exceeds its per-process limit
Default is typically 1024 (soft) on many Linux distributions

APIs:

getrlimit(RLIMIT_NOFILE, &rlim) -- query limits
setrlimit(RLIMIT_NOFILE, &rlim) -- set limits
sysconf(_SC_OPEN_MAX) -- portable way to get maximum
Shell: ulimit -n (soft), ulimit -Hn (hard)

Servers (nginx, Apache, database systems) often need to raise the soft limit to handle many concurrent connections.

Pseudo Code

1. getrlimit(RLIMIT_NOFILE, &rlim) to read current limits
2. rlim.rlim_cur = desired_soft_limit (must be <= rlim.rlim_max)
3. setrlimit(RLIMIT_NOFILE, &rlim) to apply
4. For hard limit increase: need root/CAP_SYS_RESOURCE
5. sysconf(_SC_OPEN_MAX) returns current soft limit

Code

#include <sys/resource.h>
#include <unistd.h>
#include <stdio.h>

int main(void) {
    struct rlimit rl;

    /* Get current limits */
    if (getrlimit(RLIMIT_NOFILE, &rl) == -1) {
        perror("getrlimit");
        return 1;
    }
    printf("Soft limit: %llu\n", (unsigned long long)rl.rlim_cur);
    printf("Hard limit: %llu\n", (unsigned long long)rl.rlim_max);

    /* Raise soft limit to hard limit */
    rl.rlim_cur = rl.rlim_max;
    if (setrlimit(RLIMIT_NOFILE, &rl) == -1) {
        perror("setrlimit");
        return 1;
    }
    printf("New soft limit: %llu\n", (unsigned long long)rl.rlim_cur);

    /* Portable way to get max open files */
    long max_fd = sysconf(_SC_OPEN_MAX);
    printf("sysconf _SC_OPEN_MAX: %ld\n", max_fd);

    return 0;
}

Soft limit: 256
Hard limit: unlimited
New soft limit: unlimited
sysconf _SC_OPEN_MAX: unlimited

Key Points

•Soft limit is enforced; hard limit is the ceiling
•EMFILE = per-process limit hit; ENFILE = system-wide limit hit
•getrlimit/setrlimit to query/modify per-process limits
•Only root can raise the hard limit
•sysconf(_SC_OPEN_MAX) is the portable query

Gotchas

•The default soft limit of 1024 is too low for high-performance servers
•Lowering the hard limit is irreversible for the process
•select() has FD_SETSIZE (often 1024) which limits usable fds -- use poll/epoll instead
•Close-on-exec (FD_CLOEXEC) helps prevent fd leaks in long-running servers

Each process has soft and hard limits on open fds (RLIMIT_NOFILE). Use getrlimit()/setrlimit() to query/change. Soft can be raised to hard; only root raises hard. Default is often 1024.

•Soft limit enforced; hard limit is ceiling
•EMFILE = per-process, ENFILE = system-wide

#include <sys/resource.h>
#include <stdio.h>

int main(void) {
    struct rlimit rl;
    getrlimit(RLIMIT_NOFILE, &rl);
    printf("soft=%llu hard=%llu\n",
           (unsigned long long)rl.rlim_cur,
           (unsigned long long)rl.rlim_max);
    rl.rlim_cur = rl.rlim_max;  /* raise soft to hard */
    setrlimit(RLIMIT_NOFILE, &rl);
    return 0;
}

soft=256 hard=unlimited

Default 1024 is too low for servers handling many connections

Question 14 / 15

What is the difference between ANSI C FILE* streams and UNIX file descriptors?

UNIX file descriptors (fd):

Low-level, kernel-space abstraction
Small integer (0, 1, 2, ...)
System calls: open(), read(), write(), close(), lseek()
No buffering (each call is a system call)
POSIX-specific, not part of ANSI C

ANSI C FILE* streams:

High-level, user-space abstraction (defined in <stdio.h>)
FILE * pointer to an opaque struct
Functions: fopen(), fread(), fwrite(), fclose(), fseek()
Buffered I/O -- reduces system calls by batching reads/writes
Portable across all C implementations

Buffering modes:

Fully buffered (_IOFBF): flushes when buffer is full (default for regular files)
Line buffered (_IOLBF): flushes on newline (default for terminals)
Unbuffered (_IONBF): no buffering (stderr default)

Interconversion:

fileno(FILE *) -- get the fd underlying a FILE stream
fdopen(fd, mode) -- wrap an fd in a FILE stream

When to use which:

Use FILE* for text processing, formatted I/O, portability
Use fd for system programming: pipes, sockets, non-blocking I/O, dup2, fcntl

Pseudo Code

FILE* (buffered):
  fopen -> fread/fwrite/fprintf -> fclose
  Buffer: [........] -- flushed when full, on newline, or on fclose

fd (unbuffered):
  open -> read/write -> close
  Each call = one system call

Conversion:
  int fd = fileno(fp);    /* FILE* -> fd */
  FILE *fp = fdopen(fd, "r");  /* fd -> FILE* */

Code

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>

int main(void) {
    /* FILE* is buffered -- fewer syscalls */
    FILE *fp = fopen("test.txt", "w");
    if (!fp) { perror("fopen"); return 1; }
    fprintf(fp, "Line %d\n", 1);   /* stays in buffer */
    fprintf(fp, "Line %d\n", 2);   /* stays in buffer */
    fflush(fp);                     /* NOW written to disk */

    /* Get underlying fd from FILE* */
    int fd = fileno(fp);
    printf("Underlying fd = %d\n", fd);

    /* Can mix, but dangerous without fflush first */
    fflush(fp);
    write(fd, "raw write\n", 10);  /* bypasses buffer */

    fclose(fp);  /* flushes buffer and closes fd */

    /* Wrap an fd in a FILE* */
    int pipefd[2];
    pipe(pipefd);
    FILE *pipe_fp = fdopen(pipefd[0], "r");
    if (pipe_fp) {
        /* Can now use fgets, fscanf, etc. on the pipe */
        fclose(pipe_fp);  /* also closes pipefd[0] */
    }
    close(pipefd[1]);

    return 0;
}

Underlying fd = 3

Key Points

•FILE* is buffered (user-space); fd is unbuffered (kernel calls)
•FILE* is portable ANSI C; fd is POSIX-specific
•fileno() gets fd from FILE*; fdopen() wraps fd in FILE*
•Three buffering modes: fully buffered, line buffered, unbuffered
•Use fd for system programming (pipes, sockets, dup2, fcntl)

Gotchas

•Mixing FILE* and fd operations on the same stream without fflush causes data corruption/reordering
•fclose() closes the underlying fd -- do not close it separately
•After fork(), both parent and child have the same FILE* buffers -- fflush before fork to avoid duplicate output
•setvbuf() must be called before any I/O on the stream

FILE* is buffered, portable ANSI C (fopen/fread/fwrite). File descriptors are unbuffered, POSIX-specific (open/read/write). Convert with fileno() and fdopen(). Use fd for system programming, FILE* for text/formatted I/O.

•FILE* = buffered + portable; fd = unbuffered + POSIX
•fileno() and fdopen() for conversion

#include <stdio.h>
#include <unistd.h>

int main(void) {
    FILE *fp = fopen("f.txt", "w");
    int fd = fileno(fp);          /* FILE* -> fd */
    fprintf(fp, "buffered\n");    /* in buffer */
    fflush(fp);                   /* now on disk */
    write(fd, "direct\n", 7);    /* bypasses buffer */
    fclose(fp);                   /* flushes + closes fd */
    return 0;
}

Mixing FILE* and fd I/O without fflush causes data corruption

Question 15 / 15

How does record locking work with fcntl(), and what is the difference between advisory and mandatory locking?

POSIX record locking (via fcntl()) allows processes to lock byte ranges of a file to coordinate concurrent access.

struct flock:

struct flock {
    short l_type;    /* F_RDLCK, F_WRLCK, F_UNLCK */
    short l_whence;  /* SEEK_SET, SEEK_CUR, SEEK_END */
    off_t l_start;   /* starting offset */
    off_t l_len;     /* 0 = lock to EOF */
    pid_t l_pid;     /* PID of lock owner (F_GETLK only) */
};

Lock types:

F_RDLCK -- shared (read) lock: multiple readers allowed
F_WRLCK -- exclusive (write) lock: single writer, no readers
F_UNLCK -- unlock a region

fcntl commands:

F_SETLK -- set lock (non-blocking, returns EAGAIN if cannot)
F_SETLKW -- set lock (blocking, waits until lock is available)
F_GETLK -- test if a lock would be blocked (does NOT set the lock)

Advisory vs Mandatory locking:

Advisory: The default. Locks are only respected by processes that cooperate by also calling fcntl(). A process can ignore locks and read/write freely.
Mandatory: Enforced by the kernel on every read()/write(). Requires set-group-ID bit on and group-execute off (chmod g+s,g-x file). Unreliable and deprecated on Linux.

Locks are associated with (process, inode) pairs, not file descriptors. Closing ANY fd that refers to the file releases ALL locks held by that process on that file.

Pseudo Code

1. Fill struct flock: l_type, l_whence, l_start, l_len
2. F_GETLK: test if lock is available (fills l_pid if blocked)
3. F_SETLK: try to set lock, return immediately (EAGAIN if blocked)
4. F_SETLKW: try to set lock, block until available
5. To unlock: set l_type = F_UNLCK, call F_SETLK
6. Locks released on close of ANY fd to the file (or process exit)

Code

#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>

int lock_region(int fd, int type, off_t start, off_t len) {
    struct flock fl;
    fl.l_type   = (short)type;
    fl.l_whence = SEEK_SET;
    fl.l_start  = start;
    fl.l_len    = len;     /* 0 = lock to EOF */

    if (fcntl(fd, F_SETLK, &fl) == -1) {
        if (errno == EAGAIN || errno == EACCES) {
            fprintf(stderr, "Region locked by another process\n");
            return -1;
        }
        perror("fcntl F_SETLK");
        return -1;
    }
    return 0;
}

int lock_region_wait(int fd, int type, off_t start, off_t len) {
    struct flock fl;
    fl.l_type   = (short)type;
    fl.l_whence = SEEK_SET;
    fl.l_start  = start;
    fl.l_len    = len;
    return fcntl(fd, F_SETLKW, &fl);  /* blocks until lock acquired */
}

int unlock_region(int fd, off_t start, off_t len) {
    struct flock fl;
    fl.l_type   = F_UNLCK;
    fl.l_whence = SEEK_SET;
    fl.l_start  = start;
    fl.l_len    = len;
    return fcntl(fd, F_SETLK, &fl);
}

int main(void) {
    int fd = open("shared.dat", O_RDWR | O_CREAT, 0644);
    if (fd == -1) { perror("open"); return 1; }

    /* Lock first 100 bytes exclusively */
    if (lock_region(fd, F_WRLCK, 0, 100) == 0) {
        printf("Locked bytes 0-99 for writing\n");
        write(fd, "exclusive data", 14);
        unlock_region(fd, 0, 100);
    }

    /* Lock entire file (l_len=0 means to EOF) */
    if (lock_region(fd, F_RDLCK, 0, 0) == 0) {
        printf("Read-locked entire file\n");
        unlock_region(fd, 0, 0);
    }

    close(fd);
    return 0;
}

Locked bytes 0-99 for writing
Read-locked entire file

Key Points

•fcntl() provides byte-range (record) locking with F_SETLK/F_SETLKW/F_GETLK
•F_RDLCK = shared, F_WRLCK = exclusive, F_UNLCK = release
•l_len = 0 means lock from l_start to EOF (grows with file)
•Advisory locks are the default; mandatory locks are deprecated on Linux
•Locks are per (process, inode), not per file descriptor

Gotchas

•Closing ANY fd that refers to the same file releases ALL of that process's locks on it
•F_GETLK only tests if a lock WOULD be blocked; it does NOT acquire the lock
•Deadlock detection: kernel returns EDEADLK for F_SETLKW if deadlock would occur
•flock() (BSD) and fcntl() (POSIX) locks are independent and do not interact
•Locks are not inherited across fork() but are preserved across exec()

fcntl() with F_SETLK/F_SETLKW provides byte-range locking. F_RDLCK = shared, F_WRLCK = exclusive. Advisory by default (cooperative). Locks are per (process, inode) -- closing any fd releases all locks on that file.

•F_SETLK = non-blocking; F_SETLKW = blocking
•Locks tied to (process, inode) not fd

#include <fcntl.h>
#include <unistd.h>

int main(void) {
    int fd = open("data.dat", O_RDWR | O_CREAT, 0644);
    struct flock fl = { .l_type = F_WRLCK, .l_whence = SEEK_SET,
                        .l_start = 0, .l_len = 0 };
    if (fcntl(fd, F_SETLK, &fl) == -1) {
        /* locked by another process */
    }
    /* ... do work ... */
    fl.l_type = F_UNLCK;
    fcntl(fd, F_SETLK, &fl);
    close(fd);
    return 0;
}

Closing any fd to the file releases ALL your locks on it

Built by onllm.dev

Home GitHub Support awesomePrep