BASH / CHAPTER 07

Read records without corrupting them

Intermediate

Treat standard input and output as deliberate data streams.

By the end of this chapter

  • Use read without trimming or backslash processing
  • Handle an unterminated final line
  • Keep data and diagnostics on different streams

The idea

Many shell programs connect through standard input, standard output, and standard error. Design a filter so its output contains only its result data; send diagnostics to standard error. Redirection connects these streams to files or commands. Read text using a loop rather than splitting command substitution, because word splitting does not preserve line boundaries or empty lines.

The idiom IFS= read -r line avoids trimming field separators and treating backslashes as escapes. Read returns a failure status at end of input, even when a final unterminated line supplied characters, so include that remaining nonempty value when the input contract allows it. Newlines delimit these records; arbitrary filenames can themselves contain newlines and need a different delimiter, commonly a NUL-aware producer and consumer.

Bash example
#!/usr/bin/env bash
while IFS= read -r line || [[ -n $line ]]; do
  printf '[%s]\n' "$line"
done <<'DATA'
  first
back\slash
DATA
Result
[  first]
[back\slash]

The empty IFS preserves leading spaces, and -r preserves the backslash. The quoted here-document delimiter also prevents expansion inside its content.

Try it yourself

Count nonempty lines from standard input while preserving whitespace and handling a final line without a newline.

Use your own editor for this exercise. Runnable challenges on CodeQuest are available for JavaScript, Python, and SQL.

CHECK YOUR UNDERSTANDING

Chapter checkpoint

Why use read -r when reading literal text records?