BASH / CHAPTER 07
Read records without corrupting them
IntermediateTreat standard input and output as deliberate data streams.
By the end of this chapter
- Use read without trimming or backslash processing
- Handle an unterminated final line
- Keep data and diagnostics on different streams
The idea
Many shell programs connect through standard input, standard output, and standard error. Design a filter so its output contains only its result data; send diagnostics to standard error. Redirection connects these streams to files or commands. Read text using a loop rather than splitting command substitution, because word splitting does not preserve line boundaries or empty lines.
The idiom IFS= read -r line avoids trimming field separators and treating backslashes as escapes. Read returns a failure status at end of input, even when a final unterminated line supplied characters, so include that remaining nonempty value when the input contract allows it. Newlines delimit these records; arbitrary filenames can themselves contain newlines and need a different delimiter, commonly a NUL-aware producer and consumer.
#!/usr/bin/env bash
while IFS= read -r line || [[ -n $line ]]; do
printf '[%s]\n' "$line"
done <<'DATA'
first
back\slash
DATA[ first] [back\slash]
The empty IFS preserves leading spaces, and -r preserves the backslash. The quoted here-document delimiter also prevents expansion inside its content.
Try it yourself
Count nonempty lines from standard input while preserving whitespace and handling a final line without a newline.
Use your own editor for this exercise. Runnable challenges on CodeQuest are available for JavaScript, Python, and SQL.
CHECK YOUR UNDERSTANDING