Text processing and pipelines

Give each command one part of the job

A process normally receives input on stdin (stream 0), writes useful output to stdout (1), and writes diagnostics to stderr (2). A pipe, |, sends one command's stdout to the next command's stdin. It does not automatically send stderr along that pipe.

Inside ~/foundations-shell in the Linux machine, create this complete fixture. It uses whitespace-separated fields with no embedded spaces; despite its .tsv name, the simple awk examples accept spaces or tabs.

cat > requests.tsv <<'EOF'
200 /health
200 /accounts
503 /accounts
404 /missing
503 /accounts
EOF
cat requests.tsv
grep -n '^503 ' requests.tsv
awk '{print $1}' requests.tsv | sort | uniq -c
awk '$1 >= 500 {print $2}' requests.tsv | sort | uniq -c

grep selects matching lines; -n adds their input line numbers. ^ means the beginning of a line, so the pattern selects lines beginning with status 503. Expected matching line numbers are 3 and 5.

awk divides each record into fields: $1 is status and $2 path. Its single-quoted program belongs to awk, not Bash; quoting prevents Bash trying to expand $1. The first pipeline prints statuses, sorts them together, then counts adjacent equal lines. Expected counts, ignoring alignment spaces, are:

2 200
1 404
2 503

The second pipeline selects numeric statuses at least 500 and counts their paths. Expect 2 /accounts. These are toy records, not observations from a running service. An arbitrary web log has its own field and escaping rules.

Save output without hiding errors

awk '{print $1}' requests.tsv | sort | uniq -c > status-counts.txt
cat status-counts.txt
grep '^503 ' requests.tsv | tee failures.txt
sed 's|/accounts|/profile|g' requests.tsv

> replaces the output file, >> would append, and tee both prints and writes the data it receives. sed transforms the displayed stream; without an in-place option it leaves requests.tsv unchanged. Confirm that with cat requests.tsv. cut is useful for simple fixed delimiters; jq is a JSON parser. Neither awk field splitting nor a regular-expression replacement is a general JSON or quoted-CSV parser.

To process whole lines in Bash, use:

while IFS= read -r study_line; do
  printf 'record: %s\n' "$study_line"
done < requests.tsv

< supplies the file as input. Empty IFS avoids trimming separators and -r keeps backslashes literal. A shell loop is useful when each line requires separate commands; the awk pipeline is simpler for these counts. Do not replace it with a loop over $(cat file) or parsed ls output: that loses meaningful boundaries.

Check the pipeline, not just the last tool

Bash normally uses the final command's exit status for a pipeline. This controlled example shows the difference without relying on a missing real file:

bash -c 'false | cat; printf "default status: %s\n" "$?"'
bash -c 'set -o pipefail; false | cat; printf "pipefail status: %s\n" "$?"'

Expect statuses 0 and 1. false deliberately fails, while cat succeeds after receiving empty input. pipefail returns the rightmost nonzero status if any component fails; Bash's PIPESTATUS can expose individual statuses if captured immediately. Error messages still matter. Also, grep status 1 means no matching lines, not necessarily a broken input.

Why does uniq -c alone give separate groups for the two 503 records? They are not adjacent until sorted. Why can an empty report be misleading? No matches, an empty file, and a failed read are different outcomes. Keep this fixture for inspection, then remove only requests.tsv, status-counts.txt, and failures.txt when finished.

Sources

Primary references: GNU awk manual↗; Bash pipeline status↗.