Master powerful text manipulation commands for log analysis and data processing. Learn sed for search/replace, awk for column extraction, cut/sort/uniq for data filtering, and tr for transformations.

Learning Objectives

Extract columns with cut and awk

Sort and deduplicate data with sort and uniq

Transform text with tr

Search and replace with sed

Process structured log data

Step 1

Set up practice files with structured data

Create sample CSV and log files for text processing.

Commands to Run

cd ~

mkdir text-processing && cd text-processing

echo -e 'name,age,city\nAlice,30,NYC\nBob,25,LA\nAlice,30,NYC\nCharlie,35,Chicago' > data.csv

cat data.csv

echo -e '192.168.1.1 - - [01/Jan/2024:10:00:00] GET /home\n192.168.1.2 - - [01/Jan/2024:10:05:00] POST /api\n192.168.1.1 - - [01/Jan/2024:10:10:00] GET /about' > access.log

cat access.log

What This Does

We're creating CSV and log files with realistic structure. CSV has comma-separated values. Log has Apache-style format. These mirror real DevOps data you'll process.

Expected Outcome

data.csv shows 5 lines (header + 4 records) with duplicates. access.log shows 3 HTTP requests with IPs, timestamps, and endpoints.

Pro Tips

1
CSV = Comma-Separated Values (common data format)
2
\n creates newlines in echo -e
3
Real logs have structured format (IP, timestamp, request)
4
We'll learn to extract, filter, and transform this data

Common Mistakes to Avoid

⚠️Not using -e flag with echo (\n prints literally)
⚠️Forgetting quotes around multi-line strings

Was this step helpful?

202 — Text Processing Tools