Linux Text Processors

During pentests, you will often find yourself using a Linux machine without access to the Graphical User Interface (GUI). As you can imagine, this can make editing/looking through text-based files a PITA if you're looking through large files.

To help with this, here are a few popular text-processing tools made for the command line that can help you manipulate text-based files:

Grep

Grep allows you to search for a string using either plaintext or regular expressions (regex).

grep <options> [pattern] [file]

OPTIONS
-i                   Ignore Case Distinctions
-v                   Invert Match
-a                   Assume Binary Files Are Text
-G                   Pattern Is A Regular Expression

Here is an example where I'm searching for the string 'hyperbeam' in the file weelee.txt.

grep -i hyperbeam weelee.txt

Cut

Cut is handy for parsing and manipulating structured text data, such as delimited files (e.g., CSV files) or fixed-width text files.

cut <options> [file]

OPTIONS
-b number            Prints The Specified Number Of Characters From Each Line
-c number            Prints The Specified Characters By Index From Each Line
-d                   Delimiter
-f                   Select Only These Fields

Here is an example where I'm writing just the first column of pokedex.csv to a new file:

bulbasaur,grass,1
charmander,fire,4
squirtle,water,7

cut -d ',' -f 1 pokedex.txt | tee pokemon.txt

cat pokemon.txt
bulbasaur
charmander
squirtle

Sed

Sed is excellent for substituting text in a file but can be used for various other things. The syntax for Sed is based on regular expressions, which allows for complex text manipulation.

sed <options> [regex script] [file]

OPTIONS
-e script            Scripts To Be Executed
-f scriptfile        Script File To Be Executed

Here are a few examples of Sed commands you may want to use:

sed 's/hi/hello/g'           Replace All Instances Of 'hi' With 'hello'
sed 's/.$//'                 Remove Last Character From Each Line
sed 's/$/:80/'               Add ':80' To The End Of Each Line
sed 's/^/domain\\\\/'        Add 'domain\' To The Start Of Each Line
sed 's/./\u&/'               Uppercase The First Character Of Each Line

Awk

Awk is another text manipulation tool that can filter specific lines and strings in a text file.

awk <options> [program] [file]

OPTIONS
-f program file      Program Text Is Read From File
-F value             Field Separator

Here are a few examples of Awk commands you may want to use:

awk '{print$1, $NF}'               Prints The First And Last String From Each Line
awk -F',' '{ print $1, $3 }'       Print Col 1 And 2 Of A Comma-Separated CSV File
awk '{t=$1;$1=$2;$2=t;print;}'     Swap Col 1 And 2 Of A Space-Separated File

Tr

Tr is often used to perform character-level manipulation of characters in a file.

tr <options> [set1] <set2>

OPTIONS
-c                      Use The Complement Of Set1
-d                      Delete Characters In Set1
-s                      Replace Multiple Occurrences Of The Last Letter With Just One

Here are a few examples of Tr commands you may want to use:

cat file.txt | tr 'a' 'b'            Replace All Occurences Of 'a' with 'b'
cat file.txt | tr -d ':'             Delete All Occurrences of ':'
cat file.txt | tr -s ' '             Compress Consecutive Spaces To A Single Space
cat file.txt | tr 'A-Za-z' 'a-zA-Z'  Convert Upper Characters To Lower And Vice Versa