In Linux, nearly everything you deal with directly is a file, and most often these will be text files; for instance, all configuration files in Linux are text files. So to reconfigure an application, you simply open the configuration file, change the text, save the file, and then restart the application—your reconfiguration is complete.
With so many text files, manipulating text becomes crucial in managing Linux and Linux applications. In this chapter, you’ll use several commands and techniques for manipulating text in Linux.
For illustrative purposes, I’ll use files from the world’s best network intrusion detection system (NIDS), Snort, which was first developed by Marty Roesch and is now owned by Cisco. NIDSs are commonly used to detect intrusions by hackers, so if you want to be a successful hacker, you must be familiar with the ways NIDSs can deter attacks and the ways you can abuse them to avoid detection.
NOTE
If the version of Kali Linux you’re using doesn’t come preinstalled with Snort, you can download the files from the Kali repository by entering apt-get install snort.
As demonstrated in Chapter 1, the most basic text display command is probably cat, but it has its limitations. Use cat to display the Snort config file (snort.conf) found in/etc/snort (see Listing 2-1).
kali >cat /etc/snort/snort.conf
Listing 2-1: Displaying snort.conf in the terminal window
Your screen should now display the entire snort.conf file, which will stream until it comes to the end of the file, as shown here. This isn’t the most convenient or practical way to view and work with this file.
# include $SO_RULE_PATH/exploit.rules
# include $SO_RULE_PATH/exploit.rules
# include $SO_RULE_PATH/exploit.rules
# include $SO_RULE_PATH/exploit.rules
# include $SO_RULE_PATH/exploit.rules
--snip--
# event thresholding or suppressions commands...
kali >
In the following two sections, I will show you the head and tail commands, which are two methods for displaying just part of a file’s content in order to more easily view the key content.
If you just want to view the beginning of a file, you can use the head command. By default, this command displays the first 10 lines of a file. The following command, for instance, shows you the first 10 lines of snort.conf:
kali >head /etc/snort/snort.conf
#--------------------------------------------------------------
# VRT Rules Packages Snort.conf
#
# For more information visit us at:
--snip--
#Snort bugs:bugs@snort.org
If you want to see more or fewer than the default 10 lines, enter the quantity you want with the dash (-) switch after the call to head and before the filename. For example, if you want to see the first 20 lines of the file, you would enter the command shown at the top of Listing 2-2.
kali >head -20 /etc/snort/snort.conf
#-------------------------------------------------
#VRT Rule Packages Snort.conf
#
#For more information visit us at:
#.
#.
#.
#Options : --enable-gre --enable-mpls --enable-targetbased
--enable-ppm --enable-perfprofiling enable-zlib --enable-act
live-response --enable-normalizer --enable-reload --enable-react
Listing 2-2: Displaying the first 20 lines of snort.conf in the terminal window
You should see only the first 20 lines of snort.conf displayed in your terminal window.
The tail command is similar to the head command, but it’s used to view the last lines of a file. Let’s use it on snort.conf:
kali >tail /etc/snort/snort.conf
#include $SO_RULE_PATH/smtp.rules
#include $SO_RULE_PATH/specific-threats.rules
#include $SO_RULE_PATH/web-activex.rules
#include $SO_RULE_PATH/web-client.rules
#include $SO_RULE_PATH/web-iis.rules
#include $SO_RULE_PATH/web-miscp.rules
#Event thresholding and suppression commands. See threshold.conf
Notice that this command displays some of the last include lines of the rules files, but not all of them, because similar to head, the default for tail is to show 10 lines. You can display more lines by grabbing the last 20 lines of snort.conf. As with the head command, you can tell tail how many lines to display by entering a dash (-) and then the number of lines between the command and the filename, as shown in Listing 2-3.
kali >tail -20 /etc/snort/snort.conf
#include $SO_RULE_PATH/chat.rules
#include $SO_RULE_PATH/chat.rules
#include $SO_RULE_PATH/chat.rules
--snip--
#Event thresholding or suppression commands. See theshold.conf
Listing 2-3: Displaying the last 20 lines of snort.conf in the terminal window
Now we can view nearly all the include lines of the rules files on one screen.
Sometimes—especially with very long files—we may want the file to display line numbers. Since snort.conf has more than 600 lines, line numbers would be useful here. This makes it easier to reference changes and come back to the same place within the file.
To display a file with line numbers, we use the nl (number lines) command. Simply enter the command shown in Listing 2-4.
kali >nl /etc/snort/snort.conf
612 #################################################################
613 #dynamic library rules
614 #include $SO_RULE_PATH/bad-traffic.rules
615 #include $SO_RULE_PATH/chat.rules
--snip--
630 #include $SO_RULE_PATH/web-iis.rules
631 #include $SO_RULE_PATH/web-misc.rules
Listing 2-4: Displaying line numbers in terminal output
Each line now has a number, making referencing much easier.
The command grep is probably the most widely used text manipulation command. It lets you filter the content of a file for display. If, for instance, you want to see all lines that include the word output in your snort.conf file, you could use cat and ask it to display only those lines (see Listing 2-5).
kali >cat /etc/snort/snort.conf | grep output
# 6) Configure output plugins
# Step #6: Configure output plugins
# output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
# output alert_unified2: filename merged.log, limit 128, nostamp
# output log_unified2: filename merged.log, limit 128, nostamp
# output alert_syslog: LOG_AUTH LOG_ALERT
# output log_tcpdump: tcpdump.log
Listing 2-5: Displaying lines with instances of the keyword or phrase specified by grep
This command will first view snort.conf and then use a pipe (|) to send it to grep, which will take the file as input, look for lines with occurrences of the word output, and display only those lines. The grep command is a very powerful and essential command for working in Linux, because it can save you hours of searching for every occurrence of a word or command in a file.
Let’s say you want to display the five lines immediately before a line that says # Step #6: Configure output plugins using at least four of the commands you just learned. How would you do it? (Hint: there are many more options to these commands than those we’ve discussed. You can learn more commands by using the built-in Linux command man. For example, man tail will show the help file for the tail command.)
There are many ways to solve this challenge; here, I show you which lines to change to do it one way, and your job is to find another method.
kali >nl/etc/snort.conf | grep output
34 # 6) Configure output plugins
512 # Step #6: Configure output plugins
518 # output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types
521 # output alert_unified2: filename snort.alert, limit 128, nostamp
522 # output log_unified2: filename snort.log, limit 128, nostamp
525 # output alert_syslog: LOG_AUTH LOG_ALERT
528 # output log_tcpdump: tcpdump.log
We can see that the line # Step #6: Configure output plugins is line 512, and we know we want the five lines preceding line 512 as well as line 512 itself (that is, lines 507 to 512).
kali >tail -n+507 /etc/snort/snort.conf | head -n 6
nested_ip inner, \
whitelist $WHITE_LIST_PATH/white_list.rules, \
blacklist $BLACK_LIST_PATH/black_list.rules
###################################################
# Step #6: Configure output plugins
Here, we use tail to start at line 507 and then output into head, and we return just the top six lines, giving us the five lines preceding the Step #6 line, with that line included.
The sed command lets you search for occurrences of a word or a text pattern and then perform some action on it. The name of the command is a contraction of stream editor, because it follows the same concept as a stream editor. In its most basic form, sed operates like the Find and Replace function in Windows.
Search for the word mysql in the snort.conf file using grep, like so:
kali >cat /etc/snort/snort.conf | grep mysql
include $RULE_PATH/mysql.rules
#include $RULE_PATH/server-mysql.rules
You should see that the grep command found two occurrences of mysql.
Let’s say you want sed to replace every occurrence of mysql with MySQL (remember, Linux is case sensitive) and then save the new file to snort2.conf. You could do this by entering the command shown in Listing 2-6.
kali >sed s/mysql/MySQL/g /etc/snort/snort.conf > snort2.conf
Listing 2-6: Using sed to find and replace keywords or phrases
The s command performs the search: you first give the term you are searching for (mysql) and then the term you want to replace it with (MySQL), separated by a slash (/). The g command tells Linux that you want the replacement performed globally. Then the result is saved to a new file named snort2.conf.
Now, when you use grep with snort2.conf to search for mysql, you’ll see that no instances were found, but when you search for MySQL, you’ll see two occurrences.
kali >cat snort2.conf | grep MySQL
include $RULE_PATH/MySQL.rules
#include $RULE_PATH/server-MySQL.rules
If you wanted to replace only the first occurrence of the term mysql, you would leave out the trailing g command.
kali >sed s/mysql/MySQL/ snort.conf > snort2.conf
You can also use the sed command to find and replace any specific occurrence of a word rather than all occurrences or just the first occurrence. For instance, if you want to replace only the second occurrence of the word mysql, simply place the number of the occurrence (in this case, 2) at the end of the command:
kali >sed s/mysql/MySQL/2 snort.conf > snort2.conf
This command affects only the second occurrence of mysql.
Although cat is a good utility for displaying files and creating small files, it certainly has its limitations when displaying large files. When you use cat with snort.conf, the file scrolls through every page until it comes to the end, which is not very practical if you want to glean any information from it.
For working with larger files, we have two other viewing utilities: more and less.
The more command displays a page of a file at a time and lets you page down through it using the ENTER key. It’s the utility that the man pages use, so let’s look at it first. Open snort.conf with the more command, as shown in Listing 2-7.
kali >more /etc/snort/snort.conf
--snip--
# Snort build options:
# Options: --enable-gre --enable-mpls --enable-targetbased
--enable-ppm --enable-perfprofiling enable-zlib --enable-active
-response --enable-normalizer --enable-reload --enable-react
--enable-flexresp3
#
--More--(2%)
Listing 2-7: Using more to display terminal output one page at a time
Notice that more displays only the first page and then stops, and it tells us in the lower-left corner how much of the file is shown (2 percent in this case). To see additional lines or pages, press ENTER. To exit more, enter q (for quit).
The less command is very similar to more, but with additional functionality—hence, the common Linux aficionado quip, “Less is more.” With less, you can not only scroll through a file at your leisure, but you can also filter it for terms. As in Listing 2-8, open snort.conf with less.
kali >less /etc/snort/snort.conf
--snip--
# Snort build options:
# Options: --enable-gre --enable-mpls --enable-targetbased
--enable-ppm --enable-perfprofiling enable-zlib --enable-active
-response --enable-normalizer --enable-reload --enable-react
/etc/snort/snort.conf
Listing 2-8: Using less to both display terminal output a page at a time and filter results
Notice in the bottom left of the screen that less has highlighted the path to the file. If you press the forward slash (/) key, less will let you search for terms in the file. For instance, when you first set up Snort, you need to determine how and where you want to send your intrusion alert output. To find that section of the configuration file, you could simply search for output, like so:
# Snort build options:
# Options: --enable-gre --enable-mpls --enable-targetbased
--enable-ppm --enable-perfprofiling enable-zlib --enable-active
-response --enable-normalizer --enable-reload --enable-react
/output
This will immediately take you to the first occurrence of output and highlight it. You can then look for the next occurrence of output by typing n (for next).
# Step #6: Configure output plugins
# For more information, see Snort Manual, Configuring Snort - Output Modules
#####################################################################
#unified2
# Recommended for most installs
# output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
output unified2: filename snort.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
# Additional configuration for specific types of installs
# output alert_unified2: filename snort.alert, limit 128, nostamp
# output log_unified2: filename snort.log, limit 128, nostamp
# syslog
# output alert_syslog: LOG_AUTH LOG_ALERT
:
As you can see, less took you to the next occurrence of the word output and highlighted all the search terms. In this case, it went directly to the output section of Snort. How convenient!
Linux has numerous ways of manipulating text, and each way comes with its own strengths and weaknesses. We’ve touched on a few of the most useful methods in this chapter, but I suggest you try each one out and develop your own feel and preferences. For example, I think grep is indispensable, and I use less widely, but you might feel different.