How to use the most popular command in Unix - Grep

grep is by far the most popular command that exists in Unix. Though some may argue about that, but once you begin using grep, it would always be present in all your complex commands that you think of executing at the shell prompt. grep stands for 'global regular expression printer' . Which makes no sense to most.. In sensible words grep extracts those lines from a given text that match the conditions set by the user.

Basically grep lets you enter a pattern of text and then it searches for this pattern within then text that you provide it. It would return all the lines that have the given pattern in them. grep can be used in 2 ways - Either on its own or along with pipes

Using grep on its own

$ grep '12.00' /home/david/backup/log.txt

This command basically shows how you can use grep to extract lines containing a particular string from a text file. (Text files need not necessarily end in .txt) The above command searches for the string 12.00 in the text file specified in the command, and displays all the lines that have this string in them.
The above command could be used to find out all the backups that took place at 12.00 (In case you have a log.txt file in that directory with a list of all the timings for the backups that you have made).

$ grep -v '12.00' /home/david/backup/log.txt

The above command would now show you all the lines in the text file except those that have the string 12.00 in them.

$ grep -l 'delay' /code/*.c

The above command searches for those files that end with a '.c' (within the /code directory) and in which the text 'delay' is present. It only returns the names of these files and not the lines where it found the string.

$ grep -w '\' *

The above commands search for text in a more refined way. The first command searches for those lines where any word in that line begins with the letters 'bay' and the second command searches for those lines where any word in that line ends with the letter 'watch'
-

Using grep with pipes

$ ls -l | grep rwxrwxrwx

As you must be knowing ls -l displays the directory listing for any directory. The grep rwxrwxrwx part of the command extracts only those lines which display the files having their read,write,execute permissions set for user, group and others also. Thus instead of getting a listing of all the files in the directory, you would only see those files that have their r,w,x permissions set for all everybody.

The output of grep can also be piped to another program as follows

$ du | grep 'mp3' | more

You should be able to figure out what the above command does..

$ grep '^#' /home/david/script1 | more

The above command would display those lines (from the file /home/david/script1) that begin with a '#'. The term '^#' means that # should be present as the first character on a line. The more part of the command should be known to you. If not, more basically displays the output a page at a time incase the output exceeds one page.

$ grep -v '^[0-9]' /home/david/backup/log.txt | more

This command searches for lines having any of the numbers from 0-9 in them as the first character on the line. It then prints all the lines except the ones it found initially.

Important : It's necessary to enclose patterns (as used in the above 2 commands) in single quotes so that the shell understands it correctly.Otherwise, the shell may interpret it in another method.

Some extra options for grep

-v
Reverses the normal behaviour of the grep command - Instead of selecting lines, it rejects the lines that match the given criteria.

-c
It supresses the normal output and only prints the total count of matching lines instead of the actual lines.

-i
Ignores the case of the text when matching the given pattern.

-w
Checks if the given pattern is a word by itself and not a part of another word. Thus if you search for 'bay' and the word 'baywatch' is present in a file, the particular line conatining that word would not be returned in the result.

-l
Only gives the names of the files in which the given pattern was found.

-r
Checks for the given pattern , recursively within the directory that you specify after the -r option

I hope this tutorial helps you get started with grep. grep is defintely one of the tools that gives Linux the advantage over other Operating Systems. Using grep effectively along with other tools gives the user a lot of power in Unix.

Grabbing a screenshot without X

There are plenty of screen-capture tools, but a lot of them are based on X. This leads to a problem when running an X application would interfere with the application you wanted to grab - perhaps a game or even a Linux installer. If you use the venerable ImageMagick import command though, you can grab from an X session via the console. Simply go to a virtual terminal (Ctrl+Alt+F1 for example) and enter the following:

chvt 7; sleep 2; import -display :0.0 -window root sshot1.png; chvt 1;

The chvt command changes the virtual terminal, and the sleep command gives it a while to redraw the screen. The import command then captures the whole display and saves it to a file before the final chvt command sticks you back in the virtual terminal again. Make sure you type the whole command on one line.

This can even work on Linux installers, many of which leave a console running in the background - just load up a floppy/CD with import and the few libraries it requires for a first-rate run-anywhere screen grabber.

du - the disk usage linux command

du stands for disk usage. This simple linux command provides a summary of harddisk or storage space disk usage. It has many parameter arguments that can provide results in many screen format. du command can also summarize files and directories in a recursive manner.

Here are several usage of to use the du (disk usage) command.

# cd /home/vertito

To list the files and directories from there

# ls -la
-rw-r--r-- 1 root root 29 2007-08-11 11:57 file.txt
drwxr-xr-x 2 root root 4096 2007-08-11 11:57 folder1

Show summary in bytes

# du -b
4096 ./folder1
8221 .
# du -a
4 ./file.txt
4 ./folder1
12 .

Now, let us get a more human readable results

# du -ah
4.0K ./file.txt
4.0K ./folder1
12K .

The above shows that my file.txt has about 4K of filesize rounded to nearest power of 1024K including . an ..

Now, let us it in bytes
# du -ab
29 ./file.txt
4096 ./folder1
8221 .

The above is the same results you get from issuing ls -la command. 8221 is . and ..

Now let us do it once again in human readable form

# du -abh
29 ./file.txt
4.0K ./folder1
8.1K .

You can also exclude file glob pattern or shell expression for files like so

# du -abh --exclude='file.txt'
# du -abh --exclude='*.txt'
4.0K ./folder1
8.0K .

Recursive directory disk usage summary can also be achieved by doing the default usage without any parameters

# cd /home
# du

You can also limit the recursive search dept like so

# du --max-depth=2

which search on the 2nd level of directory only and ignores any folder found above the 2nd level folders.

Getting the summarized return in a human readable form

# du -sh

Alternatively if you wish to get the last time modification

# du -ah --time
4.0K 2007-08-11 11:57 ./file.txt
4.0K 2007-08-11 11:57 ./folder1
12K 2007-08-11 11:57 .

If you are using mbox type of mail storage handling, these commands can be handy checking and reporting partition and/or folder disk usage when incorporated inside a shell scripts. Furthermore, you can create and generate your TOP 10 users with largest mails on monthly or weekly basis that could give you more detailed email report and alerts from it..

At regular interval and again using a script, you can also watch and monitor folder/partition usage changes and alerts you for certain specified thresholds like for /home or /var/ftp or /tmp.

Copy and paste from the command line

Add the following alias and function to your profile to be able to copy and paste files at the command line:

ccopy(){ cp $1 /tmp/ccopy.$1; }
alias cpaste="ls /tmp/ccopy* | sed 's|[^\.]*.\.||' | xargs -I % mv /tmp/ccopy.% ./%"

You can see below how this can be used:

blackbird:~/tst tks1$ ls
1.txt 2.txt t1.tss t2.tss t3.tss
blackbird:~/tst tks1$ ccopy 1.txt
blackbird:~/tst tks1$ ccopy 2.txt
blackbird:~/tst tks1$ cd ../tst2
blackbird:~/tst2 tks1$ ls
blackbird:~/tst2 tks1$ cpaste
blackbird:~/tst2 tks1$ ls
1.txt 2.txt

Revert files with changed mode

Sometimes (due to mismanagement!) end up with files in a git repo which have had their modes changed, but not their content. This one-liner lets me revert the mode changes, while leaving changed-content files be, so I can commit just the actual changes made.

git diff --numstat | awk '{if ($1 == "0" && $1 == "0") print $3}'  | xargs git checkout HEAD

network copy with ssh and tar

You can use ssh in conjunction with tar to pull an entire directory tree from a remote machine into your current directory:

ssh tar cf - -C . | tar xvf - 

For example, let's say you have a "bsmith" account on a host called "apple". You want to copy those files into your "bobsmith" account on a host called "pear". You'd log into your "bobsmith@pear" account and type the following:

ssh bsmith@apple tar cf - -C /home/bsmith . | tar xvf - 

This technique is useful when you have insufficient disk space on the source machine to make an intermediate tarball

md5sum: Remove duplicate files

The script below will find duplicate files (files with the same md5sum) in a specified directory and output a new shell script containing commented-out rm statements for deleting them. You can then edit this output to decide which to keep.

OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
find "$@" -type f -print0 |
xargs -0 -n1 md5sum |
sort --key=1,32 | uniq -w 32 -d --all-repeated=separate |
sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF; ls -l $OUTF

Remote network restart

I guess anyone who's administered several remote boxes has had the unfortunate problem of (when not thinking straight) taking down the network card on a machine you have no physical access to. The result being that the ssh session you used to connect dies. The typical mistake is to do something like (as root):

ifconfig eth0 down; ifconfig eth0 inet 123.4.5.6; ifconfig eth0 up

The unfortunate result being that the first statement disconnects your session and hangs up the chain resulting in the network not coming back up. A nice way around this is to use the bash "disown" builtin command, ie:

(sleep 5; ifconfig eth0 inet 123.4.5.6; ifconfig eth0 up)& disown -h $! ; ifconfig eth0 down

In this case you launch a backgrounded task that is disconneced from the session (meaning the ssh session dying won't kill the process) which sleeps for 5 seconds (to give the down a chance to happen) then configures the network card as appropriate and brings it back up. As soon as this launches and is disowned, then immediately takes the network card down. If the configuration change keeps the IP address the same, you'll find that after 5 seconds your bash prompt just comes back and the session resumes.

Debug errors in configure procedures

There are a lot of parameters to adjust the installation. Unfortunately the configure doesn't end with success. A dubios error message appears and I don't find out what's wrong.
Also the modern oracle 'google' doesen't give a hint. But after hours I find a quick and easy solution:

sh -x ./configure ... configure_options ...

First there is a long rolling of letters, but afterwards, when the error appears, you can easy find out what the mistake is, because the debug modus tells you.

IPTABLES - blocking IPs simplified

You can block an IP from IPTABLES by using

iptables -I INPUT -s 192.168.0.88 -j DROP

You can simplify things a little bit by using a shell script and a predefined text file containing the IPs we want to be blocked. Either create a new file or add the following at the end of the script which activates the firewall:

#!/bin/sh
for i in $(< bad_hosts.lst) ; do
iptables -I INPUT -i eth1 -s “$i” -j DROP
done

Now create a new file in the same directory and name it bad_hosts.lst and add a new IP to be blocked on every single line, like in the example below:
192.168.2.99
192.168.2.67
86.138.2.7

Check laptop battery in Command Line

If you run Linux off a laptop, you might want to check your battery status alt least once a year:

grep -F capacity: /proc/acpi/battery/BAT0/info

This will output something like

design capacity: 7800 mAh
last full capacity: 6414 mAh

Set the clock from command line

If you want to set the date from the Linux command line, you can use thedate command:

date nnddhhmmyy

where
nn is the month
dd is the day
hh is the hour
mm is the minute
yy is the year
For example, date 080717292009 would set the date to August the 7th, 17:29, 2009

What RAM type you have in Linux

To check what RAM memory type yo have installed (and also see other useful information about your system), do a

#dmidecode

Depending on the version of dmidecode you have installed and the hardware configuration you have, each hardware device will have a certain type number assigned t it. On my machine, the RAM has type 6. So to see what RAM type and speed you have, do a

#dmidecode --type 6

and the output will be something like

# dmidecode 2.9
SMBIOS 2.3 present.

Handle 0×0008, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: ROW-0
Bank Connections: 1 0
Current Speed: 800
Type: DIMM SDRAM
Installed Size: 256 MB (Double-bank Connection)
Enabled Size: 256 MB (Double-bank Connection)
Error Status: OK

Find details ISO images from the Linux command line

If you ever wondered what that mysterious ISO file you downloaded two years ago is, and don’t want to burn the image, you can view details of the volume with a simple command:

isoinfo -d -i filename.iso

You’ll be given details like the volume size, if the image file is a bootable one and other valuable information.

Update Twitter status from the Linux command line

You can easily update your Twitter status from the CLI by using this one simple command:

curl -u user:password -d status=”Your status message” http://twitter.com/statuses/update.xml

where user is your username and password is your Twitter password entered in plaintext. Replace the text Your status message with anything you wish

Change MAC address in Linux

If you wish to change your MAC address in Linux, all you have to do is bring the interface down then use the hw ether switch:

#ifconfig eth0 down
#ifconfig eth0 hw ether 02:01:02:03:04:08
#ifconfig eth0 up

but if you want your pc to change its MAC address on boot add that to a script in /etc/init.d/ folder, and also add symbolic link(ln) to /etc/rc2.d, /etc/rc3.d, /etc/rc4.d, /etc/rc5.d which refers to the script in /init.d/

#!/bin/bash

ifconfig eth0 down
ifconfig eth0 hw ether 02:01:02:03:04:08
ifconfig eth0 up
/etc/init.d/networking restart

Backing up the bootsector

Afraid you might mess up your MBR? Back up the bootsector:

dd if=/dev/hda of=bootsector.img bs=512 count=1

If anything goes wrong, you can boot from a LiveCD and restore the bootsector with

dd if=bootsector.img of=/dev/hda

Using a swapfile to increase the swap space on your system

This can be a very handy function if we want to use a file, instead of a partition, and with modern filesystems the performance is almost on par with using a dedicated partition for your swap area.

1. Using dd lets make a zero’d file for the swap

dd if=/dev/zero of=/swapfile bs=1048576 count=1000

This example would create a swapfile of 1 gig using a blocksize of a megabyte (1048576).

2. Make file as a swapfile

mkswp /swapfile

3. Activate swapfile

swapon /swapfile

4. Verify that our swapfile has been activated

swapon -s

We should see something like this in the list ..

Filename Type Size Used Priority
/swapfile file 9999992 0 -2

If you want to have this a permanent solution, then adding the entry to fstab would probally be a better idea,
open up /etc/fstab in your favourite editor and add the following line.

/swapfile swap swap defaults 0 0

set your default gateway using route

Want to quickly set the route for your machine from the terminal.

Start by opening up a terminal window, and type the following, just make sure you have the ip address of your gateway to do so. In this case the gateway of 10.3.156.1 is being used.

route add default gw 10.3.156.1 eth0

We can see our changes by typing

route -n

man | System Help

man (short for “manual”) is a traditional form of online documentation in Unix and Linux operating systems. Specially formatted files, “man pages”, are written for most commands and distributed with the software. Running man somecommand will display the man page for (naturally) the command or program somecommand.

Because there are so many of them, man pages are grouped into enumerated sections. This system has been around so long that you will often see commands, programs, and even programming library functions referred to with their man section number. For instance, you might see man(1). This tells you that man is documented in section 1 (user commands); you can specify that you want the section 1 man page for “man” with the command man 1 man. Specifying the section that man should look in is useful in the case of multiple items with the same name.

Table 2-1. Man Page Sections

Section	Contents
Section 1	user commands (intro only)
Section 2	system calls
Section 3	C library calls
Section 4	devices (e.g., hd, sd)
Section 5	file formats and protocols (e.g., wtmp, /etc/passwd, nfs)
Section 6	games (intro only)
Section 7	conventions, macro packages, etc. (e.g., nroff, ascii)
Section 8	system administration (intro only)

In addition to man(1), there are the commands whatis(1) and apropos(1), whose shared purpose is to make it easier to find information in the man system. whatis gives a very brief description of system commands, somewhat in the style of a pocket command reference. apropos is used to search for a man page containing a given keyword.

See their man pages for details.

The /usr/doc Directory

The source for most packages that we build comes with some sort of documentation. README files, usage instructions, license files… any sort of documentation that comes with the source is included and installed on your system in the /usr/doc directory.

If man pages don’t provide enough information, /usr/doc should be your next stop.

HOWTOs and mini-HOWTOs

It is the true spirit of community that brings you the HOWTO/mini-HOWTO collection. These files are exactly what they sound like– documents describing how to do stuff. If you install the HOWTO collection package, HOWTOs will be installed to /usr/doc/Linux-HOWTOs and the mini-HOWTOs to /usr/doc/Linux-mini-HOWTOs.

Also included in the same package is a collection of FAQs (Frequently Asked Questions lists– with answers) which are installed to the same place.

These files are well worth reading whenever you’re not quite sure how to proceed with something. An amazing range of topics are covered in sometimes surprising detail.