du - the disk usage linux command

du stands for disk usage. This simple linux command provides a summary of harddisk or storage space disk usage. It has many parameter arguments that can provide results in many screen format. du command can also summarize files and directories in a recursive manner.

Here are several usage of to use the du (disk usage) command.

# cd /home/vertito

To list the files and directories from there

# ls -la
-rw-r--r-- 1 root root 29 2007-08-11 11:57 file.txt
drwxr-xr-x 2 root root 4096 2007-08-11 11:57 folder1

Show summary in bytes

# du -b
4096 ./folder1
8221 .
# du -a
4 ./file.txt
4 ./folder1
12 .

Now, let us get a more human readable results

# du -ah
4.0K ./file.txt
4.0K ./folder1
12K .

The above shows that my file.txt has about 4K of filesize rounded to nearest power of 1024K including . an ..

Now, let us it in bytes
# du -ab
29 ./file.txt
4096 ./folder1
8221 .

The above is the same results you get from issuing ls -la command. 8221 is . and ..

Now let us do it once again in human readable form

# du -abh
29 ./file.txt
4.0K ./folder1
8.1K .

You can also exclude file glob pattern or shell expression for files like so

# du -abh --exclude='file.txt'
# du -abh --exclude='*.txt'
4.0K ./folder1
8.0K .

Recursive directory disk usage summary can also be achieved by doing the default usage without any parameters

# cd /home
# du

You can also limit the recursive search dept like so

# du --max-depth=2

which search on the 2nd level of directory only and ignores any folder found above the 2nd level folders.

Getting the summarized return in a human readable form

# du -sh

Alternatively if you wish to get the last time modification

# du -ah --time
4.0K 2007-08-11 11:57 ./file.txt
4.0K 2007-08-11 11:57 ./folder1
12K 2007-08-11 11:57 .

If you are using mbox type of mail storage handling, these commands can be handy checking and reporting partition and/or folder disk usage when incorporated inside a shell scripts. Furthermore, you can create and generate your TOP 10 users with largest mails on monthly or weekly basis that could give you more detailed email report and alerts from it..

At regular interval and again using a script, you can also watch and monitor folder/partition usage changes and alerts you for certain specified thresholds like for /home or /var/ftp or /tmp.

Display the top ten running processes sorted by memory usage

$ ps aux | sort -nk +4 | tail

This is certainly not the best way to display the top ten processes that consume the most memory, but, hey, it works.

It takes the output of ps aux, sorts it by 4th column numerically and then uses tail to output the last then lines which happen to be the processes with the biggest memory consumption.

If I was to find out who consumes the most memory, I'd simply use htop or top and not ps.

Top for files

$ watch -d -n 1 'df; ls -FlAt /path'

This one-liner watches for file changes in directory /path. It uses the watch command that executes the given command periodically. The -d flag tells watch to display differences between the command calls (so you saw what files get added or removed in /path). The -n 1 flag tells it to execute the command every second.

The command to execute is df; ls -FlAt /path that is actually two commands, executed one after other. First, df outputs the filesystem disk space usage, and then ls -FlAt lists the files in /path. The -F argument to ls tells it to classify files, appending */=>@| to the filenames to indicate whether they are executables *, directories /, sockets =, doors >, symlinks @, or named pipes |. The -l argument lists all files, -A hides . and .., and -t sorts the files by time.

Special note about doors - they are Solaris thing that act like pipes, except they launch the program that is supposed to be the receiving party. A plain pipe would block until the other party opens it, but a door launches the other party itself.

Actually the output is nicer if you specify -h argument to df so it was human readable. You can also join the arguments to watch together, making them -dn1. Here is the final version:

$ watch -dn1 'df -h; ls -FlAt /path'

Copy and paste from the command line

Add the following alias and function to your profile to be able to copy and paste files at the command line:

ccopy(){ cp $1 /tmp/ccopy.$1; }
alias cpaste="ls /tmp/ccopy* | sed 's|[^\.]*.\.||' | xargs -I % mv /tmp/ccopy.% ./%"

You can see below how this can be used:

blackbird:~/tst tks1$ ls
1.txt 2.txt t1.tss t2.tss t3.tss
blackbird:~/tst tks1$ ccopy 1.txt
blackbird:~/tst tks1$ ccopy 2.txt
blackbird:~/tst tks1$ cd ../tst2
blackbird:~/tst2 tks1$ ls
blackbird:~/tst2 tks1$ cpaste
blackbird:~/tst2 tks1$ ls
1.txt 2.txt

Remote network restart

I guess anyone who's administered several remote boxes has had the unfortunate problem of (when not thinking straight) taking down the network card on a machine you have no physical access to. The result being that the ssh session you used to connect dies. The typical mistake is to do something like (as root):

ifconfig eth0 down; ifconfig eth0 inet 123.4.5.6; ifconfig eth0 up

The unfortunate result being that the first statement disconnects your session and hangs up the chain resulting in the network not coming back up. A nice way around this is to use the bash "disown" builtin command, ie:

(sleep 5; ifconfig eth0 inet 123.4.5.6; ifconfig eth0 up)& disown -h $! ; ifconfig eth0 down

In this case you launch a backgrounded task that is disconneced from the session (meaning the ssh session dying won't kill the process) which sleeps for 5 seconds (to give the down a chance to happen) then configures the network card as appropriate and brings it back up. As soon as this launches and is disowned, then immediately takes the network card down. If the configuration change keeps the IP address the same, you'll find that after 5 seconds your bash prompt just comes back and the session resumes.

IPTABLES - blocking IPs simplified

You can block an IP from IPTABLES by using

iptables -I INPUT -s 192.168.0.88 -j DROP

You can simplify things a little bit by using a shell script and a predefined text file containing the IPs we want to be blocked. Either create a new file or add the following at the end of the script which activates the firewall:

#!/bin/sh
for i in $(< bad_hosts.lst) ; do
iptables -I INPUT -i eth1 -s “$i” -j DROP
done

Now create a new file in the same directory and name it bad_hosts.lst and add a new IP to be blocked on every single line, like in the example below:
192.168.2.99
192.168.2.67
86.138.2.7

What RAM type you have in Linux

To check what RAM memory type yo have installed (and also see other useful information about your system), do a

#dmidecode

Depending on the version of dmidecode you have installed and the hardware configuration you have, each hardware device will have a certain type number assigned t it. On my machine, the RAM has type 6. So to see what RAM type and speed you have, do a

#dmidecode --type 6

and the output will be something like

# dmidecode 2.9
SMBIOS 2.3 present.

Handle 0×0008, DMI type 6, 12 bytes
Memory Module Information
Socket Designation: ROW-0
Bank Connections: 1 0
Current Speed: 800
Type: DIMM SDRAM
Installed Size: 256 MB (Double-bank Connection)
Enabled Size: 256 MB (Double-bank Connection)
Error Status: OK

Change MAC address in Linux

If you wish to change your MAC address in Linux, all you have to do is bring the interface down then use the hw ether switch:

#ifconfig eth0 down
#ifconfig eth0 hw ether 02:01:02:03:04:08
#ifconfig eth0 up

but if you want your pc to change its MAC address on boot add that to a script in /etc/init.d/ folder, and also add symbolic link(ln) to /etc/rc2.d, /etc/rc3.d, /etc/rc4.d, /etc/rc5.d which refers to the script in /init.d/

#!/bin/bash

ifconfig eth0 down
ifconfig eth0 hw ether 02:01:02:03:04:08
ifconfig eth0 up
/etc/init.d/networking restart

Linux House vs Microsoft House

This is the second article in my series about GNU/Linux security for the GNU/Linux curious and new GNU/Linux user. The first article is here: http://blog.eracc.com/2009/10/23/2009/10/18/gnulinux-security-ubuntu-has-been-cracked/

There are many attempts to explain the differences between GNU/Linux and Microsoft products when it comes to security. In this article I am going to make yet another attempt. I want to make this as simple as I can for the non-technical users out there. Especially those that are using Microsoft products and cannot conceive of anything that is more secure by default. If you are a technogeek god then ignore the fact that the explanations here are very simple. If you, in your great geekness, want to expound further then feel free to post a comment.

At base the Microsoft products all go back to a core that is built on the MS-DOS concept of a single task, on a single computer for a single user. There is little need to be concerned about security with such a design. This is a fine concept if one never attempts to use such a system for anything other than a single task, on a single computer for a single user. But that is not what Microsoft has done. The Microsoft products simply kept that single user, single computer base technology and added on multi-tasking (Running many programs at one time.) and networking (Connecting many computers together for sharing data, printers and so on.) Later multi-user capability (More than one user on a computer at the same time.) was added on top of this single user, single tasking core. Granted the multi-user capability is not really present in Microsoft desktop products, so we can ignore the fact that one may create multiple user accounts on a modern Microsoft based desktop system. I will call the Microsoft model a one-one-one model.


The problem with adding on these multi-tasking, networking and multi-user capabilities to the Microsoft one-one-one products is that there appears to originally have been no concern for securing these systems. The security concern only began once people began to see systems being cracked and exploited “in the wild”. However, there was a serious problem with securing these systems. To correctly raise the security bar for Microsoft systems “out of the box” the core of the operating system should have been redesigned from scratch. The backwards compatibility that has its roots in that single task, single user, single computer model would have to go away at some point. Apparently the high and mighty Muckity Mucks at Microsoft made an executive decision to not do that, ever. So, today we have Microsoft Windows 7 released and containing roots going back to that insecure one-one-one operating system design.

How is GNU/Linux different? A GNU/Linux desktop system is designed from the ground up along the Unix model of multiple tasks with multiple users among multiple computers on a network. I will call this a many-many-many design. As such the basic design also includes consideration for securing the operating system and data on same when many users may have access to the same system simultaneously. Therefore, when a GNU/Linux computer is taken out of the box for the first time it already has a higher security capability. This is because of the many-many-many design that included consideration for security from the beginning.

How does this apply in a real world scenario? Okay, because of the original flawed design decisions by Microsoft many third party software packages require that a user be running as a system administrator with full access rights to the computer, including to system files. So, by default when one pulls out a new computer with a Microsoft system installed the users are created as “administrator” users. This is a problem because now this administrator user can browse to an infected web page and see a pop-up with an “anti-virus” warning. Then our poor user will click the close button on the pop-up and become infested with “Antivirus 2010″ or other fake anti-virus program that at minimum is irritating but may also have broader security implications by then installing other malware (Malicious Software) that can steal personal information. Because the user is an administrator with full access to the operating system’s files the malware that starts from the web page also has full administrator access and can install itself with impunity.

How can I blame Microsoft for these third party software packages and/or users being set up as administrators? Why not blame these third party software designers? Well, I do blame poorly written software that requires administrator access to work correctly. But I also blame Microsoft. Because Microsoft made the poor decision to stay with their one-one-one design and just “improve” it. At first the only way for any software to work correctly with these “improvements” was to have administrator access. Over the years this has changed, but rewriting all software to these new, more secure specifications is a slow and expensive process for the software companies involved. Microsoft should have scrapped that one-one-one model and redesigned the core operating system from scratch. That redesign should have looked something like Unix … or like GNU/Linux.

The GNU/Linux many-many-many system on the other hand works just fine when a plain user who is not an administrator uses programs on it. So, no software run by the user can affect system files. Further, no software on GNU/Linux is designed to automatically allow software to run from a web browser or e-mail application without the user’s knowledge. No open source developers I know are silly enough to think having such “capabilities” is a good idea. So, when our dear user browses to an infected web site and sees a pop-up about an anti-virus infection she can safely close that pop-up without worrying that an infection will occur in the background that will take over her computer. It is very unlikely that a web based malware script written with GNU/Linux as the target could find a way to even infect the user’s home directory. Why? Well, software that is downloaded from a browser instance is not set as executable. So, even if a browser could be made to download a file without the user knowing it the user would have to make changes to the file permissions to make it executable. There are no .EXE, .COM, .BAT or other files on GNU/Linux that can be run just because of their file extension. A file has to be a compiled application or a script and be set as executable before it will run. This automatically makes it much more difficult to infect a GNU/Linux system behind the user’s back. The effort required is much greater than with Microsoft based systems where the file extension makes the application or script able to be run.

I created a script and uploaded it to my web site to demonstrate this. Here is what a “ls -l” file listing of that script looks like when first downloaded:

-rw-r–r– 1 gene users 73 2009-10-23 22:28 a_script_for_you

See that “-rw-r–r–”? That means the owner of the file, the “gene” shown after the “1″, can read it and write to it but not execute it, “rw-”. The group, the “users” shown following “gene’, and everyone else, not shown but implied, can read but not write and not execute the script, “r–r–”. The dashes are placeholders for the bits that allow writing, “w”, and executing, “x”, of files. Now I will change the permissions on the script by hand and run it:

[gene@era4 ~]$ chmod 700 a_script_for_you
[gene@era4 ~]$ ./a_script_for_you
I can only run if you use the command ‘chmod 700 ./a_script_for_you’ or similar!

See? I had to explicitly intervene to make that script run. I would have to do the same if I downloaded a program from a web site. Browsers on GNU/Linux have no ability to change the script to be executable on my system without my knowledge. I have to be involved in the process, so I have to be convinced that making this program or script executable is a good idea. If this script comes from the “Joe’s Bar and Grill” web site and purports to be an upgrade for Firefox I am going to be very suspicious about making it where it will run on my computer. So should you. Social engineering attacks, where the bad guys convince a user to do something stupid, can still occur with GNU/Linux. So beware and be informed about those. But automated attacks that get system level malware installed through the browser or through e-mail are quite impossible on GNU/Linux.

This brings me to my illustration of the Linux House versus the Microsoft House. The Linux House is built with bullet-proof windows that are closed and locked. There are thick steel bar grills over all the windows. The Linux House has thick concrete walls, roof and floors. The Linux House has thick solid steel, bunker doors that bolt at both sides, the top and the bottom. Any thief that wants to get in and steal your family heirlooms is going to have to have some serious means of breaking and entering, like a bazooka or a tank. Yet all the security of the Linux House is behind beautiful and functional facades and the typical resident can be blissfully unaware of it most of the time. On the other hand the Microsoft House is pretty much like your house you live in now. It is quite adequate for day to day living but it is no serious impediment to a thief that wants to get in and steal your jewelry. It has plain old Windows. The thief can pretty much just break those Windows and climb in at will. You see, plain old Windows are no real way to stop a thief.

Can Microsoft operating systems be secured? Yes, they can, up to a point. But the starting point to secure Microsoft operating systems is far lower than the starting point for GNU/Linux systems. However, the flawed original design of Microsoft operating systems that underlie all modern versions of Microsoft operating systems keeps them more amenable to attack even when as locked down as possible. Of course, in reality, the only truly secure computer is one that is never used, by anyone. But then again, no one is going to spend money on a computer that cannot be used.

Any of you serious security types that want to share more information about GNU/Linux and its security by design model or have better illustrations than mine, please leave a comment.

source: http://blog.eracc.com/2009/10/23/gnulinux-security-linux-house-vs-microsoft-house/

pop3s server using stunnel

it’s not a secret that POP3 (Post Office Protocol) is unsecure protocol. All data, including passwords (!) between POP3 server and PC is going in plain text. POP3, like some other protocols (telnet, ftp, etc) absolutely not resistant to sniffing attacks. Even if you sure that your network or PC is secure you can’t trust your ISP network, that potentially can be compromised. Below you can find how-to secure your POP3 server in few easy steps. This how-to is actual for people who can’t or don’t want to change their POP3 server software, but want be secure.

First, download and install the latest stunnel version. I have use freebsd port of stunnel, but you can also get it on stunnel.org. Next you need generate SSL certificate:

cd /etc/ssl
openssl req -new -x509 -days 9999 \
-nodes -config openssl.cnf -out mail.pem \
-keyout mail.pem
…

Country Name (2 letter code) [AU]:ISO_Country_Code_Here
State or Province Name (full name) [Some-State]:State
Locality Name (eg, city) []:City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Org_Name
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:your_pop3_server_fqdn (important)
Email Address []:Some_Email

I prefer to use official SSL certificates, because with self-signed certificate e-mail clients always show warning message (that little annoying)

Now we need create config for stunnel. Here is my working config:

cert = /etc/ssl/mail.pem
sslVersion = all
exec = /usr/local/libexec/popa3d
output = /var/log/stunnel.log
ciphers = HIGH
debug = 6

If you have a lot of POP3 accounts it’s better to set sslVersion value to “all” because some old POP3 clients may not understand SSLv3/TLSv1

As last step you need shutdown your pop3 client and start new, secure version. In my case i need to remove pop3 from inetd.conf and add new, pop3s service:

# grep pop3s /etc/inetd.conf

pop3s stream tcp nowait root /usr/local/sbin/stunnel stunnel /etc/mail/stunnel.conf

After that you just need to restart inetd.
From client side in mail client settings must be choosen SSL or TLS connection and port 995 instead of 110. Secure Authentication must be unset, because we don’t really have POP3S server, there is only secure tunnel between PC and POP3 server.

Stay secure!

Convert wma to ogg

Write small shell script to perform batch conversion. However it works only with files from current directory, and with files what has only one period in file name (for example ASOT-280.wma)

Following script will convert all wma files to ogg format:

#!/bin/bash
for i in *.wma ;
do
ii=`echo $i | cut -d'.' -f1`
`mplayer -vo null -vc dummy -af resample=44100 \
-ao pcm:waveheader:file=$ii.wav $i`
`oggenc -q 5 $ii.wav`
done

(!) Do not forget to manually remove original wma and temporarily created wav files.

I will not explain here all mplayer and oggenc params. If you would like to know more, please look to man pages

If you would like to convert wma to mp3 you need replace line:

`oggenc -q 5 $ii.wav

with

`lame -m j -h --vbr-new -b 192 $ii.wav -o $ii.mp3`

That’s all.

Monitoring Processes with Kill

If you have a process ID but aren’t sure whether it’s valid, you can use the most unlikely of candidates to test it: the kill command. If you don’t see any reference to this on the kill(1) man page, check the info pages. The man/info page states that signal 0 is special and that the exit code from kill tells whether a signal could be sent to the specified process (or processes).

So kill -0 will not terminate the process, and the return status can be used to determine whether a process is running. For example:

$ echo $$ # show our process id
12833
$ /bin/bash # create new process
$ echo $$ # show new process id
12902
$ kill -0 12902
$ echo $? # exists, exit code is 0
0
$ exit # return to previous shell
$ kill -0 12902
bash: kill: (12902) - No such process
$ echo $? # doesn’t exist, exit code is 1
1

Many UNIX dæmons store their process IDs in a file in /var/run when they are started. Using kill -0 to test the pid is a lot easier than parsing ps output. For example, to test whether cron is running, do the following:

# kill -0 $(cat /var/run/cron.pid)
# echo $?
0

Validating an IP Address in a Bash Script

Here’s a more useful example of using them to test IP addresses for validity.

To belabor the obvious: IP addresses are 32 bit values written as four numbers (the individual bytes of the IP address) separated by dots (periods). Each of the four numbers has a valid range of 0 to 255.

The following bash script contains a bash function which returns true if it is passed a valid IP address and false otherwise. In bash speak true means it exits with a zero status, anything else is false. The status of a command/function is stored in the bash variable “$?”.

#!/bin/bash

# Test an IP address for validity:
# Usage:
# valid_ip IP_ADDRESS
# if [[ $? -eq 0 ]]; then echo good; else echo bad; fi
# OR
# if valid_ip IP_ADDRESS; then echo good; else echo bad; fi
#
function valid_ip()
{
local ip=$1
local stat=1

if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
OIFS=$IFS
IFS=’.’
ip=($ip)
IFS=$OIFS
[[ ${ip[0]} -le 255 && ${ip[1]} -le 255 \
&& ${ip[2]} -le 255 && ${ip[3]} -le 255 ]]
stat=$?
fi
return $stat
}

# If run directly, execute some tests.
if [[ "$(basename $0 .sh)" == 'valid_ip' ]]; then
ips=’
4.2.2.2
a.b.c.d
192.168.1.1
0.0.0.0
255.255.255.255
255.255.255.256
192.168.0.1
192.168.0
1234.123.123.123
‘
for ip in $ips
do
if valid_ip $ip; then stat=’good’; else stat=’bad’; fi
printf “%-20s: %s\n” “$ip” “$stat”
done
fi

If you save this script as “valid_ip.sh” and then run it directly it will run some tests and prints the results:

# sh valid_ip.sh
4.2.2.2 : good
a.b.c.d : bad
192.168.1.1 : good
0.0.0.0 : good
255.255.255.255 : good
255.255.255.256 : bad
192.168.0.1 : good
192.168.0 : bad
1234.123.123.123 : bad

In the function valid_ip, the if statement uses a regular expression to make sure the subject IP address consists of four dot separated numbers:

if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then

If that test passes then the code inside the if statement separates the subject IP address into four parts at the dots and places the parts in an array:

OIFS=$IFS
IFS=’.’
ip=($ip)
IFS=$OIFS

It does this by momentarily changing bash’s Internal Field Separator variable so that rather than parsing words as whitespace separated items, bash parses them as dot separated. Putting the value of the subject IP address inside parenthesis and assigning it to itself thereby turns it into an array where each dot separated number is assigned to an array slot. Now the individual pieces are tested to make sure they’re all less than or equal to 255 and the status of the test is saved so that it can be returned to the caller:

[[ ${ip[0]} -le 255 && ${ip[1]} -le 255 \
&& ${ip[2]} -le 255 && ${ip[3]} -le 255 ]]
stat=$?

Note that there’s no need to test that the numbers are greater than or equal to zero because the regular expression test has already eliminated any thing that doesn’t consist of only dots and digits.