Various Tools

This section lists some of the most commonly used Linux tools and other concepts. There is much more to learn, and the more of these little building blocks are at your disposal the more productive you will be.

df

The df command (disk free) shows the available space on all devices:

% df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 61169696 56601044 1461328 98% /
udev 1986456 4 1986452 1% /dev
tmpfs 798580 864 797716 1% /run
none 5120 0 5120 0% /run/lock
none 1996444 24 1996420 1% /run/shm
/dev/sdb1 4874192 1246720 3627472 26% /media/mystick

The first column shows the device files used by the system to access the hardware. The last column shows the corresponding mount points. The first partition on the fixed hard disk /dev/sda1 is mounted as / meaning the root of the file system.

Removable devices such as USB sticks are mounted to directories automatically. Here a removable drive is mounted via device file /dev/sdb1 as /media/mystick. This directory is accessible as long as the stick is plugged in.

Remember to unmount it (usually via the file manager) before unplugging the device, otherwise data written to the stick might be lost.

Note that in some distributions the mount directory is /media/username/mystick

As always, the hard disk is almost completely full. Sooner or later this will cause problems. There should be some percentage reserved for root which is not contained in the Use% above. However, if for some reason the Linux system runs out of space completely, strange situations can occur, such as users not being able to log in. Check with the following command:

% sudo tune2fs -l /dev/sda1
...
Last mounted on: /
...
Filesystem state: clean
...
Block count: ..
Reserved block count: ..

The filesystem is clean (if not, we are in trouble! boot from rescue/install disk/usb stick and fsck). Reserved block count divided by Block count should be about 5%, the default reserved for root.

Sometimes it is not entirely clear which types of filesystems are actually in use on the various mount points. The following command lists the filesystem for each partition, including encrypted ones:

sudo blkid

top

The top command provides an overview of system activity and various data on the machine, such as

Press 1 to get a listing for each core. Exit the listing by pressing q.

Most processes are in the sleep mode (S) most of the time, consuming very little resources. R means running, D means dead, and sometimes Z for zombie can be seen.

grep

The grep command searches text files for lines containing the given string and prints them out:

grep h2 *.html

This will print all headings of level 2 in all HTML files in the current directory (if markup and text is on the same line).

wc

grep printf *.java | wc -l

This command counts the number of usages of the printf statement in all Java programs.

sort

grep -l printf *.java | sort -u

This command prints the file names rather than the matching line, and the sort command with the unique option drops doubles. The result is a list of the files that contain at least one printf statement.

du

du -s tmp/* | sort -n

Disk usage for all files and directories in tmp/ piped to sort numerically. Note that disk usage and file size are usually not the same.

find

Directory hierarchies can develop into huge structures that are difficult to search interactively. The find command searches all directories starting at the given point(s) for files matching some condition:

find . -name '*.zip' -size +1000k

Here we look for ZIP files bigger than 1 MB.

find . -name '*.java' -exec grep -l printf {} \; | sort -u

In this case we perform the grep from the previous section on all Java files in the current directory, or any subdirectory. Note the {} syntax to access the currently found file name, and the \; at the end.

find . -name '*.zip' -size +1000k -exec ls -s {} \; | sort -n

Again, search for big ZIP files, list them with block usage, and sort the result numerically, i.e. by file size.

file

Looks at file content and tries to detect type and other info; -i for mime type.

% file -i *.html
shell.html: text/html; charset=us-ascii
tools.html: text/html; charset=us-ascii

Rsync for Backup

There are many backup solutions, all with their own quirks; mostly they use their own formats, and restores are often more unreliable than expected. This has caused a lot of grief for countless users.

The rsync command provides a very simple and elegant solution. It creates a full copy of the given directory tree(s) on the backup medium, allowing direct access to files without a restore operation. It is also very fast, since it only copies those files that have changed since the last backup. Consider the following shell script:

#!/bin/sh
 
# removable media
dest=/media/usbdisk/bak/
 
# rsync options
OPTS="-av"
 
# check media is plugged in
if [ ! -x $dest ]; then
    echo "Backup media not mounted"
    exit 1
fi
 
# local host
cd
rsync $OPTS vw publ projects "$dest/pc"
 
# remote host
rsync $OPTS balrog:www/le "$dest/balrog/www"

Our backup medium is a USB stick or external hard disk with a partition labelled usbstick. This partition contains a directory bak at the top level. It may contain other data, which remains unaffected.

Editors

Here are the more commonly used editors; they often come installed with the system.

make

The basic development framework of Unix is the makefile; newer versions include helpful features such as automated configuring, but the initial concept is still valid. The makefile specifies the dependencies among source files and binaries, and the actions to be taken when source files have changed. The make command consults the makefile and acts accordingly. Here is a sample makefile:

JAVAC=/usr/lib/jvm/java-6-sun-1.6.0.03/bin/javac
JAVA=/usr/lib/jvm/java-6-sun-1.6.0.03/bin/java
 
Hello.class: Hello.java
        $(JAVAC) Hello.java
        $(JAVA) Hello
 
hello: hello.c
        cc -o hello hello.c
        ./hello

In this case the Java compiler and runtime are defined, and dependencies and compile actions for a simple project are stated; similarly for a sample project in the C language.

Compiling applications from source

Additional applications that are not provided via the package manager can be compiled and installed. This involves downloading the source code, often from a git repository. Look for files called README and INSTALL for instructions on how to compile.

You probably need to install development libraries or even the compiler itself first, as they are usually not included in desktop distributions. Use your package manager to do that.

Typically a configure script will be included with the software to check dependencies and set options so that the make command will succeed with compilation. In some cases the configure script itself is generated by autogen.sh. If all goes well, the following sequence of commands will compile and install the application:

./configure
make
sudo make install

However, typically some additional libraries or other pieces of code must be downloaded, compiled, and installed first. Of course, the process can continue recursively.

The package manager is the preferred way to install software. Only compile and install from source if

Compiling and installing end-user applications that are not depended on by other components is usually not problematic. Otherwise, be very careful:

Passwords and Passphrases

Good old simple passwords like Netw0rk or g0ldf1sh can be cracked almost instantly nowadays. Graphics cards can be used for massive parallel processing and password cracking; any password of up to 7 random characters (upper case, lower case, and digits) can be cracked in about 18 minutes with a humble Radeon 5770 [1]. With 9 characters the same card would slave for about 48 days. However, parallel and distributed processing with graphics cards and other (possibly hijacked) devices achieve much higher speeds.

A strong password should be

Obviously such passwords are rather hard to remember. Therefore, passphrases are often used instead of passwords.

A common approach for creating secure passphrases is to chose several random words from a list of several thousand. A sufficiently long passphrase is both harder to crack and easier to remember than random passwords; e.g.

Method Combinations
five word passphrase from a list of 5000 words 5000^5 = 3,125,000,000,000,000,000
10 character password, upper and lower case plus digits 62^10 = 839,299,365,868,340,224

Note that the well-known passphrase "correct horse battery staple" is considered too short nowadays, five or even six words are recommended. A passphrase generator can be found e.g. at fourmilab.ch.

Keyring

A keyring holds a number of keys that are provided to applications automatically once the keyring is unlocked. The keyring 'default' will be created the first time you check the option 'Remember forever' when you enter a passphrase.

The keyring holds various keys, e.g. for ssh, email access, and encrypted USB devices. When unlocked the keyring provides those keys to all applications for the duration of the session. Therefore, once you have entered your keys into the keyring and saved them permanently (option Remember Forever) it reduces passphrase entry to one per session.

Even more conveniently, the keyring can be unlocked with the login password, if it is identical to the keyring passphrase, but this requires some setup depending on your distribution; e.g. on Linux Mint 17 the startup applications 'Certificate and Key Storage' and 'Secret Storage Service' must be added in Settings/Session and Startup.

LUKS Linux Unified Key Setup

USB sticks get lost, and notebooks get stolen. Do you want strangers and criminals to access your data? LUKS and the cryptsetup package provide a simple yet reliable method of encryption:

The following commands show the setup for encryption on a USB stick and a FAT 32 file system:

  1. backup everything currently on the stick, all data will be lost during formatting.
  2. root permission is only necessary for setup, not for use

    sudo bash

  3. install the cryptsetup package, unless you already have it

    apt-get install cryptsetup

  4. Check:

    lsmod | grep crypt

  5. No line with dm_crypt means the module is missing; simple solution: reboot
  6. plug in the USB Stick and look for the corresponding device file:

    $ df
    Filesystem ... Mounted on
    ...
    /dev/sdc1 ... /media/DISK_IMG

  7. Make sure you identify the device file correctly; it is usually /dev/sdb1 or /dev/sdc1; almost never /dev/sda1. Check the size, and the label under Mounted on.
  8. release the directory from the mount point:

    $ umount /dev/sdc1

  9. LUKS format with pass phrase (CORRECT device file...careful!!)

    $ cryptsetup luksFormat /dev/sdc1

  10. LUKS open creates a mapping in /dev/mapper/ with the given name as device label

    cryptsetup luksOpen /dev/sdc1 mystick

  11. create a FAT file system with the volume label

    mkfs.vfat -F 32 -n mystick /dev/mapper/mystick

    Or, if you are only using the external drive on Linux computers, create a proper ext4 Linux file system

    mkfs.ext4 -m 0 -L mystick /dev/mapper/mystick

    For external disks over 1 TB save some space on inodes with

    mkfs.ext4 -m 0 -L mydisk -T huge /dev/mapper/mydisk

    The number of inodes determines the maximum number of files, and inodes need space, too; the usage type huge reduces the number of inodes by a factor of four, and on a 2 TB disk this results in about 23 GB more usable space, with still an ample maximum of 30 million files.

Done. Remove the stick and plug it in again. You should be asked for the passphrase. Check the option Remember Forever. That way you only have to enter the passphrase for your keyring once at the begin of the session, not for every device you plug in.

SSH

The secure shell provides access to remote hosts with a high level of security. It uses public key cryptography to provide secure communication over an insecure network.

In the most simple version the public/private keys are generated automatically to encrypt the communication:

ssh otherhost

If your userid on the remote host is different use the following form:

ssh userid@otherhost

The password will be queried interactively. It is more convenient and actually safer to use a manually generated public/private key pair, which can be generated with

ssh-keygen

Choose a strong passphrase. It is used to encrypt the private key. Do NOT leave the passphrase empty. The directory .ssh on the local host will now contain a file id_rsa.pub which holds your public key. The public key needs to be transferred to the remote host. The command

ssh-copy-id otherhost

should work, but if it fails just use text editors on both ends and copy and paste the public key into a file .ssh/authorized_keys on the remote host. Create the directory .ssh if necessary.

Now you login to the remote host via ssh. Current Linux distributions should be set up so that you are asked the passphrase the first time, then you can save it in your keyring (option 'Remember Forever').

Man

All installed programs should have a manual page:

man chmod
man ls
man cp

Use the PageUp/PageDown keys to navigate, and the q key to exit.

References

[1] How a cheap graphics card could crack your password in under a second (June 2011)