Releasing The Potential Of GNU Core Utilities, Part 1

One often overlooked feature of Linux are the coreutils developed by GNU. Most all Linux users have used these core utility programs, such as dir, mkdir, rm, and the like, without ever knowing that they are part of a bigger family of programs. Taking another look at all the wonderful programs available in coreutils may open doors of new potential for the Linux user.

Introduction to the Core Utilities

The coreutils contain a lot of basic tools used on unix-like operating systems. Packages that were previously made, such as textutils, shellutils, and fileutils, are now combined into the coreutils package. All the programs have common options and can be used syntactically in the same way. Let’s take a look at some key programs in each category as shown on Wikipedia and look at a few general use cases that may not be commonly known.

File Utilities

1. mkfifo

This is a fun command that has a lot of potential, but few know about it. Essentially it creates a named pipe to redirect standard output. To show the concept of what it does, let’s take a basic example. Open a terminal, make a temporary directory and cd into it, then do the following:

mkfifo mypipe
cat > mypipe

Open a second terminal, cd into that same temporary directory and do the following:

cat mypipe

Then go back to the first terminal and begin typing words into it and you will see them appear in the second terminal. This is what a custom pipe does, it redirects standard output for you.

Now, let’s look at a useful example. Let’s say you are on a system that is being intensively used for I/O processing. In order to minimize the number of reads and writes we can use pipes to bypass writing temporary data. So, for example, let’s look at how to use the pipe to uncompress data before loading it into MySQL (example on Wikipedia).

First, make the pipe in /tmp so we know where to access it from:

mkfifo /tmp/mypipe

Then, gzip the file to the pipe in that same terminal:

gzip --stdout -d mydata.sql.gz > /tmp/mypipe

In another terminal, log into MySQL and import the file:

LOAD DATA INFILE '/tmp/mypipe' INTO TABLE mytable

For critical servers with large load volume, learning to use mkfifo is a must.

2. dd

This also is a little known command with great potential. Essentially it copies a block of data, be it a a file, directory, partition, or specific sectors of a drive, bit-by-bit to another location with the option of converting it along the way. Most commonly its name is understood to mean “data dump,” but it also flies by the name of “data destroyer,” or more affectionately, “death and destruction.” Let’s just say that you should know what you’re doing when you use this command.

Again, let’s look at the basic concept of what it’s doing and then go to a more useful case of it. A basic file copy looks like the following:

dd if=myfile.txt of=myfile2.txt

Notice that it does not use the standard syntax of -option value or --option=value like the others, instead it uses option=value. The parameters are “input file” (if) and “output file” (of). There are all kinds of fun things you can do with this command, like write random data on the whole drive:

dd if=/dev/urandom of=/dev/sda1

Fill up your entire hard drive with 0’s:

dd if=/dev/zero of=/home/user/HugeFile.txt

(Note: use the count flag to limit the amount of data.)

Or, make an ISO of a music or video disc:

dd if=/dev/cdrom of=/home/user/Music/FavoriteAlbum.iso

The best part of dd is perhaps the fact that it will work when all other copy commands don’t. If a hard drive is going bad and a portion of a file you need is corrupt there is not a single command line utility that will work. If the file is in a stable format, like plain text or csv, then at least some, if not nearly all, the data can be recovered by doing the following:

dd if=/home/user/Documents/file.txt of=/dev/hda1 conv=noerror

These are just two of many, but hopefully you can begin to see the use of looking into these coreutils a little more closely. Be sure to take a look at their online documentation for more information.