Distributed compilation with distcc (x86 vs R-Pi)

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

With the todays performance of our computer, the speed of compilation has been improved to the point that distributed building could be slower than local builds (it would take much more to throw your files and their dependency over your network and back than to compile those locally).

However, when working with large projects or when building on a old (or not so powerful) hardware, it would help if the PCs’ brotherhood would give you a hand.

If you have 2+ similar systems (i.e. x86_64 vs. x86_64), the initial setup of distcc would be a breeze since both GCC compilers are for the same architecture and should have the same CC prefix (eg. x86_64-pc-linux-gnu-). This kind of setup is widely spread and explained over the Internet.

The problem comes when you attempt to setup different architectures, such as an ARM (on a BCM2835/BCM2708 hardware) and a x86_64 for instance. Since this was a real experiment I’ve done, in the following I will refer to it as following:

  • The ARM will be the system you are compiling for (the HOST)
  • the x86_64 will be the system where you distribute the building task (the SLAVE)

First drawback with this setup is that, having (two) different hardware architectures between the HOST and SLAVE(s), you are going to need a cross-compiler on SLAVE(s) capable to produce code for the HOST architecture. Note that the cross-compiler you will install on the SLAVE(s) will have its own CC prefix (eg. armv6-rpi-linux-gnueabi-).

Other drawback I’ve encountered was that on the HOST the CC prefix is armv6j-hardfloat-linux-gnueabi- while on SLAVE it is armv6-rpi-linux-gnueabi-. Every time the HOST will ask the SLAVE(s) to give it a hand with some files, the HOST will instruct the SLAVE(s) which program to use (eg. hardfloat-linux-gnueabi-gcc). As the SLAVE(s) does not have such a program it will fail and furthermore will return to the HOST with an error message like:

distcc ERROR: compile <file> on <SLAVE> failed with exit code 110

In the following I am going to present my setup , step by step. Note that it’s valid only on Gentoo, for other Linux distros the process should be similar but not exactly the same:

  1. install the distcc on the HOST
  2. configure the distcc on the HOST
  3. install the cross-compiler on the SLAVE
  4. build & configure the cross-compiler on the SLAVE
  5. install the distcc on the SLAVE
  6. configure the distcc on the SLAVE
  7. if your HOST are on Gentoo Linux, configure the HOST‘s Portage to work with distcc
  8. compile the kernel while on R-Pi via distcc (not cross-compile, natively)
  9. does it worth it? good question!

1. Install the distcc on the HOST

Depending on the Linux distribution one might use, the installation of distcc could be done in several ways. The old way is via download+configure+make+install and it works every time :).
However, if you would like to take the advantage of your distribution packaging tool then you should definitely use that, whatever it calls: apt, debian, portage, packman, rpm, etc.
In Gentoo I use Portage and the installation of sys-devel/distcc package supposes to run the following command:

root@rpi-gentoo ~ $ emerge distcc

2. Configure the distcc on the HOST

If you plan to use your HOST as a slave for other systems that want to distribute their building work over the network, you should instruct your HOST system about how to run the distccd service. Edit the file /etc/conf.d/distccd and make sure it contains the following settings (note that these environment variables are well documented in your default configuration file):

DISTCCD_OPTS="--user nobody"
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
DISTCCD_OPTS="${DISTCCD_OPTS} --allow <others-ip/netmask>"

We should inform the /usr/bin/distcc application what SLAVE to use when comes to distribute the compilation jobs across the network. For that we have to edit the file /etc/distcc/hosts as following:


where SLAVE*-IP is either the ip or the host name for the SLAVE system(s) that will accept our distributed compiling jobs. We put the localhost at the end of the list in order to instruct distcc to distribute first on the network and only then to use the localhost CPU resources.

We want that whenever the C compiler is called on HOST, the distcc to “hijack” that call and to distribute that compiling job(s) to the configured distcc hosts. To achieve that we have to make two more adjustments:

  • remove the original symlinks: /usr/lib/distcc/bin/{c++,cc,g++,gcc}
  • create a wrapper script
  • recreate those 4 symlinks to point to our newly created wrapper script
  • adjust the global PATH variable such that distcc install directory to be the first one when the system is searching for C compilers

Run as root user the following commands:

GCC_WRAPPER=$DISTCC_BIN/`gcc -dumpmachine`-wrapper
rm $DISTCC_BIN/{c++,cc,g++,gcc}

cat << EOT > ${GCC_WRAPPER}
exec ${DISTCC_BIN}/`gcc -dumpmachine`-g${0:$[-2]} "$@"

for prog in c++ cc g++ gcc;do
  ln -s $GCC_WRAPPER $DISTCC_BIN/$prog

Edit the /etc/profile on your HOST system and add replace the “export PATH” line with the one below:

export PATH="/usr/lib/distcc/bin:${PATH}"

As you can see the distcc programs would be found first on the PATH in spite of the original gcc which comes later on the path. To activate this path right now, make sure to re-sync the environment global variable by running the following command:

root@rpi-gentoo ~ $ source /etc/profile

3. Install the cross-compiler on the SLAVE

Also, depending on your Linux distribution this step might have different approaches. On Gentoo, one might choose to install sys-devel/crossdev or sys-devel/ct-ng. Since I haven’t succeeded with crossdev, in the following I will present only how I succeeded with ct-ng tool.
If you want to install the ct-ng like in the old days then follow the instructions available at crosstool-ng.org . If your SLAVE is a Gentoo system then simply run:

user@x86_64-gentoo ~ $ emerge ct-ng

4. Build & configure the cross-compiler on the SLAVE

Since ct-ng is only a tool that automates the process of building a cross-compiler toolchain, it means that you don’t have (yet) the cross-compiler needed to help the HOST with the (distributed) compilation tasks.
When building a compiler for a target platform (such as ARM, for instance) you should know what’s your target platform (here HOST). One simple way of finding that would be to look up on your HOST for its CC prefix:

user@x86_64-gentoo ~ $ gcc -dumpmachine

Let’s suppose that the command above returned armv6jhardfloat-linux-gnueabi. It means that we should look up for a ct-ng sample as close as possible to this prefix.

Now that we have an idea about the target platform we can ask ct-ng to show us its pre-configured list of samples (hopefully we’ll find something similar with ours):

user@x86_64-gentoo ~ $ ct-ng list-samples

This would produce an output like:

user@x86_64-gentoo ~ $ ct-ng list-samples
Status  Sample name
[G.X]   alphaev56-unknown-linux-gnu
[G.X]   alphaev67-unknown-linux-gnu
[G.X]   arm-bare_newlib_cortex_m3_nommu-eabi
[G.X]   arm-cortex_a15-linux-gnueabi
[G..]   arm-cortex_a8-linux-gnueabi
[G..]   arm-davinci-linux-gnueabi
[G..]   arm-unknown-eabi
[G..]   arm-unknown-linux-gnueabi
[G.X]   arm-unknown-linux-uclibcgnueabi
[G..]   armeb-unknown-eabi
[G.X]   armeb-unknown-linux-gnueabi
[G.X]   armeb-unknown-linux-uclibcgnueabi
[G.X]   armv6-rpi-linux-gnueabi
[G.X]   avr32-unknown-none
[G..]   bfin-unknown-linux-uclibc
[G..]   i586-geode-linux-uclibc
[G.X]   i586-mingw32msvc,i686-none-linux-gnu
[G.X]   i686-nptl-linux-gnu
[G.X]   i686-unknown-mingw32
[G.X]   m68k-unknown-elf
[G.X]   m68k-unknown-uclinux-uclibc
[G.X]   mips-ar2315-linux-gnu
[G.X]   mips-malta-linux-gnu
[G..]   mips-unknown-elf
[G.X]   mips-unknown-linux-uclibc
[G.X]   mips64el-n32-linux-uclibc
[G.X]   mips64el-n64-linux-uclibc
[G..]   mipsel-sde-elf
[G..]   mipsel-unknown-linux-gnu
[G..]   powerpc-405-linux-gnu
[G..]   powerpc-860-linux-gnu
[G.X]   powerpc-e300c3-linux-gnu
[G.X]   powerpc-e500v2-linux-gnuspe
[G..]   powerpc-unknown-linux-gnu
[G..]   powerpc-unknown-linux-uclibc
[G..]   powerpc-unknown_nofpu-linux-gnu
[G.X]   powerpc64-unknown-linux-gnu
[G.X]   s390-ibm-linux-gnu
[G.X]   s390x-ibm-linux-gnu
[G..]   sh4-unknown-linux-gnu
[G..]   x86_64-unknown-linux-gnu
[G..]   x86_64-unknown-linux-uclibc
[G.X]   x86_64-unknown-mingw32
 L (Local)       : sample was found in current directory
 G (Global)      : sample was installed with crosstool-NG
 X (EXPERIMENTAL): sample may use EXPERIMENTAL features
 B (BROKEN)      : sample is currently broken

The sample (with bold) above looks pretty good to me since the HOST and the sample listed above represents the same CPU architecture (armv6) and they both are compatible with the GNU EABI (cool!).
If you want to know what library it would install if you would choose that profile, run the following command:

user@x86_64-gentoo ~ $ ct-ng show-armv6-rpi-linux-gnueabi
[G.X]   armv6-rpi-linux-gnueabi
    OS             : linux-3.6.11
    Companion libs : gmp-5.0.2 mpfr-3.1.0 ppl-0.11.2 cloog-ppl-0.15.11 mpc-0.9
    binutils       : binutils-2.22
    C compiler     : gcc-linaro-4.7-2013.01 (C,C++)
    C library      : eglibc-2_16 (threads: nptl)

Once you’ve decided configure the ct-ng to use that sample/profile. It would be configured by default at ${HOME}/x-tools/armv6-rpi-linux-gnueabi. Note that you should run this commands with a regular user (not root!):

user@x86_64-gentoo ~ $ ct-ng armv6-rpi-linux-gnueabi

If you want to fine-tune this configuration via a ncurses menu (like the one you have when you are configuring the Linux kernel) then run also this command:

user@x86_64-gentoo ~ $ ct-ng menuconfig

The menu will looks like the one below (that’s why I prefer this tool rather than others, plus that it works!):

click to zoom

click to zoom

When ready you can start building the cross-compiler toolchain as following (note that it will take a while, like dozens of minutes):

user@x86_64-gentoo ~ $ ct-ng build

If you want to use this compiler at the HOST level, outside of the distcc scope, then make sure that your shell environment PATH variable gets updated with the path of this new toolkit. Either edit your ~/.profile or the global /etc/profile file and add the following lines, then run either “source ~/.profile” or “source /etc/profile” to update your local/global PATH environment variable:

CROSS_ROOT="${HOME}/x-tools/armv6-rpi-linux-gnueabi" #if you use this in /etc/profile then use absolute path instead $HOME variable
export PATH="${CROSS_PATH}:${PATH}"

In order to allow distcc to use this toolkit, you must set the distcc environment PATH variable in the /etc/conf.d/distcc file (check my comments here).

Note: replace the 4.7.3 version above with the real one that fits your build.

Because the SLAVE cross-compiler CC prefix is different than the one on our HOST (i.e. armv6-rpi-linux-gnueabi vs. armv6j-hardfloat-linux-gnueabi), when the HOST will ask the SLAVE to compile a <file> with the program armv6j-hardfloat-linux-gnueabi-* the SLAVE will fail and will return in exchange the error shown at the beginning (code 110).

The solution for this would be to create the following symlinks on the SLAVE:

for prog in c++ cc g++ gcc;do

Now, every time that the HOST will ask the SLAVE to compile a using the armv6j-hardfloat-linux-gnueabi tool, the SLAVE will obey and cheat in the same time, because it will redirect that command to the local installed cross-compiler (eg. armv6-rpi-linux-gnueabi).
If you want to test your new installed ARM cross-compiler just grab a “Hello world” C program and compile it locally like below (hopefully output.o is created, everybody’s happy):

armv6-rpi-linux-gnueabi-gcc -c -o output.o source.c

You should also read the installation/usage instruction on crosstool-ng.org website.

5. Install the distcc on the SLAVE

Repeat the same procedure like in the step 1 above.

6. Configure the distcc on the SLAVE

Edit the file /etc/conf.d/distccd and make sure it contains the following settings (note that these environment variables are well documented in your default configuration file):

DISTCCD_OPTS="--user nobody"
DISTCCD_OPTS="${DISTCCD_OPTS} --log-level critical"
DISTCCD_OPTS="${DISTCCD_OPTS} --allow <HOST-ip/netmask>"

Note: adjust the 4.7.3 version with the real one that fits your build. Replace also the X with the number of cores that your SLAVE can provide. To make sure that the SLAVE is busy while the R-Pi HOST is struggling with its IO bottleneck I set this at least 2-5 times more than the number of cores of the SLAVE system. Make sure that the HOST also will attempt to use all those available sockets (if you run make manually make sure you use the -j switch, too).

Make sure that you restart the distccd service after these changes.

7. Configure the HOST Gentoo’s Portage with distcc

Portage knows to take advantage of the distcc tool out of the box. To activate the usage of distcc we should edit the /etc/make.conf (or /etc/portage/make.conf nowadays) and to configure the following options:

DISTCC_HOSTS="SLAVE1 SLAVE2 ... localhost"

Note: the X above represents the total number of CPU of your localhost + SLAVE1 + SLAVE2 + … and the SLAVE* represents the ip or the host name for those SLAVE system(s) where the HOST will distribute the compilation jobs.

8. Compile the kernel while on R-Pi via distcc

Although one might (cross)compile the kernel on a powerful system and later deploy kernel image on the R-Pi SD-CARD, it’s also possible to compile the kernel while running the R-Pi by the R-Pi itself.

Using the distcc as described above will not work out of the box because the Linux source Makefile is insisting to work with gcc/g++ and therefore local (HOST) distcc will get the request to distribute the compilation of some kernel file by the remote gcc/g++ installed on the SLAVE (instead of some cross-compiler armv6-* on the SLAVE).

Because at the SLAVE level runs your SLAVE architecture gcc (eg.: x86_64-pc-linux-gnu-*), obviously it will compile every source received from the R-Pi distcc using that native compiler, resulting a native SLAVE object binary (that runs only on SLAVE architecture, such as x86_64) instead a binary compiled for ARM architecture that is understandable by a armv6j cpu.

Although there might be many other solutions, one that I came up with and which does not alter the previous(1-7) configuration, is to disable temporary the gcc on the SLAVE. On my SLAVE (x86_64) system I just renamed (temporarily) the /usr/x86_64-pc-linux-gnu folder (/usr/x86_64-pc-linux-gnu.old) and after the kernel compilation end I renamed back as it was before. It’s a dirty solution, I know, but till I find a better solution I have to leave with that.

Note: when compiling the kernel on R-Pi don’t forget to make use of -jX switch, where X is the number of cores of your SLAVE + 1.

Does it worth all this effort?

Does it worth it?

To answer that question I’ve tried to compile the Linux kernel directly from R-Pi (@700MHz, no overclocking), with and without distcc , having the help of an HP workstation with a Core 2 Duo CPU E8400 @ 3.00GHz and 4G RAM (see the [!] below):

  • make -j10 with distcc (-j10) on (HP) SLAVE: 4080 seconds
  • make without distcc (only 1 job on R-Pi): 10524 seconds
  • The HOST /etc/distcc/hosts defines the following: SLAVE-IP/10,lzo –localslots=1

The bottom line: \frac{4080 s}{10524 s} = 0.39 i.e. with only one SLAVE did the distcc compilation succeeded 2.6 times faster. If you are adding other SLAVEs to this setup, they will definitely make the process faster. This result can be even improved if you are going to play a little with distcc host specifications -LIMIT option (see distcc manual).

[!] The above compilation was done with this kernel .config file. If you choose to use it then remember to remove the CONFIG_CROSS_COMPILE parameter as it was not meant to be used in this project but rather in this one.

In another test I’ve used other kernel .config which is even more cut-down than the original one (I’ve eliminated the sound, wireless, USB support for other things than keyboard, mouse, storage, cryptography algorithms not used, NLS pages not used, etc) and I’ve done a similar test as above, as following:

  • make -j4 with distcc (-j10) on (HP) SLAVE: 3028 seconds
  • make without distcc (only 1 job on R-Pi): 8035 seconds
  • The HOST /etc/distcc/hosts defines the following: SLAVE-IP/4 –localslots=1

So basically, if you use your R-Pi only for some experiments (not as gaming/media console) and if you use this simplified kernel configuration, it’s possible to compile the R-Pi kernel from R-Pi itself in 2h (without any extra-help) or less or equal than 50 minutes with help of distcc distributed compilation.

Final note

The process described above does not apply only to ARM vs X86-* architecture. A setup like this should also work when you have an old/slow i386 PC and a powerful X86-* system and you want to distribute the compilation across the network such the i386 get a hand from its network fellows. Working with heterogeneous environments it’s a science, but not an exact science. You have to grab the bull by the horns, sometime you fix a problem after many trials and errors, a hard job but somebody have to do it.
It worths reading this blog, too: http://rostedt.homelinux.com/distcc/.

Posted in distro, linux, Raspberry Pi | Tagged , , , , | Leave a comment

Transparently download from Google Drive

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

Google Drive is a file storage and synchronization service provided by Google which enables user cloud storage, file sharing and collaborative editing.

The storage limit for the free accounts is, at the time of writing, about 5GB.

If you have a blog, like I have, and sometimes you write an article and you want to share a files rather than inserting its contents into the articles page, you could use your free Google account and the Google Drive service to store and share that file.

The problem

The problem that I have encountered regarding this matter was that, when sharing a file you get a public share link, like the one below:


If you provide this address as your HTML link reference then you will not be prompted by the “Save as” window, instead the Google Drive Viewer application will be shown and this will allow you, of course, to download that file:

click to zoom

a file shown in Google Drive Viewer

As we can see in the link provided above, it is structured as following:


  • the secure connection protocol : https
  • the (sub)domain name : docs.google.com
  • other application/directories within the Google website : file/d/
  • the file unique identifier : 0B95k2kr1bG9feGVhd0p3QkVCSVE
  • the action/command to execute about that file: view (or edit, etc)

If I would write an article and I wish to insert a link to a shared file (stored on Google Drive) I would not be happy to let my readers/visitors to open the Google Drive Viewer then to put them to click one more time, and so on. What I would like to achieve would be to get directly that Save as windows:

click to zoom

the “Save As” window when clicking on a file link

Obviously there are (at least) two types of downloads where you really need the direct download link:

  1. nice to have: on a link inserted on a HTML page (such as a wordpress.com article)
  2. must to have: on a link where you are supposed to download the file programmatically

In the first case you could live even without the direct link, maybe the Google Drive Viewer it’s not such a big deal at the end.

In the second case, if you don’t have a direct link, you could end downloading a content which is not your file but the Google Drive Viewer html content for the requested file:

<!DOCTYPE html><html><head><meta name="google" content="notranslate">
<meta http-equiv="X-UA-Compatible" content="IE=edge;">
<meta name="fragment" content="!"><title>.config.rpi $
c&&0<b&&(c.tick("_tbnd",void 0,window.chrome.csi().startE),

My solution(s)

Obviously, by trial an error, I found solutions for both cases.

  1. For the link you would insert into your web page (such as a wordpress.com article) you could use the following pattern:
Ex: https://docs.google.com/uc?export=download&id=0B95k2kr1bG9feGVhd0p3QkVCSVE
  1. For the link where you are supposed to download the file programmatically the pattern above will not work, instead you will get an error message like below:
root@rpi-gentoo ~ $ --2013-02-23 10:33:32--  https://docs.google.com/uc?export=download
Resolving docs.google.com...,,, ...
Connecting to docs.google.com||:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2013-02-23 10:33:32 ERROR 500: Internal Server Error.

I’ve found that the following pattern will work like a charm:

Ex: https://googledrive.com/host/0B95k2kr1bG9feGVhd0p3QkVCSVE

Note that in the second case (the one above) the file name you will be prompted to save (programmatically or not) will not be the original shared file name (like in the first example) but the file hashed identifier (such as 0B95k2kr1bG9feGVhd0p3QkVCSVE). Since this method provides a direct link (by the file-id) to download the file programmatically (eg: wget -O <output-file> URL), it would not be a challenge for any programmer to choose whatever name he/she likes for the output-file.

Posted in Uncategorized | Tagged , , , , | 13 Comments

Boost Linux kernel compilation for Raspberry Pi

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

I do a lot of experiments with my new RPi nowadays. It’s like in the old days, in the early of ’90s, when I’ve got my HC-91, the Romanian ZX-Spectrum clone.

There is one small difference, though: the HC-91 has been delivered with a case an ROM-stored BASIC interpretor. The RPi has no case nor a pre-installed OS, so you have to take care of both.

To compile the Linux kernel for Raspberry Pi (i.e. ARM architecture) one should accomplish at least 8 different steps:

  • fetching the (last) Linux kernel source code (git://github.com/raspberrypi/linux.git)
  • fetching the (last) RPi firmware (git://github.com/raspberrypi/firmware.git)
  • configuring the kernel according with the target architecture
  • compiling the kernel and its modules
  • copying the kernel image to the boot partition
  • copying the RPi firmware to the boot partition
  • (optional) configuring the kernel boot command-line and its start-up configuration
  • last but not least, copying the RPi firmware utilities to the target /opt/ directory

If you have to do often this task then it would be better to automate those steps. Because I did that many times and because I love to create my own scripts that makes my live easier (in the future), I wrote an automation script that accomplish all these steps.

This script assumes that you have already a crossdev environment prepared so you can use your cross-arm-compiler right away:

Raspberry Pi kernel compilation automation script.
Usage    : /usr/sbin/rpi-build [options]
Options  :
      -w : the working directory (mandatory)
      -c : CC prefix for the TARGET architecture
      -f : the path to your predefined Linux kernel .config file
      -p : the name of the boot partition of your RPi
      -t : the type of the partition specified by -p parameter
      -h : print this help message
Report bugs to eugenmihailescux at gmail dot com

The <working directory> is the place where the Linux kernel and firmware source code will be downloaded, where the build.log will be saved, where the final output directory will be deployed.

The <CC prefix> is the full path of the crossdev toolkit that will be used to compile the kernel for the TARGET architecture. For instance, if your cross compiler toolkit is installed at ~/x-tools/armv6-rpi-linux-gnueabi/bin/ and the compiler prefix is something like armv6-rpi-linux-gnueabi, then the CC prefix you specify should be ~/x-tools/armv6-rpi-linux-gnueabi/bin/armv6-rpi-linux-gnueabi- (note the dash at the end of the prefix!).

The <.config> is the path to a predefined Linux kernel .config file. You can use a predefined .config Linux kernel configuration file so that you don’t have to configure the kernel every time you run this script. For my RPi model B I have already created a cut-down RPi configuration and can be found here.

The <boot partition> is the name of your RPi boot partition on the SD Card. If nothing is supplied then /dev/mmcblk0p2 will be assumed by default.

The <partition type> is the type of the <boot partition>. If nothing is supplied then ext4 will be assumed by default.

The script can be downloaded from here.

Posted in kernel, linux, Raspberry Pi | Tagged , , , | Leave a comment

The true price of Raspberry Pi. The cheapest setup

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

After getting the long awaited Raspberry Pi board I started to google for the required equipments in order to boot-up the board:

The bottom line of this story is that, using the equipments that one might already have it at home + buying some extra but cheap equipments from eBay, the true price of Raspberry Pi does not exceed 50$ (or 38 ‚ā¨ or 33 ¬£).

The bord along with a 2G SD card:

click to zoom

click to zoom

The silicon rubber USB keyboard:

click to zoom

click to zoom

The RCA composite video cable:

click to zoom

click to zoom

The mini to micro USB charger adaptor converted:

click to zoom

click to zoom

Last but not least my zero dollar cardboard “pibow replica” case, not so artistic, not so gorgeous like the original, but at least it’s cheap and it’s eco-friendly:


The case seen on the top

The case seen from the front

The case seen from the front


The case seen from one side


The case seen from the other side

Since the Raspberry Pi project aims to provides a cheap computing solution, I thought that it would be nonsense to spend more on accessories that on the computing unit itself. The solution I’ve found sounds/looks sensible.

Posted in Hardware, Raspberry Pi | Tagged , , , , , | Leave a comment

Disk usage in bash

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

I run several headless systems and sometimes I need to know the disk usage of a particular directory. We have several choices, from the ls command to the ncurses ncdu utility.

For what I need ‘ls’ does not provide support out of the box. The ncdu looks more alike but I don’t need ncurses dependencies or other tools added to my toolbox. I need a 10 line bash script, eventually written by myself ūüôā (let’s reinvent the wheel, don’t we?)

My solution

Create a bash script called “du1” (or whatever you like) with the following content:

# Script for displaying disk usage
# Syntax: du1 FOLDER <FILTER>
#	where FOLDER is the folder to show
#	and FILTER is a float {0..100} that allows you to filter
# 	only those folders which have a usage >= FILTER %
# Author        : Eugen Mihailescu
# Last change   : 21.Feb.2013
# E-mail        : eugenmihailescux at gmail dot com
# Tested on     : Linux 3.6.10-gentoo-2.1 x86_64 GenuineIntel

if [ -z $PARENT_DIR ];then

if [ -n "$2" ];then

# print what will be shown
s="Show usage of $PARENT_DIR"
result=`expr $FILTER_SIZE \> 0`
if [ "$result" -eq "1" ]; then
	s="$s, for subfolders usage >= $FILTER_SIZE%"
echo $s

# get the total size of the parent directory
TOTAL=$(du -sB1 $PARENT_DIR 2>/dev/null|cut -f1)

# print each subfolder disk usage
find $PARENT_DIR -mindepth 1 -maxdepth 1 -type d -print0|du -sB1 --files0-from=- 2>/dev/null|sort -gr|awk -v TOTAL=$TOTAL -v fSIZE=$FILTER_SIZE '{bsize=$1;perc=100*bsize/TOTAL;if (perc>=fSIZE){size=bsize; unit="B"; if (size>=1024){size=size/1024;unit="KB";} if (size>=1024){size=size/1024; unit="MB";} if (size>=1024){size=size/1024;unit="GB";} printf("%8.2f %-2s [%-4.1f%%]\t%s\n",size,unit,perc,$2)}}'

# print a subtotal line separator
printf "%${COLS}s\n" |tr " " "-"

# print the total disk usage
echo $TOTAL $PARENT_DIR|awk -v TOTAL=$TOTAL '{bsize=$1;size=bsize; unit="B"; if (size>=1024){size=size/1024;unit="KB";} if (size>=1024){size=size/1024; unit="MB";} if (size>=1024){size=size/1024;unit="GB";} printf("%8.2f %-2s [%-4.1f%%]\t%s\n",size,unit,100*bsize/TOTAL,$2)}'

The result

rpi-gentoo ~ # du1 / 0.1
Show usage of /, for subfolders usage >= 0.1%
    1.12 GB [94.7%]    /usr
   34.38 MB [2.8 %]    /opt
    9.81 MB [0.8 %]    /var
    9.07 MB [0.7 %]    /lib
    6.34 MB [0.5 %]    /bin
    2.88 MB [0.2 %]    /sbin
    1.19 GB [100.0%]    /

That’s why Linux can be fun!

Posted in linux, shell | Tagged , | Leave a comment

*nix tip of the day: what ‘Linux version’ am I using?

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

How to tell the Linux distro version you are using?

Since Linux is not exactly the whole ecosystem of computer programs that you are using on your desktop/server but rather the kernel that allocates the physical hardware resources for the computer programs that you are running, it would not be syntactically nor technically correct to name the version of your OS by the version of your OS kernel.

Who is who?

What we call it generically ‘Linux’ is in fact:

  • a kernel (that project started by Linus Torvalds back in ’90s and lately supported by a bunch of programmers) – an essential part of OS
  • OS programs that run in top of the kernel and which, if are missing, the kernel is just a beautiful but useless piece of furniture.

So the OS is in fact this combination of GNU with Linux added, so called GNU/Linux.

If I run the uname command I got the whole version info, like:

rpi-gentoo ~ # uname -a
Linux rpi-gentoo 3.6.11-cutdown+ #17 PREEMPT Mon Feb 18 14:27:02 CET 2013 armv6l ARMv6-compatible processor rev 7 (v6l) BCM2708 GNU/Linux

Also, the full and the correct name would be GNU/Linux.

In order to tell what ‘Linux version” am I using one should look over the version of its Linux distribution. And that piece of information comes from a file that is located on your /etc folder and which might be called: redhat-release, SuSE-release, debian_version, arch-release, gentoo-release, slackware-version, mandriva-release, etc.

So basically to tell what version is your Linux distribution one should run this command:

rpi-gentoo ~ # cat /etc/*-release /etc/*/-version 2>/dev/null
Gentoo Base System release 2.1
Posted in distro, linux, shell | Tagged , | Leave a comment

Screen is overlooked

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

Yes, screen is overlooked! I’m not talking about your cathode ray tube nor your computer monitor but the GNU Screen application that, if you are a computer geek or just an everyday system administrator, it’s kind of busybox of virtual console multiplexing.

click to zoom

click to zoom

Until I found it my live was a nightmare, I really was in the Stone Age. Now, making my first steps in this new virtual console multiplexing world, I feel already being part of the New Stone Age, the Neolithic.

GNU Screen is, according to their website:

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

So what? What is it good for? Why should I need another window manager, it’s not just (good) enough the GNOME, KDE, XFce, LXDE, etc ? Well, that’s another story…

Let my first put this question: have you ever opened a SSH connection on a remote terminal, started a long running job when suddenly your connection just died and so your task (when it was almost 99% done) ?

May I ask something else? How do you connect a remote terminal via SSH and work into two-three different console windows? I used to open two-three different SSH sessions, each on its own console window so I could run multiple commands/jobs simultaneously. Not anymore, now I’m in Neolithic, remember?

I will try to explain with my bad English, but in a plain English, what is this program good for:

  • you establish only one remote connection but work on multiple terminal windows simultaneously
  • you can switch between these windows, you can name them, you can split them (so you display 3-4 windows on the same terminal, seeing what’s going on on each of them)
  • you can leave the job[s] running in these windows while you are “detaching” from them, (even) logoff from your remote (eg. SSH) connection, come back after few hours/days (eventually from your home/office/other terminal), “reattaching” to those windows just to see that everything keeps working (and hopefully with the job[s] completed)
  • of course you can copy/paste between these windows, you can deliberately close/kill/exit each of them, you can
  • you can log and monitor the windows activity, lock the screen (unlock by password)
  • other few dozens of useful functions

As you can see in the screenshot above I have only one remote SSH connection and a terminal where are shown three different windows (splitted vertically): one which runs a ping command, the second with runs a top command and the last one which runs an system update. Each window has its own status bar where you can see its name, the current/focused one is shown on bottom as a red button.

If you just google a bit I’m sure you will find at least few hundred pages that praise this magnificent tool but I would recommend the “Using GNU Screen” article on Debian Administration weblog.

Once you have installed the tool and opened on a remote system I would recommend you just to press CTRLa and ?. That way you will invoke the help which in turn will show you the key bindings. Before running this tool it worth reading the man pages or even the /etc/screenrc configuration file, because all the key bindings (+ other options) are defined there.

In Gentoo the package is called app-misc/screen and is very well documented here.

Posted in linux, shell | Tagged , , , | 2 Comments

Gentoo on Raspberry Pi

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

Few weeks ago I received my long awaited Raspberry Pi board. For those that (yet) don’t know what a Raspberry-Pi is, “The Raspberry Pi is a credit-card sized computer that plugs into your TV and a keyboard. It‚Äôs a capable little PC which can be used for many of the things that your desktop PC does, like spreadsheets, word-processing and games. It also plays high-definition video” (source: www.raspberrypi.org).

click to zoom

If someone wonders “what can be done with such a small little toy?” then maybe you should answer this question first: “what is a smart-phone?”
Anyway, to get a grasp of what this might offer, one should check out the board layout:

I’m not going to develop more this subject, additional info can be found at: http://elinux.org/RPi_Hub.

I’m going to tell you how I have:

Note: make sure you are logged as root so we don’t have to ‘sudo’ all the time.

Create a disk image for your Raspberry Pi

First of all you should know how large is your SDHC card (eg, 2GB, 4GB,…, 32GB). Let’s suppose that your SHDC card is 2GB. When you get this info all you have to do is to run the following command (remember: 1GB is 1024*1024*1024 is 1073741824 bytes):

export RPI_IMG=/path-to-image/my-image.img
dd if=/dev/zero of=${RPI_IMG} bs=4096 count=$((2*1000*1000*1000/4096))

This image file is your (r-pi) disk image file, also it will contain your r-pi “hard disk” image. So all we have to do is to mount this disk, to create partitions and the necessary file system.

Create partitions, file-systems, mount the file-systems

The following will setup a loop device linked to your image file:

export RPI_DEV=$(losetup -f --show -P ${RPI_IMG})

Now that we have the disk device, first thing we create the disk partitions with a similar layout like this one (note that our disk device is called /dev/loop1):

Disk /dev/loop1: 2000 MB, 2000000000 bytes
255 heads, 63 sectors/track, 243 cylinders, total 3906250 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x58f30fd1

      Device Boot      Start         End      Blocks   Id  System
/dev/loop1p1   *        2048       69631       33792    c  W95 FAT32
/dev/loop1p2           69632      133119       31744   82  Linux swap
/dev/loop1p3          133120     3906249     1886565   83  Linux

Once your disk is partitioned, make sure that you can see/access all those /dev/loopXpY partitions (where X is your loop device, Y is the number of partition: 1,2,3,..). If they are not visible (they should be, see -P option in losetup utility) then you can force the OS to update its partition table:

partprobe ${RPI_DEV}

Once the partitions are ready we need to format them with a file-system so that we can store files on disk, right?

mkfs.vfat -F 32 -n RpiBoot ${RPI_DEV}p1
mkswap ${RPI_DEV}p2 && swapon ${RPI_DEV}p2
mkfs.ext4 -L RpiRoot ${RPI_DEV}p3

Note: if you have a 2GB image then those 2000MB of ${RPI_DEV}p3 will be insufficient so I recommend specifying the number of inodes to create:

mkfs.ext4 -T small -L RpiRoot ${RPI_DEV}p3

Now that we have an working disk image we should mount its partitions as local mount points where we are going to deploy Gentoo’s files:

export RPI_ROOT=/mnt/rpi
mkdir -p $RPI_ROOT && mount ${RPI_DEV}p3 $RPI_ROOT
mkdir $RPI_ROOT/boot && mount ${RPI_DEV}p1 $RPI_ROOT/boot
wget -O - http://distfiles.gentoo.org/releases/arm/autobuilds/current-stage3-armv6j_hardfp/stage3-armv6j_hardfp-YYYYMMDD.tar.bz2|tar xpjf -
wget -O - http://distfiles.gentoo.org/snapshots/portage-latest.tar.bz2| tar xpjf - -C usr/

Gentoo basic setup pre-first-boot

To make sure that you boot with a functional network, you can login with the root, you have an working ssh daemon, etc, we have to make some small adjustments first:

  • edit $RPI_ROOT/etc/shadow and reset root password: root::10770:0:::::
  • edit $RPI_ROOT/etc/conf.d/net and add the following lines:
  • edit $RPI_ROOT/etc/fstab and adjust the mount points to match the virtual disk (/dev/sda):
# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>
/dev/sda1               /boot           auto            noauto,noatime  1 2
/dev/sda3               /               ext4            noatime         0 1
/dev/sda2               none            swap            sw              0 0
  • edit $RPI_ROOT/etc/conf.d/hostname and adjust the system name: hostname=”rpi-gentoo”
  • edit $RPI_ROOT/etc/conf.d/hwclock and adjust the guest clock according to your host hwclock
  • copy your timezone to $RPI_ROOT/etc/: cp $RPI_ROOT/usr/share/zoneinfo/XXX etc/localtime, where XXX coresponds to your timezone (eg. Europe/Stockholm); edit also /etc/timezone and add the zone name (eg. Europe/Stockholm)
  • if necessary adjust you keymaps at $RPI_ROOT/etc/conf.d/keymaps
  • comment “s0:12345:respawn…” console in $RPI_ROOT/etc/inittab

Note: if you are going to deploy this image on a SDHC card then you should replace the /dev/sda/* (on $RPI_ROOT/etc/fstab) with /dev/mmcblk0/*.

Compile the kernel

So far we have deployed the required Gentoo files. Next step will be to build a qemu kernel for our ARM cpu that will allow us to run this Linux on a qemu emulator. If you plan to compile the kernel for the real thing (BCM2835 hardware) the procedure is quite the same (I will make a note where they differ). For this we need a cross-toolchain  that allows us to compile ARM code on a X86 platform.

At first I have tried the sys-devel/crossdev toolchain builder but every time there was something that didn’t worked out of the box. I thought that maybe it’s something about my system so I’ve tried to install a a fresh copy of Gentoo (on a virtual machine, of course :o) but nothing helped. If you instead prefer doing the old way (i.e. manually) then try this link first.

Finally I found an working alternative to those one, it’s called sys-devel/ct-ng and it works like a charm:

mkdir /tmp/work && cd /tmp/work
ct-ng list-samples # the list of samples/profiles
ct-ng show-armv6-rpi-linux-gnueabi # pick your sample, I chose Raspberry Pi
ct-ng armv6-rpi-linux-gnueabi # configure the sample we are using
# ct-ng menuconfig
ct-ng build # build a cross toolchain based on our chosen sample
export PATH=${PATH}:${HOME}/x-tools/armv6-rpi-linux-gnueabi/bin
armv6-rpi-linux-gnueabi-gcc -v # test it!

Note: if the toolchain sample you choose is not pre-configured as you wish, then before running “ct-ng build” you can run “ct-ng menuconfig” and tune your toolchain as needed. More here.

Once your cross-toolchain is built we are ready to compile the kernel. But wait! First we have to grab its source from Internet/github.

cd /tmp/
git clone https://github.com/raspberrypi/linux.git
wget -O /tmp/linux-arm.patch http://xecdesign.com/downloads/linux-qemu/linux-arm.patch
patch -p1 -d ./ < /tmp/linux-arm.patch

Now that we have the kernel source code we are ready to compile the kernel (make sure that you check also kernel Device Drivers -> Real Time Clock settings) :

mkdir /tmp/modules && cd /tmp/linux
make ARCH=arm versatile_defconfig
make ARCH=arm menuconfig
make ARCH=arm -j2
make ARCH=arm INSTALL_MOD_PATH=../modules modules_install
export RPI_KERNEL=arch/arm/boot/zImage

Note: if you are going to compile the kernel for the real thing then you should use bcmrpi_cutdown_defconfig default configuration instead of versatile_defconfig above. See more on your linux/arch/arm/configs/ folder.

Test it inside the QEMU emulator

Once the kernel is built you can use it right away (recommended at least on first boot):

qemu-system-arm -M versatilepb -cpu arm1176 -hda $RPI_DEV -kernel $RPI_KERNEL -append "root=/dev/sda3 panic=1" -m 256 -net nic -net user -redir tcp:2222::22

Note: in the above we have just started a qemu emulator with for a system with an ARM1176 cpu, 256 MB RAM, using our custom kernel, and our disk image, with a fully functional network (read more) where we can connect by ssh from the local host like:

ssh root@localhost -p 2222 # the local 2222 was previously redirected to the qemu port 22

You can even connect the qemu via VNC, usually when it starts is shows you the VNC server:port to use. If instead you want no VNC but all the output redirected to your terminal console then start the qemu like this:

qemu-system-arm -M versatilepb -cpu arm1176 -hda $RPI_DEV -kernel $RPI_KERNEL -append "root=/dev/sda3 panic=1 console=ttyAMA0" -m 256 -net nic -net user -redir tcp:2222::22 -nographic

Moreover, you can even monitor the qemu and communicate with qemu, such as adding a new device, ejecting a device, pausing/resuming the VM, etc:

qemu-system-arm -M versatilepb -cpu arm1176 -hda $RPI_DEV -kernel $RPI_KERNEL -append "root=/dev/sda3 panic=1 console=ttyAMA0" -m 256 -net nic -net user -redir tcp:2222::22 -nographic -monitor telnet:localhost:4444,server,nowait

, then you can access the qemu monitor by:

telnet localhost 4444

Here you can find some extra info regarding using qemu.

Gentoo basic setup post-first-boot

Now that the system just boot-up without problems, connect the guest terminal via VNC, enter the login name root (no password will be required; remember /etc/shadow ?). We have few things to setup so that next boot everything will be in place and working:

  • edit /etc/locale.gen and set your locale then run locale-gen command
  • run the following command: cd /etc/init.d && ln -s net.lo net.eth0 && rc-update add net.eth0 default
  • rc-update add sshd default
  • setup the root password by running the following command: passwd
  • run command “eselect profile list” and check out the profile list then “eselect profile set XX” where XX is the identifier for the profile that fits you

Info: If it’s happening that at first boot some services like netmount (or those that depends on net) refuse to start, maybe it’s a good idea to “emerge -qDuN world” and reboot the system.

For more about possible settings read this.

Transfer the kernel to boot partition

So the kernel it’s working and we want to transfer it to the boot partition, on our disk image. The kernel is named Image and can be found on the same path as zImage, i.e. arch/arm/boot/Image. Just copy it to our mount point (remember?) $RPI_ROOT/boot/ with a name like kernel.img.

You should also create a file called /boot/cmdline.txt with the following content:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 kernel=kernel.img root=/dev/mmcblk0p3 rootfstype=ext4 elevator=deadline rootwait

Pay attention to the kernel name, root partition name and file system type. They should match those names/type that you have configured at the previous steps.

You could also have a file called /boot/config.txt where you can define various settings for your real R-Pi machine (doesn’t work with qemu emulator, though):

# uncomment if you get no picture on HDMI for a default "safe" mode

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border

# uncomment to force a console size. By default it will be display's size minus
# overscan.

# uncomment if hdmi display is not detected and composite is being output

# uncomment to force a specific HDMI mode (this will force VGA)

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display

# uncomment for composite PAL

#uncomment to overclock the arm. 700 MHz is the default.

# for more options see http://elinux.org/RPi_config.txt

We need to transfer the kernel modules too, so just copy the folder ../modules/lib to the root mount point ($RPI_ROOT):

cp -r ../modules/lib/ $RPI_ROOT

We need one more thing: the Raspberry Pi firmware. Once you got it we have to copy these three files {bootcode.bin,fixup.dat,start.elf} in the $RPI_ROOT/boot:

cp firmware/boot/{bootcode.bin,fixup.dat,start.elf} $RPI_ROOT/boot/

Next we have to copy the VC library from the firmware to the /opt folder inside the rpi:

cp -r firmware/hardfp/opt $RPI_ROOT/

Transfer the disk image to your SDHC

Since your disk image is ready, we have to transfer it to your SDHC card. First, make sure that all mount points to that disk image are unmounted then run the following command:

dd if=$RPI_IMG of=/dev/sdX bs=4096 count=$((2*1000*1000*1000/4096)) #where X is your SHDC device

BTW: if you don’t have a SD-CARD reader on your system (where you’ve prepared the R-Pi disk image) like me, you could get an USB2 SD/MMC/RS-MMC SDHC memory card reader (1..16 GB) from eBay for only 1$.

First live test

At this point our Raspberry Pi is ready for a live test! If you want to tune the R-Pi even more then check this link.

If everything went fine then you should be able to get a first impression of your Pi:

rpi-gentoo ~ # cat /proc/cpuinfo 
Processor    : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS    : 697.95
Features    : swp half thumb fastmult vfp edsp java tls 
CPU implementer    : 0x41
CPU architecture: 7
CPU variant    : 0x0
CPU part    : 0xb76
CPU revision    : 7

Hardware    : BCM2708
Revision    : 000f
Serial      : 00000000xxxxxxxx

If 700MHz is not enough for you then you might try to overclock you R-Pi.

Since R-Pi does not have an Real Time Clock (source Wikipedia):

The Raspberry Pi does not come with a real-time clock,[7] so an OS must use a network time server, or ask the user for time information at boot time to get access to time and date for file time and date stamping. However, a real-time clock (such as the DS1307) with battery backup can be added via the I²C interface.

we have to provide a mean by which Linux will synchronize its date/time every time you boot the R-Pi (unless you wish to enter that manually every time you boot the R-Pi). I am using openntpd at this time and its installation/setup is straightforward:

emerge -q openntpd

After installation make sure to edit /etc/conf.d/ntpd and to uncomment the NTPD_OPTS=”-s” option. Moreover,¬† edit the /etc/ntpd.conf file and add some Internet public NTP Pool Servers to it such way that your little daemon can feed itself with the current date/time every time you reboot your system:

server 0.se.pool.ntp.org
server 1.se.pool.ntp.org
server 2.se.pool.ntp.org
server 3.se.pool.ntp.org

Once everything is done, remember to add the ntpd service to the RC-init system such it starts automatically at boot:

rc-update add ntpd default

Make sure you have installed the sudo program, if not then install it right away:

emerge -qDuN sudo

Don’t forget to add your user to /etc/sudoers file:

my-user-name ALL=(ALL) ALL

That would be all. Have fun!

Btw: one interesting thing about this Gentoo on R-Pi is that with the following services it eats only 24MB of RAM, while the SDHC card is only 1.3G:

 udev-mount              [  started  ]
 sysfs                   [  started  ]
 devfs                   [  started  ]
 dmesg                   [  started  ]
 udev                    [  started  ]
 hwclock                 [  started  ]
 swap                    [  started  ]
 modules                 [  started  ]
 fsck                    [  started  ]
 root                    [  started  ]
 mtab                    [  started  ]
 localmount              [  started  ]
 sysctl                  [  started  ]
 bootmisc                [  started  ]
 urandom                 [  started  ]
 net.lo                  [  started  ]
 termencoding            [  started  ]
 tmpfiles.setup          [  started  ]
 hostname                [  started  ]
 keymaps                 [  started  ]
 procfs                  [  started  ]
 swapfiles               [  started  ]
 net.eth0                [  started  ]
 netmount                [  started  ]
 sshd                    [  started  ]
 local                   [  started  ]

This sound very promising and it means that I still have ~480M RAM for my projects (games, X11, etc). I made a copy its disk (2G SDHC) image and I shared to the public domain. You can download it from here, it is only 475M (lzma compression; root pwd=rpi) and includes, in addition to the default Gentoo stage3, sudo, screen, gentoolkit, openntpd, distcc, ncdu (which I found them very useful by default).

Next stop…Python!

Posted in distro, kernel, kvm, linux, Raspberry Pi | Tagged , , | 3 Comments

Download and untar on the fly

Please note that this blog has been moved.

Now it has its own domain: mynixworld.info ūüôā

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

Many times I download an archive, extract it somewhere then remove the archive file from the disk (it’s just garbage). I have wondered if it’s possible to do everything on the fly, by one command only:

wget -O - http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.3.tar.bz2|tar -xjf -

So what I did was to download the tar.bz2 archive (wget) to the stdout file (-O -) and then, by using the Linux pipe (|), I’ve redirected the standard output and I’ve used it like an input (-) for the tar command.

Posted in linux | Tagged , , | Leave a comment