Place Ghantoos

This article has been updated on July 2nd 2009.

As I worked on packaging a project I am working on (Limited Shell) I found my self reading many tutorials on how to build debian packages. But none where related to distutils setup.py. As a nice setup.py natively knows how to package RPM, I thought about a way to include setup.py in a .deb generation.

Here is a small guide to help generate a debian package from a distutils/setuptools setup.py.

The main steps of this procedure are:
1- make sure that your setup.py is functional
2- generate a GPG key to sign your package and a public key to put on your server or wherever
3- create the files that are useful for the debian packaging:
* debian/changelog
* debian/compat
* debian/control
* debian/copyright
* debian/rules
* debian/watch
4- write the Makefile
5- generate your debian package
6- check your files’ licensing
7- check your new package using lintian
8- Conclusion
9- Links

BEFORE STARTING: All the files needed by setup.py and setup.py itself are placed in a directory called project:

zaatar:~/projects/myproject-0.2.0$ ls -l
total 104
-rwxr-xr-x 1 ghantoos ghantoos  1502 2008-10-19 16:28 CHANGES
-rw-r--r-- 1 ghantoos ghantoos 35147 2008-10-19 16:28 COPYING
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory1
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory2
-rw-r--r-- 1 ghantoos ghantoos    30 2008-10-19 16:28 file1.py
-rw-r--r-- 1 ghantoos ghantoos   135 2008-10-19 16:28 MANIFEST.in
-rw-r--r-- 1 ghantoos ghantoos  5444 2008-10-19 16:28 README
-rwxr-xr-x 1 ghantoos ghantoos   989 2008-10-19 16:28 setup.py

1- Make sure that your setup.py is functional

In this part, we will assume that this part is already done and functional.
You can test your setup.py using the sdist parameter

zaatar:~/projects/myproject-0.2.0$ python setup.py sdist

This should generate a tar.gz file in the dist/ directory.
To check the consistency of the tar file just issue:

zaatar:~/projects/myproject-0.2.0$ tar tvfz dist/yourproject-version.tar.gz

and make sure all your files are present.

For more information about how to write a setup.py file please refer to: http://www.python.org/doc/2.5.2/dist/setup-script.html
or just go to code.google.com type in ‘setup.py’ and have fun.

You could also take a look at the setup.py I use for Limited Shell here: http://lshell.cvs.sourceforge.net/viewvc/lshell/lshell/

2- Generate a GPG key

The generation of your GPG key will be used to sign the packages you will create.
Note that you can skip this part, and still generate your package. You just have some warnings from dpkg-buildpackage telling you:

dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [builddeb] Error 1

No need to worry, the debian package will be created anyways. BUT, let’s try to do it right, and have our own GPG key (classy!).
To generate your key, you’ll have to answer some basic questions:

zaatar:~/projects/myproject-0.2.0$ gpg --gen-key
gpg (GnuPG) 1.4.6; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1  <-- default
What keysize do you want? (2048) <-- default
Please specify how long the key should be valid.
      0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) <-- default
Key does not expire at all
Is this correct? (y/N) y <--
Real name: Ignace Mouzannar -ghantoos- <-- BE CAREFUL HERE
Email address: ghantoos@ghantoos.org <-- BE CAREFUL HERE
Comment:
You selected this USER-ID:
    "Ignace Mouzannar -ghantoos- "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
Enter passphrase:  <-- I say you better put one so that none can generate a package using your ID

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
++++++++++.++++++++++++++++++++++++++++++++++++++++.++++++++++++++++++++++++++++++.+++++.+++++++++
+++++++++++++++++++++.+++++.+++++++++++++++...............>..+++++......................<+++++....
.>+++++......................................................>.+++++.....+++++

(...)
public and secret key created and signed.

IMPORTANT NOTE: You must remember exactly what you have entered in ‘Real name’ and ‘Email address’ fields as they *HAVE* to be identical to the ones you will have to enter in the last entry of the changelog.

To accelerate the GPG generation, make your PC work on something else this gives the random number generator a better chance to gain enough entropy.
Once your done, you can see your GPG key in ~/.gnugp/:

zaatar:~/projects/myproject-0.2.0$ ls ~/.gnupg/
pubring.gpg  random_seed  secring.gpg  trustdb.gpg

You will now be able to sign your newly created debian package. In order to distribute your .deb you need to created a public key that you put wherever (send it by mail, put in on your webserver, bla.):

zaatar:~/projects/myproject-0.2.0$ gpg --armor --output .gnupg/ghantoos-pubkey.gpg --export 'Ignace Mouzannar -ghantoos-'

Note that I am exporting the ‘Ignace Mouzannar -ghantoos-‘ key.

In order to install your package, users will have to import your GPG key, as root issue the following:

cat ghantoos-pubkey.gpg | gpg --import

3- Create the files that are useful for the debian packaging

This the heart of the subject. As setup.py does a big part of the installing job, we will just have to edit/create the mandatory files needed by packaging process:
– debian/changelog
– debian/compat
– debian/control
– debian/copyright
– debian/rules
– debian/watch

Before starting, let’s install the necessary applications:

zaatar:~$ sudo apt-get install build-essential dpkg-dev debhelper devscripts fakeroot cdbs

The next thing to do is create a debian directory in order to place our debian packaging files there:

zaatar:~/projects/myproject-0.2.0$ mkdir debian

At the end of this section, the debian directory must only contain the 5 files mentioned below:

zaatar:~/projects/myproject-0.2.0$ ls -l debian/
total 20
-rw-r--r-- 1 ghantoos ghantoos 1923 2008-10-19 16:28 changelog
-rw-r--r-- 1 ghantoos ghantoos    2 2008-10-19 16:28 compat
-rw-r--r-- 1 ghantoos ghantoos  482 2008-10-19 16:28 control
-rw-r--r-- 1 ghantoos ghantoos 1091 2008-10-19 16:28 copyright
-rwxr-xr-x 1 ghantoos ghantoos 1910 2008-10-19 16:28 rules

debian/changelog

This file is very important. As I noted in the previous section, the name of the maintainer must be exactly the same as the one entered for the GPG key.
To create a generic debian changelog file, you must:

zaatar:~/projects/myproject-0.2.0$ DEBFULLNAME=ghantoos; export DEBFULLNAME
zaatar:~/projects/myproject-0.2.0$ DEBEMAIL=ghantoos@ghantoos.org; export DEBEMAIL
zaatar:~/projects/myproject-0.2.0$ debchange --create --package myprojectname -v 0.2.0

This will generate the following file:

zaatar:~/projects/myproject-0.2.0$ cat debian/changelog
myprojectname (0.2.0-1) UNRELEASED; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- ghantoos   Sun, 19 Oct 2008 16:23:47 +0200

You can continue using debchange (man debchange) to edit you changelog or just edit it manually.
In case you chose to edit changlog manually, make sure your indentation is correct (2 spaces before the *, and 1 space before the signature+date) as this file is used by the building process.
To change the ‘UNRELEASED’ to the proper distribution naming, just issue from inside the project folder:

zaatar:~/projects/myproject-0.2.0$ debchange -r
zaatar:~/projects/myproject-0.2.0$ cat debian/changelog
myprojectname (0.2.0-1) unstable; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- ghantoos   Sun, 19 Oct 2008 16:39:48 +0200

In case you are working on an official debian package, the initial release must close the ITP bug number.

For more details concerning the changelog file, please refer to the debian policy:
http://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog

debian/compat

The compat file must only contain a single line mentionning the debhelper compatibility level of the package. In my case ‘5’. If you want to know which version of debhelper you have, look at the end of its manpage:

zaatar:~/projects/myproject-0.2.0$ dpkg -p debhelper | grep Version
Version: 7.2.8
zaatar:~/projects/myproject-0.2.0$ cat debian/compat
7

debian/control

As it is written in the debian-policy, the debian/control file contains the most vital (and version-independent) information about the source package and about the binary packages it creates.
In this file, you will have to fill in the following package metadata information: Source, Section, Priority, Maintainer, Homepage, Build-Depends, Standards-Version, Package, Architecture, Depends, Description. Of course, this list is not exhaustive.
Here is what this file should look like:

zaatar:~/projects/myproject-0.2.0$ cat debian/control
Source: myprojectname
Section: shells
Priority: optional
Maintainer: Ignace Mouzannar (ghantoos) 
Build-Depends: debhelper (>=7.0.50~), python-support (>= 0.6), cdbs (>= 0.4.49), python-all-dev
XS-Python-Version: >=2.4
Standards-Version: 3.8.4

Package: myprojectname
Architecture: all
Homepage: http://yourprojectshomepade.com/
XB-Python-Version: ${python:Versions}
Depends: ${misc:Depends}, ${python:Depends}
Description: Blah python module that adds blah and bleh (this must not exceed a single line (i.e 60 characters)
 Here you should put a long description your package. This line *MUST* (this is
 faor default printing) begin with a single space.
 Look for more information about this:
 http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Description

All the fields above are quite intuitive. I am just going to comment the Standards-Version field. As written in the debian policy, it is the most recent version of the standards (the policy manual and associated texts) with which the package complies.
To know which version of the newest debian-policy, issue the following:

zaatar:~/projects/myproject-0.2.0$ apt-cache show debian-policy | grep Version
Version: 3.8.1.0

For the rest, you can refer to:
http://www.debian.org/doc/debian-policy/ch-controlfields.html
http://debathena.mit.edu/packaging/#control

debian/copyright

I didn’t go into many different copyright cases, just my beloved GPL. Here is the content of the Limited Shell copyright file:

zaatar:~/projects/myproject-0.2.0$ cat debian/copyright
This package was debianized by Ignace Mouzannar (ghantoos)
 on Sat, 18 Oct 2008 20:13:44 +0200.

It was downloaded from http://sourceforge.net/projects/yourproject

Upstream Author:

    Ignace Mouzannar (ghantoos) 

Files: *
Copyright:
    2008-2009, Ignace Mouzannar
License: GPL

Files: debian/*
Copyright:
    2008-2009, Ignace Mouzannar
License: GPL

License: GPL
   This package is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 2 of the License, or
   (at your option) any later version.

   This package is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this package; if not, see .

On Debian systems, the complete text of the GNU General
Public License can be found in `/usr/share/common-licenses/GPL-3'.

Note: if using you are using a GPL license, a COPYING file containing the GPL license text should be available in your source package. Be careful not to add it in your debian package, as debian has local copies of the licenses. As seen at the end of the copyriht file above, it should point to the local license copy (e.g. /usr/share/common-licenses/GPL-3)

For more information about the copyright file, please refer to the debian policy:
http://www.debian.org/doc/debian-policy/ch-docs.html#s-copyrightfile

debian/rules

The rules file is the debian Makefile used to generate your .deb package.
At first, I used dh_make to generate my rules template, and then removed all the parts that aren’t usefull for the setup.py .deb build.
Since debhelper (>= 7.0.50~) you can use dh7 features that greatly simplifies the rules file:

#!/usr/bin/make -f
# -*- makefile -*-

%:
    dh $@

Another option, is to use CDBS, as Tobu suggested.

You can still find the old rules file here.

Be careful, this file must be executable.
Here is its content:

#!/usr/bin/make -f
# -*- makefile -*-

DEB_PYTHON_SYSTEM := pysupport

include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/python-distutils.mk

clean::
    rm -rf build build-stamp configure-stamp build/ MANIFEST
    dh_clean

In this Makefile, I use CDBS templates that will call the debhelper functions (i.e. dh_*) needed to package distutils based python projects.
If you are curious to see what is really done, you can check out the old rules file or read carefully the output when building your package.

Note: dh_pysupport changes the setup.py initial module installation directory from /usr/lib/pythonX.X/site-packages/ to /usr/share/pyshared/lshell.py. Then it does a symbolic link of the py file in /usr/lib/pythonX.X/site-packages/ and bytecompile here (see man dh_pysupport for more information)

debian/watch

If you intend to create an official debian package, you have to create the debian/watch file which is used to scan upstream sources for new releases of software.
All the needed information can be found in the man of the uscan utility: man uscan.
Here the watch file of the lshell project:

version=3
http://sf.net/lshell/lshell-([^2].+)\.tar\.gz

It checks for all the sources that are available on the sourceforge site, that do not start with “2” (that’s because of some bad versioning of the first releases (e.g. 2008-01-31) )
To test if your watch file works:

zaatar:~/projects/myproject-0.2.0$ uscan --no-symlink --verbose --no-download
-- Scanning for watchfiles in .
-- Found watchfile in ./debian
-- In debian/watch, processing watchfile line:
   http://sf.net/myproject/myproject-(.+)\.tar\.gz
-- Found the following matching hrefs:
     myproject-0.1.0.tar.gz
     myproject-0.1.1.tar.gz
     myproject-0.1.9.tar.gz
Newest version on remote site is 0.1.9, local version is 0.2.0
 => remote site does not even have current version
-- Scan finished

4- Write the Makefile

This is the final step we’ll need to fill before building our package.
The following Makefile must be saved in your projects/projectname-0.2.0 directory (next to setup.py).
Here are the steps that will be written in the Makefile:
1- Create the source tar.gz then rename it to projectname_version.orig.tar.gz (e.g. myproject_0.2.0.orig.tar.gz)
2- run dpkg-buildpackage with -rfakeroot as argument to launch the building process as none root user (this is actually the default behaviour of dpkg-buildpackage, but I thought it was worth mentioning)

Here is the Makefile I used to package lshell:

zaatar:~/projects/myproject-0.2.0$ cat Makefile
# $Id: Makefile,v 1.6 2008/10/29 01:01:35 ghantoos Exp $
#

PYTHON=`which python`
DESTDIR=/
BUILDIR=$(CURDIR)/debian/myprojectname
PROJECT=myprojectname
VERSION=0.2.0

all:
        @echo "make source - Create source package"
        @echo "make install - Install on local system"
        @echo "make buildrpm - Generate a rpm package"
        @echo "make builddeb - Generate a deb package"
        @echo "make clean - Get rid of scratch and byte files"

source:
        $(PYTHON) setup.py sdist $(COMPILE)

install:
        $(PYTHON) setup.py install --root $(DESTDIR) $(COMPILE)

buildrpm:
        $(PYTHON) setup.py bdist_rpm --post-install=rpm/postinstall --pre-uninstall=rpm/preuninstall

builddeb:
        # build the source package in the parent directory
        # then rename it to project_version.orig.tar.gz
        $(PYTHON) setup.py sdist $(COMPILE) --dist-dir=../ --prune
        rename -f 's/$(PROJECT)-(.*)\.tar\.gz/$(PROJECT)_$$1\.orig\.tar\.gz/' ../*
        # build the package
        dpkg-buildpackage -i -I -rfakeroot

clean:
        $(PYTHON) setup.py clean
        $(MAKE) -f $(CURDIR)/debian/rules clean
        rm -rf build/ MANIFEST
        find . -name '*.pyc' -delete

Note: If you choose to use the old rules file, as Seb has commented below, the debian policy clearly states that a python debian package must not include its corresponding bytecompiled modules. For that, the –no-compile flag has been added, to prevent setup.py from generating the bytecompiled (.pyc) files, and let dh_pysupport take care of this. Thank you Seb for your very useful input.. This flag will be set to –no-compile in the old rules file (to keep the install part clean if you ever want to use it straight from the Makefile).
Note: As specified in the debian policy, a package may need to work as root to buid its environment. In order to do this without using sudo or su before building, you can add -rfakeroot to dpkg-buildpackage in the Makefile. Thank you Stephbul for the tip! (you can find more info in man dpkg-buildpackage in the -rgain-root-command section)

5- Generate your debian package

It’s now time to test the package building.
Your directory should now contain:

zaatar:~/projects/myproject-0.2.0$ ls -lR
.:
total 112
-rwxr-xr-x 1 ghantoos ghantoos  1502 2008-10-19 16:28 CHANGES
-rw-r--r-- 1 ghantoos ghantoos 35147 2008-10-19 16:28 COPYING
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 debian
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory1
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory2
-rw-r--r-- 1 ghantoos ghantoos    30 2008-10-19 16:28 file1.py
-rw-r--r-- 1 ghantoos ghantoos  1026 2008-10-19 16:28 Makefile
-rw-r--r-- 1 ghantoos ghantoos   135 2008-10-19 16:28 MANIFEST.in
-rw-r--r-- 1 ghantoos ghantoos  5444 2008-10-19 16:28 README
-rwxr-xr-x 1 ghantoos ghantoos   989 2008-10-19 16:28 setup.py

./debian:
total 28
-rw-r--r-- 1 ghantoos ghantoos 1923 2008-10-19 19:04 changelog
-rw-r--r-- 1 ghantoos ghantoos    2 2008-10-19 19:04 compat
-rw-r--r-- 1 ghantoos ghantoos  553 2008-10-19 19:04 control
-rw-r--r-- 1 ghantoos ghantoos 1023 2008-10-19 19:04 copyright
-rwxr-xr-x 1 ghantoos ghantoos 1592 2008-10-19 19:04 rules

./directory1:
(...)

./directory2:
(...)

Just run the following to build your package:

zaatar:~/projects/myproject-0.2.0$ make builddeb
`which python` setup.py sdist --dist-dir=../ --prune
running sdist
reading manifest template 'MANIFEST.in'
writing manifest file 'MANIFEST'
(..)
dh_installdirs -plshell
cd . && python setup.py install --root=/home/ghantoos/Desktop/python/lshell/lshell-0.9.4/debian/lshell/ --no-compile -O0
(...)
You need a passphrase to unlock the secret key for
user: "Ignace Mouzannar "
1024-bit DSA key, ID 85C218BA, created 2008-10-19
(...)
dpkg-buildpackage: full upload (original source is included)

If you haven’t created a GPG key, warning will appear, but the deb will be created anyways.
If you have added a passphrase to your GPG key you will be prompt twice: to sign the .dsc et .changes files.

Your debian package should appear in the parent directory (i.e. ../):

zaatar:~/projects/myproject-0.2.0$ ls -l ../
drwxr-xr-x 11 ghantoos ghantoos   4096 2009-05-25 17:06 myprojectname_0.2.0
-rw-r--r--  1 ghantoos ghantoos  25114 2009-05-25 17:06 myprojectname_0.2.0-1_all.deb
-rw-r--r--  1 ghantoos ghantoos   6411 2009-05-25 17:06 myprojectname_0.2.0-1.diff.gz
-rw-r--r--  1 ghantoos ghantoos   1033 2009-05-25 17:06 myprojectname_0.2.0-1.dsc
-rw-r--r--  1 ghantoos ghantoos   1734 2009-05-25 17:06 myprojectname_0.2.0-1_i386.changes
-rw-r--r--  1 ghantoos ghantoos  31822 2009-05-25 17:06 myprojectname_0.2.0.orig.tar

Note: the diff.tar.gz file should (only) contain the content of the debian/* files.

In order to clean up the files generated by the building process, just issue:

zaatar:~/projects/myproject-0.2.0$ make clean
`which python` setup.py clean
running clean
make -f /home/ghantoos/projects/myproject-0.2.0/debian/rules clean
make[1]: Entering directory `/home/ghantoos/projects/myproject-0.2.0'
test -x debian/rules
dh_clean
cd . && python setup.py clean -a
running clean
removing 'build/lib' (and everything under it)
'build/bdist.linux-i686' does not exist -- can't clean it
removing 'build/scripts-2.5' (and everything under it)
removing 'build'
rm -rf debian/python-module-stampdir
find . -name '*.pyc' -exec rm '{}' ';'
rm -rf build build-stamp configure-stamp build/ MANIFEST
dh_clean
make[1]: Leaving directory `/home/ghantoos/projects/myproject-0.2.0'
rm -rf build/ MANIFEST
find . -name '*.pyc' -delete

6- Check your files’ licensing

If you intend to create an official package, you must make sure all license statements are set correctly.
Debian offers a tool to check the files included in you package.

zaatar:~/projects$ licensecheck -r *
myproject-0.2.0/example.py: GPL (v3 or later)
myproject-0.2.0/setup.py: GPL (v3 or later)
myproject-0.2.0/myproject.py: GPL (v3 or later)

7- Check your new package using lintian

In order to check possible errors/warning in the building process, the tool to use is lintian.
It will look for common packaging errors, and output the warnings. Just issue the following:
If you want to be lazy (this is the minimum):

zaatar:~/projects/myproject-0.2.0$ lintian -Ivi ../myprojectname_0.2.0_i386.changes

If you want to make it clean:

zaatar:~/projects/myproject-0.2.0$ lintian -IviE --display-experimental --pedantic -L ">=wishlist" --color auto --show-overrides --checksums ../myprojectname_0.2.0_i386.changes

This is the last step towards finalizing your debian package.

8- Conclusion

This is just a small howto that can be used as a first step into debian packaging.
For more complete/up-to-date information you should take a look at the the debian-policy, the debian new maintainers’ guide and the developer’s reference.

I hope this can be helpful.

Cheers,

Ignace M -ghantoos-

9- Links

Other than the sources mentionned inline, here are some other very useful links:
http://www.madboa.com/geek/gpg-quickstart/
http://www.debian-administration.org/articles/336
http://www.debian.org/doc/debian-policy/index.html
http://www.debian.org/doc/manuals/maint-guide/
http://www.debian.org/doc/manuals/developers-reference/
https://wiki.ubuntu.com/PackagingGuide/Basic#Packaging%20From%20Scratch
http://www.debian-administration.org/articles/174
Limited shell CVS where you can find all my building files
http://build-common.alioth.debian.org/cdbs-doc.html