Place Ghantoos

As I worked on packaging a project I am working on (Limited Shell) I found my self reading many tutorials on how to build debian packages. But none where related to distutils setup.py. As a nice setup.py natively knows how to package RPM, I thought about a way to include setup.py in a .deb generation.

Here is a small guide to help generate a debian package from a distutils/setuptools setup.py.

The main steps of this procedure are:
1- make sure that your setup.py is functional
2- generate a GPG key to sign your package and a public key to put on your server or wherever
3- create the files that are useful for the debian packaging:
* changelog
* compat
* control
* copyright
* rules
4- write the Makefile
5- generate your debian package
6- check your new package using lintian

BEFORE STARTING: All the files needed by setup.py and setup.py itself are placed in a directory called project:

ghantoos@raoul-home:~/projects/myproject$ ls -l
total 104
-rwxr-xr-x 1 ghantoos ghantoos  1502 2008-10-19 16:28 CHANGES
-rw-r--r-- 1 ghantoos ghantoos 35147 2008-10-19 16:28 COPYING
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory1
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory2
-rw-r--r-- 1 ghantoos ghantoos    30 2008-10-19 16:28 file1.py
-rw-r--r-- 1 ghantoos ghantoos   135 2008-10-19 16:28 MANIFEST.in
-rw-r--r-- 1 ghantoos ghantoos  5444 2008-10-19 16:28 README
-rwxr-xr-x 1 ghantoos ghantoos   989 2008-10-19 16:28 setup.py

1- Make sure that your setup.py is functional

In this part, we will assume that this part is already done and functional.
You can test your setup.py using the sdist parameter

ghantoos@raoul-home:~/projects/myproject$ python setup.py sdist

This should generate a tar.gz file in the dist/ directory.
To check the consistency of the tar file just issue:

ghantoos@raoul-home:~/projects/myproject$ tar tvfz dist/yourproject-version.tar.gz

and make sure all your files are present.

For more information about how to write a setup.py file please refer to: http://www.python.org/doc/2.5.2/dist/setup-script.html
or just go to code.google.com type in ’setup.py’ and have fun.

You could also take a look at the setup.py I use for Limited Shell here: http://lshell.cvs.sourceforge.net/viewvc/lshell/lshell/

2- Generate a GPG key

The generation of your GPG key will be used to sign the packages you will create.
Note that you can skip this part, and still generate your package. You just have some warnings from dpkg-buildpackage telling you:

dpkg-buildpackage: warning: Failed to sign .dsc and .changes file
make: *** [builddeb] Error 1

No need to worry, the debian package will be created anyways. BUT, let’s try to do it right, and have our own GPG key (classy!).
To generate your key, you’ll have to answer some basic questions:

ghantoos@raoul-home:~/projects/myproject$ gpg --gen-key
gpg (GnuPG) 1.4.6; Copyright (C) 2006 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? 1  <-- default
What keysize do you want? (2048) <-- default
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) <-- default
Key does not expire at all
Is this correct? (y/N) y <--
Real name: Ignace Mouzannar -ghantoos- <-- BE CAREFUL HERE
Email address: ghantoos@ghantoos.org <-- BE CAREFUL HERE
Comment:
You selected this USER-ID:
    "Ignace Mouzannar -ghantoos- "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
Enter passphrase:  <-- I say you better put one so that none can generate a package using your ID

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
++++++++++.++++++++++++++++++++++++++++++++++++++++.++++++++++++++++++++++++++++++.+++++.+++++++++
+++++++++++++++++++++.+++++.+++++++++++++++...............>..+++++......................<+++++....
.>+++++......................................................>.+++++.....+++++

(...)
public and secret key created and signed.

IMPORTANT NOTE: You must remember exactly what you have entered in ‘Real name’ and ‘Email address’ fields as they *HAVE* to be identical to the ones you will have to enter in the last entry of the changelog.

To accelerate the GPG generation, make your PC work on something else this gives the random number generator a better chance to gain enough entropy.
Once your done, you can see your GPG key in ~/.gnugp/:

ghantoos@raoul-home:~/projects/myproject$ ls ~/.gnupg/
pubring.gpg  random_seed  secring.gpg  trustdb.gpg

You will now be able to sign your newly created debian package. In order to distribute your .deb you need to created a public key that you put wherever (send it by mail, put in on your webserver, bla.):

ghantoos@raoul-home:~/projects/myproject$ gpg --armor --output .gnupg/ghantoos-pubkey.gpg --export 'Ignace Mouzannar -ghantoos-'

Note that I am exporting the ‘Ignace Mouzannar -ghantoos-’ key.

In order to install your package, users will have to import your GPG key, as root issue the following:

cat ghantoos-pubkey.gpg | gpg --import

3- Create the files that are useful for the debian packaging

This the heart of the subject. As setup.py does a big part of the installing job, we will just have to edit/create the mandatory files needed by packaging process:
- changelog
- compat
- control
- copyright
- rules

Before starting, let’s install the necessary applications:

ghantoos@raoul-home:~$ sudo apt-get install build-essential dpkg-dev debhelper devscripts fakeroot

The next thing to do is create a debian directory in order to place our debian packaging files there:

ghantoos@raoul-home:~/projects/myproject$ mkdir debian

At the end of this section, the debian directory must only contain the 5 files mentioned below:

ghantoos@raoul-home:~/projects/myproject$ ls -l debian/
total 20
-rw-r--r-- 1 ghantoos ghantoos 1923 2008-10-19 16:28 changelog
-rw-r--r-- 1 ghantoos ghantoos    2 2008-10-19 16:28 compat
-rw-r--r-- 1 ghantoos ghantoos  482 2008-10-19 16:28 control
-rw-r--r-- 1 ghantoos ghantoos 1091 2008-10-19 16:28 copyright
-rwxr-xr-x 1 ghantoos ghantoos 1910 2008-10-19 16:28 rules

debian/changelog

This file is very important. As I noted in the previous section, the name of the maintainer must be exactly the same as the one entered for the GPG key.
To create a generic debian changelog file, you must:

ghantoos@raoul-home:~/projects/myproject$ DEBFULLNAME=ghantoos; export DEBFULLNAME
ghantoos@raoul-home:~/projects/myproject$ DEBEMAIL=ghantoos@ghantoos.org; export DEBEMAIL
ghantoos@raoul-home:~/projects/myproject$ debchange --create --package myprojectname -v 0.2

This will generate the following file:

ghantoos@raoul-home:~/projects/myproject$ cat debian/changelog
myprojectname (0.2) UNRELEASED; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- ghantoos   Sun, 19 Oct 2008 16:23:47 +0200

You can continue using debchange (man debchange) to edit you changelog or just edit it manually.
In case you chose to edit changlog manually, make sure your indentation is correct (2 spaces before the *, and 1 space before the signature+date) as this file is used by the building process.
To change the ‘UNRELEASED’ to the proper distribution naming, just issue from inside the project folder:

ghantoos@raoul-home:~/projects/myproject$ debchange -r
ghantoos@raoul-home:~/projects/myproject$ cat debian/changelog
myprojectname (0.2) unstable; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- ghantoos   Sun, 19 Oct 2008 16:39:48 +0200

In case your are using Ubuntu, the distribution name would be the name of your ubuntu release (e.g. hardy)

For more details concerning the changelog file, please refer to the debian policy:
http://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog

debian/compat

The compat file must only contain a single line mentionning the debhelper compatibility level of the package. In my case ‘5′. If you want to know what version of debhelper you have, look at the end of its manpage:

ghantoos@raoul-home:~/projects/myproject$ man debhelper
(...)
5.0.42                  2006-11-12                      debhelper(7)
ghantoos@raoul-home:~/projects/myproject$ cat debian/compat
5

debian/control

As it is written in the debian-policy, the debian/control file contains the most vital (and version-independent) information about the source package and about the binary packages it creates.
In this file, you will have to fill in the following package metadata information: Source, Section, Priority, Maintainer, Homepage, Build-Depends, Standards-Version, Package, Architecture, Depends, Description. Of course, this list is not exhaustive.
Here is what this file should look like:

ghantoos@raoul-home:~/projects/myproject$ cat debian/control
Source: myprojectname
Section: shells
Priority: extra
Maintainer: Ignace Mouzannar -ghantoos-
Build-Depends: python (>=2.4), debhelper, python-central
XS-Python-Version: >=2.4
Standards-Version: 3.7.2

Package: myprojectname
Architecture: any
XB-Python-Version: ${python:Versions}
Depends: python (>=2.4), python-central
Description: Limited Shell (this must not exceed a single line (i.e 60 characters)
 Here you should put a long description your package. This line *MUST* (this is faor default printing) begin with a single space.
 Look for more information about this: http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Description

All the fields above are quite intuitive. I am just going to comment the Standards-Version field. As written in the debian policy, it is the most recent version of the standards (the policy manual and associated texts) with which the package complies.
To know what version your system is running, issue the following:

ghantoos@raoul-home:~/projects/myproject$ apt-cache show debian-policy | grep Version
Version: 3.7.3.0

For the rest, you can refer to:
http://www.debian.org/doc/debian-policy/ch-controlfields.html
http://debathena.mit.edu/packaging/#control

debian/copyright

I didn’t go into many different copyright cases, just my beloved GPL. Here is the content of the Limited Shell copyright file:

ghantoos@raoul-home:~/projects/myproject$ cat debian/copyright
This package was debianized by Ignace Mouzannar -ghantoos-  on
Sat, 18 Oct 2008 20:13:44 +0200.

License:

   This package is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 2 of the License, or
   (at your option) any later version.

   This package is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with this package; if not, see .

On Debian systems, the complete text of the GNU General
Public License can be found in `/usr/share/common-licenses/GPL'.

The Debian packaging is (C) 2008, Ignace Mouzannar -ghantoos-  and
is licensed under the GPL, see above.

Of course, if using you are using a GPL license, a COPYING file containing the GPL license text should be available in your package. This should be included in your setup.py.

For more information about the copyright file, please refer to the debian policy:
http://www.debian.org/doc/debian-policy/ch-docs.html#s-copyrightfile

debian/rules

The rules file is the debian makefile used to generate your .deb package.
I used dh_make to generate my rules template, and then removed all the parts that aren’t usefull for the setup.py .deb build.

You can download it here: http://ghantoos.org/misc/scripts/debian/rules
Be careful, this file must be executable.
Here is its content:

#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.
#
# $Id: rules,v 1.5 2008/10/25 14:22:05 ghantoos Exp $
#
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1

configure: configure-stamp
configure-stamp:
	dh_testdir
	# Add here commands to configure the package.

	touch configure-stamp

build: build-stamp

build-stamp: configure-stamp
	dh_testdir

	# Add here commands to compile the package.
	$(MAKE)

	touch $@

clean:
	dh_testdir
	rm -f build-stamp configure-stamp
	dh_clean 

install: build
	# Add here commands to install the package into debian/myprojectname
	# refer to Makefile
	$(MAKE) DESTDIR=${DESTDIR} COMPILE=--no-compile install

# Build architecture-independent files here.
binary-indep: build install
# We have nothing to do by default.

# Build architecture-dependent files here.
binary-arch: build install
	dh_testroot
	dh_installchangelogs CHANGES
	dh_pycentral
	dh_link
	dh_strip
	dh_compress
	dh_fixperms
	dh_installdeb
	dh_shlibdeps
	dh_gencontrol
	dh_md5sums
	dh_builddeb

binary: binary-indep binary-arch
.PHONY: build clean binary-indep binary-arch binary install configure

The important line here above is:

$(MAKE) DESTDIR=$(CURDIR)/debian/YOURPROJECT COMPILE=--no-compile install

This will call the make file we are going to write in the next section forcing the destination directory.
The setup.py will be called by our make file. All your project’s files will be copied in the destination directory DESTDIR with their relative path.
The dpkg-buildpackage will then know what files need to be copied by the package and their respective correct destinations. Which will be useful for the install and uninstall processes of the debian package.

As Seb has commented below, the debian policy clearly states that a python debian package must not include its corresponding bytecompiled modules.
For that, the –no-compile flag has been added, to prevent setup.py from generating the bytecompiled (.pyc) files, and let dh_pycentral take care of this. Thank you Seb for your very useful input.

Note: dh_pycentral also changes the setup.py initial module installation directory from /usr/lib/pythonX.X/site-packages/ to /usr/share/pyshared/lshell.py. Then it does a symbolic link of the py file in /usr/lib/pythonX.X/site-packages/ and bytecompile here (see man dh_pycentral for more information)

4- Write the Makefile

This is the final step we’ll need to fill before building our package.
The following Makefile must be saved in your projects/projectname directory (next to setup.py).
Here are the steps that will be written in the makefile:
1- create the directory that will be used during the build procedure
2- run dpkg-buildpackage with -rfakeroot as argument to launch the building process as none root user

Here is the Makefile I used to package lshell:

ghantoos@raoul-home:~/projects/myproject$ cat Makefile
# $Id: Makefile,v 1.6 2008/10/29 01:01:35 ghantoos Exp $
#

PYTHON=`which python`
DESTDIR=/
BUILDIR=$(CURDIR)/debian/myprojectname
PROJECT=myprojectname
VERSION=0.2

all:
		@echo "make install - Install on local system"
		@echo "make buildrpm - Generate a rpm package"
		@echo "make builddeb - Generate a deb package"
		@echo "make clean - Get rid of scratch and byte files"

install:
		$(PYTHON) setup.py install --root $(DESTDIR) $(COMPILE)

buildrpm:
		$(PYTHON) setup.py bdist_rpm --post-install=rpm/postinstall --pre-uninstall=rpm/preuninstall

builddeb:
		mkdir -p ${BUILDIR}
		DESTDIR=$(BUILDIR) dpkg-buildpackage -rfakeroot

clean:
		$(PYTHON) setup.py clean
		$(MAKE) -f $(CURDIR)/debian/rules clean
		rm -rf build/ MANIFEST
		find . -name '*.pyc' -delete

Note: To refrain setup.py from including the .pyc files inside the built source distribution I’ve added the COMPILE flag to the Makefile. This flag will be set to –no-compile in the rules file (to keep the install part clean if you ever want to use it straight from the Makefile).
Note: As specified in the debian policy, a package may need to work as root to buid its environment. In order to do this without using sudo or su before building, you can add -rfakeroot to dpkg-buildpackage in the Makefile. Thank you Stephbul for the tip! (you can find more info in man dpkg-buildpackage in the -rgain-root-command section)

5- Generate your debian package

It’s now time to test the package building.
Your directory should now contain:

ghantoos@raoul-home:~/projects/myproject$ ls -lR
.:
total 112
-rwxr-xr-x 1 ghantoos ghantoos  1502 2008-10-19 16:28 CHANGES
-rw-r--r-- 1 ghantoos ghantoos 35147 2008-10-19 16:28 COPYING
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 debian
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory1
drwxr-xr-x 3 ghantoos ghantoos  4096 2008-10-19 16:28 directory2
-rw-r--r-- 1 ghantoos ghantoos    30 2008-10-19 16:28 file1.py
-rw-r--r-- 1 ghantoos ghantoos  1026 2008-10-19 16:28 Makefile
-rw-r--r-- 1 ghantoos ghantoos   135 2008-10-19 16:28 MANIFEST.in
-rw-r--r-- 1 ghantoos ghantoos  5444 2008-10-19 16:28 README
-rwxr-xr-x 1 ghantoos ghantoos   989 2008-10-19 16:28 setup.py

./debian:
total 28
-rw-r--r-- 1 ghantoos ghantoos 1923 2008-10-19 19:04 changelog
-rw-r--r-- 1 ghantoos ghantoos    2 2008-10-19 19:04 compat
-rw-r--r-- 1 ghantoos ghantoos  553 2008-10-19 19:04 control
-rw-r--r-- 1 ghantoos ghantoos 1023 2008-10-19 19:04 copyright
-rwxr-xr-x 1 ghantoos ghantoos 1592 2008-10-19 19:04 rules

./directory1:
(...)

./directory2:
(...)

Just run the following to build your package:

ghantoos@raoul-home:~/projects/myproject$ make builddeb
`which python` setup.py sdist
running sdist
reading manifest file 'MANIFEST'
(..)
mkdir -p deb/myprojectname-0.2/debian
cp dist/myprojectname-0.2.tar.gz deb
cd deb && tar xfz myprojectname-0.2.tar.gz
mv deb/myprojectname-0.2.tar.gz deb/myprojectname-0.2/
cp debian/* deb/myprojectname-0.2/debian/
cd deb/myprojectname-0.2 && dpkg-buildpackage
(...)
dpkg-buildpackage: full upload; Debian-native package (full source is included)

If you haven’t created a GPG key, warning will appear, but the deb will be created anyways.
If you have added a passphrase to your GPG key you will be prompt twice: to sign the .dsc et .changes files.

Your debian package should appear in the upper directory (../) (default binary directory):

ghantoos@raoul-home:~/projects/myproject$ ls -l ../
total 104
-rw-r--r-- 1 root root   563 2008-10-19 19:13 myprojectname_0.2.dsc
-rw-r--r-- 1 root root  1347 2008-10-19 19:13 myprojectname_0.2_i386.changes
-rw-r--r-- 1 root root 33200 2008-10-19 19:13 myprojectname_0.2_i386.deb
-rw-r--r-- 1 root root 51055 2008-10-19 19:13 myprojectname_0.2.tar.gz

In order to clean up the files generated by the building process, just issue:

ghantoos@raoul-home:~/projects/myproject$ make clean
`which python` setup.py clean
running clean
make -f /home/ghantoos/projects/myprojectname/debian/rules clean
make[1]: Entering directory `/home/ghantoos/projects/myprojectname'
dh_testdir
rm -f build-stamp configure-stamp
dh_clean
make[1]: Leaving directory `/home/ghantoos/projects/myprojectname'
rm -rf build/ MANIFEST
find . -name '*.pyc' -delete

6- Check your new package using lintian

In order to check possible errors/warning in the building process, the tool to use is lintian
Just issue the following:

ghantoos@raoul-home:~/projects/myproject$ lintian ../myprojectname_0.2_i386.changes

Then google the errors/warnings one by one, every warning is nicely documented. This is the last step towards finalizing your debian package.

Conclusion

I hope this can be helpful.

Cheers,

ghantoos

Links

Other than the sources mentionned inline, here are some other very useful links:
http://www.madboa.com/geek/gpg-quickstart/
http://www.debian-administration.org/articles/336
http://www.debian.org/doc/debian-policy/index.html
https://wiki.ubuntu.com/PackagingGuide/Basic#Packaging%20From%20Scratch
http://www.debian-administration.org/articles/174
Limited shell CVS where you can find all my building files