Page 1 of 6 123 ... LastLast
Results 1 to 10 of 53

Thread: Building packages on a clean-room environmend

  1. #1
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Building packages on a clean-room environment

    This HOWTO tries to solve the common problems related to package building
    • Prevent installing unwanted cruft on your system by doing the build process on a chroot (as a sandboxed, jailed) environment
    • Making sure that the package works for other people too who are trying it
    • Automatically downloaded dependencies make process smoother
    • (Easier to automate larger scale building process and testing)



    pbuilder is a tool which is "used for creating and maintaining chroot environment and building Debian package in the chroot environment".




    l'll be doing all the configuration locally only, by overriding the defaults. Another way is to make pbuilder settings to /etc/pbuilderrc
    and create your hooks on /etc/pbuilder/hook.d accordingly



    Installing
    1. Install necessary packages
      Code:
      sudo aptitude install pbuilder devscripts
    2. Create the pbuilder environment
      Code:
      sudo pbuilder create
      Now we have ready chroot environment for package building using Ubuntu defaults from /etc/pbuilderrc

      (You can test environment by doing sudo pbuilder login. Command exit will leave the environment)



    3. Override the default pbuilder settings
      Code:
      nano ~/.pbuilderrc
      These are the file contents.
      Code:
      ## Overrides /etc/pbuilderrc
      
      # Default distribution
      #DISTRIBUTION=feisty
      COMPONENTS="main restricted universe multiverse"
      
      # Repositories
      MIRRORSITE=http://archive.ubuntu.com/ubuntu
      #OTHERMIRROR="deb ${MIRRORSITE} ${DISTRIBUTION}-updates ${COMPONENTS}|deb ${MIRRORSITE} ${DISTRIBUTION}-security ${COMPONENTS}"
      
      # For build results
      BUILDRESULT=${HOME}/pbuilder/result
      
      # Hooks for chroot environment
      HOOKDIR=${HOME}/pbuilder/hook.d
      
      # Mount directories inside chroot environment
      BINDMOUNTS=${BUILDRESULT}
      
      # Bash prompt inside pbuilder
      export debian_chroot="pbuild$$"
      
      # For D70results hook
      export LOCALREPO=${BUILDRESULT}
      
      # Always include source package
      #DEBBUILDOPTS="-sa"
      
    4. Create a hook that allows to include local packages as dependencies. This feature is ofter necessary
      when the package you're building depends on other package(s) which is not found on Ubuntu repositories.
      Code:
      mkdir -p ~/pbuilder/result
      mkdir ~/pbuilder/hook.d
      nano ~/pbuilder/hook.d/D70results
      These are the file contents
      Code:
      #!/bin/sh
      
      echo "Executing hook: $0"
      
      cd ${LOCALREPO}
      dpkg-scanpackages . /dev/null > Packages
      
      echo "deb file:${LOCALREPO} ./" >> /etc/apt/sources.list
      apt-get update
      This code block creates a list of the packages in ${LOCALREPO} folder for APT to use as repository.
      Hook is always executed inside the chroot environment before the build phase, and packages are available
      for APT to use on dependency resolving.

    5. Make the hook file executable
      Code:
      chmod +x ~/pbuilder/hook.d/D70results

    6. Update the chroot environment, so that new modules (repositories) are included too.
      (Also, If you later need to change your chroot environment, use this command)
      Code:
      sudo pbuilder update --override-config




    Building packages
    • Building the packages should be as easy as
      Code:
      mkdir -p ~/packages/mypackage
      cd ~/packages/mypackage
      apt-get source mypackage
      Then either
      Code:
      sudo pbuilder build mypackage_x.x.x.dsc
      or
      Code:
      cd mypackage-x.x.x
      pdebuild --use-pdebuild-internal
    • Succefully built packages are placed on ${BUILDRESULT} folder
    • These packages are also used as dependencies, so if you get weird behaviour when building other packages,
      clean the contents of this folder and try again




    Updating the base build environment
    • You should update the build environment when new updates are released for the base system
      Code:
      sudo pbuilder update --autocleanaptcache




    A real life example: nautilus

    Code:
    # enable Ubuntu source repositories on /etc/apt/sources.list
    
    mkdir -p ~/packages/nautilus
    cd ~/packages/nautilus
    apt-get source nautilus
    sudo pbuilder build nautilus_2.14.1-0ubuntu9.dsc
    
    #or alternatively do pdebuild --use-pdebuild-internal on nautilus folder


    Notes
    • Prefer pbuilder instead of pdebuild wrapper
    • pbuilder uses .orig.tar.gz and .diff.gz to build the package.
    • If you change anything inside the source folder, source package must be rebuilt for pbuilder (or use pdebuild instead).
    • To rebuild source package
      Code:
      dpkg-source -b folder
    • /usr/share/doc/pbuilder/examples contains some nice example hooks, especially C10shell which is invoked when something goes wrong on build process.
      You can find the build target files in /tmp/buildd.
    • If you want to pass DEB_BUILD_OPTIONS to pbuilder and you're using sudo, you must prevent reset of certain environment variables. Using visudo, add definition
      Code:
      Defaults  env_keep+="DEB_* BUILD* PATH"
    • Default dependency resolution logic using apt-get is quite slow. To make it way more faster, change the resolver to gdebi. Add to your ~/.pbuilderrc
      Code:
      # use gdebi for dependency resolution
      PBUILDERSATISFYDEPENDSCMD=/usr/lib/pbuilder/pbuilder-satisfydepends-gdebi
      

    • pdebuild doesn't support hooks.
    • pdebuild ignores BUILDRESULT variable set on /etc/pbuilderrc. You can specify it using --buildresult. see pdebuild man page for details.
    • if you want to improve pbuilder's speed, check out package called cowdancer
    • Useful way to clean out obsolete packages from pbuilder cache
      Code:
      sudo pbuilder update --autocleanaptcache




    References




    [edit]
    • Debian version of pbuilder doesn't currently have support for COMPONENTS option
    • added OTHERMIRROR to ~/.pbuilderrc. This allows to use additional Ubuntu repositories. Not enabled by default.
      If you enable it, use pbuilder update --overrideconfig to make the changes apply
    • added PBUILDERSATISFYDEPENDSCMD to ~/.pbuilderrc. Makes dependency resolution very fast by using gdebi instead (In Ubuntu package only)
    Last edited by mlind; June 25th, 2007 at 03:49 PM.

  2. #2
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environment

    As second usecase example, we'll test how self-buit packages which are not in official repositories
    work with this pbuilder setup.



    Picture a scenario where you would want to build a package that's not on any Ubuntu repository,
    but Debian upsteam has it available. You've cheked out the changelogs and verified that no major
    dependencies have changed. If library is on main Ubuntu repository, think twice before actually upgrading it.




    I won't be inserting any changelog info on the packages (or changing package naming), but you may do so if you want.
    Use dch -iDdapper on the directory where sources were extracted (contains debian/ subdirectory).
    Code:
    cd mypackage-x.xx
    dch -iDdapper


    gmpc
    • Add debian unstable to your /etc/apt/sources.list
      Code:
      deb-src http://ftp.uk.debian.org/debian unstable main contrib non-free
    • Get the sources
      Code:
      mkdir -p ~/packages/gmpc
      cd ~/packages/gmpc
      sudo aptitude update
      apt-get source gmpc
    • Try to build it first time
      Code:
      sudo pbuilder build gmpc_0.13.0-2.dsc
      When you fist time try to build this package, you'll notice that it fails miserably because some dependency is missing.
      This is usually caused by missing module (respository) on pbuilderrc or like in this case, unmet version number although package is available.
      Code:
      -> Considering  libmpd-dev (>= 0.12.0)
            Tried versions: 0.01-3
        -> Does not satisfy version, not trying
      Poking the Ubuntu package info (or use apt tools for this) for libmpd you notice that it's really too old for satisfying
      the build-time requirement. Debian upstream luckily provides updated version.




    libmpd
    • Get libmpd source from debian unstable
      Code:
      mkdir -p ~/packages/libmpd
      cd ~/packages/libmpd
      apt-get source libmpd
    • Build libmpd
      Code:
      sudo pbuilder build libmpd_0.12.0-2.dsc
      When build process is finished, new packages are available on /var/cache/pbuilder/result. Good thing is that
      pbuilder chroot environment that was set previously, can see these packages too (using the provided hook).



    gmpc (2nd try..)
    • Back to building gmpc
      Code:
      cd ~/packages/gmpc
      sudo pbuilder build gmpc_0.13.0-2.dsc




    Install
    • Install builded packages
      Code:
      cd /var/cache/pbuilder/result
      sudo dpkg -i libmpd0_0.12.0-2_i386.deb gmpc_0.13.0-2_i386.deb
    Last edited by mlind; June 30th, 2006 at 03:16 PM.

  3. #3
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environment

    mplayer



    Initial setup
    • Disable any multiverse source repositories from /etc/apt/sources.list
    • Add new source repository to /etc/apt/sources.list
      Code:
      deb-src http://www.debian-multimedia.org sid main
    • Update package list
      Code:
      sudo aptitude update




    ffmpeg
    • Build ffmpeg first
      Code:
      mkdir -p ~/packages/ffmpeg
      cd ~/packages/ffmpeg
      apt-get source libavcodeccvs51-dev
    • NOTE if you get error "can't find a register in class 'GENERAL_REGS' while reloading 'asm'" you must edit debian/rules and
      change CFLAGS to appropriate values. For my system I had to use
      Code:
      CFLAGS = -Wall
      LDCONFIG =
      
      ifneq (,$(findstring noopt,$(DEB_BUILD_OPTIONS)))
              CFLAGS += -O0 -fPIC -DPIC
      else
              CFLAGS += -pipe -fomit-frame-pointer
    • If you edited anything inside the source folder, you must regenerate the diffs against the original package. (Argument is the source folder)
      Code:
      dpkg-source -b ffmpegcvs-20060625
    • Build
      Code:
      sudo pbuilder build ffmpegcvs_20060625-0.0.dsc


    libdivxdecore0
    • Build libdivxdecore0
      Code:
      mkdir -p ~/packages/libdivxdecore0
      cd ~/packages/libdivxdecore0
      apt-get source libdivxdecore0
      sudo pbuilder build divx4linux_5.0.1-1.dsc


    libvstream
    • Build libvstream
      Code:
      mkdir -p ~/packages/libvstream
      cd ~/packages/libvstream
      apt-get source libvstream-dev
      sudo pbuilder build vstream-client_1.2-0.1.dsc




    mplayer
    • Get source first
      Code:
      mkdir -p ~/packages/mplayer
      cd ~/packages/mplayer
      apt-get source mplayer
    • I had to change mplayer-skin (virtual) package dependency to mplayer-skins. To do this, modify debian/control.marillat file
      on mplayer source folder and either remove mplayer-skin dependency from mplayer-custom Package or change it to mplayer-skins
      Code:
      Package: mplayer
      Architecture: any
      Depends: ${shlibs:Depends}, mplayer-skins
    • Regenerate the diffs against the original package (if you changed dependecies)
      Code:
      dpkg-source -b mplayer-1.0-pre8
    • Build
      Code:
      DEB_BUILD_OPTIONS=marillat sudo pbuilder build mplayer_1.0-pre8-0.0.dsc



    Install
    • Install mplayer and dependencies
      Code:
      sudo aptitude install libjack0.100.0-0 libdirectfb-0.9-22 libsvga1
      cd /var/cache/pbuilder/result/
      sudo dpkg -i mplayer_1.0-pre8-0.0_i386.deb libavcodeccvs51_20060625-0.0_i386.deb libdivxdecore0_5.0.1-1_i386.deb libavutilcvs49_20060625-0.0_i386.deb
    Last edited by mlind; July 24th, 2006 at 04:57 PM.

  4. #4
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environment

    pbuilder can also be used to test your build scripts or other stuff on a sandboxed environment.
    Code:
    cp -R ~/packages/mypackage /tmp
    sudo pbuilder login --bindmount /tmp/mypackage
    
    apt-get install nano
    (start editing)
    Now /tmp/mypackage is accessible/visible on chrooted environment (on / folder). Be aware that changes you make on --bindmounted
    folders are persisted and because you're logged as root, you'll get some unwanted permission issues. It may be
    better to use temporary folders.

    Every package that you install, or any other change you make (excluding --bindmount folders) are not preserved.
    So you can safely do almost anything you want.


    If you login using --save-after-login, then all changes you make on the environment are persisted.
    It's convinient way to install nano for example, which is not included by default.
    Last edited by mlind; July 3rd, 2006 at 07:08 PM.

  5. #5
    Join Date
    Aug 2005
    Location
    California
    Beans
    78
    Distro
    Ubuntu 6.06

    Re: Building packages on a clean-room environmend

    These guides are great mlind. Would it be possible to take the debian folder from say a older version of a program and use it for the latest cvs? How do you recommend going about compiling things without source debs?
    Last edited by PsychoTrauma; July 4th, 2006 at 09:36 AM.

  6. #6
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environmend

    Quote Originally Posted by PsychoTrauma
    These guides are great mlind. Would it be possible to take the debian folder from say a older version of a program and use it for the latest cvs? How do you recommend going about compiling things without source debs?
    Yes it's possible and you should do it like that. Let's say you have source package
    foo-1.0.3 that's already been been debianized (it contains the debian folder)
    and you'd like use that with new version foo-1.0.4 that you've downloaded.

    Code:
    ~/packages/foo$  ls
    foo-1.0.3            foo_1.0.3-1.dsc        foo-1.0.4.tar.gz
    foo_1.0.3-1.diff.gz  foo_1.0.3.orig.tar.gz
    To apply foo-1.0.3 debian folder contents (it will actually patch using .diff.gz file)
    Code:
    cd foo-1.0.3
    uupdate ../foo-1.0.4.tar.gz

    Quote Originally Posted by PsychoTrauma
    After building files with the method above how would you go about removing everything so your system is the same before you started?
    What method do you mean? If you use pbuilder then your system stays the same.
    Everything is installed on chrooted environment which is always untarred from
    same archive for each pbuilder session.


    Downloaded packages are stored on /var/cache/pbuilder/aptcache, which you
    can clean using pbuilder clean.

    Sometimes, if you ctrl-c the pbuilder process /proc is not correctly umounted from
    chroot environment and as result, /var/cache/pbuilder/build/ may contain folders
    left behind by the build process. To get rid of these, see /etc/mtab and umount
    everything pbuilder related, then rm -rf build folder's contents.


    If you wanted to remove everything that this guide made you to install, then
    Code:
    sudo aptitude purge pbuilder fakeroot devscripts
    and make sure /var/cache/pbuilder doesn't exist.
    Building on Ubuntu using [pbuilder]
    Other resources [Upgrading from CVS, Gaim2, Anjuta, Rhythmbox]

  7. #7
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environment

    Quote Originally Posted by PsychoTrauma
    How do you recommend going about compiling things without source debs?
    You should read http://www.debian.org/doc/maint-guide/

    To debianize the source, use debhelper dh_make (see man debhelper). This will
    create debian folder with initial scripts inside, maintainer guide explains briefly what
    files are for. I also suggest to check out cdbs which is intended to do package maintaining easier.


    You can test your packaging rules on pbuilder environment by doing
    • pbuilder login --bindmounts yourtestfolder
    • test by executing debian/rules build (test install and clean targets too)
    • after executing install target, make sure debian/yourpackage/ contents look okay
    Last edited by mlind; July 13th, 2006 at 02:02 PM.
    Building on Ubuntu using [pbuilder]
    Other resources [Upgrading from CVS, Gaim2, Anjuta, Rhythmbox]

  8. #8
    Join Date
    Feb 2006
    Location
    Brasil
    Beans
    244
    Distro
    Xubuntu 7.04 Feisty Fawn

    Re: Building packages on a clean-room environment

    This is really great mlind !

    Now I don't need to install tons of -dev packages on my main system : D
    Great How-To!

    I wish there was a GUI to debianize source packages tho (yeah, I'm lazy)

  9. #9
    Join Date
    Nov 2005
    Beans
    2,322
    Distro
    Ubuntu Intrepid Ibex (testing)

    Re: Building packages on a clean-room environment

    Quote Originally Posted by LKRaider View Post
    This is really great mlind !

    Now I don't need to install tons of -dev packages on my main system : D
    Great How-To!

    I wish there was a GUI to debianize source packages tho (yeah, I'm lazy)
    Thanks, that was why I wanted to look for alternative solutions, keep unwatend cruft out and be sure that package would actually work for others too.

    I've modified proposed pbuilder configuration a bit. My BUILDRESULT, where builded packages appear is BUILDRESULT=/home/myusername/pbuilder/result, which makes package handling little easier.

    You can set aliases to pbuilder commands to make invoking little non-verbosive, but working gui would be nice too.



    Here's few tips too
    • If you are testing packages using pbuilder login --bindmounts /path/to/mypackage, you can copy /usr/lib/pbuilder/pbuilder-satisfydepends to your hookdir and call it inside chroot environment to automatically fetch and install all dependencies.

    • Hooks are found on /tmp/hooks/ inside chroot environment, and may be called explicitly too.

    • You can run X-server applications "the easy, but not right" -way inside chroot, by bindmounting your /tmp folder to chroot environment. This contains necessary files for applications to use X. You also need to grant privileges for root user. This can be done by using xhost on normal environment.
      Code:
      xhost +local:root
      (Remember to umount bindmounted /tmp before exiting the environment)
    Last edited by mlind; July 22nd, 2006 at 06:57 PM.
    Building on Ubuntu using [pbuilder]
    Other resources [Upgrading from CVS, Gaim2, Anjuta, Rhythmbox]

  10. #10
    Join Date
    Aug 2005
    Location
    California
    Beans
    78
    Distro
    Ubuntu 6.06

    Re: Building packages on a clean-room environmend

    I seem to be having trouble with the new changes. Pbuilder no longer sees the files I have compiled previously. A example is I compiled libx264-dev for the newer version on ffmpeg. I go to compile ffmpeg and it can't find the libx264-dev deb in ~/pbuilder/result.

    I also should add this is on a fresh install so I never had the previous method set up yet.

Page 1 of 6 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •