First off, let me be very clear up-front: normally, I write my blog articles to be approachable by readers of varying levels of technical background (or none at all). This will not be one of those. This will be a deep dive into the very bowels of the sausage factory.
This blog post assumes that the reader is aware of the Fedora Modularity Initiative and would like to learn how to build their very own modules for inclusion into the Fedora Project. I will guide you through the creation of a simple module built from existing Fedora Project packages on the “F26” branch.
To follow along, you will need a good working knowledge of the git source-control system (in particular, Fedora’s “dist-git“) as well as being generally comfortable around Fedora system tools such as dnf and python.
Setting up the Module Build Service
For the purposes of this blog, I am going to use Fedora 25 (the most recent stable release of Fedora) as the host platform for this demonstration and Fedora 26 (the current in-development release) as the target. To follow along, please install Fedora 25 Server on a bare-metal or virtual machine with at least four processors and 8 GiB of RAM.
First, make sure that the system is completely up-to-date with all of the latest packages. Then we will install the “module-build-service” package. We will need version 1.3.24 or later of the
module-build-service RPM and version 1.2.0 or later of
python2-modulemd, which at the time of this writing requires installing from the “updates-testing” repository. (EDIT 2017-06-30: version 1.3.24 requires the
mock-scm package for local builds but doesn’t have a dependency on it.)
dnf install --enablerepo=updates-testing module-build-service python2-modulemd mock-scm
This may install a considerable number of dependency packages as well. Once this is installed, I recommend modifying
/etc/module-build-service/config.py to change
NUM_CONCURRENT_BUILDS to match the number of available processors on the system.
Leave the rest of the options alone at this time. The default configuration will interact with the production Fedora Project build-systems and is exactly what we want for the rest of this tutorial.
In order to perform builds locally on your machine, your local user will need to be a member of the
mock group on the system. To do this, run the following command:
usermod -a -G mock <yourloginname>
Then you will need to log out of the system and back in for this to take effect (since Linux only adds group memberships at login time).
Gathering the module dependencies
So now that we have a build environment, we need something to build. For demonstration purposes, I’m going to build a module to provide the
libtalloc library used by the Samba and SSSD projects. This is obviously a trivial example and would never become a full module on its own.
The first thing we need to do is figure out what runtime and build-time dependencies this package has. We can use
dnf repoquery to accomplish this, starting with the runtime dependencies:
dnf repoquery --requires libtalloc.x86_64 --resolve
Which returns with:
glibc-0:2.25-4.fc26.i686 glibc-0:2.25-4.fc26.x86_64 libcrypt-0:2.25-4.fc26.x86_64 libcrypt-nss-0:2.25-4.fc26.x86_64
There are two libcrypt implementations that will satisfy this dependency, so we can pick one a little later. For glibc, we only want the one that will operate on the primary architecture, so we’ll ignore the
Next we need to get the build-time dependencies with:
dnf repoquery --requires --enablerepo=fedora-source --enablerepo=updates-source libtalloc.src --resolve
Which returns with:
docbook-style-xsl-0:1.79.2-4.fc26.noarch doxygen-1:1.8.13-5.fc26.x86_64 libxslt-0:1.1.29-1.fc26.i686 libxslt-0:1.1.29-1.fc26.x86_64 python2-devel-0:2.7.13-8.fc26.i686 python2-devel-0:2.7.13-8.fc26.x86_64 python3-devel-0:3.6.1-6.fc26.i686 python3-devel-0:3.6.1-6.fc26.x86_64
OK, that’s not bad. Similar to the runtime dependencies above, we will ignore the
.i686 versions. So now we have to find out which of these packages are provided already by the base-runtime module or the shared-userspace module, so we don’t need to rebuild them. Unfortunately, we don’t have a good reference location for getting this data yet (it’s coming a little ways into the future), so for the time being we will need to look directly at the module metadata YAML files:
When reading the YAML, the section that we are interested in is the api->rpms section. This part of the metadata describes the set of packages whose interfaces are public and can be consumed directly by the end-user or other modules. So, looking through these two foundational modules, we see that the base-runtime provides
python3-devel and shared-userspace provides
python2-devel and common-build-dependencies provides
doxygen. So in this case, all of the dependencies are satisfied by these three core modules. If they were not, we’d need to recurse through the dependencies and figure out what additional packages we would need to include in our module to support
libtalloc or see if there was another module in the collection that provided it.
So, the next thing we’re going to need to do is decide which version of
libtalloc we want to package. What we want to do here is check out the
libtalloc module from Fedora dist-git and then find a git commit has that matches the build we want to add to our module. We can check out the
libtalloc module by doing:
fedpkg clone --anonymous rpms/libtalloc && cd libtalloc
Once we’re in this git checkout, we can use the
git log command to find the commit hash that we want to include. For example:
[sgallagh@sgallagh540:libtalloc (master)]$ git log -1 commit f284a27d9aad2c16ba357aaebfd127e4f47e3eff (HEAD -> master, origin/master, origin/f26, origin/HEAD) Author: Lukas Slebodnik <email@example.com> Date: Tue Feb 28 09:03:05 2017 +0100 New upstream release - 2.1.9 rhbz#1401225 - Rename python packages to match packaging guidelines https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages
The string of hexadecimal characters following the word “commit” is the git commit hash. Save it somewhere, we’re going to need it in the next section.
Creating a new module
The first thing to be aware of is that the module build-service has certain constraints. The build can only be executed from a directory that has the same name as the module and will look for a file named
modulename.yaml in that directory. So in our case, I’m going to name the module
talloc, which means I must create a directory called
talloc and a module metadata file called
talloc.yaml. Additionally, the module-build-service will only work within a git checkout, so we will initialize this directory with a blank metadata file.
mkdir talloc && cd talloc touch talloc.yaml git init git add talloc.yaml git commit -m "Initial setup of the module"
Now we need to edit the module metadata file
talloc.yml and define the contents of the module. A module metadata file’s basic structure looks like this:
document: modulemd version: 1 data: summary: Short description of this module description: Full description of this module license: module: - LICENSENAME references: community: Website for the community that supports this module documentation: Documentation website for this module tracker: Issue-tracker website for this module dependencies: buildrequires: base-runtime: f26 shared-userspace: f26 common-build-dependencies: f26 requires: base-runtime: f26 shared-userspace: f26 api: rpms: - rpm1 - ... filter: rpms: - filteredrpm1 - ... components: rpms: rpm1: rationale: reason to include rpm1 ref:
Let’s break this down a bit. First, the document type and version are fixed values. These determine the version of the metadata format. Next comes the “data” section, which contains all the information about this module.
The summary, description and references are described in the sample. The license field should describe the license of the module, not its contents which carry their own licenses.
apisection is a list of binary RPMs that are built from the source RPMs in this module whose presence you want to treat as “public”. In other words, these are the RPMs in this module that others can expect to be available for their use. Other RPMs may exist in the repository (to satisfy dependencies or simply because they were built as a side-effect of generating these RPMs that you need), but these are the ones that consumers should use.
On the flip side of that, we have the
filter section. This is a place to list binary RPM packages that explicitly must not appear in the final module so that no user will try to consume them. The main reason to use this would be if a package builds a subpackage that is not useful to the intended audience and requires additional dependencies which are not packaged in the module. (For example, a module might contain a package that provides a plugin for another package and we don’t want to ship that other package just for this reason).
Each of the components describes a source RPM that will be built as part of this module. The rationale is a helpful comment to explain why it is needed in this module. The
ref field describes any reference in the dist-git repository that can be used to acquire these sources. It is recommended to use an exact git commit here so that the results are always repeatable, but you can also use tag or branch names.
talloc module should look like this:
document: modulemd version: 1 data: summary: The talloc library description: A library that implements a hierarchical allocator with destructors. stream: '' version: 0 license: module: - LGPLv3+ references: community: https://talloc.samba.org/ documentation: https://talloc.samba.org/talloc/doc/html/libtalloc__tutorial.html tracker: http://bugzilla.samba.org/ dependencies: buildrequires: base-runtime: f26 shared-userspace: f26 common-build-dependencies: f26 requires: base-runtime: f26 api: rpms: - libtalloc - libtalloc-devel - python-talloc - python-talloc-devel - python3-talloc - python3-talloc-devel components: rpms: libtalloc: rationale: Provides a hierarchical memory allocator with destructors ref: f284a27d9aad2c16ba357aaebfd127e4f47e3eff
You will notice I omitted the “filter” section because we want to provide all of the subpackages here to our consumers. Additionally, while most modules will require the shared-userspace module at runtime, this particular trivial example does not.
So, now we need to commit these changes to the local git repository so that the module build service will be able to see it.
git commit talloc.yaml -m "Added module metadata"
Now, we can build this module in the module build service. Just run:
The build will proceed and will provide a considerable amount of output telling you what it is doing (and even more if you set
LOG_LEVEL = 'debug' in the
/etc/module-build-service/config.py file). The first time it runs, it will take a long time because it will need to download and cache all of the packages from the base-runtime and shared-userspace modules to perform the build. (Note: due to some storage-related issues in the Fedora infrastructure right now, you may see some of the file downloads time out, canceling the build. If you restart it, it will pick up from where it left off and retry those downloads.)
The build will run and deposit results in the
~/modulebuild/builds directory in a subdirectory named after the module and the timestamp of the git commit from which it was built. This will include mock build logs for each individual dependency, which will show you if it succeeded or failed.
When the build completes successfully, the module build service will have created a yum repository in the same results directory as the build logs containing all of the produced RPMs and repodata (after filtering out the undesired subpackages).
And there you have it! Go off and build modules!
Edit 2017-06-30: Switched references from
NUM_CONCURRENT_BUILDS and updated the minimum MBS requirement to 1.3.24. Added notes about needing to be in the ‘mock’ group.