OpenShift and SSSD Part 1: Basic LDAP Authentication

Overview

OpenShift provides a fairly simple and straightforward authentication provider for use with LDAP setups. It has one major limitation, however: it can only connect to a single LDAP server. This can be problematic if that LDAP server becomes unavailable for any reason. When this happens, end-users get very unhappy.

Enter SSSD. Originally designed to manage local and remote authentication to the host OS, it can now be configured to provide identity, authentication and authorization services to web services like OpenShift as well. It provides a multitude of advantages over the built-in LDAP provider; in particular it has the ability to connect to any number of failover LDAP servers as well as to cache authentication attempts in case it can no longer reach any of those servers.

These advantages don’t come without a cost, of course: the setup of this configuration is somewhat more advanced, so I’m writing up this guide to help you get it set up. Rather than adding a few lines to the master-config.yml in OpenShift and calling it a day, we are going to need to set up a separate authentication server that OpenShift will talk to. This guide will describe how to do it on a dedicated physical or virtual machine, but the concepts should also be applicable to loading up such a setup in a container as well. (And in the future, I will be looking into whether we could build such a static container right into OpenShift, but for now this document will have to suffice.) For this guide, I will use the term VM to refer to either type of machine, simply because it’s shorter to type and read.

This separate authentication server will be called the “authenticating proxy” from here on out and describes a solution that will provide a specialized httpd server that will handle the authentication challenge and return the results to the OpenShift Server. See the OpenShift documentation for security considerations around the use of an authenticating proxy.

Formatting Notes

  • If you see something in italics within a source-code block below, you should replace it with the appropriate value for your environment.
  • Source-code blocks with a leading ‘#’ character indicates a command that must be executed as the “root” user, either by logging in as root or using the sudo command.
  • Source-code blocks with a leading ‘$’ character indicates a command that may be executed by any user (privileged or otherwise). These commands are generally for testing purposes.

Prerequisites

You will need to know the following information about your LDAP server to follow the directions below:

  • Is the directory server powered by FreeIPA, Active Directory or another LDAP solution?
  • What is the URI for the LDAP server? e.g. ldap.example.com
  • Where is the CA certificate for the LDAP server?
  • Does the LDAP server correspond to RFC 2307 or RFC2307bis for user-groups?

Prepare VMs:

  • proxy.example.com: A VM to use as the authenticating proxy. This machine must have at least SSSD 1.12.0 available, which means a fairly recent operating system. In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.
  • openshift.example.com: A VM to use to run OpenShift

(These machines *can* be configured to run on the same system, but for the purposes of this tutorial, I am keeping them separate)

Phase 1: Certificate Generation

In order to ensure that communication between the authenticating proxy and OpenShift is trustworthy, we need to create a set of TLS certificates that we will use during the other phases of this setup. For the purposes of this demo, we will start by using the auto-generated certificates created as part of running

# openshift start \
    --public-master=https://openshift.example.com:8443 \
    --write-config=/etc/origin/

Among other things, this will generate /etc/origin/master/ca.{cert|key}. We will use this signing certificate to generate keys to use on the authenticating proxy.

# mkdir -p /etc/origin/proxy/
# oadm ca create-server-cert \
    --cert='/etc/origin/proxy/proxy.example.com.crt' \
    --key='/etc/origin/proxy/proxy.example.com.key' \
    --hostnames=proxy.example.com,1.2.3.4 \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key='/etc/origin/master/ca.key' \
    --signer-serial='/etc/origin/master/ca.serial.txt'

For the hostnames, ensure that any hostnames and interface IP addresses that might need to access the proxy are listed, otherwise the HTTPS connection will fail.

Next, we will generate the API client certificate that the authenticating proxy will use to prove its identity to OpenShift (this is necessary so that malicious users cannot impersonate the proxy and send fake identities). First, we will create a new CA to sign this client certificate.

# oadm ca create-signer-cert \
  --cert='/etc/origin/proxy/proxyca.crt' \
  --key='/etc/origin/proxy/proxyca.key' \
  --name='openshift-proxy-signer@`date +%s`' \
  --serial='/etc/origin/proxy/proxyca.serial.txt'

(The date +%s in that block is used to make the  signer unique. You can use any name you prefer, however.)

# oadm create-api-client-config \
    --certificate-authority='/etc/origin/proxy/proxyca.crt' \
    --client-dir='/etc/origin/proxy' \
    --signer-cert='/etc/origin/proxy/proxyca.crt' \
    --signer-key='/etc/origin/proxy/proxyca.key' \
    --signer-serial='/etc/origin/proxy/proxyca.serial.txt' \
    --user='system:proxy'
# cat /etc/origin/proxy/system\:proxy.crt \
      /etc/origin/proxy/system\:proxy.key \
      > /etc/origin/proxy/authproxy.pem

Phase 2: Authenticating Proxy Setup

Step 1: Copy certificates

From openshift.example.com, securely copy the necessary certificates to the proxy machine:

# scp /etc/origin/proxy/master/ca.crt \
      root@proxy.example.com:/etc/pki/CA/certs/

# scp /etc/origin/proxy/proxy.example.com.crt \
      /etc/origin/proxy/authproxy.pem \
      root@proxy.example.com:/etc/pki/tls/certs/

# scp /etc/origin/proxy/proxy.example.com.key \
      root@proxy.example.com:/etc/pki/tls/private/

Step 2: SSSD Configuration

Install a new VM with a recent operating system (in order to use the mod_identity_lookup module later, it will need to be running SSSD 1.12.0 or later). In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.

First thing is to install all of the necessary dependencies:

# yum install -y sssd \
                 sssd-dbus \
                 realmd \
                 httpd \
                 mod_session \
                 mod_ssl \
                 mod_authnz_pam

This will give us the SSSD and the web server components we will need. The first step here will be to set up SSSD to authenticate this VM against the LDAP server. If the LDAP server in question is a FreeIPA or Active Directory environment, then realmd can be used to join this machine to the domain. This is the easiest way to get up and running.

realm join ldap.example.com

If you aren’t running a domain, then your best option is to use the authconfig tool (or follow the many other tutorials on the internet for configuring SSSD for identity and authentication).

# authconfig --update --enablesssd --enablesssdauth \
             --ldapserver=ldap.example.com \
             --enableldaptls \
             --ldaploadcert=http://ldap.example.com/ca.crt

This should create /etc/sssd/sssd.conf with most of the appropriate settings. (Note: RHEL 7 appears to have a bug wherein authconfig does not create the /etc/openldap/cacerts directory, so you may need to create it manually before running the above command.)

If you are interested in using SSSD to manage failover situations for LDAP, this can be configured simply by adding additional entries in /etc/sssd/sssd.conf on the ldap_uri line. Systems enrolled with FreeIPA will automatically handle failover using DNS SRV records.

Finally, restart SSSD to make sure that all of the changes are applied properly:

$ systemctl restart sssd.service

Now, test that the user information can be retrieved properly:

$ getent passwd <username>
username:*:12345:12345:Example User:/home/username:/usr/bin/bash

At this point, it is wise to attempt to log into the VM as an LDAP user and confirm that the authentication is properly set up. This can be done via the local console or a remote service such as SSH. (Later, you can modify your /etc/pam.d files to disallow this access if you prefer.) If this fails, consult the SSSD troubleshooting guide.

Step 3: Apache Configuration

Now that we have the authentication pieces in place, we need to set up Apache to talk to SSSD. First, we will create a PAM stack file for use with Apache. Create the /etc/pam.d/openshift file and add the following contents:

auth required pam_sss.so
account required pam_sss.so

This will tell PAM (the pluggable authentication module) that when an authentication request is issued for the “openshift” stack, it should use pam_sss.so to determine authentication and access-control.

Next we will configure the Apache httpd.conf. (Taken from the OpenShift documentation and modified for SSSD.) For this tutorial, we’re only going to set up the challenge authentication (useful for logging in with oc login and similar automated tools). A future entry in this series will describe setup to use the web console.

First, create the new file openshift-proxy.conf in /etc/httpd/conf.d (substituting the correct hostnames where indicated):

LoadModule request_module modules/mod_request.so
LoadModule lookup_identity_module modules/mod_lookup_identity.so
# Nothing needs to be served over HTTP.  This virtual host simply redirects to
# HTTPS.
<VirtualHost *:80>
  DocumentRoot /var/www/html
  RewriteEngine              On
  RewriteRule     ^(.*)$     https://%{HTTP_HOST}$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
  # This needs to match the certificates you generated.  See the CN and X509v3
  # Subject Alternative Name in the output of:
  # openssl x509 -text -in /etc/pki/tls/certs/proxy.example.com.crt
  ServerName proxy.example.com

  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/proxy.example.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/proxy.example.com.key
  SSLCACertificateFile /etc/pki/CA/certs/ca.crt

  # Send logs to a specific location to make them easier to find
  ErrorLog logs/proxy_error_log
  TransferLog logs/proxy_access_log
  LogLevel warn
  SSLProxyEngine on
  SSLProxyCACertificateFile /etc/pki/CA/certs/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /etc/pki/tls/certs/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}:8443/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://openshift.example.com:8443/oauth/authorize
    AuthType Basic
    AuthBasicProvider PAM
    AuthPAMService openshift
    Require valid-user
  </Location>

  <ProxyMatch /oauth/authorize>
    AuthName openshift
    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER
  </ProxyMatch>
</VirtualHost>

RequestHeader unset X-Remote-User

 

Then we need to tell SELinux that it’s acceptable for Apache to contact the PAM subsystem, so we set a boolean:

# setsebool -P allow_httpd_mod_auth_pam on

At this point, we can start up Apache.

# systemctl start httpd.service

Phase 3: OpenShift Configuration

This describes how to set up an OpenShift server from scratch in an “all in one” configuration. For more complicated (and interesting) setups, consult the official OpenShift documentation.

First, we need to modify the default configuration to use the new identity provider we just created. We’ll start by modifying the /etc/origin/master/master-config.yaml file. Scan through it and locate the identityProviders section and replace it with:

  identityProviders:
  - name: any_provider_name
    challenge: true
    login: false
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxy/proxyca.crt
      headers:
      - X-Remote-User

Now we can start openshift with the updated configuration:

# openshift start \
    --public-master=https://openshift.example.com:8443 \
    --master-config=/etc/origin/master/master-config.yaml \
    --node-config=/etc/origin/node-node1.example.com/node-config.yaml

Now you can test logins with

oc login https://openshift.example.com:8443

It should now be possible to log in with only valid LDAP credentials. Stay tuned for further entries in this series where I will teach you how to set up a “login” provider for authenticating the web console, how to retrieve extended user attributes like email address and full name from LDAP, and also how to set up automatic single-sign-on for users in a FreeIPA or Active Directory domain.

 Updates 2016-05-27: There were some mistakes in the httpd.conf as originally written that made it difficult to set up Part 2. They have been retroactively corrected. Additionally, I’ve moved the incomplete configuration of extended attributes out of this entry and will reintroduce them in a further entry in this series.

Self-Signed SSL/TLS Certificates: Why They are Terrible and a Better Alternative

A Primer on SSL/TLS Certificates

Many of my readers (being technical folks) are probably already aware of the purpose and value of certificates, but in case you are not familiar with them, here’s a quick overview of what they are and how they work.

First, we’ll discuss public-key encryption and public-key infrastructure (PKI). It was realized very early on in human history that sometimes you want to communicate with other people in a way that prevents unauthorized people from listening in. All throughout time, people have been devising mechanisms for obfuscating communication in ways that only the intended recipient of the code would be able to understand. This obfuscation is called encryption, the data being encrypted is called plaintext and the encrypted data is called ciphertext. The cipher is the mathematical transformation that is used to turn the plaintext into the ciphertext and relies upon one or more keys known only to trusted individuals to get the plaintext back.

Early forms of encryption were mainly “symmetric” encryption, meaning that the cipher used the same key for both encryption and decryption. If you’ve ever added a password to a PDF document or a ZIP file, you have been using symmetric encryption. The password is a human-understandable version of a key. For a visual metaphor, think about the key to your front door. You may have one or more such keys, but they’re all exactly alike and each one of them can both lock and unlock the door and let someone in.

Nowadays we also have forms of encryption that are “asymmetric”. What this means is that one key is used to encrypt the message and a completely different key is used to decrypt it. This is a bit harder for many people to grasp, but it works on the basic mathematical principle that some actions are much more complicated to reverse than others. (A good example I’ve heard cited is that it’s pretty easy to figure out the square of any number with a pencil and a couple minutes, but most people can’t figure out a square-root without a modern calculator). This is harder to visualize, but the general idea is that once you lock the door with one key, only the other one can unlock it. Not even the one that locked it in the first place.

So where does the “public” part of public-key infrastructure come in? What normally happens is that once an asymmetric key-pair is generated, the user will keep one of those two keys very secure and private, so that only they have access to it. The other one will be handed out freely through some mechanism to anyone at all that wants to talk to you. Then, if they want to send you a message, they simply encrypt their message using your public key and they know you are the only one who can decrypt it. On the flip side, if the user wanted to send a public message but provide assurance that it came from them, they can also sign a message with the private key, so that the message will contain a special signature that can be decrypted with their public key. Since only one person should have that key, recipients can trust it came from them.

Astute readers will see the catch here: how do users know for certain that your public key is in fact yours? The answer is that they need to have a way of verifying it. We call this establishing trust and it’s exceedingly important (and, not surprisingly, the basis for the rest of this blog entry). There are many ways to establish trust, with the most foolproof being to receive the public key directly from the other party while looking at two forms of picture identification. Obviously, that’s not convenient for the global economy, so there needs to be other mechanisms.

Let’s say the user wants to run a webserver at “www.mydomain.com”. This server might handle private user data (such as their home address), so a wise administrator will set the server up to use HTTPS (secure HTTP). This means that they need a public and private key (which in this case we call a certificate). The common way to do this is for the user to contact a well-known certificate authority and purchase a signature from them. The certificate authority will do the hard work of verifying the user’s identity and then sign their webserver certificate with the CA’s own private key, thus providing trust by way of a third-party. Many well-known certificate authorities have their public keys shipped by default in a variety of operating systems, since the manufacturers of those systems have independently verified the CAs in turn. Now everyone who comes to the site will see the nice green padlock on their URL bar that means their communications are encrypted.

A Primer on Self-Signed Certificates

One of the major drawbacks to purchasing a CA signature is that it isn’t cheap: the CAs (with the exception of Let’s Encrypt) are out there to make money. When you’re developing a new application, you’re going to want to test that everything works with encryption, but you probably aren’t going to want to shell out cash for every test server and virtual machine that you create.

The solution to this has traditionally been to create what is called a self-signed certificate. What this means is that instead of having your certificate signed by a certificate authority, you instead use the certificates public key to add a signature to the private key. The problem with this approach is that web browsers and other clients that verify the security of the connection will be unable to verify that the server is who it says it is. In most cases, the user will be presented with a warning page that informs them that the server is pretending to be the one you went to. When setting up a test server, this is expected. Unfortunately, however, clicking through and saying “I’m sure I want to connect” has a tendency to form bad habits in users and often results in them eventually clicking through when they shouldn’t.

It should be pretty obvious, but I’ll say it anyway: Never use a self-signed certificate for a production website.

One of the problems we need to solve is how to avoid training users to ignore those warnings. One way that people often do this is to load their self-signed certificate into their local trust store (the list of certificate authorities that are trusted, usually provided by the operating system vendor but available to be extended by the user). This can have some unexpected consequences, however. For example, if the test machine is shared by multiple users (or is breached in a malicious attack), then the private key for the certificate might fall into other hands that would then use it to sign additional (potentially malicious) sites. And your computer wouldn’t try to warn you because the site would be signed by a trusted authority!

So now it seems like we’re in a Catch-22 situation: If we load the certificate into the trusted authorities list, we run the risk of a compromised private key for that certificate tricking us into a man-in-the-middle attack somewhere and stealing valuable data. If we don’t load it into the trust store, then we are constantly bombarded by a warning page that we have to ignore (or in the case of non-browser clients, we may have to pass an option not to verify the client) in which case we could still end up in a man-in-the-middle attack, because we’re blindly trusting the connection. Neither of those seems like a great option. What’s a sensible person to do?

Two Better Solutions

So, let’s take both of the situations we just learned about and see if we can locate a middle ground somewhere. Let’s go over what we know:

  • We need to have encryption to protect our data from prying eyes.
  • Our clients need to be able to trust that they are talking to the right system at the other end of the conversation.
  • If the certificate isn’t signed by a certificate in our trust store, the browser or other clients will warn or block us, training the user to skip validation.
  • If the certificate is signed by a certificate in our trust store, then clients will silently accept it.
  • Getting a certificate signed by a well-known CA can be too expensive for an R&D project, but we don’t want to put developers’ machines at risk.

So there are two better ways to deal with this. One is to have an organization-wide certificate authority rather than a public one. This should be managed by the Information Technologies staff. Then, R&D can submit their certificates to the IT department for signing and all company systems will implicitly trust that signature. This approach is powerful, but can also be difficult to set up (particularly in companies with a bring-your-own-device policy in place). So let’s look at a another solution that’s closer to the self-signed approach.

The other way to deal with it would be to create a simple site-specific certificate authority for use just in signing the development/test certificate. In other words, instead of generating a self-signed certificate, you would generate two certificates: one for the service and one to sign that certificate. Then (and this is the key point – pardon the pun), you must delete and destroy the private key for the certificate that did the signing. As a result, only the public key of that private CA will remain in existence, and it will only have ever signed a single service. Then you can provide the public key of this certificate authority to anyone who should have access to the service and they can add this one-time-use CA to their trust store.

Now, I will stress that the same rule holds true here as for self-signed certificates: do not use this setup for a production system. Use a trusted signing authority for such sites. It’s far easier on your users.

A Tool and a Tale

I came up with this approach while I was working on solving some problems for the Fedora Project. Specifically, we wanted to come up with a way to ensure that we could easily and automatically generate a certificate for services that should be running on initial start-up (such as Cockpit or OpenPegasus). Historically, Fedora had been using self-signed certificates, but the downsides I listed above gnawed at me, so I put some time into it and came up with the private-CA approach.

In addition to the algorithm described above, I’ve also built a proof-of-concept tool called sscg (the Simple Signed Certificate Generator) to easily enable the creation of these certificates (and to do so in a way that never drops the CA’s private key onto a filesystem; it remains in memory). I originally wrote it in Python 3 and that version is packaged for use in Fedora today. This past week as a self-assigned exercise to improve my knowledge of Go, I rewrote the sscg in that language. It was a fun project and had the added benefit of removing the fairly heavyweight dependency on the Python 3 version. I plan to package the golang version for Fedora 25 at some point in the near future, but if you’d like to try it out, you can clone my github repository. Patches and suggestions for functionality are most welcome.

Sausage Factory: Multiple Edition Handling in Fedora

First off, let me be very clear up-front: normally, I write my blog articles to be approachable by readers of varying levels of technical background (or none at all). This will not be one of those. This will be a deep dive into the very bowels of the sausage factory.

The Problem

Starting with the Fedora.next initiative, the Fedora Project embarked on a journey to reinvent itself. A major piece of that effort was the creation of different “editions” of Fedora that could be targeted at specific user personas. Instead of having a One-Size-Fits-Some Fedora distribution, instead we would produce an operating system for “doers” (Fedora Workstation Edition), for traditional infrastructure administrators (Fedora Server Edition) and for new, cloudy/DevOps folks (Fedora Cloud Edition).

We made the decision early on that we did not want to produce independent distributions of Fedora. We wanted each of these editions to draw from the same collective set of packages as the classic Fedora. There were multiple reasons for this, but the most important of them was this: Fedora is largely a volunteer effort. If we started requiring that package maintainers had to do three or four times more work to support the editions (as well as the traditional DIY deployments), we would quickly find ourselves without any maintainers left.

However, differentiating the editions solely by the set of packages that they deliver in a default install isn’t very interesting. That’s actually a problem that could have been solved simply by having a few extra choices in the Anaconda installer. We also wanted to solve some classic arguments between Fedora constituencies about what the installed configuration of the system looks like. For example, people using Fedora as a workstation or desktop environment in general do not want OpenSSH running on the system by default (since their access to the system is usually by sitting down physically in front of a keyboard and monitor) and therefore don’t want any potential external access available. On the other hand, most Fedora Server installations are “headless” (no input devices or monitor attached) and thus having SSH access is critical to functionality. Other examples include the default firewall configuration of the system: Fedora Server needs to have a very tightened default firewall allowing basically nothing in but SSH and management features, whereas a firewall that restrictive proves to be harmful to usability of a Workstation.

Creating Per-Edition Default Configuration

The first step to managing separate editions is having a stable mechanism for identifying what edition is installed. This is partly aesthetic, so that the user knows what they’re running, but it’s also an important prerequisite (as we’ll see further on) to allowing the packaging system and systemd to make certain decisions about how to operate.

The advent of systemd brought with it a new file that describes the installed system called os-release. This file is considered to be authoritative for information identifying the system. So this seemed like the obvious place for us to extend to include information about the edition that was running as well. We therefore needed a way to ensure that the different editions of Fedora would produce a unique (and correct) version of the os-release file depending on the edition being installed. We did this by expanding the os-release file to include two new values: VARIANT and VARIANT_ID. VARIANT_ID is a machine-readable unique identifier that describes which version of Fedora is installed. VARIANT is a human-readable description.

In Fedora, the os-release file is maintained by a special RPM package called fedora-release. The purpose of this package is to install the files onto the system that guarantee this system is Fedora. Among other things, this includes os-release, /etc/fedora-release, /etc/issue, and the systemd preset files. (All of those will become interesting shortly).

So the first thing we needed to do was modify the fedora-release package such that it included a series of subpackages for each of the individual Fedora editions. These subpackages would be required to carry their own version of os-release that would supplant the non-edition version provided by the fedora-release base package. I’ll circle back around to precisely how this is done later, but for now accept that this is true.

So now that the os-release file on the system is guaranteed to contain appropriate VARIANT_ID, we needed to design a mechanism by which individual packages could make different decisions about their default configurations based on this. The full technical details of how to do this are captured in the Fedora Packaging Guidelines, but the basic gist of it is that any package that wants to behave differently between two or more editions must read the VARIANT_ID from os-release during its %posttrans (post-transaction) phase of package installation and place a symlink to the correct default configuration file in place. This needs to be done in the %posttrans phase because, due to the way that yum/dnf processes the assorted RPMs, there is no other way to guarantee that the os-release file has the right values until that time. This is because it’s possible for a package to install and run its %post script between the time that the fedora-release and fedora-release-EDITION package gets installed.

That all assumes that the os-release file is correct, so let’s explore how that is made to happen. First of all, we created a new directory in /usr/lib called /usr/lib/os.release.d/ which will contain all of the possible alternate versions of os-release (and some other files as well, as we’ll see later). As part of the %install phase of the fedora-release package, we generate all of the os-release variants and then drop them into os.release.d. We will then later symlink the appropriate one into /usr/lib/os-release and /etc/os-release during %post.

There’s an important caveat here: the /usr/lib/os-release file must be present and valid in order for any package to run the %systemd_post scripts to set up their unit files properly. As a result, we need to take a special precaution. The fedora-release package will always install its generic (non-edition) os-release file during its %post section, to ensure that the %systemd_post scripts will not fail. Then later if a fedora-release-EDITION package is installed, it will overwrite the fedora-release one with the EDITION-specific version.

The more keen-eyed reader may have already spotted a problem with this approach as currently described: What happens if a user installs another fedora-release-EDITION package later? The short answer was that in early attempts at this: “Bad Things Happened”. We originally had considered that installation of a fedora-release-EDITION package atop a system that only had fedora-release on it previously would result in converting the system to that edition. However, that turned out to A) be problematic and B) violate the principle of least surprise for many users.

So we decided to lock the system to the edition that was first installed by adding another file: /usr/lib/variant which is essentially just a copy of the VARIANT_ID line from /etc/os-release. In the %post script of each of the fedora-release subpackages (including the base subpackage), it is checked for its contents. If it does not exist, the %post script of a fedora-release-EDITION package will create it with the appropriate value for that edition. If processing reaches all the way to the %posttrans script of the fedora-release base package (meaning no edition package was part of the transaction), then it will write the variant file at that point to lock it into the non-edition variant.

There remains a known bug with this behavior, in that if the *initial* transaction actually includes two or more fedora-release-EDITION subpackages, whichever one is processed first will “win” and write the variant. In practice, this is effectively unlikely to happen since all of the install media are curated to include at most one fedora-release-EDITION package.

I said above that this “locks” the system into the particular release, but that’s not strictly true. We also ship a script along with fedora-release that will allow an administrator to manually convert between editions by running `/usr/sbin/convert-to-edition -e <edition>`. Effectively, this just reruns the steps that the %post of that edition would run, except that it skips the check for whether the variant file is already present.

Up to now, I’ve talked only about the os-release file, but the edition-handling also addresses several other important files on the system, including /etc/issue and the systemd presets. /etc/issue is handled identically to the os-release file, with the symlink being created by the %post scripts of the fedora-release-EDITION subpackages or the %posttrans of the fedora-release package if it gets that far.

The systemd presets are a bit of a special case, though. First of all, they do not replace the global default presets, but the do supplement them. This means that what we do is symlink in an edition-specific preset into the /usr/lib/systemd/system-preset/ directory. These presets can either enable new services (as in the Server Edition, where it turns on Cockpit and rolekit) or disable them (as in Workstation Edition where it shuts off OpenSSH). However, due to the fact that systemd only processes the preset files during its %post phase, we need to force systemd to reread them after we add the new values.

We need to be careful when doing this, because we only want to apply the new presets if the current transaction is the initial installation of the fedora-release-EDITION package. Otherwise, an upgrade could override choices that the user themselves have made (such as disabling a service that defaults to enabled). This could lead to unexpected security issues, so it has to be handled carefully.

In this implementation, instead of just calling the command to reprocess all presets, we instead parse the preset files and just process only those units that are mentioned in them. (This is to be overcautious in case any other package is changing the default enabled state besides systemd, such as some third-party RPMs that might have `systemctl enable httpd.service` in their %post section, for example.)

Lastly, due to the fact that we are using symlinks to manage most of this, we had to write the %post and %posttrans scripts in the built-in Lua implementation carried by RPM. This allowed us to call posix.symlink() without having to add a dependency on coreutils to do so in bash (which resulted in a circular dependency and broken installations). We wrote this as a single script that is imported by the RPM during the SRPM build phase. This script is actually coped by rpmbuild into the scriptlet sections verbatim, so the script must be present in the dist-git checkout on its own and not even as part of the exploded tarball. So when modifying the Lua script, it’s important to make sure to modify the copy in dist-git as well as the copy upstream.

Remote group merging for Fedora

The Problem

One of the major features of the Fedora Server Edition is the Cockpit administrative console. This web-based interface provides administrators with a powerful set of tools for controlling their system. Cockpit relies upon low-level tools like polkit and sudo to make authorization decisions to determine what a user is permitted to do. By default, most operations on a Fedora system are granted to users in the ‘wheel’ group. People granted administrator access to Cockpit (and other tools through shell access) are generally added to the wheel group in the /etc/group file.

This works reasonably well for single-user systems or very small environments where manual edits to /etc/group are maintainable, but in larger deployments, it becomes very unwieldy to manage lots of entries in /etc/group. In these cases, most environments switch over to using some form of a domain controller (such as FreeIPA, Microsoft Active Directory or a custom LDAP setup). These domain controllers allow users to be managed centrally, allowing administrators to make changes in a single place and have this be automatically picked up by all enrolled systems.

However, there is a problem: historically the group processing on Fedora (provided by glibc) has forced users to choose between using centrally managed groups (such as those provided by a domain and maintained by SSSD) or groups maintained on the local system in the /etc/group file. The behavior of glibc is specified in /etc/nsswitch.conf to decide which of the two mechanisms will “win” in the event of a conflict. This means that administrators need to decide up front whether their groups must all come from a domain controller or some locally.

The Solution

Over the last few months, I worked on adding a new feature to the glibc name-service functionality to enable “group merging”. The net effect is that now for all lookups of a group, glibc can be configured to check both the local files and the remote service and (if the group appears in both), combine the list of member users for both representations of the group into a single response.

Thus, it becomes possible to provide both local and central administrators into the wheel group. This can come in handy for example if an administrator wants to keep one or more local accounts available to do disaster recovery in the event that the machine loses access to the remote users (such as a bad update resulting in SSSD not starting).

Of course, this functionality does not come without a cost: because all merging lookups will try both data sources, it can result in a performance hit when operating against groups that otherwise would have been answered only by the local /etc/group file. With caching services like SSSD, this impact should be minimized.

Fedora and glibc upstream

The group merging patch has been submitted to the upstream glibc project but has not yet been merged into a release. It narrowly missed the 2.23 merge window, so it is currently slated for inclusion into glibc 2.24.

However, Carlos O’Donell has taken the patch and applied it to glibc in Fedora Rawhide (which will become Fedora 24), so it will be possible to take advantage of these features first in Fedora 24, before anyone else. (For anyone interested from other distributions, the patch should apply cleanly on 2.23 and likely with minimal effort atop 2.22 as well, since little changed besides this.)

My Git Workflow

It’s been a long while since I published a new entry on this blog, and I keep meaning to improve on that. Today, I was having a conversation with one of my colleagues today and discussing how I set up my local git checkouts. It occurs to me that this might be of interest, so I figure I’ll restart this blog by describing it.

This blog will describe specifically my workflow when dealing with upstreams hosted on Github. There are only minor changes to it when discussing non-Github projects (mainly in the public forking process).

Initial Setup

First, I have to find a project to get involved in. For some people, this is a difficult process involving a great deal of contemplation. For me, on the other hand, I seem to pick up new projects to get involved like they were falling from the sky. Most open-source projects these days seem to be hosted on Github (or at least have a presence there), so my workflow has become reasonably Github-centric.

I will skip over the part where I sign up for a Github account and set up two-factor authentication and uploaded my public SSH key, but rest assured that I have done all of those things. (If you aren’t using two-factor authentication anywhere that is even remotely important, fix that now. I also highly recommend the use of the open-source FreeOTP as a software token for either iOS or Android devices over Google Authenticator; it works anywhere Google Authenticator does.) You may also assume that I am properly logged in to Github at this point.

I’ll use the imaginary package “bar” created by Github user “foo” as my representative example. So I would browse to https://github.com/foo/bar and then click on the github-fork button. (Since I also belong to several Github organizations, this prompts me for which entity I am cloning to, but if you only have a personal account, it will probably skip this phase).

Local Repository Clone

Now that I have a public fork of the “bar” project, I want to be able to work with it. This means that I need to clone my repository to the local machine so I can operate on its contents. Github provides a handy way to identify the git URL needed for the cloning operation. When cloning my personal Github fork, I will want to clone using the “SSH” URL, which allows both reading from it and pushing changes. (I’ll talk about the “HTTPS” URL in a moment). To find the “SSH” URL, look on the main toolbar of the project’s Github main page. If you don’t see it, check for github-HTTPS and click on it, then select “SSH”. After that, it should look like github-SSH and there will be a URL in the text box to the right of it. It should look something like:

git@github.com:sgallagher/bar.git

Now we will open a terminal window, change to an appropriate containing directory and run:

git clone git@github.com:sgallagher/bar.git

This will pull down a copy of the repository onto the local system, ready to work with. I can make whatever local changes I want and run `git push` to submit them to my public fork. However, we are not finished. The next step will be to create an additional “git remote” that points at the original upstream repository. This I do in order to be able to track other changes that are happening upstream (particularly so I can rebase atop others’ work and ensure that my work still applies atop the upstream code). So in this case, I would do the following: first, I would browse to https://github.com/foo/bar again and

cd bar.git
git remote add upstream https://github.com/foo/bar.git
git remote update

This means that I am adding a new remote name (“upstream”) and associating it with the “HTTPS” (read-only) URL for the original project. (The `git remote update` piece causes me to pull the latest bits from upstream and store them locally.)

Special Case: Upstream Committer

In the special case where I also have commit privileges to the upstream repository, I also add another git remote called “upstream-push” using the “SSH” URL. Then, when I have patches ready to go upstream, I can ready them in a proper branch and then run

git push upstream-push local_branch_name:remote_branch_name

The reason for this additional upstream is to avoid accidental pushes to master (which is very easy to do if you have created a branch from e.g. upstream/remote_branch_name).

Appendix: Useful Git Aliases

In addition to my workflow convention above, I have also created a number of very useful git aliases that I store in my ~/.gitconfig file.

[alias]
 patch = format-patch -M -C --patience --full-index
 patches = format-patch -M -C --patience --full-index @{upstream}..
 up = remote update
 new = log ..@{upstream}

git patch

This handy alias is essentially a wrapper around `git format-patch`, which is useful for generating comprehensive patch files for `git send-email` (among other uses). It creates patches that auto-detect file copies and renames, diffs using the “patience” algorithm (which tends to be more human-readable than other algorithms) and displays git indices using their un-shortened forms (to avoid ambiguities).

This alias needs to be invoked with additional `git format-patch` arguments, such as a specification of which commits to generate patches. Which leads us to:

git patches

This is a variant of the above, except it will automatically generate all patches on the branch that are not present in the upstream branch being tracked. (So if you are tracking e.g. upstream/master, this will output all of the patches atop master).

git up

This is a quick shorthand for `git remote update` to automatically pull all of the new data from every remote configured in the checkout. This is particularly useful before a `git rebase -i upstream/branch` right before submitting a patch for code-review. (Which you should always do, to make the reviewer’s life easier.)

git new

This is another quick shorthand essentially for a variant of `git log` that only shows you your own commits atop the remote branch.

 

Rolekit (or “How I learned to stop thinking in terms of packages”)

What’s the problem?

Let’s start with a simplification and discuss the lifecycle of software at a high-level:

  1. Research and Development – In this phase, the software is designed, coded and (hopefully) tested.
  2. Packaging – Here, we take the compiled, tested bits of the software and bundle it up into some sort of package that can be used to deliver it to a user.
  3. Deployment – An end-user takes the package and does something interesting with it (for the purists out there, I’m lumping the test, staging and production environments into the “deployment” category).

Despite the brevity of the list above, there are a lot of moving parts here. I’m going to use the Fedora process to illustrate how this all works in a pre-rolekit world and then talk a little bit about the limitations, some of the alternatives and finally how rolekit addresses the issue. First, though, I’ll answer the question I posited in the header: “What’s the problem?”

The problem to be solved is how to get useful software up and running in an end-user’s environment with the least amount of difficulty for the user. The first and most important rule in software is this: software is a means to an end, not an end unto itself. People install a piece of software in order to achieve a goal. This goal could be something relatively simple, such as “I want to listen to this MP3 I bought” or as complex as “I run the IT department for a multinational manufacturing company and I want to keep track of all my products, the rate of their sales and margins as well as what my competitors are doing”. The job of software is to enable the user to get to that desired state. To that end, I would argue this: it is far more important to help the user get started than it is to offer them every possible feature.

Some of you may interject: “But if you don’t have the feature they need, won’t they go to someone who does?”. Sure, sometimes that will happen. But you will probably discover that people will make a different tradeoff than you might think: “I can get 90% of what I need and get it set up in a few weeks” is a far more compelling statement to make to a financial decision-maker than “This product provides everything we need, but I’ll need two more full-time people to get it running next year”.

What are we doing today?

Open source development is fairly unique compared to traditional software development. One of its major advantages for development can also become its biggest challenge to deployment. Because of the breadth of open source projects out there, there is almost always someone who has done at least a piece of what you want to do already. These other projects, such as coding libraries, web application frameworks, video game engines, etc. all provide the building blocks to start your work. The great thing here is that you can pick up the pieces that you need from somewhere else and then focus your attention only on the parts that make your project unique or exciting.

However, the challenge starts happening when you get to the packaging phase. Now that you have something you want to share with the world, you need to package it in a manner that allows them to use it. There are generally two schools of thought on how to do this, each with their own strengths and weaknesses.

  1. Grab the source code (or pre-built binaries) for everything that you depend on for your project to work and package them all together in a single deliverable.
  2. Package all of your dependencies separately in their own deliverables

I’m not going to go into the details of why, but the Fedora Project has policies that require the second option. (If you’re interested in the reasoning, I strongly recommend reading the Fedora Packaging Guidelines page on the subject). Fedora then provides a dependency-resolution mechanism that simplifies this case by ensuring that when you attempt to retrieve the package you want, it also automatically installs all of the packages that it depends on (and so on, recursively until they are all satisfied).

How do we deploy it now?

There are two schools of thought on this subject, which I will refer to as the “Fedora Approach” and the “Debian Approach”, since those two Linux distributions best represent them. (Note: my understanding of the Debian Approach is second-hand, so if I get any of the subtleties incorrect, please feel free to leave a comment and I’ll correct it).

The Debian Approach

In Debian and its derivatives (such as Ubuntu, Mint, etc.), when the package resolution is completed and the packages are downloaded, the user is required to indicate at that time their explicit decision on how the package must behave. Through a system called “debconf”, package installation is directly tied to deployment; the package installation cannot conclude without it being explicitly configured at that time. If the installation is non-interactive (such as if the installation was initiated by another service, rather than the user), the configuration must either be specified by an “answer file” (a configuration file passed to debconf stating the answers in advance) or else the package must provide a sensible set of defaults to automatically deploy it.

 The Fedora Approach

In Fedora and its derivatives (such as Red Hat Enterprise Linux, CentOS, Scientific Linux, etc.), when the package resolution is completed and the packages are downloaded, that’s it. In the vast majority of cases, the software is now on the system, but it is not configured to do anything at all. (There are a few specific exceptions which have been granted by the Fedora Engineering Steering Committee for things like the firewall). On these systems, nothing will happen until the user takes an explicit action to configure and start the services.

“That sounds like the Debian Approach is better!” you may say. However, there are concerns to be had here. For one, the above explanation I made about dependency-resolution comes into play; you as a user may not be fully aware of what packages are going to be pulled in by your dependencies (even accidentally). Furthermore, just because you installed a web-server package, it doesn’t mean that you necessarily want it running immediately. So, Fedora forces you to make these decisions explicitly, rather than implicitly. So when you’re ready, you configure the software and then start it up.

Where does this fall down?

The real problem is that the concept of “packages” derives very much from the engineering side of things. A package is a logical bundling of software for the developers. Not all problems can be solved with a single package, though. For example, the FreeIPA identity-management solution requires many top-level packages including an LDAP directory server, a certificate authority server, a DNS server and others. In this, the concept of a “package” gets more than a little fuzzy. In this particular case (as has been common historically), the solution was “Let’s make another package that glues them together!”. So the FreeIPA package just adds those other packages to its dependency chain.

But just adding more packages doesn’t necessarily solve the end-user concern: How do I easily deploy this?

Enter rolekit

Rolekit was designed to be specifically for handling the deployment situation and shield end-users from the concept of project-level packages. Instead, complete solutions will be “packaged” as Server Roles. Users will come to rolekit and declare a machine to be e.g. a Domain Controller, providing the minimum information necessary to set it up (today, that’s just an admin password in the Domain Controller example). Rolekit will handle all of the other necessary work under the hood, which involves downloading the appropriate packages, installing them on the system, setting up the configuration, starting the appropriate services and carefully opening up the firewall to allow access to it.

There are a lot of moving parts involved in deploying a role, but the user doesn’t really need to know what they are. If they can be shielded from much of the noise and churn inherent in package installation, configuration, service management and firewall settings, then they get back much of their time for solving the problems unique to their environments.

Fedora and Server Roles

As of Fedora 21, we have implemented the first release of the rolekit framework as well as a single representative Role: the Domain Controller. For Fedora 22, we’re working with the Cockpit project to produce a simple and powerful graphical interface to deploy the Domain Controller Role as well as building a new Database Server Role. As the project progresses, we very much hope that others will come forward to help us build more solutions. A few that I’d love to see (but don’t have time to start on yet):

  • A fileserver role that manages Samba and NFS file-shares (maybe [s]ftp as well).
  • A mail and/or groupware server role built atop something like Kolab
  • A backup server

Welcome to the post-package world, my friends!

Flock to Fedora: Day One (Afternoon)

Overview

The afternoon today has been absolutely filled with excellent talks. As with this morning, I’ve captured most of them for your entertainment and edification below. These blog entries are getting very long; perhaps I need an index.

Sessions

Fedora Workstation: Goals, Philosophy and Future

“The desktop is not dying!” cries Christian Schaller as he dives into his talk on the Fedora Workstation. In a throwback to the old saw about the Year of Linux on the Desktop, he points out that 35% of laptop sales are Chromebooks and that there are now over six hundred games available on Steam for Linux.

So what is the Fedora Workstation trying to achieve? What are its driving principles? First of all, Fedora Workstation is the only desktop out there that isn’t trying to sell you (in other words, capture information about you to sell to someone else). Fedora Workstation will consider end-user privacy one of its highest goal. By default, Workstation will not share your data.

The recent switch to focusing on delivering specific Fedora Products gave the Workstation an opportunity to start defining standards and maintaining them within a particular Product. Traditionally, Fedora was effectively a collection of exactly what upstreams provided. With the Workstation, we’ll be able to curate this code and integrate it into a cohesive whole (Author’s note: bingo!). One of the key points here is that by providing a known base, we give developers a better idea of what they can rely upon when writing their applications and services.

Christian continues on to discuss more about the future plans of the Workstation. The first of these was a discussion about how containers will fit into the Workstation world. Significant research is going into figuring out how to use container images to isolate and secure applications from one another, which will offer both flexibility and security in the long run. With some of the features offered by Wayland, this will eventually provide an excellent (and safe!) experience for users. Christian also notes that their goal is to make this all seamless from the users’ perspectives. If they notice it’s been done (without being told), they failed.

The Workstation Working group has been doing extensive research on Docker as a container implementation but has yet to make a decision on whether to standardize on Docker or else use a customized namespacing solution.

Christian then talks a bit about how Workstation is going to try to grow the platform. He acknowledges that there has been a slow decline of usage in Fedora and wants to put effort into drawing people back in. Some of the plans for this involve increased publicity through blogging and the Fedora Marketing groups. It will also involve finding ways to reduce the split between applications that run on Fedora and Red Hat Enterprise Linux. In part, that involves making sure that newer APIs are available in RHEL and older interfaces remain available in Fedora.

Another key piece of Fedora Workstation’s long-term strategy involves the creation and maintenance of a true ABI at all levels of the stack. This will help reduce the “moving target” aspect of Fedora and provide more guarantees about long-term sustainability of an application written for Fedora.

Christian takes a moment to talk about the target users. A lot of the conversation so far has been about making Fedora better for developers, but from his perspective it is also necessary to build a platform that’s useful to creators of all sorts. “Creators” may include video editors, 3D-modeling, music mixing and all sorts of other creation tasks.

After this, the conversation moved on to discuss a little bit of the challenges they face. Marketing and PR is going to be a significant challenge, particularly correcting some existing negative press that we’ve gathered over the last few years. We’ve also got to convince ourselves. There have been a lot of big changes in Fedora lately (the Three Product Plan being the most visible) and it’s clear that there’s going to be a period of adjustment as we course-correct and really figure out how we’re going to move forward.

Christian then talks about the advent of web-based applications and how that relates to the Workstation. He notes that there are users (even at Red Hat) that never open any local application other than their web browser. From this perspective, it’s very clear that we need Fedora Workstation to be a powerful mechanism for running web applications. So one of the things being worked on is ways that web applications can be tied into the desktop environment in a more integrated way.

Comments during and after the talk focused a great deal on the publicity and marketing aspects of things. It was noted that the more targeted user groups lends itself better to more controlled messaging. This will bring in more users in those particular groups who will in turn pass their good experience on by word-of-mouth.

Several questions were also asked about specific feature enhancements to things like Gnome Online Accounts (particularly around the earlier discussion of web applications). Christian indicated that the next  immediate efforts on that front will likely be focused around a better calendaring experience.

Evolving the Fedora Updates Process

I next attended a talk about the Fedora update process given by Luke Macken, author and maintainer of the Bodhi update system.

Once upon a time in Fedora Core 1 through Fedora Core 3, updates were handled via a manual process involving emails to release engineering. Starting with Fedora Core 4, a private internal updating system that was available only to Red Hat employees.

The modern world of Bodhi began in Fedora 7 at the same time that Fedora Core and Fedora extras were merged. It introduced the concept of Karma and it was written in TurboGears 1.x and it is still in production today, seven years and many revisions later.

Bodhi does a lot of things behind the scenes, being both extremely intricate and very inefficient. Luke described a number of issues that have cropped up over the years, including inflexible SQL routines and the karma process outliving its usefulness.

Luke next took a little side-trip to tell us about some of the more entertaining glitches that have cropped up over the years, including the infamous Fedora 9 GPG re-keying and numerous crashes during update pushes.

After that, he moved on to discussing the plans for the Bodhi2 project. The plan is to have it land sometime after the Fedora 21 release. We don’t want to rely on it for zero-day updates, but we’ll phase it in soon after and it should hopefully be a graceful transition.

Some of the major changes in Bodhi2 will be a comprehensive REST API, a new and improved command-line tool and major changes to the UI that should provide a better experience for the users.

Another great feature of Bodhi2 is that it will integrate with fedmsg and the new Fedora Message Service to reduce the amount of “spam” email that Bodhi sends out. Luke dives in a bit to talk about the datagrepper and datanommer mining tools that power the notification service and the set of filters that you can opt into.

Luke showed off how Bodhi2 will be tightly integrated with Taskotron to perform automated testing on updates, as well as the integration with the Fedora Badges (there are lots of them available for Bodhi!) and then on to the feedback system. He called out both the fedora-easy-karma command-line tool and the fedora-gooey-karma GUI tool for managing karma updates on Bodhi1 (and noted that they will be working together to support Bodhi2 as well).

Then he went and left me slack-jawed with the new submitter process, automating almost everything and making it almost unbelievably simple. Adding to that, the new karma system allows the submitter to select fine-grained karma controls, so they can request that specific tests have to pass karma before accepting it into the stable repository.

The talk finished up with some prognosticating about the future, particularly talking about being able to run AMI and other cloud image updates through as well.

State of the Fedora Kernel

The next stop on my whirlwind tour of the wide world of Fedora was Josh Boyer’s annual discussion on Fedora’s treatment of the kernel. First up on the agenda was an overview of the release process. Fedora focuses on having a common base kernel across all stable releases (which means that bugs and features are shared). Fedora rebases to the latest upstream kernel on a regular basis, staggering the updates back to the two stable releases.

Josh described how the old process for kernel updates was to be more conservative on updates on older Fedora releases. However, a few years ago the process changed and updates are now faster and keeps Fedora much closer to the kernel upstream.

The talk then moved on to discussing the state of open bugs in the kernel. During this talk in 2013, there were 854 open bugs against the Fedora Kernel. After last year, the kernel maintainers sat down and spent a lot of time to knock down the bugs. Today it’s down to 533, but this is still not good enough. There will be a talk on Saturday about some ways to address this.

Josh pointed out several consistent problem areas: WiFI support, suspend/resume, video playback and brightness settings, platform-specific drivers (like Fn keys) and bluetooth. “All the stuff that involves ACPI is usually broken on some platform, somewhere”.

He then moved on to talking about how they handle bugs. He pointed out that if someone files a bug, they’re contacted (the bug is set NEEDINFO) every time the kernel is rebased. If after two weeks the reporter doesn’t confirm a fix (or a continuing bug), the bug is closed INSUFFICIENT_INFO.

What does all this mean? In short, it means that the Fedora kernel maintainers are permanently saturated, the work is never-ending and they would really appreciate if people would take vacations at different times than the maintainers so they don’t always return to 200+ extra bugs. Additionally, they really need help with triage, but it’s difficult to find anyone to do so, mainly because bug triage is admittedly boring. Some steps have been made in the last couple years that really helps, particularly the ABRT bug reports and the retrace server that helps narrow down which bugs are having the widest impact. The retrace server in particular keeps statistics on the number of reports, so that helps in prioritizing the fixing efforts.

After the bug discussion, Josh moved on to talking about the situation with Fedora 21. The plan is to release either kernel 3.16 or 3.17 at release, depending on schedule slips. During the Fedora 21 process, the kernel maintenance has actually been fairly calm, despite a set of new packaging changes.

During the Fedora.next process, the Fedora Cloud Working Group made requests of the kernel team to shrink down its size. There are a lot of optional components built into the kernel and many of these weren’t actually needed in a cloud environment. So the kernel team went and broke out the available modules into a core set and a common set above that. This made the minimal installation set much smaller and reduced the space on the cloud images substantially. In addition to this, they found ways to compress the kernel modules so the storage on disk shrank quite a bit as well.

Another useful feature that was added to packaging in Fedora 21 is support for automatically-generated RPM “Provides” listing the set of modules that are present in each of the packages. This will make it easier for packagers to specify dependencies on the appropriate package (and will continue working if modules move around).

The last major change in Fedora 21 is support for 64-bit ARM hardware (aarch64), which as was noted by an audience member is now available for general purchase. It works fairly well (thanks in large part to a herculean effort by Red Hat ARM engineers) and may be promoted to a primary architecture in Fedora 22 or 23. As a side-effect of this work, it’s going to be possible to replace the slow armv7hl builders in Koji with the new aarch64 builders that will be vastly more performant.

Josh then moved on to discuss the new kernel Playground, which is a new unsupported COPR containing some new experimental features. It tracks Fedora Rawhide and 21 and provides today the Overlayfs v23 (a union filesystem) and kdbus (the high-performance kernel D-BUS implementation). These are fairly stable patches to the kernel that are still out of the main tree and therefore not really suitable for Fedora proper (yet).

In the future, it may include other features such as kpatch and kgraft (the in-kernel infrastructure for supporting live-patching the kernel).

Advocating Fedora.next

After taking a short break to catch my breath and give my fingers a rest (these blog entries are a lot of work!), I went along to Christoph Wickert’s session Advocating Fedora.next. This session was largely (but not exclusively) directed at Fedora Ambassadors, informing them how best to talk about the Fedora.next initiative to the public.

He began his talk by addressing the question of “Why?”. Why did we need to change things so substantially? After providing the delightfully glib answer “Why not?”, Cristoph described a bit about Fedora’s history. He pointed out quite eloquently how Fedora has always been about change. Fedora has never been afraid to try something new to improve.

He then tackled some of the non-reasons behind Fedora.next, specifically the rumors of our demise post-GNOME 3 and similar. The truth is that we have a strong brand with a large contributor base that is extremely closely linked to upstream development (more so than many, if not all, of the other distributions). We’ve had a decade of successes and Fedora 20 has been very positively reviewed overall.

Another such rumor was that the new products are a benefit only to Red Hat. The obvious rebuttal here is that separating products makes good sense, focusing on specific user sets. Also, Red Hat has no particular interest in a consumer workstation product.

A final common criticism is that the new working groups are a power-grab by Red Hat. The older governance has not changed (and remains community-elected). Furthermore, all of the working groups were self-nominated and contains numerous non-Red Hat community members, including Christoph himself.

Christoph then ruminates, “If we didn’t do it for those reasons, why did we do it?”. The first answer is that distributions have become boring. All major distributions have declining market share. The general impression is that the distro is a commodity and that the users don’t care which one they’re running. The user really only cares that their applications run. Things like cloud environments and containers blur this line as well.

Continuing on, Christoph calls out to the Fedora Mission statement: “The Fedora Project’s mission is to lead the advancement of free and open source software and content as a collaborative community” and asks whether we feel like we’ve been living it. With that in mind, he defines Fedora.next for us: an umbrella term for the changes in the way that we create and release Fedora.

Fedora.next was born of two proposals that were first publicised at last year’s Flock conference: “The Architecture for a More Agile Fedora” by Matthew Miller and “The Fedora Crystal Ball: Where are we going for the next five years?” by yours truly, Stephen Gallagher. As the last year has passed, the Fedora.next that we now know has become a merge of these two proposals.

Matthew’s proposal involved having different policies depending on how low in the stack that a package lived, with core functionality having stricter guidelines than packages up in the application layer. My proposal was around having three different development streams (Server, Workstation and Cloud) and possibly different release cycles. The modern vision will be a combination of the two, with three products.

Christoph also warned the Ambassadors to be aware that the Fedora installation DVD will be retired in Fedora 21, but notes that it was never truly well-maintained and that its replacement netinstalls, spins and live media should be at least sufficient.

“What is a product?”, Christoph asks, then answers. A Product is more than simply a Fedora Spin. Each product has a defined target audience, a mission statement, a product requirements document (PRD) and a technical specification. The mission statement, PRD and technical statement were all defined and discussed publicly by the Product Working Groups and ratified by the Board and FESCo. Each product contains features not present in older Fedoras and has its own working group with their own governance models.

Christoph stresses that this is not a power-grab but instead the opposite: it’s an opportunity to give more power to the specific people who are building the Products. As a member of the Workstation Working Group, he calls out to its Mission Statement and then discusses a few of the Workstation-specific cool features. He notes that what Fedora Workstation will look like in Fedora 21 will not be a huge difference from the classic Fedora Live image, but this will change over time as the Workstation comes into its own life.

He then continues on to discuss Fedora Server a bit, calling out the exciting new Cockpit system management console.

Moving on to the Fedora Cloud, Christoph asks for Matthew Miller in the audience to comment further on it. Matthew describes the pets vs. cattle metaphor and explains that Fedora Cloud is really meant to fill the “cattle” side of that metaphor. Matthew notes the work towards the Fedora Base Image and the Fedora Atomic effort.

Christoph notes that this is an excellent example of why Spins are not enough. For example, when distributing cloud images, it doesn’t really meet the definition of a Spin because it doesn’t install via anaconda.

The talk then moves on to discussing the remaining two working groups: “Base Design” and the “Environments and Stacks” groups. Talking about the “Base Design”, he stresses that the idea is for the base to be as small as possible and provide a common framework for the Products to build on. The “Environments and Stacks” working groups are focused on making developers’ lives easier, providing the ability to install known software (development) stacks, possibly different versions side-by-side.

Christoph summarizes that there has been a great deal of misinformation put out there and he calls out to the Ambassadors and everyone else to explain what’s really going on, how it works and why it’s a positive change in Fedora. The message must be positive, because the change is exciting and there’s much more to come. He cautions “it’s not just ‘the next fedora’, it’s ‘Fedora.next’.”

Daily Summary

It’s really hard to pick out one specific thing to say about Flock’s first day. Every one of the speakers I listened to today were excited, engaging and clearly love Fedora, warts and all. I think I’ll leave it there for today.