Base Runtime and the Generational Core

A Quick Primer on Modularity

Modularity (formerly, Modularization) is an ongoing initiative in Fedora to resolve the issue of divergent, occasionally conflicting lifecycles of different components. A module provides functionality (such as a web server) and includes well-integrated and well-tested components (such as Apache httpd and the libraries on which it depends). It can be deployed into production in various ways: as “classic” RPM packages or a container image, and is updated as a whole. Different modules can emphasize new features, stability, security, etc. differently.

Modules differ from traditional packaging in certain important ways. Perhaps most importantly, they allow us to separate internal implementation details from the exposed interfaces of the module. Historically in Fedora, if a packager wanted to deliver a new web application, that would also often mean that they needed to package and carry the framework or other libraries used by that application. This tended to be a double-edged sword: on the one hand, those libraries were now available for anyone to pick up and use in Fedora. However, in many cases, this meant that the primary maintainer of that package might actually have no specific knowledge or understanding of it except that its lack would mean their application didn’t work. This can be a problem if a person is carrying around a library for the use of a single helper function and don’t want to be responsible for issues in the rest of the library.

With a modular approach, the module itself will provide a definition of what public interfaces are stable for use by other projects. This way, they can opt to contain an internal-only implementation of some libraries.

A good metaphor for Modularity might be urban planning: sections of the Earth are selected for establishing housing, businesses and other construction projects. Each of these projects would be effectively a module.

Base Runtime: The Bedrock

The first thing that a construction company would look at when establishing a new building site would be the ground on which it is to be constructed. It is essential that this location be sturdy, reliable and capable of supporting the weight of the projects being built atop it.

In the Fedora Project, the bedrock upon which the other modules will be built is called the Base Runtime. The definition of this project has gone through a number of revisions over the last few months, but at this point it is fairly settled down to this:

The Base Runtime contains the software necessary to boot the system to a running kernel and the runtime libraries for the most rudimentary operation of the system.

In practical terms, this means that the Base Runtime is

• Not installable by itself. It can only boot to a kernel; it has no init system or running applications.
• Not self-hosting. It does not contain the packages necessary to rebuild itself from source. Other modules will need to provide this.
• Limited to packages with an extremely stable public API. The Base Runtime needs to be swappable at any time out from under the running system without impacting the operation of applications running atop it.

System Runtime: Urban Infrastructure

 

Once you have a location chosen and have built some homes and some businesses, you need a certain amount of infrastructure to connect it all together: roads, plumbing, electricity, etc.

In the case of a computer operating system, this means things like service control and monitoring, filesystem operations, command shells and other basic tools for operating on and maintaining the system. In Fedora, this means essentially the bash shell environment, login services and the standard POSIX command utilities.

The primary reason for separating the System Runtime from the Base Runtime is to allow these two modules to carry different API lifecycle guarantees. While the Base Runtime will need to remain backwards compatible for an extended period, it may be permissible for the System Runtime to see revisions at a higher rate (where it makes sense to provide new functionality faster). For example, Fedora may wish to update the systemd project (used for control, monitoring and management of system services) at a much higher rate than the low-level C runtime library.

Shared Components: Building Materials

In order to build up your city, you naturally need a set of raw materials. Wood, stone, metal and essential workers.

The third and final piece of the puzzle is the Shared Components module. This special module  is comprised of the set of low-level libraries common to both the Base Runtime and the System Runtime. Some of these libraries may be made available for other services to consume, but the majority of them will be held privately within the modules.

Generational Core: Local Government

After building up the town, it is important to have a mechanism in place for maintaining it, improving it and making sure to adapt to changing conditions in the world around it. In the urban planning world, this would mean that the local government would establish policies and groups capable of performing these tasks. They would be given access to all of the tools used to create the city in the first place and would continue to monitor things and address issues as they arise. This is effectively what the Generational Core is: the tying together of all those disparate components as a single entity.

While the Base Runtime, System Runtime and Shared Components modules will be built separately and maintained for independent lifecycles, they will be delivered to end-users as part of a single combined module stack called the Generational Core. (Defining “Generational” in the sense of “genealogy” as opposed to “creation”).

Unlike the Base Runtime and System Runtime, the Generational Core will be installable and very similar to the “minimal install” of previous releases of Fedora. It will be somewhat more stripped down even than those. For example, the Generational Core does not need to provide network management services, remote login capabilities or network storage connectivity. These features will be provided by additional modules and module stacks built atop the Generational Core.

Fedora Server: Expanding Throughout the Galaxy

History

Three years ago, Fedora embarked on a new initiative that we collectively refer to as Fedora.next. As part of this initiative, we decided to start curating deliverable artifacts around specific use-cases rather than the one-size-fits-all approach of Fedora 20 and earlier. One of those specific use-cases was to meet the needs of “server administrators”. And thus, the Fedora Server Edition was born.

One of the earliest things that we did after creating the Fedora Server Working Group (Server WG from here on) was to perform what in the corporate world might be called a “gap analysis”. What this means is that we looked at Fedora from the perspective of the server administrator “personas” we had created and tried to understand their pain points (particularly in contrast to how things function on competitive platforms such as Microsoft Windows Server).

The most obvious gap that we identified was the relative difficulty of getting started with Fedora Server Edition at all. With Microsoft Windows Server, the first experience after logging in is to be presented with a tool called Server Manager that provides basic (graphical) information about the system as well as presenting the user with a list of things that they might want this server to do. It then walks them through a guided installation of those core features (such as domain controller services, remote desktop services and so on). With Fedora, a default install would get you a bash prompt with no guidance; typing “help” would only lead to the extremely expert-focused help documentation for the bash shell itself.

OK, advantage Windows here. So how do we address that? Server WG had agreed early on that we were not willing require a desktop environment for server systems. We instead set our sights on a fledgling project called Cockpit, which was gaining traction and looked to provide an excellent user experience without requiring a local display – it’s a web-based admin console and so can be accessed by users running the operating system of their choice.

Once Cockpit was established as the much-friendlier initial experience for Fedora Server, we started to look at the second part of the problem that we needed to solve: that of simplified deployment of key infrastructure components. To that end, we started the development of a tool that we could integrate with the Cockpit admin console and provide the actual deployment implementation. What we came up with was a python project that we called rolekit that would provide a fairly simple local D-BUS API that Cockpit would be able to call out to in order to deploy the requested services.

While our intentions were good, rolekit faced two serious problems:

• The creation of the roles were complicated and manual, requiring careful curation and attention to make sure that they continued to work from release to release of Fedora.
• The Cockpit Project became very popular and its limited resources became dedicated to serving the needs of their other consumers, leaving us unable to get the integration of rolekit completed.

The second of these issues remains and will likely need to be addressed, but that will be a topic for another day. The remainder of this blog entry will discuss our plans for how to improve the creation and maintenance of roles.

Ansible Galaxy

Ansible Galaxy describes itself as “[Y]our hub for finding, reusing and sharing the best Ansible content”. What this means is that the Ansible project runs a public software service enabling the sharing of Github repositories containing useful Ansible roles and playbooks for deploying software services.

The Galaxy hub contains literally thousands of pre-built server roles for Fedora, Red Hat Enterprise Linux and other systems with more being added every day. With such a large community made available to us, the Server WG has decided to explore the use of Ansible Galaxy as the back-end for our server role implementation, replacing rolekit’s custom (and redundant) implementation.

As part of this effort, I attended the Ansible Contributor Conference and AnsibleFest this week in Brooklyn, New York. I spent a great deal of time talking with Chris Houseknecht about ways in which we could enhance Ansible Galaxy to function for our needs.

Required Galaxy Enhancements

There are a few shortcomings to Galaxy that we will need to address before we can implement a complete solution. The first of these is assurance: there is currently no way for a consumer of a role to indicate its suitability. Specifically, we will want there to be a way for Fedora to elevate a set of roles (and specific versions of those roles) to a “recommended” state. In order to do this, Galaxy will be adding support for third-party signing of role versions. Fedora will become a “signing authority” for Ansible Galaxy, indicating that certain roles and their versions should be consumed by users of Fedora.

We will also add filtering to the Galaxy API to enable clients to limit their searches to only those roles that have been signed by a particular signing authority. This will be useful for limiting the list that we expose to users in Cockpit.

The other remaining issue with Ansible is that there is currently no way to execute an Ansible script through an API; at present it must be done via execution of the Ansible CLI tool. Fedora will be collaborating with Ansible on this (see below).

Required Fedora Enhancements

In Fedora, we will need to provide a useful UI for this new functionality. This will most likely need to happen in the Cockpit project, and we will have to find resources to handle this.

Specifically, we will need:

• UI to handle searching the Galaxy API using the role signatures and other tag filtering.
• UI for an “answer file” for satisfying required variables in the roles.
• UI for interrogating a system for what roles have been applied to it.

In addition to Cockpit UI work, we will need to provide infrastructure within the Fedora Project to provide our signatures. This will mean at minimum having secure, audited storage of our private signing key and a tool or service that performs the signing. In the short term, we can allow a set of trusted users to do this signing manually, but in the longer term we will need to focus on setting up a CI environment to enable automated testing and signing of role updates.

Lastly, as mentioned above, we will need to work on an API that Cockpit can invoke to fire off the generated Ansible playbook. This will be provided by Fedora (likely under the rolekit banner) but may be eventually absorbed into the upstream Ansible project once it matures.

Flocking to Kraków

In less than five days, the fourth annual Flock conference will take place in Kraków, Poland. This is Fedora’s premier contributor event each year, alternately taking place in North America and Europe. Attendance is completely free for anyone at all, so if you happen to be in the area (maybe hanging around after World Youth Day going on right now), you should certainly stop in!

This year’s conference is shaping up to be a truly excellent one, with a massive amount of exciting content to see. The full schedule has been available for a while, and I’ve got to say: there are no lulls in the action. In fact, I’ve put together my schedule of sessions I want to see and there are in fact no gaps in it. That said, here are a few of the sessions that I suspect are going to be the most exciting:

Aug. 2 @11:00 – Towards an Atomic Workstation

For a couple of years now, Fedora has been at the forefront of developing container technologies, particularly Docker and Project Atomic. Now, the Workstation SIG is looking to take some of those Project Atomic technologies and adopt them for the end-user workstation.

Aug. 2 @17:30 – University Outreach

I’ve long held that one of Fedora’s primary goals should always be to enlighten the next generation of the open source community. Over the last year, the Fedora Project began an Initiative to expand our presence in educational programs throughout the world. I’m extremely interested to see where that has taken us (and where it is going next).

Aug. 3 @11:00 – Modularity

This past year, there has been an enormous research-and-development effort poured into the concept of building a “modular” Fedora. What does this mean? Well it means solving the age-old Too Fast/Too Slow problem (sometimes described as “I want everything on my system to stay exactly the same for a long time. Except these three things over here that I always want to be running at the latest version.”). With modularity, the hope is that people will be able to put together their ideal operating system from parts bigger than just traditional packages.

Aug. 3 @16:30 – Diversity: Women in Open Source

This is a topic that is very dear to my heart, having a daughter who is already finding her way towards an engineering future. Fedora and many other projects (and companies) talk about “meritocracy” a lot: the concept that the best idea should always win. However the technology industry in general has a severe diversity problem. When we talk about “meritocracy”, the implicit contract there is that we have many ideas to choose from. However, if we don’t have a community that represents many different viewpoints and cultures, then we are by definition only choosing the best idea from a very limited pool. I’m very interested to hear how Fedora is working towards attracting people with new ideas.

 

OpenShift and SSSD Part 3: Extended LDAP Attributes

Overview

This is the third post in a series on setting up advanced authentication mechanisms with OpenShift Origin. This entry will build upon the foundation created earlier, so if you haven’t already gone through that tutorial, start here and continue here.

Configuring Extended LDAP Attributes

Prerequisites

• SSSD 1.12.0  or later. This is available on Red Hat Enterprise Linux 7.0 and later.
• mod_lookup_identity 0.9.4 or later. The required version is not yet available on any released version of Red Hat Enterprise Linux 7, but RPMs for this platform are available from upstream at this COPR repository until they arrive in Red Hat Enterprise Linux.

Configuring SSSD

First, we need to ask SSSD to look up attributes in LDAP that it normally doesn’t care about for simple system-login use-cases. In the OpenShift case, there’s really only one such attribute: email. So we need to modify the [domain/DOMAINNAME] section of /etc/sssd/sssd.conf on the authenticating proxy and add this attribute:

[domain/example.com]
...
ldap_user_extra_attrs = mail

Next, we also have to tell SSSD that it’s acceptable for this attribute to be retrieved by apache, so we need to add the following two lines to the [ifp] section of /etc/sssd/sssd.conf as well:

[ifp]
user_attributes = +mail
allowed_uids = apache, root

Now we should be able to restart SSSD and test this configuration.

# systemctl restart sssd.service

# getent passwd <username>
username:*:12345:12345:Example User:/home/username:/usr/bin/bash

# gdbus call \
        --system \
        --dest org.freedesktop.sssd.infopipe \
        --object-path /org/freedesktop/sssd/infopipe/Users/example_2ecom/12345 \
        --method org.freedesktop.DBus.Properties.Get \
        "org.freedesktop.sssd.infopipe.Users.User" "extraAttributes"
(<{'mail': ['username@example.com']}>,)

Configuring Apache

Now that SSSD is set up and successfully serving extended attributes, we need to configure the web server to ask for them and to insert them in the correct places.

First, we need to install and enable the mod_lookup_identity module for Apache (See note in the “Prerequisites” setting for installing on RHEL 7):

# yum -y install mod_lookup_identity

Second, we need to enable the module so that Apache will load it. We need to modify /etc/httpd/conf.modules.d/55-lookup_identity.conf and uncomment the line:

LoadModule lookup_identity_module modules/mod_lookup_identity.so

Next, we need to let SELinux know that it’s acceptable for Apache to connect to SSSD over D-BUS, so we’ll set an SELinux boolean:

# setsebool -P httpd_dbus_sssd on

Then we’ll edit /etc/httpd/conf.d/openshift-proxy.conf and add the following lines (bolded to show the additions) inside the <ProxyMatch /oauth/authorize> section:

  <ProxyMatch /oauth/authorize>
    AuthName openshift

    LookupOutput Headers
    LookupUserAttr mail X-Remote-User-Email
    LookupUserGECOS X-Remote-User-Display-Name

    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER
 </ProxyMatch>

Then restart Apache to pick up the changes.

# systemctl restart httpd.service

Configuring OpenShift

The proxy is now all set, so it’s time to tell OpenShift where to find these new attributes during login. Edit the /etc/origin/master/master-config.yaml file and add the following lines to the identityProviders section (new lines bolded):

  identityProviders:
  - name: sssd
  challenge: true
  login: true
  mappingMethod: claim
  provider:
    apiVersion: v1
    kind: RequestHeaderIdentityProvider
    challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
    loginURL: "https://proxy.example.com/login-proxy/oauth/authorize?${query}"
    clientCA: /etc/origin/master/proxy/proxyca.crt
    headers:
    - X-Remote-User
    emailHeaders:
    - X-Remote-User-Email
    nameHeaders:
    - X-Remote-User-Display-Name

Go ahead and launch OpenShift with this updated configuration and log in to the web as a new user. You should see their full name appear in the upper-right of the screen. You can also verify with oc get identities -o yaml that both email addresses and full names are available.

Debugging Notes

OpenShift currently only saves these attributes to the user at the time of the first login and doesn’t update them again after that. So while you are testing (and only while testing), it’s advisable to run oc delete users,identities --all to clear the identities out so you can log in again.

OpenShift and SSSD Part 2: LDAP Form Authentication

Overview

This is the second post in a series on setting up advanced authentication mechanisms with OpenShift Origin. This entry will build upon the foundation created earlier, so if you haven’t already gone through that tutorial, start here. Note that some of the content on that page has changed since it was first published to ensure that this second part is easier to set up, so make sure to double-check your configuration.

Configuring Form-based Authentication

In this tutorial, I’m going to describe how to set up form-based authentication to use when signing into the OpenShift Origin web console. The first step is to prepare a login page. The OpenShift upstream repositories have a handy template for forms, so we will copy that down to our authenticating proxy on proxy.example.com.

# curl -o /var/www/html/login.html \
    https://raw.githubusercontent.com/openshift/openshift-extras/master/misc/form_auth/login.html

You may edit this login HTML however you prefer, but if you change the form field names, you will need to update those in the configuration below as well.

Next, we need to install another Apache module, this time for intercepting form-based authentication.

# yum -y install mod_intercept_form_submit

Then we need to modify /etc/httpd/conf.modules.d/55-intercept_form_submit.conf and uncomment the LoadModule line.

Next, we’ll add a new section to our openshift-proxy.conf inside the <VirtualHost *:443> block.

  <Location /login-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://openshift.example.com:8443/oauth/authorize

    InterceptFormPAMService openshift
    InterceptFormLogin httpd_username
    InterceptFormPassword httpd_password

    RewriteCond %{REQUEST_METHOD} GET
    RewriteRule ^.*$ /login.html [L]
  </Location>

This tells Apache to listen for POST requests on the /login-proxy/oauth/authorize and pass the username and password over to the openshift PAM service, just like in the challenging-proxy example in the first entry of this series. This is all we need to do on the Apache side of things, so restart the service and move back over to the OpenShift configuration.

In the master-config.yaml, update the identityProviders section as follows (new lines bolded):

  identityProviders:
  - name: any_provider_name
    challenge: true
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
      loginURL: "https://proxy.example.com/login-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxy/proxyca.crt
      headers:
      - X-Remote-User

Now restart OpenShift with the updated configuration. You should be able to browse to https://openshift.example.com:8443 and use your LDAP credentials at the login form to sign in.

OpenShift and SSSD Part 1: Basic LDAP Authentication

Overview

OpenShift provides a fairly simple and straightforward authentication provider for use with LDAP setups. It has one major limitation, however: it can only connect to a single LDAP server. This can be problematic if that LDAP server becomes unavailable for any reason. When this happens, end-users get very unhappy.

Enter SSSD. Originally designed to manage local and remote authentication to the host OS, it can now be configured to provide identity, authentication and authorization services to web services like OpenShift as well. It provides a multitude of advantages over the built-in LDAP provider; in particular it has the ability to connect to any number of failover LDAP servers as well as to cache authentication attempts in case it can no longer reach any of those servers.

These advantages don’t come without a cost, of course: the setup of this configuration is somewhat more advanced, so I’m writing up this guide to help you get it set up. Rather than adding a few lines to the master-config.yml in OpenShift and calling it a day, we are going to need to set up a separate authentication server that OpenShift will talk to. This guide will describe how to do it on a dedicated physical or virtual machine, but the concepts should also be applicable to loading up such a setup in a container as well. (And in the future, I will be looking into whether we could build such a static container right into OpenShift, but for now this document will have to suffice.) For this guide, I will use the term VM to refer to either type of machine, simply because it’s shorter to type and read.

This separate authentication server will be called the “authenticating proxy” from here on out and describes a solution that will provide a specialized httpd server that will handle the authentication challenge and return the results to the OpenShift Server. See the OpenShift documentation for security considerations around the use of an authenticating proxy.

Formatting Notes

• If you see something in italics within a source-code block below, you should replace it with the appropriate value for your environment.
• Source-code blocks with a leading ‘#’ character indicates a command that must be executed as the “root” user, either by logging in as root or using the sudo command.
• Source-code blocks with a leading ‘$’ character indicates a command that may be executed by any user (privileged or otherwise). These commands are generally for testing purposes.

Prerequisites

You will need to know the following information about your LDAP server to follow the directions below:

• Is the directory server powered by FreeIPA, Active Directory or another LDAP solution?
• What is the URI for the LDAP server? e.g. ldap.example.com
• Where is the CA certificate for the LDAP server?
• Does the LDAP server correspond to RFC 2307 or RFC2307bis for user-groups?

Prepare VMs:

• proxy.example.com: A VM to use as the authenticating proxy. This machine must have at least SSSD 1.12.0 available, which means a fairly recent operating system. In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.
• openshift.example.com: A VM to use to run OpenShift

(These machines *can* be configured to run on the same system, but for the purposes of this tutorial, I am keeping them separate)

Phase 1: Certificate Generation

In order to ensure that communication between the authenticating proxy and OpenShift is trustworthy, we need to create a set of TLS certificates that we will use during the other phases of this setup. For the purposes of this demo, we will start by using the auto-generated certificates created as part of running

# openshift start \
    --public-master=https://openshift.example.com:8443 \
    --write-config=/etc/origin/

Among other things, this will generate /etc/origin/master/ca.{cert|key}. We will use this signing certificate to generate keys to use on the authenticating proxy.

# mkdir -p /etc/origin/proxy/
# oadm ca create-server-cert \
    --cert='/etc/origin/proxy/proxy.example.com.crt' \
    --key='/etc/origin/proxy/proxy.example.com.key' \
    --hostnames=proxy.example.com,1.2.3.4 \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key='/etc/origin/master/ca.key' \
    --signer-serial='/etc/origin/master/ca.serial.txt'

For the hostnames, ensure that any hostnames and interface IP addresses that might need to access the proxy are listed, otherwise the HTTPS connection will fail.

Next, we will generate the API client certificate that the authenticating proxy will use to prove its identity to OpenShift (this is necessary so that malicious users cannot impersonate the proxy and send fake identities). First, we will create a new CA to sign this client certificate.

# oadm ca create-signer-cert \
  --cert='/etc/origin/proxy/proxyca.crt' \
  --key='/etc/origin/proxy/proxyca.key' \
  --name='openshift-proxy-signer@`date +%s`' \
  --serial='/etc/origin/proxy/proxyca.serial.txt'

(The date +%s in that block is used to make the  signer unique. You can use any name you prefer, however.)

# oadm create-api-client-config \
    --certificate-authority='/etc/origin/proxy/proxyca.crt' \
    --client-dir='/etc/origin/proxy' \
    --signer-cert='/etc/origin/proxy/proxyca.crt' \
    --signer-key='/etc/origin/proxy/proxyca.key' \
    --signer-serial='/etc/origin/proxy/proxyca.serial.txt' \
    --user='system:proxy'
# cat /etc/origin/proxy/system\:proxy.crt \
      /etc/origin/proxy/system\:proxy.key \
      > /etc/origin/proxy/authproxy.pem

Phase 2: Authenticating Proxy Setup

Step 1: Copy certificates

From openshift.example.com, securely copy the necessary certificates to the proxy machine:

# scp /etc/origin/proxy/master/ca.crt \
      root@proxy.example.com:/etc/pki/CA/certs/

# scp /etc/origin/proxy/proxy.example.com.crt \
      /etc/origin/proxy/authproxy.pem \
      root@proxy.example.com:/etc/pki/tls/certs/

# scp /etc/origin/proxy/proxy.example.com.key \
      root@proxy.example.com:/etc/pki/tls/private/

Step 2: SSSD Configuration

Install a new VM with a recent operating system (in order to use the mod_identity_lookup module later, it will need to be running SSSD 1.12.0 or later). In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.

First thing is to install all of the necessary dependencies:

# yum install -y sssd \
                 sssd-dbus \
                 realmd \
                 httpd \
                 mod_session \
                 mod_ssl \
                 mod_authnz_pam

This will give us the SSSD and the web server components we will need. The first step here will be to set up SSSD to authenticate this VM against the LDAP server. If the LDAP server in question is a FreeIPA or Active Directory environment, then realmd can be used to join this machine to the domain. This is the easiest way to get up and running.

realm join ldap.example.com

If you aren’t running a domain, then your best option is to use the authconfig tool (or follow the many other tutorials on the internet for configuring SSSD for identity and authentication).

# authconfig --update --enablesssd --enablesssdauth \
             --ldapserver=ldap.example.com \
             --enableldaptls \
             --ldaploadcert=http://ldap.example.com/ca.crt

This should create /etc/sssd/sssd.conf with most of the appropriate settings. (Note: RHEL 7 appears to have a bug wherein authconfig does not create the /etc/openldap/cacerts directory, so you may need to create it manually before running the above command.)

If you are interested in using SSSD to manage failover situations for LDAP, this can be configured simply by adding additional entries in /etc/sssd/sssd.conf on the ldap_uri line. Systems enrolled with FreeIPA will automatically handle failover using DNS SRV records.

Finally, restart SSSD to make sure that all of the changes are applied properly:

$ systemctl restart sssd.service

Now, test that the user information can be retrieved properly:

$ getent passwd <username>
username:*:12345:12345:Example User:/home/username:/usr/bin/bash

At this point, it is wise to attempt to log into the VM as an LDAP user and confirm that the authentication is properly set up. This can be done via the local console or a remote service such as SSH. (Later, you can modify your /etc/pam.d files to disallow this access if you prefer.) If this fails, consult the SSSD troubleshooting guide.

Step 3: Apache Configuration

Now that we have the authentication pieces in place, we need to set up Apache to talk to SSSD. First, we will create a PAM stack file for use with Apache. Create the /etc/pam.d/openshift file and add the following contents:

auth required pam_sss.so
account required pam_sss.so

This will tell PAM (the pluggable authentication module) that when an authentication request is issued for the “openshift” stack, it should use pam_sss.so to determine authentication and access-control.

Next we will configure the Apache httpd.conf. (Taken from the OpenShift documentation and modified for SSSD.) For this tutorial, we’re only going to set up the challenge authentication (useful for logging in with oc login and similar automated tools). A future entry in this series will describe setup to use the web console.

First, create the new file openshift-proxy.conf in /etc/httpd/conf.d (substituting the correct hostnames where indicated):

LoadModule request_module modules/mod_request.so
LoadModule lookup_identity_module modules/mod_lookup_identity.so
# Nothing needs to be served over HTTP.  This virtual host simply redirects to
# HTTPS.
<VirtualHost *:80>
  DocumentRoot /var/www/html
  RewriteEngine              On
  RewriteRule     ^(.*)$     https://%{HTTP_HOST}$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
  # This needs to match the certificates you generated.  See the CN and X509v3
  # Subject Alternative Name in the output of:
  # openssl x509 -text -in /etc/pki/tls/certs/proxy.example.com.crt
  ServerName proxy.example.com

  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/proxy.example.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/proxy.example.com.key
  SSLCACertificateFile /etc/pki/CA/certs/ca.crt

  # Send logs to a specific location to make them easier to find
  ErrorLog logs/proxy_error_log
  TransferLog logs/proxy_access_log
  LogLevel warn
  SSLProxyEngine on
  SSLProxyCACertificateFile /etc/pki/CA/certs/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /etc/pki/tls/certs/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}:8443/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://openshift.example.com:8443/oauth/authorize
    AuthType Basic
    AuthBasicProvider PAM
    AuthPAMService openshift
    Require valid-user
  </Location>

  <ProxyMatch /oauth/authorize>
    AuthName openshift
    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER
  </ProxyMatch>
</VirtualHost>

RequestHeader unset X-Remote-User

 

Then we need to tell SELinux that it’s acceptable for Apache to contact the PAM subsystem, so we set a boolean:

# setsebool -P allow_httpd_mod_auth_pam on

At this point, we can start up Apache.

# systemctl start httpd.service

Phase 3: OpenShift Configuration

This describes how to set up an OpenShift server from scratch in an “all in one” configuration. For more complicated (and interesting) setups, consult the official OpenShift documentation.

First, we need to modify the default configuration to use the new identity provider we just created. We’ll start by modifying the /etc/origin/master/master-config.yaml file. Scan through it and locate the identityProviders section and replace it with:

  identityProviders:
  - name: any_provider_name
    challenge: true
    login: false
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxy/proxyca.crt
      headers:
      - X-Remote-User

Now we can start openshift with the updated configuration:

# openshift start \
    --public-master=https://openshift.example.com:8443 \
    --master-config=/etc/origin/master/master-config.yaml \
    --node-config=/etc/origin/node-node1.example.com/node-config.yaml

Now you can test logins with

oc login https://openshift.example.com:8443

It should now be possible to log in with only valid LDAP credentials. Stay tuned for further entries in this series where I will teach you how to set up a “login” provider for authenticating the web console, how to retrieve extended user attributes like email address and full name from LDAP, and also how to set up automatic single-sign-on for users in a FreeIPA or Active Directory domain.

 Updates 2016-05-27: There were some mistakes in the httpd.conf as originally written that made it difficult to set up Part 2. They have been retroactively corrected. Additionally, I’ve moved the incomplete configuration of extended attributes out of this entry and will reintroduce them in a further entry in this series.

Self-Signed SSL/TLS Certificates: Why They are Terrible and a Better Alternative

A Primer on SSL/TLS Certificates

Many of my readers (being technical folks) are probably already aware of the purpose and value of certificates, but in case you are not familiar with them, here’s a quick overview of what they are and how they work.

First, we’ll discuss public-key encryption and public-key infrastructure (PKI). It was realized very early on in human history that sometimes you want to communicate with other people in a way that prevents unauthorized people from listening in. All throughout time, people have been devising mechanisms for obfuscating communication in ways that only the intended recipient of the code would be able to understand. This obfuscation is called encryption, the data being encrypted is called plaintext and the encrypted data is called ciphertext. The cipher is the mathematical transformation that is used to turn the plaintext into the ciphertext and relies upon one or more keys known only to trusted individuals to get the plaintext back.

Early forms of encryption were mainly “symmetric” encryption, meaning that the cipher used the same key for both encryption and decryption. If you’ve ever added a password to a PDF document or a ZIP file, you have been using symmetric encryption. The password is a human-understandable version of a key. For a visual metaphor, think about the key to your front door. You may have one or more such keys, but they’re all exactly alike and each one of them can both lock and unlock the door and let someone in.

Nowadays we also have forms of encryption that are “asymmetric”. What this means is that one key is used to encrypt the message and a completely different key is used to decrypt it. This is a bit harder for many people to grasp, but it works on the basic mathematical principle that some actions are much more complicated to reverse than others. (A good example I’ve heard cited is that it’s pretty easy to figure out the square of any number with a pencil and a couple minutes, but most people can’t figure out a square-root without a modern calculator). This is harder to visualize, but the general idea is that once you lock the door with one key, only the other one can unlock it. Not even the one that locked it in the first place.

So where does the “public” part of public-key infrastructure come in? What normally happens is that once an asymmetric key-pair is generated, the user will keep one of those two keys very secure and private, so that only they have access to it. The other one will be handed out freely through some mechanism to anyone at all that wants to talk to you. Then, if they want to send you a message, they simply encrypt their message using your public key and they know you are the only one who can decrypt it. On the flip side, if the user wanted to send a public message but provide assurance that it came from them, they can also sign a message with the private key, so that the message will contain a special signature that can be decrypted with their public key. Since only one person should have that key, recipients can trust it came from them.

Astute readers will see the catch here: how do users know for certain that your public key is in fact yours? The answer is that they need to have a way of verifying it. We call this establishing trust and it’s exceedingly important (and, not surprisingly, the basis for the rest of this blog entry). There are many ways to establish trust, with the most foolproof being to receive the public key directly from the other party while looking at two forms of picture identification. Obviously, that’s not convenient for the global economy, so there needs to be other mechanisms.

Let’s say the user wants to run a webserver at “www.mydomain.com”. This server might handle private user data (such as their home address), so a wise administrator will set the server up to use HTTPS (secure HTTP). This means that they need a public and private key (which in this case we call a certificate). The common way to do this is for the user to contact a well-known certificate authority and purchase a signature from them. The certificate authority will do the hard work of verifying the user’s identity and then sign their webserver certificate with the CA’s own private key, thus providing trust by way of a third-party. Many well-known certificate authorities have their public keys shipped by default in a variety of operating systems, since the manufacturers of those systems have independently verified the CAs in turn. Now everyone who comes to the site will see the nice green padlock on their URL bar that means their communications are encrypted.

A Primer on Self-Signed Certificates

One of the major drawbacks to purchasing a CA signature is that it isn’t cheap: the CAs (with the exception of Let’s Encrypt) are out there to make money. When you’re developing a new application, you’re going to want to test that everything works with encryption, but you probably aren’t going to want to shell out cash for every test server and virtual machine that you create.

The solution to this has traditionally been to create what is called a self-signed certificate. What this means is that instead of having your certificate signed by a certificate authority, you instead use the certificates public key to add a signature to the private key. The problem with this approach is that web browsers and other clients that verify the security of the connection will be unable to verify that the server is who it says it is. In most cases, the user will be presented with a warning page that informs them that the server is pretending to be the one you went to. When setting up a test server, this is expected. Unfortunately, however, clicking through and saying “I’m sure I want to connect” has a tendency to form bad habits in users and often results in them eventually clicking through when they shouldn’t.

It should be pretty obvious, but I’ll say it anyway: Never use a self-signed certificate for a production website.

One of the problems we need to solve is how to avoid training users to ignore those warnings. One way that people often do this is to load their self-signed certificate into their local trust store (the list of certificate authorities that are trusted, usually provided by the operating system vendor but available to be extended by the user). This can have some unexpected consequences, however. For example, if the test machine is shared by multiple users (or is breached in a malicious attack), then the private key for the certificate might fall into other hands that would then use it to sign additional (potentially malicious) sites. And your computer wouldn’t try to warn you because the site would be signed by a trusted authority!

So now it seems like we’re in a Catch-22 situation: If we load the certificate into the trusted authorities list, we run the risk of a compromised private key for that certificate tricking us into a man-in-the-middle attack somewhere and stealing valuable data. If we don’t load it into the trust store, then we are constantly bombarded by a warning page that we have to ignore (or in the case of non-browser clients, we may have to pass an option not to verify the client) in which case we could still end up in a man-in-the-middle attack, because we’re blindly trusting the connection. Neither of those seems like a great option. What’s a sensible person to do?

Two Better Solutions

So, let’s take both of the situations we just learned about and see if we can locate a middle ground somewhere. Let’s go over what we know:

• We need to have encryption to protect our data from prying eyes.
• Our clients need to be able to trust that they are talking to the right system at the other end of the conversation.
• If the certificate isn’t signed by a certificate in our trust store, the browser or other clients will warn or block us, training the user to skip validation.
• If the certificate is signed by a certificate in our trust store, then clients will silently accept it.
• Getting a certificate signed by a well-known CA can be too expensive for an R&D project, but we don’t want to put developers’ machines at risk.

So there are two better ways to deal with this. One is to have an organization-wide certificate authority rather than a public one. This should be managed by the Information Technologies staff. Then, R&D can submit their certificates to the IT department for signing and all company systems will implicitly trust that signature. This approach is powerful, but can also be difficult to set up (particularly in companies with a bring-your-own-device policy in place). So let’s look at a another solution that’s closer to the self-signed approach.

The other way to deal with it would be to create a simple site-specific certificate authority for use just in signing the development/test certificate. In other words, instead of generating a self-signed certificate, you would generate two certificates: one for the service and one to sign that certificate. Then (and this is the key point – pardon the pun), you must delete and destroy the private key for the certificate that did the signing. As a result, only the public key of that private CA will remain in existence, and it will only have ever signed a single service. Then you can provide the public key of this certificate authority to anyone who should have access to the service and they can add this one-time-use CA to their trust store.

Now, I will stress that the same rule holds true here as for self-signed certificates: do not use this setup for a production system. Use a trusted signing authority for such sites. It’s far easier on your users.

A Tool and a Tale

I came up with this approach while I was working on solving some problems for the Fedora Project. Specifically, we wanted to come up with a way to ensure that we could easily and automatically generate a certificate for services that should be running on initial start-up (such as Cockpit or OpenPegasus). Historically, Fedora had been using self-signed certificates, but the downsides I listed above gnawed at me, so I put some time into it and came up with the private-CA approach.

In addition to the algorithm described above, I’ve also built a proof-of-concept tool called sscg (the Simple Signed Certificate Generator) to easily enable the creation of these certificates (and to do so in a way that never drops the CA’s private key onto a filesystem; it remains in memory). The tool is built in the C language and has very few dependencies (including OpenSSL). If you’d like to contribute, you can clone my github repository. Patches and suggestions for functionality are most welcome.

Sausage Factory: Multiple Edition Handling in Fedora

First off, let me be very clear up-front: normally, I write my blog articles to be approachable by readers of varying levels of technical background (or none at all). This will not be one of those. This will be a deep dive into the very bowels of the sausage factory.

The Problem

Starting with the Fedora.next initiative, the Fedora Project embarked on a journey to reinvent itself. A major piece of that effort was the creation of different “editions” of Fedora that could be targeted at specific user personas. Instead of having a One-Size-Fits-Some Fedora distribution, instead we would produce an operating system for “doers” (Fedora Workstation Edition), for traditional infrastructure administrators (Fedora Server Edition) and for new, cloudy/DevOps folks (Fedora Cloud Edition).

We made the decision early on that we did not want to produce independent distributions of Fedora. We wanted each of these editions to draw from the same collective set of packages as the classic Fedora. There were multiple reasons for this, but the most important of them was this: Fedora is largely a volunteer effort. If we started requiring that package maintainers had to do three or four times more work to support the editions (as well as the traditional DIY deployments), we would quickly find ourselves without any maintainers left.

However, differentiating the editions solely by the set of packages that they deliver in a default install isn’t very interesting. That’s actually a problem that could have been solved simply by having a few extra choices in the Anaconda installer. We also wanted to solve some classic arguments between Fedora constituencies about what the installed configuration of the system looks like. For example, people using Fedora as a workstation or desktop environment in general do not want OpenSSH running on the system by default (since their access to the system is usually by sitting down physically in front of a keyboard and monitor) and therefore don’t want any potential external access available. On the other hand, most Fedora Server installations are “headless” (no input devices or monitor attached) and thus having SSH access is critical to functionality. Other examples include the default firewall configuration of the system: Fedora Server needs to have a very tightened default firewall allowing basically nothing in but SSH and management features, whereas a firewall that restrictive proves to be harmful to usability of a Workstation.

Creating Per-Edition Default Configuration

The first step to managing separate editions is having a stable mechanism for identifying what edition is installed. This is partly aesthetic, so that the user knows what they’re running, but it’s also an important prerequisite (as we’ll see further on) to allowing the packaging system and systemd to make certain decisions about how to operate.

The advent of systemd brought with it a new file that describes the installed system called os-release. This file is considered to be authoritative for information identifying the system. So this seemed like the obvious place for us to extend to include information about the edition that was running as well. We therefore needed a way to ensure that the different editions of Fedora would produce a unique (and correct) version of the os-release file depending on the edition being installed. We did this by expanding the os-release file to include two new values: VARIANT and VARIANT_ID. VARIANT_ID is a machine-readable unique identifier that describes which version of Fedora is installed. VARIANT is a human-readable description.

In Fedora, the os-release file is maintained by a special RPM package called fedora-release. The purpose of this package is to install the files onto the system that guarantee this system is Fedora. Among other things, this includes os-release, /etc/fedora-release, /etc/issue, and the systemd preset files. (All of those will become interesting shortly).

So the first thing we needed to do was modify the fedora-release package such that it included a series of subpackages for each of the individual Fedora editions. These subpackages would be required to carry their own version of os-release that would supplant the non-edition version provided by the fedora-release base package. I’ll circle back around to precisely how this is done later, but for now accept that this is true.

So now that the os-release file on the system is guaranteed to contain appropriate VARIANT_ID, we needed to design a mechanism by which individual packages could make different decisions about their default configurations based on this. The full technical details of how to do this are captured in the Fedora Packaging Guidelines, but the basic gist of it is that any package that wants to behave differently between two or more editions must read the VARIANT_ID from os-release during its %posttrans (post-transaction) phase of package installation and place a symlink to the correct default configuration file in place. This needs to be done in the %posttrans phase because, due to the way that yum/dnf processes the assorted RPMs, there is no other way to guarantee that the os-release file has the right values until that time. This is because it’s possible for a package to install and run its %post script between the time that the fedora-release and fedora-release-EDITION package gets installed.

That all assumes that the os-release file is correct, so let’s explore how that is made to happen. First of all, we created a new directory in /usr/lib called /usr/lib/os.release.d/ which will contain all of the possible alternate versions of os-release (and some other files as well, as we’ll see later). As part of the %install phase of the fedora-release package, we generate all of the os-release variants and then drop them into os.release.d. We will then later symlink the appropriate one into /usr/lib/os-release and /etc/os-release during %post.

There’s an important caveat here: the /usr/lib/os-release file must be present and valid in order for any package to run the %systemd_post scripts to set up their unit files properly. As a result, we need to take a special precaution. The fedora-release package will always install its generic (non-edition) os-release file during its %post section, to ensure that the %systemd_post scripts will not fail. Then later if a fedora-release-EDITION package is installed, it will overwrite the fedora-release one with the EDITION-specific version.

The more keen-eyed reader may have already spotted a problem with this approach as currently described: What happens if a user installs another fedora-release-EDITION package later? The short answer was that in early attempts at this: “Bad Things Happened”. We originally had considered that installation of a fedora-release-EDITION package atop a system that only had fedora-release on it previously would result in converting the system to that edition. However, that turned out to A) be problematic and B) violate the principle of least surprise for many users.

So we decided to lock the system to the edition that was first installed by adding another file: /usr/lib/variant which is essentially just a copy of the VARIANT_ID line from /etc/os-release. In the %post script of each of the fedora-release subpackages (including the base subpackage), it is checked for its contents. If it does not exist, the %post script of a fedora-release-EDITION package will create it with the appropriate value for that edition. If processing reaches all the way to the %posttrans script of the fedora-release base package (meaning no edition package was part of the transaction), then it will write the variant file at that point to lock it into the non-edition variant.

There remains a known bug with this behavior, in that if the *initial* transaction actually includes two or more fedora-release-EDITION subpackages, whichever one is processed first will “win” and write the variant. In practice, this is effectively unlikely to happen since all of the install media are curated to include at most one fedora-release-EDITION package.

I said above that this “locks” the system into the particular release, but that’s not strictly true. We also ship a script along with fedora-release that will allow an administrator to manually convert between editions by running `/usr/sbin/convert-to-edition -e <edition>`. Effectively, this just reruns the steps that the %post of that edition would run, except that it skips the check for whether the variant file is already present.

Up to now, I’ve talked only about the os-release file, but the edition-handling also addresses several other important files on the system, including /etc/issue and the systemd presets. /etc/issue is handled identically to the os-release file, with the symlink being created by the %post scripts of the fedora-release-EDITION subpackages or the %posttrans of the fedora-release package if it gets that far.

The systemd presets are a bit of a special case, though. First of all, they do not replace the global default presets, but the do supplement them. This means that what we do is symlink in an edition-specific preset into the /usr/lib/systemd/system-preset/ directory. These presets can either enable new services (as in the Server Edition, where it turns on Cockpit and rolekit) or disable them (as in Workstation Edition where it shuts off OpenSSH). However, due to the fact that systemd only processes the preset files during its %post phase, we need to force systemd to reread them after we add the new values.

We need to be careful when doing this, because we only want to apply the new presets if the current transaction is the initial installation of the fedora-release-EDITION package. Otherwise, an upgrade could override choices that the user themselves have made (such as disabling a service that defaults to enabled). This could lead to unexpected security issues, so it has to be handled carefully.

In this implementation, instead of just calling the command to reprocess all presets, we instead parse the preset files and just process only those units that are mentioned in them. (This is to be overcautious in case any other package is changing the default enabled state besides systemd, such as some third-party RPMs that might have `systemctl enable httpd.service` in their %post section, for example.)

Lastly, due to the fact that we are using symlinks to manage most of this, we had to write the %post and %posttrans scripts in the built-in Lua implementation carried by RPM. This allowed us to call posix.symlink() without having to add a dependency on coreutils to do so in bash (which resulted in a circular dependency and broken installations). We wrote this as a single script that is imported by the RPM during the SRPM build phase. This script is actually coped by rpmbuild into the scriptlet sections verbatim, so the script must be present in the dist-git checkout on its own and not even as part of the exploded tarball. So when modifying the Lua script, it’s important to make sure to modify the copy in dist-git as well as the copy upstream.

Remote group merging for Fedora

The Problem

One of the major features of the Fedora Server Edition is the Cockpit administrative console. This web-based interface provides administrators with a powerful set of tools for controlling their system. Cockpit relies upon low-level tools like polkit and sudo to make authorization decisions to determine what a user is permitted to do. By default, most operations on a Fedora system are granted to users in the ‘wheel’ group. People granted administrator access to Cockpit (and other tools through shell access) are generally added to the wheel group in the /etc/group file.

This works reasonably well for single-user systems or very small environments where manual edits to /etc/group are maintainable, but in larger deployments, it becomes very unwieldy to manage lots of entries in /etc/group. In these cases, most environments switch over to using some form of a domain controller (such as FreeIPA, Microsoft Active Directory or a custom LDAP setup). These domain controllers allow users to be managed centrally, allowing administrators to make changes in a single place and have this be automatically picked up by all enrolled systems.

However, there is a problem: historically the group processing on Fedora (provided by glibc) has forced users to choose between using centrally managed groups (such as those provided by a domain and maintained by SSSD) or groups maintained on the local system in the /etc/group file. The behavior of glibc is specified in /etc/nsswitch.conf to decide which of the two mechanisms will “win” in the event of a conflict. This means that administrators need to decide up front whether their groups must all come from a domain controller or some locally.

The Solution

Over the last few months, I worked on adding a new feature to the glibc name-service functionality to enable “group merging”. The net effect is that now for all lookups of a group, glibc can be configured to check both the local files and the remote service and (if the group appears in both), combine the list of member users for both representations of the group into a single response.

Thus, it becomes possible to provide both local and central administrators into the wheel group. This can come in handy for example if an administrator wants to keep one or more local accounts available to do disaster recovery in the event that the machine loses access to the remote users (such as a bad update resulting in SSSD not starting).

Of course, this functionality does not come without a cost: because all merging lookups will try both data sources, it can result in a performance hit when operating against groups that otherwise would have been answered only by the local /etc/group file. With caching services like SSSD, this impact should be minimized.

Fedora and glibc upstream

The group merging patch has been submitted to the upstream glibc project but has not yet been merged into a release. It narrowly missed the 2.23 merge window, so it is currently slated for inclusion into glibc 2.24.

However, Carlos O’Donell has taken the patch and applied it to glibc in Fedora Rawhide (which will become Fedora 24), so it will be possible to take advantage of these features first in Fedora 24, before anyone else. (For anyone interested from other distributions, the patch should apply cleanly on 2.23 and likely with minimal effort atop 2.22 as well, since little changed besides this.)

My Git Workflow

It’s been a long while since I published a new entry on this blog, and I keep meaning to improve on that. Today, I was having a conversation with one of my colleagues today and discussing how I set up my local git checkouts. It occurs to me that this might be of interest, so I figure I’ll restart this blog by describing it.

This blog will describe specifically my workflow when dealing with upstreams hosted on Github. There are only minor changes to it when discussing non-Github projects (mainly in the public forking process).

Initial Setup

First, I have to find a project to get involved in. For some people, this is a difficult process involving a great deal of contemplation. For me, on the other hand, I seem to pick up new projects to get involved like they were falling from the sky. Most open-source projects these days seem to be hosted on Github (or at least have a presence there), so my workflow has become reasonably Github-centric.

I will skip over the part where I sign up for a Github account and set up two-factor authentication and uploaded my public SSH key, but rest assured that I have done all of those things. (If you aren’t using two-factor authentication anywhere that is even remotely important, fix that now. I also highly recommend the use of the open-source FreeOTP as a software token for either iOS or Android devices over Google Authenticator; it works anywhere Google Authenticator does.) You may also assume that I am properly logged in to Github at this point.

I’ll use the imaginary package “bar” created by Github user “foo” as my representative example. So I would browse to https://github.com/foo/bar and then click on the  button. (Since I also belong to several Github organizations, this prompts me for which entity I am cloning to, but if you only have a personal account, it will probably skip this phase).

Local Repository Clone

Now that I have a public fork of the “bar” project, I want to be able to work with it. This means that I need to clone my repository to the local machine so I can operate on its contents. Github provides a handy way to identify the git URL needed for the cloning operation. When cloning my personal Github fork, I will want to clone using the “SSH” URL, which allows both reading from it and pushing changes. (I’ll talk about the “HTTPS” URL in a moment). To find the “SSH” URL, look on the main toolbar of the project’s Github main page. If you don’t see it, check for  and click on it, then select “SSH”. After that, it should look like  and there will be a URL in the text box to the right of it. It should look something like:

git@github.com:sgallagher/bar.git

Now we will open a terminal window, change to an appropriate containing directory and run:

git clone git@github.com:sgallagher/bar.git

This will pull down a copy of the repository onto the local system, ready to work with. I can make whatever local changes I want and run `git push` to submit them to my public fork. However, we are not finished. The next step will be to create an additional “git remote” that points at the original upstream repository. This I do in order to be able to track other changes that are happening upstream (particularly so I can rebase atop others’ work and ensure that my work still applies atop the upstream code). So in this case, I would do the following: first, I would browse to https://github.com/foo/bar again and

cd bar.git
git remote add upstream https://github.com/foo/bar.git
git remote update

This means that I am adding a new remote name (“upstream”) and associating it with the “HTTPS” (read-only) URL for the original project. (The `git remote update` piece causes me to pull the latest bits from upstream and store them locally.)

Special Case: Upstream Committer

In the special case where I also have commit privileges to the upstream repository, I also add another git remote called “upstream-push” using the “SSH” URL. Then, when I have patches ready to go upstream, I can ready them in a proper branch and then run

git push upstream-push local_branch_name:remote_branch_name

The reason for this additional upstream is to avoid accidental pushes to master (which is very easy to do if you have created a branch from e.g. upstream/remote_branch_name).

Appendix: Useful Git Aliases

In addition to my workflow convention above, I have also created a number of very useful git aliases that I store in my ~/.gitconfig file.

[alias]
 patch = format-patch -M -C --patience --full-index
 patches = format-patch -M -C --patience --full-index @{upstream}..
 up = remote update
 new = log ..@{upstream}

git patch

This handy alias is essentially a wrapper around `git format-patch`, which is useful for generating comprehensive patch files for `git send-email` (among other uses). It creates patches that auto-detect file copies and renames, diffs using the “patience” algorithm (which tends to be more human-readable than other algorithms) and displays git indices using their un-shortened forms (to avoid ambiguities).

This alias needs to be invoked with additional `git format-patch` arguments, such as a specification of which commits to generate patches. Which leads us to:

git patches

This is a variant of the above, except it will automatically generate all patches on the branch that are not present in the upstream branch being tracked. (So if you are tracking e.g. upstream/master, this will output all of the patches atop master).

git up

This is a quick shorthand for `git remote update` to automatically pull all of the new data from every remote configured in the checkout. This is particularly useful before a `git rebase -i upstream/branch` right before submitting a patch for code-review. (Which you should always do, to make the reviewer’s life easier.)

git new

This is another quick shorthand essentially for a variant of `git log` that only shows you your own commits atop the remote branch.