We are (still) not who we are

This article is a reprint. It first appeared on my blog on January 24, 2013. Given the recent high-profile hack of Germany’s defense minister, I decided it was time to run this one again.


In authentication, we generally talk about three “factors” for determining identity. A “factor” is a broad category for establishing that you are who you claim to be. The three types of authentication factor are:

  • Something you know (a password, a PIN, the answer to a “security question”, etc.)
  • Something you have (an ATM card, a smart card, a one-time-password token, etc.)
  • Something you are (your fingerprint, retinal pattern, DNA)

Historically, most people have used the first of these three forms most commonly. Whenever you’ve logged into Facebook, you’re entering something you know: your username and password. If you’ve ever used Google’s two-factor authentication to log in, you probably used a code stored on your smartphone to do so.

One of the less common, but growing, authentication methods are the biometrics. A couple years ago, a major PC manufacturer ran a number of television commercials advertising their laptop models with a fingerprint scanner. The claim was that it was easy and secure to unlock the machine with a swipe of a finger. Similarly, Google introduced a service to unlock an Android smart phone by using facial recognition with the built-in camera.

Pay attention folks, because I’m about to remove the scales from your eyes. Those three factors I listed above? I listed them in decreasing order of security. “But how can that be?” you may ask. “How can my unchangeable physical attributes be less secure than a password? Everyone knows passwords aren’t secure.”

The confusion here is due to subtle but important definitions in the meaning of “security”. Most common passwords these days are considered “insecure” because people tend to use short passwords which by definition have a limited entropy pool (meaning it takes a smaller amount of time to run through all the possible combinations in order to brute-force the password or run through a password dictionary). However, the pure computational complexity of the authentication mechanism is not the only contributor to security.

The second factor above, “something you have” (known as a token), is almost always of significantly higher entropy than anything you would ever use as a password. This is to eliminate the brute-force vulnerability of passwords. But it comes with a significant downside as well: something you have is also something that can be physically removed from you. Where a well-chosen password can only be removed from you by social engineering (tricking you into giving it to an inappropriate recipient), a token might be slipped off your desk while you are at lunch.

Both passwords and tokens have an important side-effect that most people never think about until an intrusion has been caught: remediation. When someone has successfully learned your password or stolen your token, you can call up your helpdesk and immediately ask them to reset the password or disable the cryptographic seed in the token. Your security is now restored and you can choose a new password and have a new token sent to you.

However, this is not the case with a biometric system. By its very nature, it is dependent upon something that you cannot change. Moreover, the nature of its supposed security derives from this very fact. The problem here is that it’s significantly easier to acquire a copy of someone’s fingerprint, retinal scan or even blood for a DNA test than it is to steal a password or token device and in many cases it can even be done without the victim knowing.

Many consumer retinal scanners can be fooled by a simple reasonably-high-resolution photograph of the person’s eye (which is extremely easy to accomplish with today’s cameras). Some of the more expensive models will also require a moving picture, but today’s high-resolution smartphone cameras and displays can defeat many of these mechanisms as well. It’s well-documented that Android’s face-unlock feature can be beaten by a simple photograph.

These are all technological limitations and as such it’s plausible that they can be overcome over time with more sensitive equipment. However, the real problem with biometric security lies with its inability to replace a compromised authentication device. Once someone has a copy of your ten fingerprints, or a drop of your blood from a stolen blood-sugar test or a close-up video of your eye from a scoped video camera, there is no way to change this data out. You can’t ask helpdesk to send you new fingers, an eyeball or DNA. Therefore, I contend that I lied to you above. There is no full third factor for authentication, because, given a sufficient amount of time, any use of biometrics will eventually degenerate into a non-factor.

Given this serious limitation, one should never under any circumstances use biometrics as the sole form of authentication for any purpose whatsoever.

One other thought: have you ever heard the argument that you should never use the same password on multiple websites because if it’s stolen on one, they have access to the others? Well, the same is true of your retina. If someone sticks malware on your cellphone to copy an image of your eye that you were using for “face unlock”, guess what? They can probably use that to get into your lab too.

The moral of the story is this: biometrics are minimally useful, since they are only viable until the first exposure across all sites where they are used. As a result, if you are considering initiating a biometric-based security model, I encourage you to save your money (those scanners are expensive!) and look into a two-factor solution involving passwords and a token of some kind.

Base Runtime and the Generational Core

A Quick Primer on Modularity

lego_chicago_city_view_2001Modularity (formerly, Modularization) is an ongoing initiative in Fedora to resolve the issue of divergent, occasionally conflicting lifecycles of different components. A module provides functionality (such as a web server) and includes well-integrated and well-tested components (such as Apache httpd and the libraries on which it depends). It can be deployed into production in various ways: as “classic” RPM packages or a container image, and is updated as a whole. Different modules can emphasize new features, stability, security, etc. differently.

Modules differ from traditional packaging in certain important ways. Perhaps most importantly, they allow us to separate internal implementation details from the exposed interfaces of the module. Historically in Fedora, if a packager wanted to deliver a new web application, that would also often mean that they needed to package and carry the framework or other libraries used by that application. This tended to be a double-edged sword: on the one hand, those libraries were now available for anyone to pick up and use in Fedora. However, in many cases, this meant that the primary maintainer of that package might actually have no specific knowledge or understanding of it except that its lack would mean their application didn’t work. This can be a problem if a person is carrying around a library for the use of a single helper function and don’t want to be responsible for issues in the rest of the library.

With a modular approach, the module itself will provide a definition of what public interfaces are stable for use by other projects. This way, they can opt to contain an internal-only implementation of some libraries.

A good metaphor for Modularity might be urban planning: sections of the Earth are selected for establishing housing, businesses and other construction projects. Each of these projects would be effectively a module.

Base Runtime: The Bedrock

The first thing that a construction company would look at when establishing a new building site would be the ground on which it is to be constructed. It is essential that this location be sturdy, reliable and capable of supporting the weight of the projects being built atop it.

In the Fedora Project, the bedrock upon which the other modules will be built is called the Base Runtime. The definition of this project has gone through a number of revisions over the last few months, but at this point it is fairly settled down to this:

The Base Runtime contains the software necessary to boot the system to a running kernel and the runtime libraries for the most rudimentary operation of the system.

In practical terms, this means that the Base Runtime is

  • Not installable by itself. It can only boot to a kernel; it has no init system or running applications.
  • Not self-hosting. It does not contain the packages necessary to rebuild itself from source. Other modules will need to provide this.
  • Limited to packages with an extremely stable public API. The Base Runtime needs to be swappable at any time out from under the running system without impacting the operation of applications running atop it.

System Runtime: Urban Infrastructure


los_angeles_-_echangeur_autoroute_110_105Once you have a location chosen and have built some homes and some businesses, you need a certain amount of infrastructure to connect it all together: roads, plumbing, electricity, etc.

In the case of a computer operating system, this means things like service control and monitoring, filesystem operations, command shells and other basic tools for operating on and maintaining the system. In Fedora, this means essentially the bash shell environment, login services and the standard POSIX command utilities.

The primary reason for separating the System Runtime from the Base Runtime is to allow these two modules to carry different API lifecycle guarantees. While the Base Runtime will need to remain backwards compatible for an extended period, it may be permissible for the System Runtime to see revisions at a higher rate (where it makes sense to provide new functionality faster). For example, Fedora may wish to update the systemd project (used for control, monitoring and management of system services) at a much higher rate than the low-level C runtime library.

Shared Components: Building Materials


In order to build up your city, you naturally need a set of raw materials. Wood, stone, metal and essential workers.

The third and final piece of the puzzle is the Shared Components module. This special module  is comprised of the set of low-level libraries common to both the Base Runtime and the System Runtime. Some of these libraries may be made available for other services to consume, but the majority of them will be held privately within the modules.

Generational Core: Local GovernmentВыступление Михаила Горбачева на сессии Генеральной ассамблеи ООН

After building up the town, it is important to have a mechanism in place for maintaining it, improving it and making sure to adapt to changing conditions in the world around it. In the urban planning world, this would mean that the local government would establish policies and groups capable of performing these tasks. They would be given access to all of the tools used to create the city in the first place and would continue to monitor things and address issues as they arise. This is effectively what the Generational Core is: the tying together of all those disparate components as a single entity.

While the Base Runtime, System Runtime and Shared Components modules will be built separately and maintained for independent lifecycles, they will be delivered to end-users as part of a single combined module stack called the Generational Core. (Defining “Generational” in the sense of “genealogy” as opposed to “creation”).

Unlike the Base Runtime and System Runtime, the Generational Core will be installable and very similar to the “minimal install” of previous releases of Fedora. It will be somewhat more stripped down even than those. For example, the Generational Core does not need to provide network management services, remote login capabilities or network storage connectivity. These features will be provided by additional modules and module stacks built atop the Generational Core.

Fedora Server: Expanding Throughout the Galaxy


Three years ago, Fedora embarked on a new initiative that we collectively refer to as Fedora.next. As part of this initiative, we decided to start curating deliverable artifacts around specific use-cases rather than the one-size-fits-all approach of Fedora 20 and earlier. One of those specific use-cases was to meet the needs of “server administrators”. And thus, the Fedora Server Edition was born.

One of the earliest things that we did after creating the Fedora Server Working Group (Server WG from here on) was to perform what in the corporate world might be called a “gap analysis”. What this means is that we looked at Fedora from the perspective of the server administrator “personas” we had created and tried to understand their pain points (particularly in contrast to how things function on competitive platforms such as Microsoft Windows Server).

The most obvious gap that we identified was the relative difficulty of getting started with Fedora Server Edition at all. With Microsoft Windows Server, the first experience after logging in is to be presented with a tool called Server Manager that provides basic (graphical) information about the system as well as presenting the user with a list of things that they might want this server to do. It then walks them through a guided installation of those core features (such as domain controller services, remote desktop services and so on). With Fedora, a default install would get you a bash prompt with no guidance; typing “help” would only lead to the extremely expert-focused help documentation for the bash shell itself.

OK, advantage Windows here. So how do we address that? Server WG had agreed early on that we were not willing require a desktop environment for server systems. We instead set our sights on a fledgling project called Cockpit, which was gaining traction and looked to provide an excellent user experience without requiring a local display – it’s a web-based admin console and so can be accessed by users running the operating system of their choice.

Once Cockpit was established as the much-friendlier initial experience for Fedora Server, we started to look at the second part of the problem that we needed to solve: that of simplified deployment of key infrastructure components. To that end, we started the development of a tool that we could integrate with the Cockpit admin console and provide the actual deployment implementation. What we came up with was a python project that we called rolekit that would provide a fairly simple local D-BUS API that Cockpit would be able to call out to in order to deploy the requested services.

While our intentions were good, rolekit faced two serious problems:

  • The creation of the roles were complicated and manual, requiring careful curation and attention to make sure that they continued to work from release to release of Fedora.
  • The Cockpit Project became very popular and its limited resources became dedicated to serving the needs of their other consumers, leaving us unable to get the integration of rolekit completed.

The second of these issues remains and will likely need to be addressed, but that will be a topic for another day. The remainder of this blog entry will discuss our plans for how to improve the creation and maintenance of roles.

Ansible Galaxy

Ansible Galaxy describes itself as “[Y]our hub for finding, reusing and sharing the best Ansible content”. What this means is that the Ansible project runs a public software service enabling the sharing of Github repositories containing useful Ansible roles and playbooks for deploying software services.

The Galaxy hub contains literally thousands of pre-built server roles for Fedora, Red Hat Enterprise Linux and other systems with more being added every day. With such a large community made available to us, the Server WG has decided to explore the use of Ansible Galaxy as the back-end for our server role implementation, replacing rolekit’s custom (and redundant) implementation.

As part of this effort, I attended the Ansible Contributor Conference and AnsibleFest this week in Brooklyn, New York. I spent a great deal of time talking with Chris Houseknecht about ways in which we could enhance Ansible Galaxy to function for our needs.

Required Galaxy Enhancements

There are a few shortcomings to Galaxy that we will need to address before we can implement a complete solution. The first of these is assurance: there is currently no way for a consumer of a role to indicate its suitability. Specifically, we will want there to be a way for Fedora to elevate a set of roles (and specific versions of those roles) to a “recommended” state. In order to do this, Galaxy will be adding support for third-party signing of role versions. Fedora will become a “signing authority” for Ansible Galaxy, indicating that certain roles and their versions should be consumed by users of Fedora.

We will also add filtering to the Galaxy API to enable clients to limit their searches to only those roles that have been signed by a particular signing authority. This will be useful for limiting the list that we expose to users in Cockpit.

The other remaining issue with Ansible is that there is currently no way to execute an Ansible script through an API; at present it must be done via execution of the Ansible CLI tool. Fedora will be collaborating with Ansible on this (see below).

Required Fedora Enhancements

In Fedora, we will need to provide a useful UI for this new functionality. This will most likely need to happen in the Cockpit project, and we will have to find resources to handle this.

Specifically, we will need:

  • UI to handle searching the Galaxy API using the role signatures and other tag filtering.
  • UI for an “answer file” for satisfying required variables in the roles.
  • UI for interrogating a system for what roles have been applied to it.

In addition to Cockpit UI work, we will need to provide infrastructure within the Fedora Project to provide our signatures. This will mean at minimum having secure, audited storage of our private signing key and a tool or service that performs the signing. In the short term, we can allow a set of trusted users to do this signing manually, but in the longer term we will need to focus on setting up a CI environment to enable automated testing and signing of role updates.

Lastly, as mentioned above, we will need to work on an API that Cockpit can invoke to fire off the generated Ansible playbook. This will be provided by Fedora (likely under the rolekit banner) but may be eventually absorbed into the upstream Ansible project once it matures.

Flocking to Kraków

In less than five days, the fourth annual Flock conference will take place in Kraków, Poland. This is Fedora’s premier contributor event each year, alternately taking place in North America and Europe. Attendance is completely free for anyone at all, so if you happen to be in the area (maybe hanging around after World Youth Day going on right now), you should certainly stop in!

This year’s conference is shaping up to be a truly excellent one, with a massive amount of exciting content to see. The full schedule has been available for a while, and I’ve got to say: there are no lulls in the action. In fact, I’ve put together my schedule of sessions I want to see and there are in fact no gaps in it. That said, here are a few of the sessions that I suspect are going to be the most exciting:

Aug. 2 @11:00 – Towards an Atomic Workstation

For a couple of years now, Fedora has been at the forefront of developing container technologies, particularly Docker and Project Atomic. Now, the Workstation SIG is looking to take some of those Project Atomic technologies and adopt them for the end-user workstation.

Aug. 2 @17:30 – University Outreach

I’ve long held that one of Fedora’s primary goals should always be to enlighten the next generation of the open source community. Over the last year, the Fedora Project began an Initiative to expand our presence in educational programs throughout the world. I’m extremely interested to see where that has taken us (and where it is going next).

Aug. 3 @11:00 – Modularity

This past year, there has been an enormous research-and-development effort poured into the concept of building a “modular” Fedora. What does this mean? Well it means solving the age-old Too Fast/Too Slow problem (sometimes described as “I want everything on my system to stay exactly the same for a long time. Except these three things over here that I always want to be running at the latest version.”). With modularity, the hope is that people will be able to put together their ideal operating system from parts bigger than just traditional packages.

Aug. 3 @16:30 – Diversity: Women in Open Source

This is a topic that is very dear to my heart, having a daughter who is already finding her way towards an engineering future. Fedora and many other projects (and companies) talk about “meritocracy” a lot: the concept that the best idea should always win. However the technology industry in general has a severe diversity problem. When we talk about “meritocracy”, the implicit contract there is that we have many ideas to choose from. However, if we don’t have a community that represents many different viewpoints and cultures, then we are by definition only choosing the best idea from a very limited pool. I’m very interested to hear how Fedora is working towards attracting people with new ideas.


OpenShift and SSSD Part 3: Extended LDAP Attributes


This is the third post in a series on setting up advanced authentication mechanisms with OpenShift Origin. This entry will build upon the foundation created earlier, so if you haven’t already gone through that tutorial, start here and continue here.

Configuring Extended LDAP Attributes


  • SSSD 1.12.0  or later. This is available on Red Hat Enterprise Linux 7.0 and later.
  • mod_lookup_identity 0.9.4 or later. The required version is not yet available on any released version of Red Hat Enterprise Linux 7, but RPMs for this platform are available from upstream at this COPR repository until they arrive in Red Hat Enterprise Linux.

Configuring SSSD

First, we need to ask SSSD to look up attributes in LDAP that it normally doesn’t care about for simple system-login use-cases. In the OpenShift case, there’s really only one such attribute: email. So we need to modify the [domain/DOMAINNAME] section of /etc/sssd/sssd.conf on the authenticating proxy and add this attribute:

ldap_user_extra_attrs = mail

Next, we also have to tell SSSD that it’s acceptable for this attribute to be retrieved by apache, so we need to add the following two lines to the [ifp] section of /etc/sssd/sssd.conf as well:

user_attributes = +mail
allowed_uids = apache, root

Now we should be able to restart SSSD and test this configuration.

# systemctl restart sssd.service

# getent passwd <username>
username:*:12345:12345:Example User:/home/username:/usr/bin/bash

# gdbus call \
        --system \
        --dest org.freedesktop.sssd.infopipe \
        --object-path /org/freedesktop/sssd/infopipe/Users/example_2ecom/12345 \
        --method org.freedesktop.DBus.Properties.Get \
        "org.freedesktop.sssd.infopipe.Users.User" "extraAttributes"
(<{'mail': ['username@example.com']}>,)

Configuring Apache

Now that SSSD is set up and successfully serving extended attributes, we need to configure the web server to ask for them and to insert them in the correct places.

First, we need to install and enable the mod_lookup_identity module for Apache (See note in the “Prerequisites” setting for installing on RHEL 7):

# yum -y install mod_lookup_identity

Second, we need to enable the module so that Apache will load it. We need to modify /etc/httpd/conf.modules.d/55-lookup_identity.conf and uncomment the line:

LoadModule lookup_identity_module modules/mod_lookup_identity.so

Next, we need to let SELinux know that it’s acceptable for Apache to connect to SSSD over D-BUS, so we’ll set an SELinux boolean:

# setsebool -P httpd_dbus_sssd on

Then we’ll edit /etc/httpd/conf.d/openshift-proxy.conf and add the following lines (bolded to show the additions) inside the <ProxyMatch /oauth/authorize> section:

  <ProxyMatch /oauth/authorize>
    AuthName openshift

    LookupOutput Headers
    LookupUserAttr mail X-Remote-User-Email
    LookupUserGECOS X-Remote-User-Display-Name

    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER

Then restart Apache to pick up the changes.

# systemctl restart httpd.service

Configuring OpenShift

The proxy is now all set, so it’s time to tell OpenShift where to find these new attributes during login. Edit the /etc/origin/master/master-config.yaml file and add the following lines to the identityProviders section (new lines bolded):

  - name: sssd
  challenge: true
  login: true
  mappingMethod: claim
    apiVersion: v1
    kind: RequestHeaderIdentityProvider
    challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
    loginURL: "https://proxy.example.com/login-proxy/oauth/authorize?${query}"
    clientCA: /etc/origin/master/proxy/proxyca.crt
    - X-Remote-User
    - X-Remote-User-Email
    - X-Remote-User-Display-Name

Go ahead and launch OpenShift with this updated configuration and log in to the web as a new user. You should see their full name appear in the upper-right of the screen. You can also verify with oc get identities -o yaml that both email addresses and full names are available.

Debugging Notes

OpenShift currently only saves these attributes to the user at the time of the first login and doesn’t update them again after that. So while you are testing (and only while testing), it’s advisable to run oc delete users,identities --all to clear the identities out so you can log in again.

OpenShift and SSSD Part 2: LDAP Form Authentication


This is the second post in a series on setting up advanced authentication mechanisms with OpenShift Origin. This entry will build upon the foundation created earlier, so if you haven’t already gone through that tutorial, start here. Note that some of the content on that page has changed since it was first published to ensure that this second part is easier to set up, so make sure to double-check your configuration.

Configuring Form-based Authentication

In this tutorial, I’m going to describe how to set up form-based authentication to use when signing into the OpenShift Origin web console. The first step is to prepare a login page. The OpenShift upstream repositories have a handy template for forms, so we will copy that down to our authenticating proxy on proxy.example.com.

# curl -o /var/www/html/login.html \

You may edit this login HTML however you prefer, but if you change the form field names, you will need to update those in the configuration below as well.

Next, we need to install another Apache module, this time for intercepting form-based authentication.

# yum -y install mod_intercept_form_submit

Then we need to modify /etc/httpd/conf.modules.d/55-intercept_form_submit.conf and uncomment the LoadModule line.

Next, we’ll add a new section to our openshift-proxy.conf inside the <VirtualHost *:443> block.

  <Location /login-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://openshift.example.com:8443/oauth/authorize

    InterceptFormPAMService openshift
    InterceptFormLogin httpd_username
    InterceptFormPassword httpd_password

    RewriteCond %{REQUEST_METHOD} GET
    RewriteRule ^.*$ /login.html [L]

This tells Apache to listen for POST requests on the /login-proxy/oauth/authorize and pass the username and password over to the openshift PAM service, just like in the challenging-proxy example in the first entry of this series. This is all we need to do on the Apache side of things, so restart the service and move back over to the OpenShift configuration.

In the master-config.yaml, update the identityProviders section as follows (new lines bolded):

  - name: any_provider_name
    challenge: true
    login: true
    mappingMethod: claim
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
      loginURL: "https://proxy.example.com/login-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxy/proxyca.crt
      - X-Remote-User

Now restart OpenShift with the updated configuration. You should be able to browse to https://openshift.example.com:8443 and use your LDAP credentials at the login form to sign in.

OpenShift and SSSD Part 1: Basic LDAP Authentication


OpenShift provides a fairly simple and straightforward authentication provider for use with LDAP setups. It has one major limitation, however: it can only connect to a single LDAP server. This can be problematic if that LDAP server becomes unavailable for any reason. When this happens, end-users get very unhappy.

Enter SSSD. Originally designed to manage local and remote authentication to the host OS, it can now be configured to provide identity, authentication and authorization services to web services like OpenShift as well. It provides a multitude of advantages over the built-in LDAP provider; in particular it has the ability to connect to any number of failover LDAP servers as well as to cache authentication attempts in case it can no longer reach any of those servers.

These advantages don’t come without a cost, of course: the setup of this configuration is somewhat more advanced, so I’m writing up this guide to help you get it set up. Rather than adding a few lines to the master-config.yml in OpenShift and calling it a day, we are going to need to set up a separate authentication server that OpenShift will talk to. This guide will describe how to do it on a dedicated physical or virtual machine, but the concepts should also be applicable to loading up such a setup in a container as well. (And in the future, I will be looking into whether we could build such a static container right into OpenShift, but for now this document will have to suffice.) For this guide, I will use the term VM to refer to either type of machine, simply because it’s shorter to type and read.

This separate authentication server will be called the “authenticating proxy” from here on out and describes a solution that will provide a specialized httpd server that will handle the authentication challenge and return the results to the OpenShift Server. See the OpenShift documentation for security considerations around the use of an authenticating proxy.

Formatting Notes

  • If you see something in italics within a source-code block below, you should replace it with the appropriate value for your environment.
  • Source-code blocks with a leading ‘#’ character indicates a command that must be executed as the “root” user, either by logging in as root or using the sudo command.
  • Source-code blocks with a leading ‘$’ character indicates a command that may be executed by any user (privileged or otherwise). These commands are generally for testing purposes.


You will need to know the following information about your LDAP server to follow the directions below:

  • Is the directory server powered by FreeIPA, Active Directory or another LDAP solution?
  • What is the URI for the LDAP server? e.g. ldap.example.com
  • Where is the CA certificate for the LDAP server?
  • Does the LDAP server correspond to RFC 2307 or RFC2307bis for user-groups?

Prepare VMs:

  • proxy.example.com: A VM to use as the authenticating proxy. This machine must have at least SSSD 1.12.0 available, which means a fairly recent operating system. In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.
  • openshift.example.com: A VM to use to run OpenShift

(These machines *can* be configured to run on the same system, but for the purposes of this tutorial, I am keeping them separate)

Phase 1: Certificate Generation

In order to ensure that communication between the authenticating proxy and OpenShift is trustworthy, we need to create a set of TLS certificates that we will use during the other phases of this setup. For the purposes of this demo, we will start by using the auto-generated certificates created as part of running

# openshift start \
    --public-master=https://openshift.example.com:8443 \

Among other things, this will generate /etc/origin/master/ca.{cert|key}. We will use this signing certificate to generate keys to use on the authenticating proxy.

# mkdir -p /etc/origin/proxy/
# oadm ca create-server-cert \
    --cert='/etc/origin/proxy/proxy.example.com.crt' \
    --key='/etc/origin/proxy/proxy.example.com.key' \
    --hostnames=proxy.example.com, \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key='/etc/origin/master/ca.key' \

For the hostnames, ensure that any hostnames and interface IP addresses that might need to access the proxy are listed, otherwise the HTTPS connection will fail.

Next, we will generate the API client certificate that the authenticating proxy will use to prove its identity to OpenShift (this is necessary so that malicious users cannot impersonate the proxy and send fake identities). First, we will create a new CA to sign this client certificate.

# oadm ca create-signer-cert \
  --cert='/etc/origin/proxy/proxyca.crt' \
  --key='/etc/origin/proxy/proxyca.key' \
  --name='openshift-proxy-signer@`date +%s`' \

(The date +%s in that block is used to make the  signer unique. You can use any name you prefer, however.)

# oadm create-api-client-config \
    --certificate-authority='/etc/origin/proxy/proxyca.crt' \
    --client-dir='/etc/origin/proxy' \
    --signer-cert='/etc/origin/proxy/proxyca.crt' \
    --signer-key='/etc/origin/proxy/proxyca.key' \
    --signer-serial='/etc/origin/proxy/proxyca.serial.txt' \
# cat /etc/origin/proxy/system\:proxy.crt \
      /etc/origin/proxy/system\:proxy.key \
      > /etc/origin/proxy/authproxy.pem

Phase 2: Authenticating Proxy Setup

Step 1: Copy certificates

From openshift.example.com, securely copy the necessary certificates to the proxy machine:

# scp /etc/origin/proxy/master/ca.crt \

# scp /etc/origin/proxy/proxy.example.com.crt \
      /etc/origin/proxy/authproxy.pem \

# scp /etc/origin/proxy/proxy.example.com.key \

Step 2: SSSD Configuration

Install a new VM with a recent operating system (in order to use the mod_identity_lookup module later, it will need to be running SSSD 1.12.0 or later). In these examples, I will be using a clean install of Red Hat Enterprise Linux 7.2 Server.

First thing is to install all of the necessary dependencies:

# yum install -y sssd \
                 sssd-dbus \
                 realmd \
                 httpd \
                 mod_session \
                 mod_ssl \

This will give us the SSSD and the web server components we will need. The first step here will be to set up SSSD to authenticate this VM against the LDAP server. If the LDAP server in question is a FreeIPA or Active Directory environment, then realmd can be used to join this machine to the domain. This is the easiest way to get up and running.

realm join ldap.example.com

If you aren’t running a domain, then your best option is to use the authconfig tool (or follow the many other tutorials on the internet for configuring SSSD for identity and authentication).

# authconfig --update --enablesssd --enablesssdauth \
             --ldapserver=ldap.example.com \
             --enableldaptls \

This should create /etc/sssd/sssd.conf with most of the appropriate settings. (Note: RHEL 7 appears to have a bug wherein authconfig does not create the /etc/openldap/cacerts directory, so you may need to create it manually before running the above command.)

If you are interested in using SSSD to manage failover situations for LDAP, this can be configured simply by adding additional entries in /etc/sssd/sssd.conf on the ldap_uri line. Systems enrolled with FreeIPA will automatically handle failover using DNS SRV records.

Finally, restart SSSD to make sure that all of the changes are applied properly:

$ systemctl restart sssd.service

Now, test that the user information can be retrieved properly:

$ getent passwd <username>
username:*:12345:12345:Example User:/home/username:/usr/bin/bash

At this point, it is wise to attempt to log into the VM as an LDAP user and confirm that the authentication is properly set up. This can be done via the local console or a remote service such as SSH. (Later, you can modify your /etc/pam.d files to disallow this access if you prefer.) If this fails, consult the SSSD troubleshooting guide.

Step 3: Apache Configuration

Now that we have the authentication pieces in place, we need to set up Apache to talk to SSSD. First, we will create a PAM stack file for use with Apache. Create the /etc/pam.d/openshift file and add the following contents:

auth required pam_sss.so
account required pam_sss.so

This will tell PAM (the pluggable authentication module) that when an authentication request is issued for the “openshift” stack, it should use pam_sss.so to determine authentication and access-control.

Next we will configure the Apache httpd.conf. (Taken from the OpenShift documentation and modified for SSSD.) For this tutorial, we’re only going to set up the challenge authentication (useful for logging in with oc login and similar automated tools). A future entry in this series will describe setup to use the web console.

First, create the new file openshift-proxy.conf in /etc/httpd/conf.d (substituting the correct hostnames where indicated):

LoadModule request_module modules/mod_request.so
LoadModule lookup_identity_module modules/mod_lookup_identity.so
# Nothing needs to be served over HTTP.  This virtual host simply redirects to
<VirtualHost *:80>
  DocumentRoot /var/www/html
  RewriteEngine              On
  RewriteRule     ^(.*)$     https://%{HTTP_HOST}$1 [R,L]

<VirtualHost *:443>
  # This needs to match the certificates you generated.  See the CN and X509v3
  # Subject Alternative Name in the output of:
  # openssl x509 -text -in /etc/pki/tls/certs/proxy.example.com.crt
  ServerName proxy.example.com

  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile /etc/pki/tls/certs/proxy.example.com.crt
  SSLCertificateKeyFile /etc/pki/tls/private/proxy.example.com.key
  SSLCACertificateFile /etc/pki/CA/certs/ca.crt

  # Send logs to a specific location to make them easier to find
  ErrorLog logs/proxy_error_log
  TransferLog logs/proxy_access_log
  LogLevel warn
  SSLProxyEngine on
  SSLProxyCACertificateFile /etc/pki/CA/certs/ca.crt
  # It's critical to enforce client certificates on the Master.  Otherwise
  # requests could spoof the X-Remote-User header by accessing the Master's
  # /oauth/authorize endpoint directly.
  SSLProxyMachineCertificateFile /etc/pki/tls/certs/authproxy.pem

  # Send all requests to the console
  RewriteEngine              On
  RewriteRule     ^/console(.*)$     https://%{HTTP_HOST}:8443/console$1 [R,L]

  # In order to using the challenging-proxy an X-Csrf-Token must be present.
  RewriteCond %{REQUEST_URI} ^/challenging-proxy
  RewriteCond %{HTTP:X-Csrf-Token} ^$ [NC]
  RewriteRule ^.* - [F,L]

  <Location /challenging-proxy/oauth/authorize>
    # Insert your backend server name/ip here.
    ProxyPass https://openshift.example.com:8443/oauth/authorize
    AuthType Basic
    AuthBasicProvider PAM
    AuthPAMService openshift
    Require valid-user

  <ProxyMatch /oauth/authorize>
    AuthName openshift
    RequestHeader set X-Remote-User %{REMOTE_USER}s env=REMOTE_USER

RequestHeader unset X-Remote-User


Then we need to tell SELinux that it’s acceptable for Apache to contact the PAM subsystem, so we set a boolean:

# setsebool -P allow_httpd_mod_auth_pam on

At this point, we can start up Apache.

# systemctl start httpd.service

Phase 3: OpenShift Configuration

This describes how to set up an OpenShift server from scratch in an “all in one” configuration. For more complicated (and interesting) setups, consult the official OpenShift documentation.

First, we need to modify the default configuration to use the new identity provider we just created. We’ll start by modifying the /etc/origin/master/master-config.yaml file. Scan through it and locate the identityProviders section and replace it with:

  - name: any_provider_name
    challenge: true
    login: false
    mappingMethod: claim
      apiVersion: v1
      kind: RequestHeaderIdentityProvider
      challengeURL: "https://proxy.example.com/challenging-proxy/oauth/authorize?${query}"
      clientCA: /etc/origin/master/proxy/proxyca.crt
      - X-Remote-User

Now we can start openshift with the updated configuration:

# openshift start \
    --public-master=https://openshift.example.com:8443 \
    --master-config=/etc/origin/master/master-config.yaml \

Now you can test logins with

oc login https://openshift.example.com:8443

It should now be possible to log in with only valid LDAP credentials. Stay tuned for further entries in this series where I will teach you how to set up a “login” provider for authenticating the web console, how to retrieve extended user attributes like email address and full name from LDAP, and also how to set up automatic single-sign-on for users in a FreeIPA or Active Directory domain.

 Updates 2016-05-27: There were some mistakes in the httpd.conf as originally written that made it difficult to set up Part 2. They have been retroactively corrected. Additionally, I’ve moved the incomplete configuration of extended attributes out of this entry and will reintroduce them in a further entry in this series.