Flock 2019 Trip Report

Just flew back from Flock to Fedora in Budapest, Hungary and boy are my arms tired! As always, it was an excellent meeting of the minds in Fedora. I even had the opportunity to meet my Outreachy intern, Niharika Shrivastava!

Day One – Thursday

As usual, the conference began with Matthew Miller’s traditional “State of Fedora” address wherein he uses pretty graphs to confound and amaze us. Oh, and reminds us that we’ve come a long way in Fedora and we have much further to go together, still.

Next was a keynote by Cate Huston of Automattic (now the proud owners of both WordPress and Tumblr, apparently!). She talked to us about the importance of understanding when a team has become dysfunctional and some techniques for getting back on track.

After lunch, Adam Samalik gave his talk, “Modularity: to modularize or not to modularize?”, describing for the audience some of the cases where Fedora Modularity makes sense… and some cases where other packaging techniques are a better choice. This was one of the more useful sessions for me. Once Adam gave his prepared talk, the two of us took a series of great questions from the audience. I hope that we did a good job of disambiguating some things, but time will tell how that works out. We also got some suggestions for improvements we could make, which were translated into Modularity Team tickets: here and here.

Next, Merlin Mathesius, our official Modularity Wizard, gave his talk on “Tools for Making Modules in Fedora”, focusing on various resources that he and others have created for simplifying the module packaging process.

Next, I rushed off to my annual “State of the Fedora Server” talk. This was a difficult one for me. Fedora Server has, for some time now, been operating as a largely one-man (me) effort of just making sure that the installation media continues to function properly. It has seen very little innovation and is failing in its primary mission: to provide a development ground for the next generation of open-source servers. I gave what amounted to an obituary speech and then opened the floor to discussion. The majority of the discussion came down to this: projects can only survive if people want to work on them and there really isn’t a clear idea of what that would be in the server space. Fedora Server is going to need to adapt or dissipate. More on that in a future update.

Later that afternoon, I attended Brendan Conoboy’s talk “Just in Time Transformation” where he discussed the internal process changes that Red Hat went through in order to take Fedora and deliver Red Hat Enterprise Linux 8. Little of this was new to me, naturally, having lived through it (with scars to show), but it was interesting to hear how the non-Red Hat attendees perceived it.

For the last event of the first day, we had a round of Slideshow Karaoke. This was a lot of fun and quite hilarious. It was a great way to round out the start of Flock.

Day Two – Friday

The second day of Flock opened with Denise Dumas, VP of Platform Engineering at Red Hat, giving a talk about “Fedora, Red Hat and IBM”. Specifically: How will the IBM acquisition affect Fedora? Short answer: it won’t. Best line of this talk: “If you want to go fast, go alone. If you want to go far, go together.”

After that came a lively panel discussion where Denise Dumas, Aleksandra Fedorova, Brendan Conoboy and Paul Frields talked to us about the relationship between Fedora and Red Hat Enterprise Linux 8, particularly where it diverged and a little of what is coming next for that relationship.

After lunch, I attended Pierre-Yves Chibon’s talk on Gating rawhide packages. Now that it’s live and in production, there was a very high interest; many were unable to find seats and stood around the walls. There was a short lecture describing the plans to get more tests and support for multi-package gating.

Next up, I attended Alexander Bokovoy’s talk on the “State of Authentication and Identity Management in Fedora”. Alexander discussed a lot of deep technical topics, including the removal of old, insecure protocols from Fedora and the status of authentication tools like SSSD and Kerberos in the distribution.

I went to yet another of Brendan Conoboy’s talks after that, this time on “What Stability Means and How to Do Better”. The focus on this talk was that “stability” means many different things to different people. Engineers tend to focus on stabliity meaning “it doesn’t crash”, but stability can mean everything from that through “backwards-compatibility of ABIs” and all the way through to “the user experience remains consistent”. This was quite informative and I think the attendees got a lot out of it. I did.

The next talk I attended was given by Niharika Shrivastava (my aforementioned Outreachy intern) and Manas Mangaonkar on “Students in developing nations and FOSS contribution limitation”. It provided a very interesting (and, at times, disturbing) perspective on how open-source contribution is neglected and even dismissed by many Indian universities and businesses. Clearly we (the FOSS community) need to expend more resources in this area.

Friday concluded with a river cruise along the Danube, which was a nice chance to unwind and hobnob with my fellow Fedorans. I got a few pictures, chatted with some folks I hadn’t seen in a long time as well as got introduced to several new faces (always wonderful to see!).

Day Three – Saturday

By the time Saturday rolled around, jet-lag was catching up to me, as well as some very long days, so I was somewhat tired and zombie-like. I’ve been told that I participated in a panel during the “Fedora Summer Coding 2019 Project Showcase and Meetup”, but I have few memories of the event. Kidding aside, it was a wonderful experience. Each of the interns from Google Summer of Code, Google Code-In and Outreachy gave a short presentation of the work they had been doing over the summer. I was extremely proud of my intern, Niharika, who gave an excellent overview of the translation work that she’s been working on for the last two months. The other projects were exciting as well and I look forward to their completion. The panel went quite well and we got some excellent questions. All in all, this year was one of my most positive experiences with internships and I hope very much that it’s setting the stage for the future as well.

After lunch came the headsman… I mean the “Modularity & Packager Experience Birds-Of-A-Feather” session. We started the session by spending fifteen minutes to list all of our gripes with the current state of Modularity packaging. These were captured on a poster board and later by Langdon White into a Google Doc. We then voted, unconference-style, on the issues that people most wanted to see addressed. The top four subjects were selected and we allocated a quarter of the remaining session time for each of them.

I personally missed the first topic as I ended up in a sidebar discussing internationalization plans with one of our Fedora Translation Team members, who had been following the work that Niharika and I have been doing in that space.

The other topics that were discussed at length involved how to perform offline local module builds, creating documentation and tooling to enable non-MBS services like COPR and OBS to create modules and how to deal with rolling defaults and rolling dependencies. Langdon White took additional notes and is, I believe, planning to present a report on it as well, which I will link to once it becomes available.

This was unquestionably the most useful session at Flock for me. We were able, in a fairly short period of time, to enumerate the problems before us and work together to come up with some concrete steps for addressing them. If this had been the only session I attended at Flock, it would still have been worth the price of travel.

Day Four – Sunday

Due to a slight SNAFU scheduling my return flight,  I had to leave at 11:00 in the morning to catch my plane. I did, however, spend a while in the morning playing around with some ideas on how to offer simple module creation to OBS and COPR. I think I made some decent progress, which I’ll follow up on in a future blog post.

Conclusion

As always, Flock to Fedora was an excellent conference. As every year, I find that it revitalizes me and inspires me to get back to work and make reality out of the ideas we brainstormed there. It’s going to be an interesting year!

Flocking to Kraków

In less than five days, the fourth annual Flock conference will take place in Kraków, Poland. This is Fedora’s premier contributor event each year, alternately taking place in North America and Europe. Attendance is completely free for anyone at all, so if you happen to be in the area (maybe hanging around after World Youth Day going on right now), you should certainly stop in!

This year’s conference is shaping up to be a truly excellent one, with a massive amount of exciting content to see. The full schedule has been available for a while, and I’ve got to say: there are no lulls in the action. In fact, I’ve put together my schedule of sessions I want to see and there are in fact no gaps in it. That said, here are a few of the sessions that I suspect are going to be the most exciting:

Aug. 2 @11:00 – Towards an Atomic Workstation

For a couple of years now, Fedora has been at the forefront of developing container technologies, particularly Docker and Project Atomic. Now, the Workstation SIG is looking to take some of those Project Atomic technologies and adopt them for the end-user workstation.

Aug. 2 @17:30 – University Outreach

I’ve long held that one of Fedora’s primary goals should always be to enlighten the next generation of the open source community. Over the last year, the Fedora Project began an Initiative to expand our presence in educational programs throughout the world. I’m extremely interested to see where that has taken us (and where it is going next).

Aug. 3 @11:00 – Modularity

This past year, there has been an enormous research-and-development effort poured into the concept of building a “modular” Fedora. What does this mean? Well it means solving the age-old Too Fast/Too Slow problem (sometimes described as “I want everything on my system to stay exactly the same for a long time. Except these three things over here that I always want to be running at the latest version.”). With modularity, the hope is that people will be able to put together their ideal operating system from parts bigger than just traditional packages.

Aug. 3 @16:30 – Diversity: Women in Open Source

This is a topic that is very dear to my heart, having a daughter who is already finding her way towards an engineering future. Fedora and many other projects (and companies) talk about “meritocracy” a lot: the concept that the best idea should always win. However the technology industry in general has a severe diversity problem. When we talk about “meritocracy”, the implicit contract there is that we have many ideas to choose from. However, if we don’t have a community that represents many different viewpoints and cultures, then we are by definition only choosing the best idea from a very limited pool. I’m very interested to hear how Fedora is working towards attracting people with new ideas.

 

Flock to Fedora: Day One (Morning)

Overview

Today was quite a day! I attended many interesting sessions and it’s going to be quite difficult to keep this post to a reasonable length. I make no promises that I will succeed at that. Here’s a set of highlights, mostly my stream-of-consciousness notes from the sessions I attended. I’ll pare down the best details in a final wrap-up post on Saturday or Sunday. For now, consider this an exhaustively detailed set of live notes published as a digest.

This has gotten quite long as I write, so I’m going to break these up into morning and afternoon reports instead. Enjoy!

Sessions

Free and Open Source Software in Europe: Policies and Implementations

The first talk I attended today was the keynote by Gijs Hellenius, Free and Open Source Software in Europe: Policies and Implementations. It was an excellent overview of the successes and issues surrounding the use of Open Source software in the European public sector. The good news was that more and more politicians are identifying the value of Open Source, providing lower costs and avoidance of lock-in. The bad news is that extensive lobbying by large proprietary companies has done a lot of damage, thus preventing more widespread adoption.

That is all in terms of usage. Unfortunately, the situation soured somewhat because developers of public sector code are hesitant about making their code open-source (or contributing to other open-source projects directly). It was noted that the legal questions have all been answered (and plenty of examples of open-sourced public development exist). Unfortunately, a culture of conservatism exists and has slowed this advance.

A highlight of the talk was the Top three most visible open source implementations section. The first of these was the French Gendarmie who switched 72,000 systems over to Ubuntu Linux and LibreOffice Desktops. While this author would certainly prefer that they had selected an RPM-based distribution, I have to bow to the reported 80% cost reduction.

The second major win was the conversion of 32,000 PCs in government and healthcare in the Spain Extremadura areas (mostly Debian).

The third one was the City Administration of Munich, who also converted 14,800 workstations from Windows to Ubuntu Linux and LibreOffice. Excellent quote on his slide (paraphrased because it went by too quickly): “Bill Gates: Why are you doing this, Mr. Ude?” “Ude: To gain freedom.” “Gates: Freedom from what?” “Ude: From you, Mr. Gates.”

Fedora QA – What Why How…?

The second talk I attended today was an introduction to Fedora QA from Amita Sharma, a Red Hat senior quality engineer.

Amita started by talking about why Fedora QA is important, citing that Fedora’s fast schedule would otherwise lead to a very buggy and unpleasant system to use. She showed us a few bugs in Bugzilla to explain how easy it can be to participate in QA. In the simplest sense, it’s no more than spotting a bug and reporting it.

Next she discussed a bit about the team composition, meeting schedule and duties. This involved things like update and release testing, creation of automation and validation testing, filing bugs, weekly meetings and scheduling and execution of Fedora Test Days.

Some of the interesting bits that Amita pointed out were some of the various available resources for Fedora QA. She made sure to call out the daily Rawhide Report emails to the devel@lists.fedoraproject.org mailing list, as well as Adam Williamson’s Rawhide Watch blog and the regular Security Update test notice.

She next launched into a detailed explanation of the karma process for Fedora testing updates. She explained how it is used to gate updates to the stable repository until users or QA testers validate that it works properly.

Amita moved on to describing the bug workflow process, from how to identify a bug to reporting it on Bugzilla and following it through to resolution.

The next phase of Amita’s talk focused on the planning, organization and execution of a Fedora Test Day, from proposing a Test Day to creating test cases and managing the process on IRC.

Where’s Wayland?

I was originally planning to attend the Taskotron talk by Tim Flink, but the room was over capacity, so I went over to the Wayland talk being given by Matthias Clasen (which was my close second choice).

I arrived a few minutes late, so I missed the very beginning and it took me a bit of time to catch up. Matthias was describing Wayland’s capabilities and design decisions at a very low level, in many cases over my head, when I came in, so I really regretted missing the beginning.

One of the fundamental principles of Wayland is that clients are fully isolated. There’s no root window and there’s no concept of where your window is positioned relative to other windows, no “grabs” of input and no way for one Wayland application’s session to interfere directly with another. (Author’s note: this is a key security feature and a major win over X11). Wayland will allow certain clients to be privileged for special purposes such as taking screenshots and accessibility tools. (Author’s note: I had a conversation earlier in the day with Hans de Goede where we talked about implementing Android-style “intents” for such privileged access so that no application can perform display-server-wide actions without express user permission. I think this is a great idea and we need to work out with user-experience designers how best to pull this off.)

Another interesting piece that Wayland will offer is multiple interfaces. For example, it will be much easier to implement a multi-seat setup with Wayland than it was with X11 historically.

Matthias then moved on to describe why we would want to replace X11. The biggest reason is to clean out a lot of cruft that has accumulated in the X11 protocol over the years (much of it unused in modern systems). Having both the compositor and renderer in the same engine also means that you can avoid a lot of excess and duplicated work in each of the window managers.

One very important piece is the availability of sandboxing. In X11, any application talking to the server has access to the entire server. This potentially means that any compromised X application can read anything made available to X from any other application. With Wayland, it becomes very easy to isolate the display server information between processes, which makes it much more difficult to escalate privileges.

Matthias also covered some of the concerns (or arguable disadvantages), the biggest of which was that compositor crashes will now destroy the display session, whereas on older X11 approaches it was possible to restart the window manager and retain the display. Other issues involved driver support; for example the nVidia driver does not support Wayland. (Note: the Nouveau driver works just fine, it’s the proprietary driver that does not.)

The next topic was GNOME support of Wayland, which has come a very long way in the last two years. GNOME 3.10 and later has experimental support for running on Wayland. The remaining gaps on Fedora 21 are mainly drag-and-drop support, input configuration and WACOM support. (Author’s note: I’ve played with GNOME 3.13.3 running atop Wayland. Much of the expected functionality works stably, but I wouldn’t yet recommend it for daily use. Given the rate of improvement I’ve seen, this recommendation may change at F21 GA release).

Matthias made a rather bold statement that the goal to hit before making Wayland the default in Fedora is that end-users should not notice any difference (Author’s note: presumably negative difference) in their desktop.

Flock to Fedora: Intro

Today was a big day: we kicked off the first of the four-day Flock conference. This is the second time we’ve run a Flock, which is the largest annual gathering of contributors to the Fedora Project. There are a great many excellent talks scheduled for Flock this year, and I’ll only be able to attend a handful of them. Fortunately, all of the talks (but not the hackfests and workshops, sorry!) will be streamed live and recorded at the Flock YouTube Channel. My plan is to take a few notes on each of the talks I attend and put up a summary each day with the highlights. Come back tonight for my review of the first day!

 

Edit: See also Máirín Duffy’s blog for an absolutely excellent breakdown of how to follow along with Flock remotely.

 

Flock 2013 Con Report

My god, it’s full of stars!

So as most of you probably know, this past weekend was Flock, the first of a new breed of Fedora conference that replaces FUDCon. It was reasonably well-attended (the count I heard was about two hundred). Among those that attended were nearly all of the well-known members of the Fedora community. I’m sure some (many?) of them will also be summarizing their experiences, but I want to get my notes out there while it’s still fresh. This is not a comprehensive overview of Flock. If that’s what you’re looking for, Máirín Duffy has provided an excellent set of blog posts including the recordings and transcripts[1].

The first three days followed a pattern of talks/lectures before lunch and workshops or hackfests in the afternoon. There were a great many sessions (usually eight or nine running concurrently at any time) and this made it somewhat difficult to see all of the useful content.

There were a great many interesting talks, but I’m going to focus this report on the three that I think will have the most lasting (and widely-felt) impact on the Fedora project as a whole.

Updates Model

The first here was Tom Callaway’s proposal of changing the default updates delivery method. As he put it, currently Fedora users are “drinking from the firehose”. All updates are delivered as soon as they hit the stable updates repository and it feels to users like they are in constant flux. Tom’s proposal is that instead of a constant updates stream, we should shift to delivering important[2] updates as part of a monthly update set. Security issues should still be released in the current model, so that users get those as quickly as possible. There was some debate over whether security updates should be installed automatically or just notified, but that was left unresolved. The set of “important” updates should be rolled up as a bundled collection of updates that is prepared a week before the monthly deliverable and offered for testing as a complete set. Package updates outside this set should be made available similar to the way Microsoft does its “optional” updates. They can be selected manually (or in bulk) in an updates tool if the user wants them. We believe that, by having the ability to tests updates as a set we will be able to reduce some of the instability the arises throughout the Fedora supported lifetime as well.

Tom’s proposal dovetails closely with the next two proposals, one from Matthew Miller and one from myself. Matthew’s proposal is fairly well-known at this point. It involves redefining Fedora into a set of policy rings wherein we can tighten or relax the packaging rules in order to make it easier to run applications and SDKs in or on Fedora. I won’t go into great detail on this proposal, as it’s been discussed at great length on other lists. I will say that the Fedora community as represented by Flock attendees was receptive and very much liked the idea of relaxed packaging requirements at the higher levels of the stack, as well as being able to layer projects atop other foundational pieces (such as OpenShift Gears or Software Collections).

Fedora as Products

The final proposal was one that I came up with and discussed with Robyn Bergeron and the RHEL architects shortly before Flock. The general idea behind this proposal was to stop trying to treat Fedora as “everything for everyone”. Instead of our classic approach of building all projects in Fedora as a single collection of packages, we should instead focus on a set of specific constituencies and serve those needs directly and completely. To that end, I proposed that we should build three products from the Fedora family with very specific target audiences and then work with Matthew’s proposal to layer other software atop these products in a sensible way.

Fedora Server

Target: People for whom RHEL and CentOS don’t move fast enough, and people who want to see and influence the future of RHEL
Purpose: Build a better RHEL, and engage customers in development of RHEL, OpenStack, and other emerging technologies.

One of the key points here is that we would start making it clear that Fedora Server is the development path for RHEL 8. Disruptive changes made to the Fedora Server would be carefully scoped out ahead of time. We want this to be the place that we can engage application developers and tell them “if you want something from RHEL to enable functionality for your application, this is where you build it”.

Fedora Cloud

Target: Development; DevOps in production; OpenShift
Purpose: We do not become entirely irrelevant.
This should be mainly a proper subset of the Fedora Server, with some few additional packages for acting as a virtualization client. The goal here is to provide the definitive operating system for running in a PaaS environment like OpenShift. This is the fast-moving DevOps platform that people who are doing rapid develop-and-deploy with high resiliency would want to use in production.

Fedora Workstation[3]

Target: Creatives, Developers, Sysadmins, and other IT professionals
Purpose: Keep Linux users on Fedora (This is a niche market; we recognize that. But let’s make it our niche.)
This is the platform upon which we would want people to be doing useful work in the Fedora/RHEL ecosystem. It’s where the “makers” and IT professionals should be able to get their work done. This is also the product that would have the most leniency in making disruptive changes (the most obvious forthcoming one being Wayland). That is not to say that this should be “Rawhide in disguise”. The goal here (as with the other products) is that this should be a stable and useful workstation that enables the user. It is not a playground, nor is it a general-purpose desktop.

Tying it together

The responsibility for defining the specific requirements placed upon each of these products would be placed upon a working team (chaired by a FESCo member) for each product. The proposals presented therein would require approval and then would become essentially a constitution for that team. The specific details of how this process would work are still being worked out. All of this is contingent upon acceptance of the overall plan by the Fedora Advisory Board, who will be receiving a formal proposal in the next week or so.

Last Day

The fourth day was dedicated to hackfests. I spent the morning working on Kerberos-related hacking on the fixes for the new kernel keyring support and then the afternoon in Matthew Miller’s “Hack the Future” session discussing his and my proposals for a new, more targeted Fedora design. In this latter discussion, we came up with most of the above plans.

It is worth noting that while the overall reception of these plans was positive and encouraging, there were individuals from Fedora Release Engineering and Fedora Quality Assurance teams that fear these changes will strain their already-limited resources. This is a valid concern and will be addressed. We’re looking at ways to reduce the impact on these two teams (such as by increasing automation and re-using existing capabilities wherever possible) as well as possibilities for providing increased resourcing. No plan will go forward without consulting these two groups.

If you have read this far, I thank you. I know it’s a bit wordy.

Notes

[1] http://blog.linuxgrrl.com/category/fedora/flock/
[2] “Important” should be left intentionally vague at this time.
[3] There is debate over whether to call it “Workstation” or “Client”, but we all agree that “Desktop” is not the right name. I’m in favor of Workstation, so that’s what I’ll use here.

Flock to Fedora!

In just a little over a month’s time, Fedora will be kicking off it’s brand-new annual developer’s conference in Charleston, South Carolina. In the past, Fedora would run two or more regional conferences called “FUDCon”[1] each year. Historically, these FUDCons were run as “unconferences”, where the most devout of Fedora’s community would converge and put together an agenda on the fly during the first day. Starting this year, instead of an unconference format, we’ve opted for developing something a lot closer to the OpenStack Summit. We opened a call for participation several months ago and then had a public voting period to determine which set of proposed talks and hackfests would be scheduled.

This new conference, named “Flock” will be running for four days, from August 9th through the 12th. The plan is to have a keynote each morning, followed by nine concurrent rooms holding one-hour talks until lunch. After lunch, the rooms will be dedicated to hackfests and birds-of-a-feather discussions. The goal of Flock is to be more of a “Do-Con” (described as such by our illustrious Fedora Project Leader, Robyn Bergeron). We want people to come and learn something each morning, then get together in the afternoon and hack on it (either code or conversation). August 12th is reserved entirely for hackfests and is currently unscheduled. If you get a good start on something new and cool at the conference, you can stick around and really hack on it on Monday.

I submitted a number of proposed sessions to Flock and had a total of five (!) selected for inclusion. If my readers would like to stop in and hear what I have to say, you are welcome to come visit any of the following sessions:

• Aug. 09, 1400-1500 EDT: The Fedora Crystal Ball: Where are we going for the next five years?
• Aug. 10, 1400-1500 EDT: The Totally New Fedora Changes Process
• Aug. 11, 1200-1300 EDT: Unifying Administration with OpenLMI
• Aug. 11, 1400-1600 EDT: OpenLMI Manageability Hackfest
• Aug. 11, 1600-1800 EDT:  Conference Recap with FESCo

These are, of course, not all of the sessions I will be attending (in fact, looking at the schedule, there is not a single time slot where I do not plan to be attending at least one session). The full schedule for Flock looks to be amazing and I think that there will be a great deal of interesting discussion and work done during this long weekend.

[1] “Fedora Users and Developers Conference”, as well as being a play on words (FUDCon: Against FUD)