Sausage Factory: Modules – Fake it till you make it

Module Masquerade

Last week during Flock to Fedora, we had a discussion about what is needed to build a module outside of the Fedora infrastructure (such as through COPR or OBS). I had some thoughts on this and so I decided to perform a few experiments to see if I could write up a set of instructions for building standalone modules.

To be clear, the following is not a supported way to build modules, but it does work and covers most of the bases.

Step 1: Creating module-compatible RPMs

RPMs built as part of a module within Fedora’s Module Build Service are slightly different than RPMs built traditionally. In MBS, all RPMs built have an extra header injected into them: ModularityLabel. This header contains information about what module the RPM belongs to and is intended to help DNF avoid situations where an update transaction would attempt to replace a modular RPM with a non-modular one (due to a transient unavailability of the module metadata). This step may not be absolutely necessary in many cases. If you are trying to create a module from RPMs that you didn’t build, you can probably get away with skipping this step, provided that you don’t care if there might be unpredictable behavior if you encounter a broken repo mirror.

To create a module-compatible RPM, add the following line to your spec file for each binary RPM you are producing:

ModularityLabel: <arbitrary string>

Other than that new RPM label, you don’t need to do anything else. Just build your RPMs and then create a yum repository using the createrepo_c tool. The ModularityLabel can be any string at all. In Fedora, we have a convention to use name:stream:version:context to indicate from which build the RPM originally came from, but this is not to be relied upon. It may change at any time and it also may not be accurately reflective of the module in which it currently resides, due to component-reuse in the Module Build System.

Step 2: Converting the repo into a module

Now comes the complicated part: we need to construct the module metadata that matches the content you want in your module and then inject it into the yum repo you created above. This means that we need to generate the appropriate module metadata YAML for this repository first.

Fortunately, for this simple approach, we really only need to focus on a few bits of the module metadata specification. First, of course, we need to specify all of the required attributes: name, stream, version, context, summary, description and licenses. Then we need to look at what we want need for the artifacts, profiles and api sections.

Artifacts are fairly straightforward: you need to include the NEVRA of every package in the repository that you want to be exposed as part of the module stream. The NEVRA format is of the form examplepackage-0:0.1-5.x86_64.

Once the artifacts are all listed, you can decide if you want to create one or more profiles and if you want to identify the public API of the module.

It is always recommended to check your work with the modulemd-validator binary included in the libmodulemd package. It will let you know if you have missed anything that will break the format.

Shortcut

While drafting this walkthrough, I ended up writing a fairly simple python3 tool called repo2module. You can run this tool against a repository created as in Step 1 and it will output most of what you need for module metadata. It defaults to including everything in the api section and also creating a default profile called everything that includes all of the RPMs in the module.

Step 3: Injecting the module metadata into the repository

Once the module metadata is ready for inclusion, it can be copied into the repository from Step 1 using the following command:

modifyrepo_c --mdtype=modules modules.yaml /path/to/repodata

With that done, add your repository to your DNF/Yum configuration (or merge it into a bigger repository with mergerepo_c, provided you have version 0.13.2 or later) and run dnf module list and you should see your new module there!

 

Edit 2019-08-16: Modified the section on ModularityLabel to recognize that there is no defined syntax and that any string may be used.

Advertisements

Flock 2019 Trip Report

Just flew back from Flock to Fedora in Budapest, Hungary and boy are my arms tired! As always, it was an excellent meeting of the minds in Fedora. I even had the opportunity to meet my Outreachy intern, Niharika Shrivastava!

Day One – Thursday

As usual, the conference began with Matthew Miller’s traditional “State of Fedora” address wherein he uses pretty graphs to confound and amaze us. Oh, and reminds us that we’ve come a long way in Fedora and we have much further to go together, still.

Next was a keynote by Cate Huston of Automattic (now the proud owners of both WordPress and Tumblr, apparently!). She talked to us about the importance of understanding when a team has become dysfunctional and some techniques for getting back on track.

After lunch, Adam Samalik gave his talk, “Modularity: to modularize or not to modularize?”, describing for the audience some of the cases where Fedora Modularity makes sense… and some cases where other packaging techniques are a better choice. This was one of the more useful sessions for me. Once Adam gave his prepared talk, the two of us took a series of great questions from the audience. I hope that we did a good job of disambiguating some things, but time will tell how that works out. We also got some suggestions for improvements we could make, which were translated into Modularity Team tickets: here and here.

Next, Merlin Mathesius, our official Modularity Wizard, gave his talk on “Tools for Making Modules in Fedora”, focusing on various resources that he and others have created for simplifying the module packaging process.

Next, I rushed off to my annual “State of the Fedora Server” talk. This was a difficult one for me. Fedora Server has, for some time now, been operating as a largely one-man (me) effort of just making sure that the installation media continues to function properly. It has seen very little innovation and is failing in its primary mission: to provide a development ground for the next generation of open-source servers. I gave what amounted to an obituary speech and then opened the floor to discussion. The majority of the discussion came down to this: projects can only survive if people want to work on them and there really isn’t a clear idea of what that would be in the server space. Fedora Server is going to need to adapt or dissipate. More on that in a future update.

Later that afternoon, I attended Brendan Conoboy’s talk “Just in Time Transformation” where he discussed the internal process changes that Red Hat went through in order to take Fedora and deliver Red Hat Enterprise Linux 8. Little of this was new to me, naturally, having lived through it (with scars to show), but it was interesting to hear how the non-Red Hat attendees perceived it.

For the last event of the first day, we had a round of Slideshow Karaoke. This was a lot of fun and quite hilarious. It was a great way to round out the start of Flock.

Day Two – Friday

The second day of Flock opened with Denise Dumas, VP of Platform Engineering at Red Hat, giving a talk about “Fedora, Red Hat and IBM”. Specifically: How will the IBM acquisition affect Fedora? Short answer: it won’t. Best line of this talk: “If you want to go fast, go alone. If you want to go far, go together.”

After that came a lively panel discussion where Denise Dumas, Aleksandra Fedorova, Brendan Conoboy and Paul Frields talked to us about the relationship between Fedora and Red Hat Enterprise Linux 8, particularly where it diverged and a little of what is coming next for that relationship.

After lunch, I attended Pierre-Yves Chibon’s talk on Gating rawhide packages. Now that it’s live and in production, there was a very high interest; many were unable to find seats and stood around the walls. There was a short lecture describing the plans to get more tests and support for multi-package gating.

Next up, I attended Alexander Bokovoy’s talk on the “State of Authentication and Identity Management in Fedora”. Alexander discussed a lot of deep technical topics, including the removal of old, insecure protocols from Fedora and the status of authentication tools like SSSD and Kerberos in the distribution.

I went to yet another of Brendan Conoboy’s talks after that, this time on “What Stability Means and How to Do Better”. The focus on this talk was that “stability” means many different things to different people. Engineers tend to focus on stabliity meaning “it doesn’t crash”, but stability can mean everything from that through “backwards-compatibility of ABIs” and all the way through to “the user experience remains consistent”. This was quite informative and I think the attendees got a lot out of it. I did.

The next talk I attended was given by Niharika Shrivastava (my aforementioned Outreachy intern) and Manas Mangaonkar on “Students in developing nations and FOSS contribution limitation”. It provided a very interesting (and, at times, disturbing) perspective on how open-source contribution is neglected and even dismissed by many Indian universities and businesses. Clearly we (the FOSS community) need to expend more resources in this area.

Friday concluded with a river cruise along the Danube, which was a nice chance to unwind and hobnob with my fellow Fedorans. I got a few pictures, chatted with some folks I hadn’t seen in a long time as well as got introduced to several new faces (always wonderful to see!).

Day Three – Saturday

By the time Saturday rolled around, jet-lag was catching up to me, as well as some very long days, so I was somewhat tired and zombie-like. I’ve been told that I participated in a panel during the “Fedora Summer Coding 2019 Project Showcase and Meetup”, but I have few memories of the event. Kidding aside, it was a wonderful experience. Each of the interns from Google Summer of Code, Google Code-In and Outreachy gave a short presentation of the work they had been doing over the summer. I was extremely proud of my intern, Niharika, who gave an excellent overview of the translation work that she’s been working on for the last two months. The other projects were exciting as well and I look forward to their completion. The panel went quite well and we got some excellent questions. All in all, this year was one of my most positive experiences with internships and I hope very much that it’s setting the stage for the future as well.

After lunch came the headsman… I mean the “Modularity & Packager Experience Birds-Of-A-Feather” session. We started the session by spending fifteen minutes to list all of our gripes with the current state of Modularity packaging. These were captured on a poster board and later by Langdon White into a Google Doc. We then voted, unconference-style, on the issues that people most wanted to see addressed. The top four subjects were selected and we allocated a quarter of the remaining session time for each of them.

I personally missed the first topic as I ended up in a sidebar discussing internationalization plans with one of our Fedora Translation Team members, who had been following the work that Niharika and I have been doing in that space.

The other topics that were discussed at length involved how to perform offline local module builds, creating documentation and tooling to enable non-MBS services like COPR and OBS to create modules and how to deal with rolling defaults and rolling dependencies. Langdon White took additional notes and is, I believe, planning to present a report on it as well, which I will link to once it becomes available.

This was unquestionably the most useful session at Flock for me. We were able, in a fairly short period of time, to enumerate the problems before us and work together to come up with some concrete steps for addressing them. If this had been the only session I attended at Flock, it would still have been worth the price of travel.

Day Four – Sunday

Due to a slight SNAFU scheduling my return flight,  I had to leave at 11:00 in the morning to catch my plane. I did, however, spend a while in the morning playing around with some ideas on how to offer simple module creation to OBS and COPR. I think I made some decent progress, which I’ll follow up on in a future blog post.

Conclusion

As always, Flock to Fedora was an excellent conference. As every year, I find that it revitalizes me and inspires me to get back to work and make reality out of the ideas we brainstormed there. It’s going to be an interesting year!