The vibrancy of open source projects/products

Posted by

On 8th December 2020, the other shoe dropped: the project known as Community Enterprise Operating System (CentOS, and I pronounce it as cen-tos, ypmv), which has been under the Red Hat technology ecosystem, got redefined as CentOS Stream and also where in the ecosystem CentOS Stream would fit.

But first, some background.

In 2002, Red Hat, in response to customer needs for supportable Linux operating systems, Red Hat created Red Hat Advanced Server 2.1 which in 2003, was renamed as Red Hat Enterprise Linux 2.1. This change to a subscription based model of consumption of Red Hat Enterprise Linux, upset the community who have been consuming Red Hat Linux since the launch of RHL in 1994.

Yes, I was part of that user community that was not happy with the change. And, yes, I did try to see what I could do to continue to benefit from what RHL used to provide in the new RHEL space.

I was not alone in that pursuit. There were many efforts around the world to create a RHL-like system available at no cost to the end user but based on the RHEL code base. The code base of RHEL is available for download because Red Hat complies to and exceeds the requirements of the GNU General Public License of making available the code that goes into a product. [See here for a twitter thread on how Singtel and Aztech have failed to respect the terms of GPL license of the code running in their products]

Of all those efforts, two of them stood out – Community Enterprise Operating System and White Box Enterprise Linux. Both projects took the code of RHEL and compiled the binaries from the code so that users can freely download the ISOs and/or rpms and do as they pleased.

There is another distribution, Scientific Linux, maintained by Fermi Lab, that is primarily used by research institutions around the world and is a downstream of RHEL Naturally, anyone is free to use SL for their own uses as well.

In 2014, given the growth of OpenStack and related container-based methods of technology provisioning and consumption, the need for a stable base level operating system became critical for all the innovation that needs to happen. Going through the typical subscription acquisition process would not be useful (or desirable) for the high velocity of innovation that needs to happen. Hence, lots of the OpenStack and container roll-outs were based on CentOS.

The opportunity to help drive container adoption as well as building the next generation of a Platform-as-a-Service that ensures no cloud-vendor lock-in (aka hybrid cloud) took on added impetus. In 2014, Red Hat decided to bring within the Red Hat ecosystem, CentOS – the shoe dropped. The move has a mixed response. There were people who were dead against it (internal and external) but there was something that I felt was good about the idea, given the state of technology then. Also, this acquisition is really an acqui-hire. CentOS benefited very much from the added resources Red Hat was able to provide.

It must be noted that from the beginning of the CentOS project, CentOS was run by a relatively small group of people who essentially were making a distribution that was bug compatible with RHEL. They were not in the business of fixing anything that’s broken that was not already broken in RHEL and when a fix was made available in RHEL, it becomes available in CentOS. There was also no SLA statements or official support offered to CentOS users. They were on their own. CentOS’ website made that very clear. If a CentOS user needs official accountability, they should consider getting a RHEL subscription.

There really wasn’t an open source community of contributors to CentOS – the community was really one of consumers. Not that it is bad, but we need to understand the lay of the land and the people of that land. Let me reiterate – CentOS was never really an open source project like Fedora for example.

Yes, one could find issues in CentOS and provide bug fixes, but these rarely got fed back upstream to RHEL. In many ways, RHEL was a juggernaut that held it’s own because the engineering and QE was done with years and years of experience of doing quality work and strict adherence to stability.

I remember many times, over the years, speaking with customers (CIOs, CTOs, IT Directors etc) about their journey in adopting RHEL and the officially supported Red Hat product portfolio and telling inviting them to contribute features, bug fixes etc to the Red Hat ecosystem. They were all not entirely convinced that Red Hat would accept those contributions as they have never seen anything like that from the proprietary vendors they engage with. But we are Red Hat. And we did have contributions from various customers to the open source products we shipped them.

The same contribution to CentOS, on the other hand, was going to be hard to consider and incorporate in RHEL because there was necessarily a different mode of existence of both systems.

These customers would also ask me of the “roadmap of Red Hat’s products” and all I needed to do is to point them to the various upstreams of the Red Hat open source products like Fedora, JBoss etc with the caveat that, just because something was made available in the upstream, does not necessarily mean it will also be made available downstream in the Red Hat open source products pantheon. The best way to do that is to get engaged with the actual product teams.

I think it is useful to cite an example. Years ago, a local ISV/SI was providing a solution to a local bank that was going to be running JBoss as their middleware platform. The ISV/SI, instead of working with Red Hat and using the respective JBoss open source products, decided to work with a JBoss upstream open source project (for a discussion of OS products vs OS projects, see my talk here). The ISV/SI used a feature/functionality of the upstream project as part of their solution and when they deployed their solution into production for the bank, it failed – duh!. The ISV/SI was struggling to get is up and running by trying to add the functionality into the product that the bank had and failing. They reached out to us for help. They were fully aware that they were using upstream and that we are not obligated to help. The ISV/SI got their customer, the bank, to contact us for help. Out of goodwill and recognising that the solution being deployed is going to be a mission critical system, we agreed with the customer on the understanding that they will only use the product version in production and not the project version. It was hard to switch over completely, but we worked with both the ISV/SI and the bank customer to make it happen. Herein lies the learning: open source allows you to tinker, tweak and work with the upstream, but one has to always be mindful of what the delta that is between the upstream project and the downstream product. Failing to recognise that is not going to be good for all concerned.

Now let’s fast forward to 8th December 2020. Red Hat and CentOS announced that there will be a change in how CentOS will be working moving forward and that it will become an upstream channel of RHEL to be called CentOS Stream. RHEL will have the primary upstream, Fedora which has a constant six-month cadence in new version release and CentOS Stream side streaming, in a birectional manner, into and out of RHEL. This model will give the open source community the opportunity to contribute fixes, features etc that could get adopted in CentOS Stream and as as result in RHEL.

I think Scott McCarty‘s post does a very good job in explaining the change and I shall encourage you to read that to understand.

Free and open source software has fed my family and I, helped us move the needle in technology adoption and grown employment within Red Hat as well as enable/empower the greater IT ecosystem – globally.

When I started in 2003, we were about 700 Red Hatters globally and we had exactly one product, RHAS 2.1 aka RHEL 2.1. We were trying to figure out how to make a business selling open source software and, believe me, it was not easy.

Selling subscriptions was not an easy undertaking in the early days (and it has only become slightly easier in the last 5 to 7 years). This was because the IT industry was only familiar with the proprietary software licensing model that you paid a lump sum for the technology and then that’s it.

Keeping the proprietary software business model moving along meant that those vendors had to release new versions every now and then. Each new version was deemed a “revenue event”.

Not so for Red Hat.

As long as you are a subscriber to Red Hat, you get access to everything we have for the product you are a subscriber to including older (but not EOLed) versions and new versions as they get released. I recall Charlie Peters, a former CFO of Red Hat, recounting his frustration, during earnings calls (back when we were still listed on NYSE) why a new version release of RHEL (for example RHEL 4 to RHEL 5) was not a revenue event. He had to repeatedly explain the subscription model until they finally got it (or not).

Playing the infinite game

The days after the announcement of the CentOS Stream move has helped me look at the big picture of the vibrancy of open source projects/products. The more I look at it, the more I am convinced that in order to ensure that we in Red Hat can bring to bear quality open source products rapidly with collaboration from customers, that we need to innovate into a model that is vibrant and easy for the community/customers to consume/participate. I see the change from CentOS being downstream of RHEL into CentOS Stream being a sidestream of RHEL as an evolution of the model that is foundationally significant as was the change from freely downloadable RHL to a subscription based RHEL in 2002.

What would now fill the gap that is left by CentOS no longer being downsteam of RHEL? You could run Scientific Linux or you could take the RHEL code and compile it for yourself. Or you could look at the Rocky Linux effort.

And if you raring to get going to contribute to CentOS Stream don’t wait. Go to this wiki (https://wiki.centos.org/Contribute/CentOSStream) and follow the instructions and join us!

Leave a Reply