Meritocracy and Hierarchy at Apache

Apache is actually governed in two ways: by the meritocracy of the Apache Way, and the hierarchy of the Apache Software Foundation. We strive to govern ourselves as much by meritocracy (and consensus) as possible, and as little by hierarchy as is practical for non-profit corporation. This has a number of both practical effects, as well as a number of social corollaries.

The ASF prides itself in running as a community-based meritocracy of peers. That is, within each project community, people participate as individuals and gain merit as measured by that community. The community recognizes your merit by voting people in as committers or PMC members. A new committer can vote on certain kinds of code changes, and can checkin their own code. PMC members vote on new committers and product releases.

In Apache projects the underlying clichĂ© is correct: those who are doing the work get to make the decisions. A healthy community will seek consensus over the direction of the project, and typically only uses [VOTE]s to solve thorny technical issues that don’t come to a clear consensus. Similarly, PMC members each have one vote on elections for new committers or PMC members; decisions within projects are made by the community as a whole. Technical governance is always done by meritocracy.

Even within the Foundation – i.e. the Delaware registered non-profit corporation that makes the ASF a legal entity – we strive to govern by meritocracy as well. Nominating new members and electing the board of directors are the obvious rights of members, and these are discussed and judged based on individuals’ merit towards the various Apache projects and to the ASF as a whole.

Similarly while the board appoints individuals to serve as corporate officers with specific delegated powers, all Apache members are allowed to subscribe to any mailing lists and participate across various ASF activities. Various members step up to work on policy, corporate, legal, treasury, and many other tasks that are needed in any corporation, always working with the relevant officers. Again, those who do the work decide how it gets done.

In these organizational tasks, it’s not always as clear how to reach consensus on contentious issues. Social issues or corporate governance questions don’t often have as simple a metric to measure against as technical issues do. Code either works or it doesn’t; corporate policy often has a very different kind of judgement. Similarly, with an active membership (with many very passionate individuals!), finding consensus on non-technical questions amongst 100+ individuals is often difficult. Here, hierarchy steps in to govern.

In most cases this works smoothly. Either the relevant officer will make a policy decision and publish it, or the board will pass a resolution or clearly ask for a change. While the whole community – the membership, the various PMCs of Apache projects – may not have a consensus on the decision, it is still policy, and projects will generally work to make the change.

In some cases, this doesn’t. A key way that the board ensures there is sufficient oversight of all Apache projects is by requiring reporting: monthly for officers; quarterly for PMCs / projects. There have been cases where PMCs have not reported regularly, or have not provided sufficient details to the board’s satisfaction in their reports. In these cases, the board will require a new or updated report. This is not a question of merit, or getting a consensus: this is an order. Non-response to a board request or not following an officer’s policy will and does result in the board stepping in to force changes within the project. As they say in politics, officers serve at the pleasure of the board.

It seems that sometimes this switch from meritocracy – where everyone in the PMC has an equal say, and they reach consensus – to hierarchy – where the board says “jump” – sometimes catches people by surprise. Sometimes, it seems that some PMC members just never really thought about the fact that the ASF is a legal corporation, and must follow certain rules. Other times, this surprise is clearly exacerbated by poor communication in specific cases between the board and projects. While cases like this really aren’t a choice – the board can and will dissolve or reconstitute projects (always as a last resort!) – it is important to clearly explain the issues and why this kind of hierarchy is needed in some cases.

Has anyone else had this feeling? Of switching between meritocracy, and consensus among the community, to hierarchy, and the board simply making a decision? How can we improve the handling of future issues like this, to ensure we keep the maximum meritocratic freedom within our projects, while still ensuring the legal shield of the ASF is secure and can function for the long term?

I’ve been thinking about this lately since I recently drew up a new org chart here for the ASF as a whole. Note that the hierarchy part only comes in on organizational, legal, and similar matters – technical decisions are always in the meritocratic hands of the PMCs and projects. More reading can be found on some of these excellent slides about Apache and open source.

What do you think?