Meritocracy and Hierarchy at Apache

Apache is actually governed in two ways: by the meritocracy of the Apache Way, and the hierarchy of the Apache Software Foundation. We strive to govern ourselves as much by meritocracy (and consensus) as possible, and as little by hierarchy as is practical for non-profit corporation. This has a number of both practical effects, as well as a number of social corollaries.

The ASF prides itself in running as a community-based meritocracy of peers. That is, within each project community, people participate as individuals and gain merit as measured by that community. The community recognizes your merit by voting people in as committers or PMC members. A new committer can vote on certain kinds of code changes, and can checkin their own code. PMC members vote on new committers and product releases.

In Apache projects the underlying cliché is correct: those who are doing the work get to make the decisions. A healthy community will seek consensus over the direction of the project, and typically only uses [VOTE]s to solve thorny technical issues that don’t come to a clear consensus. Similarly, PMC members each have one vote on elections for new committers or PMC members; decisions within projects are made by the community as a whole. Technical governance is always done by meritocracy.

Even within the Foundation – i.e. the Delaware registered non-profit corporation that makes the ASF a legal entity – we strive to govern by meritocracy as well. Nominating new members and electing the board of directors are the obvious rights of members, and these are discussed and judged based on individuals’ merit towards the various Apache projects and to the ASF as a whole.

Similarly while the board appoints individuals to serve as corporate officers with specific delegated powers, all Apache members are allowed to subscribe to any mailing lists and participate across various ASF activities. Various members step up to work on policy, corporate, legal, treasury, and many other tasks that are needed in any corporation, always working with the relevant officers. Again, those who do the work decide how it gets done.

In these organizational tasks, it’s not always as clear how to reach consensus on contentious issues. Social issues or corporate governance questions don’t often have as simple a metric to measure against as technical issues do. Code either works or it doesn’t; corporate policy often has a very different kind of judgement. Similarly, with an active membership (with many very passionate individuals!), finding consensus on non-technical questions amongst 100+ individuals is often difficult. Here, hierarchy steps in to govern.

In most cases this works smoothly. Either the relevant officer will make a policy decision and publish it, or the board will pass a resolution or clearly ask for a change. While the whole community – the membership, the various PMCs of Apache projects – may not have a consensus on the decision, it is still policy, and projects will generally work to make the change.

In some cases, this doesn’t. A key way that the board ensures there is sufficient oversight of all Apache projects is by requiring reporting: monthly for officers; quarterly for PMCs / projects. There have been cases where PMCs have not reported regularly, or have not provided sufficient details to the board’s satisfaction in their reports. In these cases, the board will require a new or updated report. This is not a question of merit, or getting a consensus: this is an order. Non-response to a board request or not following an officer’s policy will and does result in the board stepping in to force changes within the project. As they say in politics, officers serve at the pleasure of the board.

It seems that sometimes this switch from meritocracy – where everyone in the PMC has an equal say, and they reach consensus – to hierarchy – where the board says “jump” – sometimes catches people by surprise. Sometimes, it seems that some PMC members just never really thought about the fact that the ASF is a legal corporation, and must follow certain rules. Other times, this surprise is clearly exacerbated by poor communication in specific cases between the board and projects. While cases like this really aren’t a choice – the board can and will dissolve or reconstitute projects (always as a last resort!) – it is important to clearly explain the issues and why this kind of hierarchy is needed in some cases.

Has anyone else had this feeling? Of switching between meritocracy, and consensus among the community, to hierarchy, and the board simply making a decision? How can we improve the handling of future issues like this, to ensure we keep the maximum meritocratic freedom within our projects, while still ensuring the legal shield of the ASF is secure and can function for the long term?


I’ve been thinking about this lately since I recently drew up a new org chart here for the ASF as a whole. Note that the hierarchy part only comes in on organizational, legal, and similar matters – technical decisions are always in the meritocratic hands of the PMCs and projects. More reading can be found on some of these excellent slides about Apache and open source.

Shane’s Apache Director Position Statement 2012

The ASF is holding it’s Annual Member’s Meeting this week, where the Membership elects a new board of directors along with other matters, like voting in new Member candidates. I am truly fortunate enough to have been nominated again for the board election, in this case by one of the founders of the ASF (JimJag). Along with believing that I can do a great deal of work to help the foundation and all Apache projects as a director, I also have to say that I have a lot of fun serving on the board. Yeah, I’m a little funny that way in liking doing all the organizing work behind the scenes.

Director candidates at the ASF write position statements about what their objectives for being a director are in preparation for the voting process. Since I write what I believe in, I also am posting my statement here, publicly. One of the biggest issues for the smooth functioning of the ASF as a home for healthy projects is doing a better job of explaining how we work – I hope this helps people understand us Apache types just a little bit better.


Shane Curcuru (curcuru)–Director position statement –v1.0

My mission statement for the ASF

“The mission of the ASF is to provide high quality, open source software for the public good at no cost, and to showcase our meritocratic and community-driven method of building sustainable software projects.”

http://shane.curcuru.name/blog/2009/04/what-i-believe-asf-mission/

Objectives as a Director

My objective as a director is to prevent unnecessary drama and help the ASF and all Apache projects work more smoothly by:

  • Acting as a calm, moderating, and explanatory voice on the board, and working to improve the board’s communication style,
  • More clearly documenting and publicizing the very few rules and requirements that we expect of Apache projects,
  • Better explaining and educating our project communities on the very many best practices and suggested guidelines we have.

I’ve learned a lot in the past few years on the board. A key lesson is that we need to minimize the number of hard rules that we have for our projects to those that are truly
requirements. We also need to much more clearly explain those rules and educate our project communities about them.

Likewise, we have an immense amount of best practices and great guidelines for running successful, collaborative projects that we’ve proven are helpful over and over again. More clearly presenting these as guidelines to our project communities and to the world at large will help all of our projects to succeed.

We’ve often said the board is a blunt instrument – but I believe that we’re experienced enough and a large enough organization that we don’t need to be. There are a number of times in the past years where the effect of the board tromping in to fix a project issue has been… unfortunate. With more professional communication and with better explanations of the rules and guidelines we expect Apache projects to follow, we can resolve project issues – and provide better advice to projects when they ask – with much less drama than we’ve had in some cases in the past.

We have immense collected wisdom on managing community led projects within the membership and the excellent director candidates for this year. It’s time to ensure we make this wisdom accessible to both our project communities and to potential contributors by better documenting it and presenting it, rather than waiting until after an issue arises and then explaining “you’re doin’ it wrong!”

Common Questions

Some members have been discussing a handful of questions about what people think about the future of the ASF, and some of our key projects and technologies, which I’ll comment on in some general ways.

The Apache Incubator is probably the most important project at the ASF today. It is where we should be teaching new podling communities both what is expected behavior and processes for Apache projects, as well as mentoring new communities on effective ways to build consensus-driven and community-built software projects. While the Incubator has been doing OK for a while, there is a lot of room for improvement. Luckily, both with Jukka as the new chair and a number of new, energetic members thinking through improvements, things are looking great. I’d encourage the IPMC to continue to simplify and normalize the sometimes wordy, pedantic, and all-over-the-place documentation for podlings – and especially to continue the great report review process changes that Jukka has begun. Similarly, we need a strong focus on ensuring that Members who sign up to mentor new podlings are truly showing up and working with their podling communities.

This is actually one of the most crucial areas that Apache Members can help. Both in terms of mentoring new podling communities within the Incubator, and in terms of helping to keep an independent and wise ear within our existing TLPs, Members are the personal element to the shared tribal knowledge of the Apache Way. Fundamentally, individuals will do what interests them; however I believe with some effort on clearer documentation around best practices, and especially some patience and effort at remaining calm and polite in our discussions (both privately and publicly), that we can improve Member activities within projects. In particular, this is a place where we need existing members to step up and provide strong mentoring to the various new Member candidates I expect will be voted into the ASF at this week’s meeting.

Discussions around new tools and arguments about technical merits don’t usually interest me that much. What does interest me is ensuring that we work to provide the tools that our projects responsibly ask for, and that for all of our tools we:

  • Ensure we have a reliable capacity to maintain the tools with our own existing infrastructure (people and servers). It is important that the ASF is in control of the tools and servers that our projects need to continue their work, since we’ve seen issues in the past where, even with the best of intentions, third parties maintaining tools for us just doesn’t work in the long term.
  • Ensure that the ways the tools are used truly supports consensus-driven and community-built software projects, in ways that keep our projects functioning as true collaborations of a community, and not just a collection of individuals.

Greg Stein – another existing Director who’s nominated for another term – came up with a great objective: to create a Foundation that can live for 50 years, hosting healthy projects. I love that idea. I think we already have the start of that, however some key things to think about are:

  • Better documenting and mentoring new members and new communities on the core values of the Apache Way. As we grow, it’s harder and harder to convey the tribal knowledge that many of our members share organically; it’s time to be more consistent at educating newcomers to all of them.
  • Much more clearly document the minimum required rules that we need to operate. I say minimum, because these should be limited to only what we truly need; however there are plenty of corporate procedures that we know how to do, but may not have written down for the next decade’s set of officers and directors to learn from.
  • Continue to evaluate new tools, methods, ideas, and incoming communities to see how we can grow and improve upon our core Apache Way – all the while without losing our way. The ASF is not a place for all projects; we have a clear ethos around how to build and distribute software that we should not bend. But there are improvements we can make to ease the way for communities of developers who do share our core values.
  • We need to get better and more frequent feedback from our regular sponsors, since their recurring donations are critical to keep our independence – through the infrastructure, press, and other services those funds allow us to provide to all Apache projects. This is never to say that sponsorship dollars ever translate into technical influence. But for many potential sponsors, it’s both what we do and how we present ourselves that is important.
  • We need to ensure that as we grow staff positions (infrastructure and other contractors) that we are careful to keep our policy decisions thinking about the future Foundation, hosting projects full of volunteers; and our paid staff and officers focused on execution.

Some members have asked about legal and trademark issues. These are both areas that work differently than our Apache projects, in that much of the work is necessarily done in private lists. This often makes it more difficult for people to understand how comfortable they are with what’s happening (even though all Apache Members are always welcome to review any mailing list, even private ones).

In both the legal and trademark areas, I agree with most directors that we have delegated execution of these areas to the respective officers (disclaimer: I am the current VP, Brand Management), and that I trust those officers will responsibly manage their affairs. I’m confident this will work, and that we’ll be able to find the resources needed, both because I know the officers, and because we have a solid and monthly reporting that the board reviews and discusses regularly. Our Conferences Committee has had some issues in the past year or so with burnout (planning ApacheCon takes a lot out of people – as I and a dozen or so other Members know far too well). But we have a new VP in the form of Nick, and some great energy from a number of new volunteers in that space, so I’m confident that while ApacheCon may be a little bit different in years to come, that it and the many other BarCamps and smaller events we’re lending our brand to will be successful in bringing newcomers into our communities.

Many members have varying opinions on growth: how much should the ASF grow, how should we grow, etc. I recently wrote about this in my set of slides About: Apache, where I reviewed the ASF and the Apache Way. A key part of my slides is: the ASF doesn’t have a technical agenda. “Apache” doesn’t have a top-down list of projects or technologies we want to “acquire”. The ASF exists to provide a home for like-minded communities who build software. While I hope that my work on Apache branding and work on improving documentation about the ASF and Apache projects helps new communities understand who we are, I don’t personally plan to directly “recruit” new projects. If people ask, I’m certainly very happy to tell them about the Apache Way, and why I think it’s so cool – but I’m not looking to proselytize.

Officers and Directors

If elected I will not accept any new officer positions this term, unless the new board wishes to appoint me Chair/Vice Chair.

About Shane

I recently had my 20th+ anniversary at IBM, where I work in the HR division as an Applications Architect. My employment and income have been unrelated to my work at the ASF for many years, and I will always clearly separate volunteer work from employer-funded work.

My involvement in the ASF is driven by a belief in, and a love of, the ASF, and is not influenced by politics or finances. I live in Arlington, MA with my wife, young daughter, and 2 cats. I view directorships and officer positions at the ASF as serious commitments.

I will continue my attendance at every board meeting if re-elected.


Reminder: “The postings on this site are my own and don’t necessarily represent the positions, strategies or opinions of my employer nor of any volunteer organization I work with.”

CamelOne 2012 Presentation – About: Apache

I was recently asked to speak at the CamelOne conference here in Boston, about The Apache Way. Here are my slides (including my speaker notes, written today).

TheApacheWay-ShaneCurcuru-CamelOne2012

This presentation is placed under the Apache License, v2.0, and was created with Apache OpenOffice v3.4 Impress application, and uses Open Sans fonts, also under the Apache License. This is not to say that I use only the Apache license, more to point out that there are plenty of different kinds of resources available today under the Apache License.

If you have suggestions, questions, or comments about this or anything else Apache, please let me know! Even better, ask about it on the Apache Community Development Mailing Lists!

Many thanks to Jim Jagielski and Justin Erenkrantz, who’s past Apache Way slides I borrowed heavily from; and also many other Apache members like Lars, J Aaron and company who have given many an Apache Way talk in the past.

Apache projects are technically independent

In the earlier essay I wrote on Apache project independence (since promoted to an article on the Apache Community Development website) I missed a critical bit of independence that I think we should add to the ComDev site.

Apache projects are technically independent

While Apache PMCs provide a report of their operations to the Board on a quarterly basis, they are technically independent of the Board. PMCs are free to collaboratively decide their project’s technical direction however they choose. While the Board holds PMC chairs and PMCs accountable for ensuring that their methods are appropriate – following the Apache Way and the few other rules at Apache – the Board does not exercise any technical direction over Apache projects.

This is a critical point to understand. There is no top-down direction at the ASF in terms of what technology we implement or what kinds of projects we seek out or support. All technical choices are made by the individual committers and PMC members within their project communities – i.e. the individuals actually doing the coding decide where the coding will go.

This applies equally to the Apache Incubator, as the gateway of new project proposals at Apache. The ASF has noagenda of what projects it wants to find, nor do we have a corporate strategy for new project acquisition. Project communities that want to come to the ASF are welcome – and will be reviewed in the Incubation process for community health and diversity, and for IP clearance and licensing. But not for technical merit.

The essence is: if there is a sufficiently diverse community who chooses to live by the Apache Way, the ASF is happy to host them. The fact that there is sufficient diverse community means that people are interested in the technology. Similarly, the Apache Attic awaits projects that may seem interesting technically, but fail to have a sufficiently diverse and active community to maintain them.

So – worthy to add to the ComDev page?

Thank you, Apache Sponsors!

I just wanted to take time to say thank you to the many financial Sponsors of the ASF.
Thank you, Apache Sponsors!

Our many regular Sponsors pledge to donate a set amount of funds to the ASF annually, and they really deserve some recognition for this. While we very, very much appreciate the many individuals and other businesses who donate funds periodically, Sponsor donations do make up the bulk of the ASF’s annual income. These donations pay for things like bandwidth for our websites and software downloads, servers and racks for our million+ Subversion revisions, hundreds of mailing lists, and other services, and our crack Infrastructure contractors who keep it all running, plus a whole host of other organizational services we provide for all Apache projects.

Some news was made about several new sponsors recently: Huwawei, GoDaddy, Twitter, and Citrix at various levels of sponsorship, and we appreciate all of them.

I wanted to recognize one great quote about one of these sponsorships. Ars Technica covered one of these sponsorship announcements, and there was a very interesting comment thread following the article.

One commenter snarked, in effect, “Twitter isn’t profitable, so you’re really giving other people’s money to Apache”. What struck my eye is the very honest and right-on-the-mark reply from caniszczyk of Twitter, who replied:

As the guy who created and runs Twitter’s Open Source Office, this is spot on. We already contribute heavily to the ASF by actually committing and contributing to the projects we use: Mesos, Cassandra, Lucene, Thrift, Hadoop, Pig, Mahout and more. There are solid engineering resources committed to some of these projects, which some of them its their full time jobs.

As we evolve as a company, we think it’s the right thing to do to give back to the foundation that supports projects which we benefit so much from. As an open source guy for many many years and on a personal note, I don’t think enough companies give back.

In the end, as we grow, we hope we can give back more in the near future.

I won’t pretend that most businesses don’t take giving money away seriously, and will look for their ROI or whatever in deciding (or not) to donate funds to open source projects. But it was just one quick and very well put quote that shows the larger value of the great software the many Apache projects put out that benefit thousands of businesses every day.

Thanks, sponsors!