You probably use contribute to several Apache projects. But do you know what goes on behind the scenes at the ASF? Besides all the work of the 200+ project communities, the ASF has an annual budget of about one $million USD to fund the services our projects use. How we manage providing these services – and governing the corporation behind the projects – continues to change and improve.
How much do you know about the Apache Software Foundation (ASF) and the many Apache projects we host? Did you know we’re holding our annual Members meeting to elect our board of directors and new Members in just a few days?
I’m often surprised by the variety of basic questions and misunderstandings I hear in the software world about how the ASF really works. We’ve written plenty of documentation about the Apache Way and our governance, but let’s try a different approach. I’d like to interview myself to try to explain some things.
So, Shane, what *is* Apache? I thought it was that web server?
The ASF is a non-profit, public charity, 501(c)3 membership corporation with the mission of producing software for the public good. The Apache HTTP Server project (to use it’s formal name) is a project community at the ASF that creates the httpd web server, which has powered more active websites than any other server since 2000.
The ASF is the corporation that provides legal, branding, press, fundraising, and infrastructure support, and proven community mentoring to the many Apache projects like the HTTP Server. Think of the ASF as a great big house, where we provide shelter for a lot of different families that write open source software.
Well how many Apache projects are there?
We have over 165 different projects, and about 40 podlings. These 200+ project communities create a wide variety of software products, including Apache Hadoop, Apache Lucene, Apache OpenOffice, Apache CloudStack, and many, many more.
You are almost certainly using multiple Apache products right now as you read this. You may not realize it, but much of the plumbing of the internet uses Apache software to keep servers organized and connected. Most browsers use various Apache products under the hood for a wide variety of utility functions. It is our project communities that actually create the software you’re using — the ASF just helps keep them organized.
How does the ASF organize all these projects?
The ASF provides all the infrastructure an open source project needs: websites, code repositories, mailing lists, bugtracking services, a crack infrastructure team. We also provide all the rest of the services that a project will want, like legal support, access to press releases or analyst contacts, and some fundraising support. The ASF also owns all Apache trademarks on behalf of our projects, to ensure they get the credit they deserve.
Most importantly, the Apache Membership and many of our 4,000+ Apache committers provide the community mentoring and support to keep our projects running smoothly, with an independent project governance. We have many passionate Members with amazing experience in making open souce projects work, and they volunteer to help keep our projects healthy and running strong.
But this is mentoring and guidance, not direction. The ASF does not direct the technical direction of our projects. We let the people doing the work — the project committers and Project Management Committees (PMCs) decide where the code should go.
So the projects direct themselves. But what is “independent project governance“? How do you enforce it?
A critical behavior for any Apache project is independent governance. That means that every project manages their code for the benefit of all users (the public good), and not just for some company or vendor. In particular, the ASF and Apache projects only recognize individuals as committers or Members — never companies.
We expect when committers are working within their Apache project, they are acting for the best interests of the project itself. But we also have checks and balances: all Apache projects report formally to the board of directors quarterly. The board reviews project health — are they acting indepenently, are they publishing software releases, are they voting in new committers. If the board sees behavior that does not show mature Apache project behavior, the board will work within that project community to help the project community correct itself. Many Apache Members also volunteer to mentor our projects in these cases. In extreme cases (very rare), where a project does not follow the Apache Way, the board will unilaterally make changes to correct their course.
Can you clarify who are the board, the Members, and how they relate to projects? Are Members part of all projects?
Imagine Apache as a condominium association with multiple condos together. The ASF as a corporation provides the building. Like some condo associations, we also define a few expected behaviors and appearances for all the condos we offer. We also offer bonus services, like help moving into your condo or fixing things up. Each Apache project lives in one of these condos. We’re happy for you, the project community, to live your own lifestyle within your condo and paint the inside whatever color you want, as long as your public behaviors when you’re here follow our community best practices.
Here, the Apache board is the board of the condo association. They set the core rules and guidelines for the building. The Membership are sort of the owners of the building — not that they can ever sell their shares or make a profit, but they are the only ones who can nominate and elect the board. The board appoints all the officers who set detailed policies and make all the operations of the building work, like trash pickup and elevator maintenance.
Every Apache project condo has at least one Apache Member involved with the community: the Incubator requires that every new project has a few Members interested in that community to help mentor it. But within each project condo, the code direction or decor choice is completely up to the whole project community to decide. Membership in Apache is not transitive to any project: Members need to be elected to your project to have a direct say in it.
The ASF offers a lot of services to projects. How does this all get paid for?
The ASF board approves an annual corporate budget of about one million dollars. Our primary income is from our formal Sponsorship program, where organizations can provide a regular annual donation. As a 501(c)3 charity, we also have many individual donors, and some authors donate royalties from their books about Apache software to the ASF.
Importantly, sponsorship of the ASF does not provide any influence over Apache operations nor the operations of any projects. The ASF’s mission is to serve the public good, and while we very much appreciate our generous sponsors, we do not serve them: we serve our project communities and software users.
Sponsors provide a variety of reasons why they sponsor the ASF, many of which relate to how we host so many different critical software product communities. As one sponsor said, “Apache builds the plumbing of the internet”. Some sponsors and donors simply want to give back to the ASF in appreciation for all the software we provide for free.
So the Sponsors can pay for Apache project development, interesting.
No! Sponsorship funds are purely undirected — we do not accept donations with ties or requirements. By policy, the ASF does not pay for core development on any Apache project. All our budget is used for the support services that allow our project communities to do their work — which is building Apache software.
Do any Apache committers get paid for their work?
Of course — but not by the ASF. Many committers are working on Apache projects on behalf of their employers, who may be software vendors providing support, hosting, or add on products for that Apache project. Some committers are independent consultants, trainers, authors, or the like, who make their own living from helping other people use Apache projects. And a lot of committer work is done simply because that person needs to fix a bug or add a feature that they need for themselves. With so many people using Apache software to run their businesses, most work is self-serving: building code that they need.
The ASF provides a vendor-neutral place where everyone who benefits from Apache software can collaborate to improve that software. The ASF does not have an agenda or direction — we rely on the people using our software to help improve it.
Well… how much do *you* get paid at Apache?
No, seriously — how much do you get paid for this?
Seriously: nothing. Zero. Zip. The ASF has never paid me for my work here, and my current dayjob is wholly unrelated to my Apache activities. I’m here purely as an unpaid volunteer.
How did you get to be at Apache? What drives you to do all this unpaid work?
I first started committing code to the newly formed Apache Xalan project shortly after the ASF was incorporated in November 1999. At that time, I was paid by Lotus/IBM, my employer, to contribute to the Xalan as part of my job. I also got to attend and speak at a few ApacheCons to try to promote our work on Xalan and Xerces.
Over time, my dayjob changed direction, and turned away from Apache, but I was still interested in how the ASF worked. At ApacheCon I had followed a friend into a conference planning meeting to see how it worked, and I walked out of the meeting with assigned tasks for the next ApacheCon. Once I started helping with events, I was hooked. In 2002 I was elected as a Member of the ASF, and got to see how the sausage was made from the inside. In 2004 my job changed to be wholly unrelated to any open source work, but I was already personally invested in the ASF and our many excellent communities.
I volunteer at Apache for several reasons:
- This is how I give back to the world. I’m lucky enough to have a healthy family, nice home, and stable job. I volunteer my extra time to help make it easier for Apache project communities to build more free software for the public good.
- I love the ASF and it’s people. I’ve met so many amazing people at ApacheCon and within our projects, and it feels like much of the Apache Membership is one big family. Sure, we fight plenty, but we also buy each other plenty of free beers and meals.
Helping open source communities get organized and keep their volunteers motivated is something I’m good at, and something I’d love to do more of if I could. Volunteering at Apache is a huge impact to my personal brand and my future job prospects.
Wow, that’s been a great interview Shane! How should we wrap this up?
It’s been a pleasure. Thinking about what motivates me, this is one of the things I love doing: explaining technology and communities to interested audiences. This is also great timing for this interview, because the ASF is having it’s annual corporate meeting where the Membership elects a new board.
We have a truly stellar list of director nominees this year: looking at the candidates the Membership has nominated shows just how talented and friendly all our candidates are, and how any of them would be a help to ensuring the smooth operations of the ASF in the year ahead.
Since I do have the microphone, I will make one short plug to note that I’m also running for a seat on the board. I’ll be posting my director nomination statement — a note detailing why I hope Members will vote for me — soon here on my open source blog, Community Over Code.
Good luck with the election Shane — it sounds like Apache is in good hands no matter who gets on the board.
Yup. While we still have a way to go to make it simple to understand Apache for newcomers, the ASF and our project communities are doing amazing work, and often having a lot of fun doing it. Apache plans to be around for the next 50 years providing a stable home for like-minded project communities of all sorts.
Thanks for the interview — this was great to talk to someone about this!
Shane volunteers as the Vice President of Brand Management for the ASF, although all content here is his own personal opinion. He is not normally an interviewer, but does sometimes play a trademark lawyer on the internet. He hopes you liked this article, which also appears on Medium, and will ask him any questions about Apache that you might have. He promises to stop writing in the third person now.
When push comes to shove and full consensus on governance matters at the ASF or at Apache projects isn’t easily found, it’s important to consider what our underlying objectives are. The mission of the ASF is to produce software for the public good. That’s a good start, but like many concise mission statements, it doesn’t tell the whole story.
There are several aspects of how we expect Apache projects to work that we believe are critical to our mission’s success and longevity. These include things like The Apache Way of: volunteer and collaborative led community built software projects; using the permissive Apache license; and having a consistent and stable brand, infrastructure services, and home for all Apache projects.
We – as in the bulk of the Apache membership and especially the directors that we elect to the board (and thus the corporate officers the board appoints to set policy) – believe that these few core organizational and community management principles are so important for our long term mission that we require these behaviors of all Apache projects. Part of having a consistent brand is showing some level of consistent behavior with the products you produce. Having a shared set of infrastructure services likewise simplifies the process for users to find and use our products. Using the permissive Apache license means that we give maximum freedom to the users of our software, meaning that more users are likely to choose it.
But beyond these core principles, ones we have carefully crafted through experience working on Apache projects and have attempted to minimize when prudent, how do we decide where the ASF should go in the future, or what other, more detailed principles might be important to us? While the officers and directors set the official policies of the ASF as an organization, we look to the constructive input of all Apache members when discussing these ideas.
One recurring meme when discussing “What else should the ASF be/do” has been: “Projects first, or Foundation first?” I.e. is the ASF here to serve the needs of our Apache projects, or have Apache projects chosen to come here to follow the direction of the ASF? Should the ASF decide what kinds of projects we want to join, and decide The One True Technology; or should we simply see what projects show up, and try to support them?
If you picture Eclipse, you can see an example of “Foundation first”: the Eclipse board and leadership – mostly through it’s paying corporate sponsors and de facto corporate-led projects – has an overall vision and drive that explicitly provides direction for it’s projects. Eclipse has full-time staff devoted to “…1) IT Infrastructure, 2) IP Management, 3) Development Process, and 4) Ecosystem Development.” Similarly, the Eclipse board is partially made of appointed members from their Strategic Developer and Strategic Consumer corporations. That leads to strong consistency and direction, primarily for the benefit of the corporations directly or indirectly funding Eclipse projects.
“Projects first” means that the ASF exists to serve the needs of any projects who choose to join us, and who willingly agree to follow our basic rules, like using our license, consensus-driven decisions, and running projects independently of corporations. Our mission is not on behalf of the many corporations who pay their employees to donate code to Apache projects. Our mission is to provide software for the public good – that is, the end users of our software, and the larger world around us.
The ASF doesn’t have an agenda beyond that mission. We don’t have a strategic plan for acquiring projects in specific technologies or from specific corporations. We rely on providing a stable and friendly home for like-minded individuals and communities to build software… for the public good. We certainly hope – and time has shown – that this model will attract vibrant communities that will build useful software. There are plenty of Apache Members and Committers who are passionate about sharing their experiences in software development with others, which is great. But as an organization, we don’t have a strategic plan for seeking out new projects: we believe the best process is to let like-minded projects seek us out.
To ensure that Apache projects focus on the public good, we also require our projects to be run independently of undue commercial influence. Apache projects don’t exist to serve as a vehicle for the corporations who may have donated technology or employee developer hours to them – they exist for the broader public good.
This independence is fundamental to the long-term health of our projects, and to the ASF. It both makes the ASF a place where potentially competing corporations can feel comfortable collaborating on technology (which gets us lots of software for our mission) and also seems to attract plenty of individuals who are interested in participating in our projects – even when changing employers. This independence means that an independent Apache project can continue to exist for the benefit of it’s users even if one of the corporations donating technology to it decides to change direction.
But beyond the core requirements here – ones designed to keep project governance in our independent communities – the ASF strives to not add any additional requirements, but rather to provide the best home to like-minded Apache projects that we can. So in that way, we try to remain focused going forward on putting “Projects first”.
Actually, I should amend that. We do have the core project requirements; ones associated with being an Apache project. So I should really say that we put “Apache projects first”. But beyond that Apacheness that the ASF has developed over 13 years and 190 project communities, we strive to not have an agenda – other than to support those Apache project communities who choose to join us as best we can.
People still reading may also be interested in learning how governance at Apache works.
The old saying goes that one of the last things done in many open source projects is the documentation. Taking this to the organizational level at the ASF – the corporate level – that’s both all too true, and not at all true.
Questions regularly come up across various ASF lists about all sorts of corporate operations, and there are a fair number of times when it takes a while to get the right answer to the questioner. Luckily, we have a lot of very experienced Members and Officers who are there to ensure the right thing does happen – but I believe that the answers come be a lot quicker if we had better documentation for these corporate operations. Namely, all the stuff that’s not dealing with code.
This is not to say we haven’t documented how the ASF as a corporation works: a lot of effort has gone into our standard operating procedures, and we’ve both covered the basic legalistic bases, as well as put serious thought into minimizing the absolute rules at Apache while providing lots of best practices. These procedures – and the discussions of pros and cons and whys explaining how they came to be that way – are all available… on the mailing lists, somewhere. 😉
The foundation of our corporate policies is The Apache Way. Understanding that, and understanding that the fundamental mission of the ASF is to produce software code for the public good, leads you to many of the policies we practice, both at the ASF and in Apache projects. To start with, here’s a brief laundry list of how to learn about Apache Corporate Policies.
- Apache Corporate Governance and org chart describe organizationally how the ASF as a corporation is run. This also makes clear the distinction between organizational governance – the board, membership, and officers as a whole; and technical governance – the fact that PMCs independently manage their projects.
- The Foundation page provides the overall corporate mission, and links to the Board and Membership pages with the nitty gritty of corporate records.
- The apache.org/dev pages are both a generic project runbook, technical issues FAQ, and partial corporate operations listing. While these are written by committers, for committers and developers, they have a lot of great information on how all sorts of things at the ASF and all Apache projects should work.
- Another key overview of organizational documentation is the Community Development project site. This aims to be a more human-friendly signpost to find the relevant technical widget or policy information elsewhere.
- The Apache Incubator is another important place where proper procedures are documented: this is where we try to explain to prospective new communities how an Apache project should be run before the ASF formally accepts them.
Here are some other principles and rules we follow that are important to understand.
- Transparency. All technical decisions on Apache projects are reflected on publicly archived mailing lists. The phrase “If it didn’t happen on the list, it didn’t happen” is not just an old saying, it’s the rule at Apache.
- Respect. One of the primary exceptions to transparency are discussions about individuals. Votes on new committers and PMC members are typically held in private lists amongst a whole PMC, since this both allows better honesty of expression, as well as prevents embarrassment in rare cases where a nominee is not voted in (yet).
- Organizational Oversight. While the ASF relies on volunteers to produce the code that we give away, we keep it organized by having oversight: the PMCs ensure that projects are managed appropriately; the board ensures that PMCs manage themselves, and corporate officers ensure that the ASF’s corporate bits are covered.
- Member Oversight. Members of the ASF have the final level of oversight of the board and the corporate officers. All members are allowed to review any private mailing list within the ASF, including any private@ list for projects. Members do not receive special status on any lists; merely the ability to oversee all operations. The only exceptions are specifically restricted mailing aliases that are used for legally protected communications or infrastructure root level issues.
Note that while Members have a broad oversight ability within all aspects of the ASF’s operations, organizational matters like this kind of oversight are not transparent outside of the ASF. While we certainly strive to make public as many of the great lessons we’ve learned about governing community-led software projects, the operations of governance are kept primarily within the Membership.
Partly, this is due to all the lawyers. The details of running a million-dollar corporation require a fair amount of legal work. We strive to publish as much information as is practical in things like board minutes; however the actual corporate processes are done privately.
Partly, this is because part of a meritocracy is allowing those who do to do the work. Individuals who are trusted and useful to Apache projects are elected as committers or PMC members, and therefore can help directly govern those projects. Similarly, individuals who show wisdom and effort towards the goals of the ASF may be elected as Members, and therefore can help elect the board who will appoint the officers to ensure the ASF continues running smoothly.
A great place to ask questions about these kinds of issues is the Community Development mailing list.
Apache is actually governed in two ways: by the meritocracy of the Apache Way, and the hierarchy of the Apache Software Foundation. We strive to govern ourselves as much by meritocracy (and consensus) as possible, and as little by hierarchy as is practical for non-profit corporation. This has a number of both practical effects, as well as a number of social corollaries.
The ASF prides itself in running as a community-based meritocracy of peers. That is, within each project community, people participate as individuals and gain merit as measured by that community. The community recognizes your merit by voting people in as committers or PMC members. A new committer can vote on certain kinds of code changes, and can checkin their own code. PMC members vote on new committers and product releases.
In Apache projects the underlying cliché is correct: those who are doing the work get to make the decisions. A healthy community will seek consensus over the direction of the project, and typically only uses [VOTE]s to solve thorny technical issues that don’t come to a clear consensus. Similarly, PMC members each have one vote on elections for new committers or PMC members; decisions within projects are made by the community as a whole. Technical governance is always done by meritocracy.
Even within the Foundation – i.e. the Delaware registered non-profit corporation that makes the ASF a legal entity – we strive to govern by meritocracy as well. Nominating new members and electing the board of directors are the obvious rights of members, and these are discussed and judged based on individuals’ merit towards the various Apache projects and to the ASF as a whole.
Similarly while the board appoints individuals to serve as corporate officers with specific delegated powers, all Apache members are allowed to subscribe to any mailing lists and participate across various ASF activities. Various members step up to work on policy, corporate, legal, treasury, and many other tasks that are needed in any corporation, always working with the relevant officers. Again, those who do the work decide how it gets done.
In these organizational tasks, it’s not always as clear how to reach consensus on contentious issues. Social issues or corporate governance questions don’t often have as simple a metric to measure against as technical issues do. Code either works or it doesn’t; corporate policy often has a very different kind of judgement. Similarly, with an active membership (with many very passionate individuals!), finding consensus on non-technical questions amongst 100+ individuals is often difficult. Here, hierarchy steps in to govern.
In most cases this works smoothly. Either the relevant officer will make a policy decision and publish it, or the board will pass a resolution or clearly ask for a change. While the whole community – the membership, the various PMCs of Apache projects – may not have a consensus on the decision, it is still policy, and projects will generally work to make the change.
In some cases, this doesn’t. A key way that the board ensures there is sufficient oversight of all Apache projects is by requiring reporting: monthly for officers; quarterly for PMCs / projects. There have been cases where PMCs have not reported regularly, or have not provided sufficient details to the board’s satisfaction in their reports. In these cases, the board will require a new or updated report. This is not a question of merit, or getting a consensus: this is an order. Non-response to a board request or not following an officer’s policy will and does result in the board stepping in to force changes within the project. As they say in politics, officers serve at the pleasure of the board.
It seems that sometimes this switch from meritocracy – where everyone in the PMC has an equal say, and they reach consensus – to hierarchy – where the board says “jump” – sometimes catches people by surprise. Sometimes, it seems that some PMC members just never really thought about the fact that the ASF is a legal corporation, and must follow certain rules. Other times, this surprise is clearly exacerbated by poor communication in specific cases between the board and projects. While cases like this really aren’t a choice – the board can and will dissolve or reconstitute projects (always as a last resort!) – it is important to clearly explain the issues and why this kind of hierarchy is needed in some cases.
Has anyone else had this feeling? Of switching between meritocracy, and consensus among the community, to hierarchy, and the board simply making a decision? How can we improve the handling of future issues like this, to ensure we keep the maximum meritocratic freedom within our projects, while still ensuring the legal shield of the ASF is secure and can function for the long term?
I’ve been thinking about this lately since I recently drew up a new org chart here for the ASF as a whole. Note that the hierarchy part only comes in on organizational, legal, and similar matters – technical decisions are always in the meritocratic hands of the PMCs and projects. More reading can be found on some of these excellent slides about Apache and open source.
In the earlier essay I wrote on Apache project independence (since promoted to an article on the Apache Community Development website) I missed a critical bit of independence that I think we should add to the ComDev site.
Apache projects are technically independent
While Apache PMCs provide a report of their operations to the Board on a quarterly basis, they are technically independent of the Board. PMCs are free to collaboratively decide their project’s technical direction however they choose. While the Board holds PMC chairs and PMCs accountable for ensuring that their methods are appropriate – following the Apache Way and the few other rules at Apache – the Board does not exercise any technical direction over Apache projects.
This is a critical point to understand. There is no top-down direction at the ASF in terms of what technology we implement or what kinds of projects we seek out or support. All technical choices are made by the individual committers and PMC members within their project communities – i.e. the individuals actually doing the coding decide where the coding will go.
This applies equally to the Apache Incubator, as the gateway of new project proposals at Apache. The ASF has noagenda of what projects it wants to find, nor do we have a corporate strategy for new project acquisition. Project communities that want to come to the ASF are welcome – and will be reviewed in the Incubation process for community health and diversity, and for IP clearance and licensing. But not for technical merit.
The essence is: if there is a sufficiently diverse community who chooses to live by the Apache Way, the ASF is happy to host them. The fact that there is sufficient diverse community means that people are interested in the technology. Similarly, the Apache Attic awaits projects that may seem interesting technically, but fail to have a sufficiently diverse and active community to maintain them.
So – worthy to add to the ComDev page?
This is the third in a series of essays describing Apache governance. The ASF is a non-profit, membership-based corporation; as such the Members of the ASF have a role similar to shareholders in a publicly traded corporation.
In this essay, the term “member” specifically means someone who is a Member of the Foundation itself, as elected per the corporate bylaws.
The ASF is a membership corporation, so Members serve a similar role as shareholders do in publicly traded corporations. Members may propose new candidate members, may cast one vote in new candidate elections, and may cast one vote in board elections.
Members are not empowered to speak officially on behalf of the ASF as a whole, nor do they have any special rights to influence technical direction of any Apache projects that they are not otherwise elected as a committer or PMC member on. While members often volunteer and serve in many capacities both at the Foundation level and within various Apache projects, their only specific organizational rights are to vote in board and member elections.
The Apache Incubator is special: any ASF member may request to be added to the Incubator PMC without a vote. Within the Incubator PMC, members may serve as official mentors or champions to incoming podlings, as well as voting on Incubator policy and releases of podlings undergoing incubation. In this way, members work to mentor podling communities and guide them to the Apache Way and eventual graduation as a top level Apache project.
Members act as shareholders of the corporation. As such, each member has a single vote on electing directors to the board; similarly members may vote on new nominees to membership. Members are eligible to nominate new candidate members and to nominate individuals to the board.
Organizationally, members do not have specific standing within any Apache projects or Incubator podlings. However most members are active as individuals within multiple Apache projects on their own technical or social merits, and once elected as a member often find more ways to get involved in more Apache projects.
While the board and relevant officers are directly responsible for providing oversight to the many Apache projects, members often work within many Apache projects to help ensure projects run smoothly and follow the Apache Way.
The central place for member-focused announcements and discussions is the privately archived members@ mailing list. This is used for a wide variety of purposes: both proposing and discussing new technical or policy ideas within the ASF, announcing marriages or births within member’s families or other major social events, and any formal announcements to the membership from the board or corporate officers.
By policy, members have the right to inspect and review the archives of all mailing lists at Apache. This policy is designed to ensure that every member can independently inspect all corporate operations, and the operations of all Apache projects. Thus members may review all private mailing lists (for example) about internal legal affairs, fundraising, security issues, as well as any private@ lists used by our Apache projects’ PMCs.
While members may send messages to any private lists, they do not automatically receive any special merit in terms of influencing the technical direction of Apache projects. Merit in Apache projects is gained within each individual project community, and membership does not convey any other special privileges.
The board typically holds an Annual Members Meeting, where a new board of directors is elected, and new member candidates may be voted upon. Meetings are currently held on a private IRC channel, as an alternative to a traditional conference call, where any reports are read and members may ask questions. The meeting has a 48 hour recess for voting, and then reconvenes to announce results and complete the meeting: this allows members who may not be able to personally attend the first portion of the meeting to attend the second half, as well as to conduct voting over email using our own secure voter tool. Members who are unable to attend any of the 2 day meeting period may provide for another member to proxy their attendance and votes.
The board often holds interim Special Members Meetings in between Annual Meetings, primarily to give members a chance to nominate new candidate members and be voted in. Members also use Special and Annual meetings to raise additional questions or issues about Foundation operations – although members usually just raise questions to the board or to any ASF officer at any time over our usual mailing lists. Members and invited guests are welcome at all Members meetings.
Given the distributed and volunteer nature of the ASF, official in-person meetings of the Membership are no longer held. All Foundation or project business is conducted on normal mailing lists – although some of the lists are private. Members do often meet in small groups in person – although this is for social reasons, and often involves a meal or drinks. Many members and committers also traditionally attend the ASF’s annual ApacheCon conference.
All members have the ability to nominate new individuals as candidates for membership. The amount and types of merit that existing members look for in Apache committers varies, but always includes some significant technical or other contribution to one or more of our projects, as well as a clear interest and understanding of the Apache Way. In most cases, these contributions and traits are displayed over a significant period of time, usually over a year of engagement in one or more Apache projects.
Note that it is not required to be a committer before being considered for membershp, however as a practical matter, in the vast majority of cases any potentially worthy individual has already become a committer on some Apache project. At least two individuals have been elected as members without being committers first, in each case for non-coding contributions in mentoring on the Apache Way or other organizational work.
Many newly elected members are surprised (pleasantly!) to be told they’ve been elected. Typically, being nominated and elected as a new member is something that happens well after the nominee has a clear track record and a positive influence on Apache projects for some time; in hindsight, a frequent comment is “isn’t So-and-So a member already? They do such good work!”. Individuals asking to be made a member (or worse, insisting you should be elected!) is culturally frowned upon. In a perhaps counter-intuitive way, being considered for membership is something that requires real effort acting over a measureable time, but without making it obvious that you’re seeking recognition.
Candidate members are voted on as a simple majority vote (more yes’es than no’s) at Annual or Special member’s meetings. Voting is performed using custom written voting software with secret ballots.
Within the Membership, merit is equal; all members have an equal vote and ability to propose change within the ASF. Membership is a notably helpful factor in being considered for the board or in appointment to officer positions at the ASF, although the most important factor is demonstrated merit within the particular project or the ASF as a whole.
The ability of members to influence the ASF is simultaneously major and immaterial. On one hand, members vote on new candidate members, and more importantly vote on the Board of Directors, which clearly affects strategic policies of the ASF. On the other hand, membership grants individuals no other special merit within any of our projects, and is not an official position within the legal corporation itself.
The perceived importance of membership is likewise a dichotomy. Many members are quite modest about their membership, and a frequently heard comment is that “oh, they really deserve to be recognized more than I did”. For many, being elected a member was not a goal or title that individuals were pursuing; rather, they were naturally doing their own work (at an Apache project), and were then recognized for it.. Many members list their affiliation on their resume (or LinkedIn, etc.); some do not.
Recognition of membership varies widely outside of the ASF: some software companies don’t seem to care what the ASF is or what membership means; a few software companies have strongly encouraged their employees to become committers or members, and gaining one of those roles is a notable prestige bonus within those companies. Likewise, a number members or officers of the ASF are frequently sought out as speakers at open source conferences.
Members share both the community of all other members, as well as often serving as bridges across the various Apache projects they are already involved in. Many members also volunteer extra time to serve as mentors, both within the ASF working with our projects and communities, and outside of it, mentoring others or speaking at conferences.
The membership as a whole and individually does not provide technical direction for any Aapche projects directly; every PMC is free to manage it’s technical direction independently. PMCs are the governing body for their project, and are expected to manage the project’s technology in the best interest of the whole project community, independent of outside commercial influence.
This is similar to how the board does not set technical direction for projects: this is a key reflection of how the ASF is intentionally structured to provide maximum freedom to it’s projects. The board and the ASF membership are happy to provide a home to any software project communities that are willing to follow the Apache Way. The mission of the ASF is to provide software for the common good: we are happy to help like-minded communities to provide that software; are confident that communities will form around software that is useful; and understand that there are many different ways to effectively and collaboratively build software.
In the case of members, nearly all members do participate in a number of Apache projects – as individuals based on their merit within the specific projects. Simply being elected a member does not confer any additional abilities in terms of other projects. Many members chose to serve as ambassadors of the Apache Way, and usually get involved in more projects as time goes on.
This is the second in a series of essays describing Apache governance. Fundamentally, the most important organization at the ASF is a PMC, or Project Management Committee. These are the actual groups that decide what software our projects release, and as such do the bulk of the actual work of the ASF.
Apache projects are managed by Project Management Committees (PMCs), comprised of committers elected within a specific project community to provide oversight for the project for the ASF and to decide the release strategy for the project. PMC members are expected to act as individuals, making decisions for the best interest of the project when acting on PMC or development lists.
Each project’s PMC is independent. PMCs are free to set community and technical direction for their project, and are directly responsible for overseeing releases and the healthy development of their communities. PMCs are responsible for ensuring their project follows certain core requirements set by the board or other corporate officers of the ASF, like following Legal, Branding, and Infrastructure related requirements, along with ensuring their community operates within the basic outline of the Apache Way.
PMC members nominate new contributors to the project as committers, and PMC members cast votes on electing new committers to the project. PMC members also have binding votes on any project matters.
The board creates new PMCs to manage specific named projects by resolution at monthly meetings. New top level projects (TLPs) are created from podlings in the Apache Incubator after a successful graduation vote from Incubation. The board appoints a Vice President – an officer of the corporation – to serve as the chair of the PMC itself when creating the PMC.
PMC Chairs – who are VPs – serve as normal PMC members, with one vote on project matters just like other PMC members. The primary duty of a PMC chair is providing quarterly reports to the board about the health and status of their project. PMC chairs are expected to subscribe to the board@ mailing list, and be aware of any board concerns about the project, and are responsible for working with the PMC as a whole to address any board concerns.
When PMCs elect new potential PMC members or existing members resign, the chair of a project requests an “ACK” from the board. Once one director ACK’s email is sent recognizing the change in the PMC membership, and after a 72 hour wait, the PMC chair may update the official roster of the members of that PMC. The ACK process is designed to ensure that all PMC changes are explicitly acknowledged by the board, and the waiting period ensures that all directors have a chance to object or comment on any improper changes. Changes in PMC chairs are only made by board resolution at monthly meetings, since this involves appointment of an officer of the ASF.
As official committees of the ASF, PMCs provide the legal oversight of project operations and releases, ensuring that software releases are made on behalf of the ASF as an organization, and not as individuals. This helps to ensures that any legal liability for the products of Apache projects is born by the ASF and not by our committers individually. All PMC chairs provide quarterly reports to the board, which ensures that the board has oversight of all the projects of the Foundation.
PMC members should always be subscribed to their project’s private@ list, and certainly should be subscribed to at least the dev@ list to be aware of the operations of the project.
Virtually all PMC communication should happen on the dev@ list or any other appropriate public mailing list. The private@ list for each PMC should only be used for matters that require confidentiality, such as discussing personnel matters or voting in new committers or PMC members, or security-related issues that have not yet been publicly disclosed. Ensuring that PMC discussions about the future of the project are held on the public dev@ list ensures that all of the community – including non-committers – can follow the discussion and comment on issues. While only PMC members or committers have binding votes on project matters, healthy PMCs certainly work with the larger community of their project.
PMCs are free to use JIRA/Bugzilla, wikis, and other infrastructure-hosted tools that provide for public view and comment, along with the normal project mailing lists. PMCs should make specific requests to the Infrastructure team for services that the project desires.
Some PMCs sponsor regular or occasional in person meetings, teleconferences, or IRC chats to brainstorm project ideas, but are expected to bring all information back to public lists for further discussion and to make final decisions. Many PMCs either directly organize, or have contributors or organizations in their technical space organize Meetups, BarCamps, or other in-person events focused on end user education and technical information sharing.
Given the distributed and volunteer nature of our projects, in-person meetings of PMCs are quite rare. All project business is conducted on normal public mailing lists.
PMCs are free to set the bar for merit within their projects, as long as decision making is done in a collaborative fashion as in the Apache Way. Healthy PMCs will regularly review contributions from non-committers – both specific code patches, bugs reported or commented on, or just helpful interaction on their project lists – to evaluate contributors as potential committers. Ensuring that PMC members are helping to mentor helpful new contributors to their projects helps to ensure a healthy and growing project community.
PMCs vary significantly in the level of commitment and work expected to be considered for a committership. Some PMCs vote in new PMC members typically from their existing committers (i.e. the progression is contributor -> vote -> committer -> vote -> PMC), while other PMCs always elect new committers into the PMC simultaneously (contributor -> vote -> committer & PMC member). PMCs are solely responsible for managing votes and granting commit privileges for new committers; however new PMC members elected must always be ACK’d by the board before they are officially added to the PMC.
Merit within one PMC is not transferable to other PMCs; each project’s committer list is independently managed by it’s own PMC. PMC members or committers with a long history on one project will often be known by their past activities by other project’s PMC members, but they are expected to build merit on each separate project they wish to become a committer on.
While only PMC members have binding votes on project releases, in general PMC members participate equally with their project committers on the dev@ list. PMC members can help to ensure a healthy and diverse project community by ensuring the project’s mailing lists are welcoming to newcomers and that community standards are promoted on it’s mailing lists.
While there are many projects with overlapping communities – and therefore some shared personal or technical relationships – fundamentally each project is it’s own community in terms of the bulk of project work. While all projects are part of the overall ASF, and as such share certain basic requirements and bits of the Apache Way, all technical direction and most of the community focus is within each project individually.
PMCs are expected to report on the health and diversity of their project’s community to the board. While there are no strict requirements on the makeup of PMC members (in terms of employment or affiliation of the individuals on the PMC), the board does expect PMCs to operate independently of outside commercial influence. PMCs that are unable or unwilling to ensure their actions are on behalf of the whole community – or act in ways targeted to specific commercial interests – will be contacted by the board and required to make corrections.
The board does not provide technical direction for any of it’s projects or activities; every PMC is free to manage it’s technical direction independently. PMCs are the governing body for their project, and are expected to manage the project’s technology in the best interest of the whole project community, independent of outside commercial influence.
While this may be a surprise to some, it is a key reflection of how the ASF is intentionally structured to provide maximum freedom to it’s projects. The board and the ASF are happy to provide a home to any software project communities that are willing to follow the Apache Way. The mission of the ASF is to provide software for the common good: we are happy to help like-minded communities to provide that software; are confident that communities will form around software that is useful; and understand that there are many different ways to effectively and collaboratively build software.
One of the most important pieces of advice I’d personally give to PMC members is to feel a sense of leadership and responsibility for their projects as independent projects. Apache relies on it’s PMC members to ensure that their projects are managed in an open and fair manner, and that their projects act to serve the public interest by providing free software under the Apache License. In many ways, PMC membership is a duty to the project as a whole, and a call to set aside outside influence to act for the best interests of the project’s active community itself.
- Project Management Committee Guide – how-to of PMC duties
- PMC Branding Responsibilities – ensuring their projects follow Apache Branding Requirements
- Podling Project Management Committee Guide – expectations for Incbator podlings’ governance
- Brief overview of what a PMC is from the How It Works
- Apache Projects are Independent
This is the first in a series of essays describing Apache governance. While there are a number of existing documents that explain some of the “hows”, I’m hoping to fill in some of the key information about the “whys” of how the Apache Software Foundation (ASF) actually works day-to-day. Look for more articles on things like the Membership, PMCs, projects, and more coming up!
The board consists of 9 directors, each serving as an individual. After each annual board election, the board chooses a Chairman from it’s ranks. This is currently done a couple of months after the board election, to give the new board a chance to get to know one another. The board also appoints executive officers for the corporation at that time.
The board meets by conference call and IRC chat monthly. The agenda is published privately to the Membership. All corporate officers make a monthly written report to the board, and every Project Management Committee (PMC) makes a quarterly written report to the board. In this way, the PMCs report directly to the board, not to the President or other corporate officers. All ASF Members and invited guests are welcome at board meetings, except during very rare executive sessions of the board.
Currently, the average monthly agenda includes 10 corporate officer reports, 40 PMC reports, and often a handful of official resolutions. To ease the review process for reports, reports are due well before the meeting begins, and many directors “preapprove” reports by initialing them in the agenda file before the meeting. Each PMC report is assigned a director to serve as “shepherd” for the month; any questions raised before or during the meeting are taken by the shepherd back to the relevant officer or PMC for an answer or resolution. PMCs that report late or where serious questions are raised during the meeting may be asked to provide an additional report at the next monthly meeting.
The President is responsible for managing the day-to-day operations, and helps to coordinate the work of the VP, Infrastructure and the sysadmin team; however VP, Infrastructure also makes a monthly report directly to the board. In this way, the board exercises oversight over all official Foundation activities.
Article V. of the ASF bylaws specifies a Board comprised of 9 Directors, elected annually by the ASF Membership. The Apache Board is much like the board of directors of any other corporation, responsible for “…management of the corporate assets (funds, intellectual property, trademarks, and support equipment) and allocation of corporate resources to projects.”
The board is responsible for appointing any officers of the ASF, including executive officers – President, Secretary, Treasurer – and other officers, both those with corporate functions (Legal, Publicity), and Vice Presidents of individual Apache projects. The board also sets overall policy for Apache projects, or in many cases, delegates the details of policy setting and implementation to specific corporate officers. Official resolutions are used to appoint or change officer positions, create new Top Level Projects (TLPs) or retire unused projects, and in certain cases to make fundamental policy decisions. Most policy is simply agreed upon by the board and any relevant officer(s), and published on our website.
With the exception of monthly conference calls, board communication is done over private mailing lists. The email@example.com mailing list is the primary place where official business is done. All directors, corporate officers and PMC chairs are expected to subscribe to this mailing list and track issues relevant to their areas of responsibility. Many ASF Members also participate, comment, and help resolve issues where appropriate.
Directors will regularly read and participate on various project mailing lists when a board-level issue or question has been raised. This participation is done wearing their board hat: i.e. as a strategic voice, addressing broad ASF policies or important community health issues. Directors participate technically in Apache projects as individuals: directors are not automatically granted committerships, PMC memberships, or even binding votes on any project matters. Directors wishing to influence the technical work of a project must gain merit and be voted in as a committer to that specific project like any other contributor.
The board exercises oversight of Apache Projects by reviewing project reports at the monthly board meeting, and by explicitly approving all changes to PMC membership. When PMCs have voted in and nominated a new committer to serve on the PMC, they ask the board for an ACK, or acknowledgment, of the addition. Once any one director sends an ACK email, the official PMC membership roster is updated after a 72 hour wait, designed to give other directors a chance to comment or question the appointment before it is officially recorded.
PMCs are expected to provide accurate and thorough reports to the board of their community health and technical and software release activities. Projects that do not report or have serious issues in their report will be contacted by that month’s shepherd or another director to find a resolution. In extreme cases, where a PMC is dysfunctional or is not following required Apache policies, the board may unilaterally make changes to the PMC or may dissolve a project to correct the issue.
Board elections are held annually; currently, each board seat is for one year, so all 9 directorships are voted on annually. ASF Members are eligible to nominate people for board seats as well as to vote in the board election. Director candidates run individually, and Member votes are currently cast by Single Transferable Vote (STV). All nominations, position statements, and voting are conducted electronically either over private mailing lists or using our own software for casting votes. Individual votes are kept anonymous, although the full STV calculations are published privately amongst the Membership.
In the past, only existing Members with some history within the ASF have been nominated. While the bylaws allow for any person to be nominated for the board, the amount of trust and expected experience with the Apache Way for directors is such that nominees outside of the existing Membership are unlikely for the time being.
For the first several years after incorporation the board was quite stable, with a majority continuing to serve repeatedly. As the ASF grew in membership around 2004, the board began to change somewhat, with a few new faces appearing each year. Jim Jagielski, one of the founders of the ASF, has remained on the board since it’s beginning.
Directors are expected to serve as individuals on the board, representing the Membership and acting in the best interests of the ASF as a whole. Directors are expected to disclose any conflicts of interest they may have due to their current employment, and recuse themselves when necessary from votes or potentially discussions on subjects they have a conflict with.
While Directors participate in a variety of communities both within and outside of the ASF, the primary community home for directors is the Membership. Most directors follow the main private Members mailing list, where any ASF Member start discussions about a wide variety of topics, spanning from new technologies, policy ideas, questions about project communities, or even personal announcements of weddings and births.
From time to time Directors participate in the communities of various Apache projects when a board-level issue or question is raised; however they do so wearing their individual Director hat. Many Directors are separately committers or PMC members on a variety of Apache projects, where they do technical and community work just like any other committer, wearing a committer hat.
Many Directors also serve in the public community; serving as spokespeople for the Apache Way or granting interviews about the ASF or our projects, or reviewing other open source or charitable foundation works.
The board does not provide technical direction for any of it’s projects or activities. The board does set broad policies – for example, requiring that all source control systems are run on ASF hardware, and controlled by ASF Infrastructure. Technical direction – either in terms of how to implement infrastructure or what code projects should work on is all delegated to the PMCs or relevant officers.
While this may be a surprise to some, it is a key reflection of how the ASF is intentionally structured to provide maximum freedom to it’s projects. The board and the ASF are happy to provide a home to any software project communities that are willing to follow the Apache Way. The mission of the ASF is to provide software for the common good: we are happy to help like-minded communities to provide that software; are confident that communities will form around software that is useful; and understand that there are many different ways to effectively and collaboratively build software.
While not all aspects of the Apache Way are practiced the same way by all projects at the ASF, there are a number of rules that Apache projects are required to follow – things like complying with PMC release voting, legal policy, brand policy, using mailing lists, etc., which are documented in various places. There are a few rules that I think may not be documented as succinctly as they should be.
A primary purpose of the basic requirements the ASF places on it’s projects are to help ensure long-lived and stable projects by having a broad enough community to maintain the project even in the potential absence of any individual volunteer or any sea change at a major vendor in that area. The Apache project governance model is explicitly based on a diverse community. This is different from other governance models, like the “benevolent dictator” idea or the often corporate-backed model that Eclipse uses.
Apache projects are independent
This is implicit in the fact that the Project Management Committee (PMC) runs the project, and the fact that PMC members are expected to contribute to the project as individuals, wearing their “PMC hat”. The concept of hats means that when a PMC member votes on project matters, they are casting their vote as an individual acting in the best interests of that PMC, and not as an employee or representative of some third party. There are also certain expectations of diversity within a PMC; the board may apply extra scrutiny to PMCs with low diversity (i.e. PMCs that are dominated by people with a common employer). Similarly, the ASF does not allow corporations to participate directly in project management, only individuals.
There are two important aspects to this independence: project management, and project use by end users.
Apache projects are managed independently
Apache projects should be managed independently, and PMCs must ensure that they are acting in the best interests of the project as a whole. Note that it is similarly important that the PMC clearly show this independence within their project community. The perception of existing and new participants within the community that the PMC is run independently and without favoring any specific third parties over others is important, to allow new contributors to feel comfortable both joining the community and contributing their work. A community that obviously favors one specific vendor in some exclusive way will often discourage new contributors from competing vendors, which is an issue for the long term health of the project.
Apache products may be used independently
All Apache projects must release their code under the Apache License, which clearly specifies the minimum restrictions that users of Apache software must agree to. Apache software is all about being able to use it for virtually whatever our users want: open source, proprietary, secret: we’re happy to have users take our software (although not our name) for virtually any purpose. While our legal guidelines allow certain other software licenses to be used for specific dependencies, the software we release always uses our license.
Extending this idea, users of Apache software should be able to find our software, learn how to use it, and actually apply it to all its common use cases solely by going to the Apache project’s own website. Apache projects should provide sufficient documentation, install features, basic user help (through mailing lists) and services for the common use cases to the user, without them having to rely on third parties. It is important that our users can both make use of our software freely – both in terms of not having to pay for the software, as well as not having to worry about IP claims or other more restrictive licenses on either the software or the configurations or other common materials required to actually use the software.
Apache projects are non-commercial
The ASF’s mission is to produce software for the public good. All Apache software is always available for free, and solely under the Apache License. While our projects manage the technical implementation of their individual software products independently, Apache software is released from the ASF, and is always meant to serve the public good.
We’re happy to have third parties, including for-profit corporations, take our software and use it for their own purposes – even when in some cases it may technically compete with Apache software. However it is important in these cases to ensure that the brand and reputation of the Apache project is not misused by third parties for their own purposes. It is important for the longevity and community health of our projects that they get the appropriate credit for producing our freely available software.
Reminder: “The postings on this site are my own and don’t necessarily represent positions, strategies or opinions of either my employer nor the ASF.”