Posts

MesosCon

Sign up for free live video streaming of all keynote sessions at MesosCon Europe.

Can’t make it to MesosCon Europe in Prague this week? The Linux Foundation is pleased to offer free live video streaming of all keynote sessions on Thursday, Oct 26 and Friday, Oct 27, 2017.

MesosCon is an annual conference organized by the Apache Mesos community, bringing together users and developers to share and learn about the project and its growing ecosystem. Users, developers, experts, and community members will convene next week.

Apache Software Foundation, Mesosphere, and Netflix are among the many organizations that will keynote next week.

The livestream will begin on Thursday, Oct. 26 at 9 a.m. CEST (Central European Summer Time). Sign up now! You can also follow our live event updates on Twitter with #MesosCon.

All keynotes will be broadcasted live, including a welcome and opening remarks by Ben Hindman, Co-Creator, Apache Mesos and Founder, Mesosphere.

Other featured keynotes include:

  • Rich Bowen, VP Conferences, Apache Software Foundation will analyze The Apache Way.
  • Katharina Probst, Netflix will talk about making and keeping Netflix highly available.
  • SMACK in the enterprise panel.
  • Pierre Cheynier, Operations Engineer, Criteo will discuss operating 600+ Mesos servers on 7 data centers.
  • And more.

View the full schedule of keynotes.

Sign up now for the free live video stream.

Once you sign up, you’ll be able to view the livestream on the same page. If you sign up prior to the livestream day/time, simply return to this page and you’ll be able to view.

 

Concepts such as decentralizing strategy, delegating direction, and fierce transparency in communication are part of the backbone of successful open source projects. In my presentation at Open Source Summit EU in Prague, I will explore how these concepts are not only applicable to volunteer-run organizations but can also help growing corporations avoid some of the coordination overhead that often comes with growing teams and organizations.

We’ll look at some of the key aspects of how project members collaborate at The Apache Software Foundation (ASF). After that, we’ll take a closer look at German FinTech company Europace AG, which decided to move toward self-organization two years ago. We’ll highlight parallels between Europace AG’s organizing approaches and those of open source projects.

Let’s start with some of the core values of ASF projects.

Community over Code

One main principle is the concept of “community over code” — which means that without a diverse and healthy team of contributors to a project, there is no project. It puts the team front and center, as highlighted in the Apache project maturity model.

Meritocracy

Another core value to Apache projects is meritocracy — essentially meaning that there is no governance by fiat. There is no way to simply buy influence into projects — you have to invest time to gain influence. This directly translates to frequently given advice for how to get started with any given project: Focus on the things you are using yourself and fix what bothers you most essentially following a scratch your own itch kind of model. There is one level of indirection here: Committers and contributors can be paid either by their employers or as part of a consulting gig to increase their motivation to work on the topics that you are interested in.

Project independence

Be aware though that there is a strong principle of people participating as individuals on projects. That level of independence means that within an Apache project, there is no way to assign tasks to project members. Apache projects by design have a very flat organization with barely any hierarchy: Titles like Project Management Committee Chair (PMC Chair) are famous for coming with additional responsibility but no entitlement to task assignment. That means projects need a process for aligning people with potentially very diverse interests and itches to scratch. A key ingredient to coming up with workable decisions is making project goals and needs public and transparent — both in terms of asking for help and in showing appreciation and rewarding contributions that help the project. Another one can be found in an approach to integrating differing opinions and arguments in a “Yes, and” instead of a “Yes, but” fashion. This plays together very closely with the next concept.

Full transparency in open source projects

“What didn’t happen on the mailing list, didn’t happen” is one of the mantras for those involved with Apache projects. A high level of transparency is what makes Apache projects so easy to participate in across boundaries — whether they are geographic, corporate, or other. At Apache, mailing lists are treated as the point of reference for any decisions made. Thus, there is full transparency into the historic record of all open source projects.

Of course, with that entirely open and transparent model comes responsibility: A lot of decisions can be taken in the open. People tend to show better behavior when under public scrutiny. However, discussions involving interpersonal issues are best kept out of public sight. This is particularly important in a digital age where deleting statements made online and mirrored worldwide is pretty much impossible.

Project autonomy

Technologically, Apache projects are very diverse. However, when unified under one organization with a common focus on community, meritocracy, and communication patterns, there is a lot of freedom and decentralized decision making.

One important piece in growing the ASF as a whole was giving autonomy to each project. Still there are a couple topics that each project has to deal with: It’s great having one central entity answering questions on, for example. trademarks, licensing, or software patents. However, these entities controlling each of the 300 or so top level open source projects won’t scale. Instead, Foundation-level services serve as guidelines for Apache open source projects to manage their own activities in these areas. As a result, each project is supposed to have some people knowledgeable in those topics.

Open source patterns in the enterprise

In the past decade, people in the open source space have successfully brought several of these concepts to their respective employers — either in the form of open development, inner source, or open organizations. Meanwhile, ideas of transforming traditional hierarchical organizations into self-organized, open organizations found their way into concepts like sociocracy and holacracy.

Europace AG decided to move toward self-organization roughly two years ago. Looking at what was established during these two years, we see quite a few similarities to how nonprofits such as the Apache Software Foundation and The Linux Foundation work with open source projects:

Decentralization and autonomy

Europace AG is a tech company experienced with XP and Scrum, and in 2015, they began a journey toward self-organization, looking into holacracy and sociocracy frameworks. Organizationally, there are roughly 200 people split into four fairly autonomous technical units, each one of which is responsible for one business product of the company. Units are autonomous in how they self-organize: Some chose fairly standard Scrum-based software development in iterations and standard rituals. Others went for sociocratic or holacratic models. By organizing in circles, employees are given more influence over how decisions are made that affect their daily work. Backing for this level of autonomy can be found in the company’s own principles, where self-organization and decentralization are outlined as one of the four important values and are actively supported by the board.

This is not unlike Apache, where open source projects operate independently and autonomously in how they develop their software and self-organize (that is, provided they make sure that project governance remains independent of one single player, security issues are being dealt with, and legalities like licensing, patent issues, and trademarks are taken care of).

Transparent decision making

Everyone working for and with Europace AG is located in one timezone, and Europace AG itself has only one site. Nonetheless, there are good reasons to look at how decisions are made in distributed open source projects: Establishing a remote friendly communication model can help support employees with family, and deal with long commute distances. Apart from these obvious benefits, there are a few effects that aren’t quite obvious:

While an asynchronous communication model won’t lead to decisions happening faster, it can help with reducing the amount of interruptions caused by face-to-face meetings: By sharing a proposal before the actual meeting happens and providing a means to exchange ideas around that proposal can help with understanding people’s opinions, issues and suggestions with respect to the proposal. It can help with making that understanding transparent to everyone interested in the topic up front. Combined with a “Yes, and” philosophy, this can help form consent in the team quickly.

Organizational Groups

Splitting roughly 200 employees into just four units creates groups that are bigger than typical “2 pizza sized teams” often deemed ideal for software development teams. As a result, each unit had to figure out how to create sub-groups that can work well as teams.

A common approach was to create cross-functional feature teams, comprising both software developers and people familiar with financial domain (and people with backgrounds in data analytics, UX, etc. as needed). Those are called feature teams in anticipation of the fact that they aren’t necessarily particularly stable: Each week, there is a check point people can use to decide to change feature teams depending on where they are needed most. Decisions related to governance as well as product strategic aspects are delegated to the team itself. As a result, team members need a good understanding of the current product direction in terms of features and usage.

For tasks and topics that tend to fall under the general topic of “organizational operations” or “decision making with dependencies,” people organize themselves in circles, which are essentially groups of people interested in participating in making a decision related to a specific topic. Despite a lot of communication still happening face to face, some circles also set up a Slack channel that is open for all to join and participate. This level of transparency is crucial in elevating trust people have in those decisions.

Each of these circles can come with dedicated roles: Secretary being the one taking over meeting minute creation, a Lead Link making sure that “things will actually get done” and communicated to the people needing the respective information. While people can volunteer for these roles, at the end of the day these roles get filled through nomination by the team. This is not unlike Apache projects where new committers, PMC members, and even Foundation members are elected after nomination. On one hand, this kind of model shows a certain amount of appreciation for the work these people are doing. On the other, it helps with promoting people into these roles who according to their own self-assessment might not have confidence to take on the role.

Put the human first 

Much like Apache puts community development front and center with its open source projects, Europace AG puts team collaboration into a central position. This means that each employee is responsible for raising issues with team climate, and they are supported by a team of professional coaches, moderators, and mediators with a deep understanding of human communication.

Conclusion

When moving toward self-organization, corporations can learn a lot from how open source projects organize themselves. Europace AG is on the journey toward moving more power away from traditional formal structures, making it available for the people in purpose-driven, circle-based organization. It will be interesting to watch how Open Source project management, governance, and communication patterns can be applied within corporate contexts.

Isabel Drost-Fromm is Open Source Strategist at Europace AG Germany. She is a member of the Apache Software Foundation, co-founder of Apache Mahout and has mentored several incubating projects. Isabel is interested in all things FOSS, search and text mining with a decent machine learning background.

It’s been two years since The Linux Foundation forged a partnership with the Apache Software Foundation to become the producer of their official ASF events. This year, ApacheCon and Apache Big Data continued to grow and gain momentum as the place to share knowledge, ideas, best practices and creativity with the rest of the Apache open source community.

As 2016 draws to a close, we looked back at some of the highlights from ApacheCon and Apache Big Data and collected the 7 videos from our most-read articles about the events in 2016.

These videos help highlight the good work the open source community accomplished for and with Apache projects this year. We hope they inspire you to participate in the community and present your work again at ApacheCon and Apache Big Data, May 16-18, 2017 in Miami. The deadline to submit proposals is February 11!

Submit an ApacheCon Proposal      

Submit an Apache: Big Data Proposal

1. IBM’s Wager on Open Source Is Still Paying Off

When IBM got involved with the Linux open source project in 1998, they were betting that giving their code and time to the community would be a worthwhile investment. Now, 18 years later, IBM is more involved than ever, with more than 62,000 employees trained and expected to contribute to open source projects, according to Todd Moore, Vice President of Open Technology at IBM, speaking at ApacheCon in Vancouver in May.

“It became apparent that open source could be the de facto standards we needed to be the engine to go out and drive things,” Moore said in his keynote. “[The contributions] were bets; we didn’t know how this was going to come out, and we didn’t know if open source would grow, we knew there would be roadblocks and things we’d have to overcome along the way, but it had promise. We thought this would be the way of the future.”

Moore reiterated IBM’s commitment to open source, highlighting projects born at IBM’s developerWorks Open (dWOpen), such as SystemML, Toree, and Quarks, and now in the Apache Incubator.

Read our coverage of Moore’s presentation, and watch the full video below.

2. Open Source is a Positive-Sum Game, Sam Ramji, Cloud Foundry

As open source software matures and is used by more and more major corporations, it is becoming clear that the enterprise software game has changed. Sam Ramji, CEO of the Cloud Foundry Foundation, believes that open source software is a positive sum game, as reflected in his ApacheCon keynote.

Invoking his love of game theory, Ramji stated emphatically that open source software is a positive-sum game, where the more contributors there are to the common good, the more good there is for everyone. This idea is the opposite of a zero-sum game, where if someone benefits or wins, then another person must suffer, or lose.

Read the full coverage and watch the video below.

3. Apache Milagro: A New Security System for the Future of the Web

With 25 billion new devices set to hit the Internet by 2025, the need for a better worldwide cryptosystem for securing information is paramount. That’s why the Apache Milagro project is currently incubating at the Apache Software Foundation. It’s a collaboration between MIRACL and Nippon Telegram and Telegraph (NTT), and Brian Spector, MIRACL CEO and Co-Founder, discussed the project in his keynote at ApacheCon in May.

Spector said the project was born in a bar on the back of a napkin after a brainstorm about how one would rebuild Internet security from the ground up. That sounds like a lot of work, but Spector believes it’s absolutely necessary: the future of the Web is going to be very different from the past.

Read the full article and watch the video below.

4. Netflix Uses Open Source Tools for Global Content Expansion

“We measured, we learned, we innovated, and we grew.”

Brian Sullivan, Director of Streaming Data Engineering & Analytics at Netflix, recited this recipe for the streaming video giant’s success several times during his keynote address at the Apache Big Data conference in Vancouver today. It was this mantra, combined with an open source toolkit, that took the stand-alone streaming product from a tiny test launch in Canada to making Netflix a global presence.

Read a summary of the presentation and watch the video, below.

5. Spark 2.0 Is Faster, Easier for App Development, and Tackles Streaming Data

It only makes sense that as the community of Spark contributors got bigger the project would get even more ambitious.  Spark 2.0 came out with three robust new features, according to Ion Stoica, the founder of Databricks.

“Spark 2.0 is about taking what has worked and what we have learned from the users and making it even better,” Stoica said.

Read our coverage of the keynote and watch the full presentation, below.

6. IBM Uses Apache Spark Across Its Products to Help Enterprise Customers

IBM is invested in Spark’s machine-learning capabilities and is contributing back to the project with its work on SystemML, which helps create iterative machine-learning algorithms. The company offers Spark-as-a-service in the cloud, and it’s building it into the next iteration of the Watson analytics platform. Basically anywhere it can, IBM is harnessing the efficient power of Apache Spark.

“We at IBM … have noted the power of Spark, and the other big data technologies that are coming in [from the Apache Software Foundation],” said Luciano Resende, an architect at IBM’s Spark Technology Center.

Read the full article and watch the presentation, below.

7. How eBay Uses Apache Software to Reach Its Big Data Goals

eBay’s ecommerce platform creates a huge amount of data. It has more than 800 million active listings, with 8.8 million new listings each week. There are 162 million active buyers, and 25 million sellers.

“The data is the most important asset that we have,” said Seshu Adunuthula, eBay’s head of analytics infrastructure, during a keynote at Apache Big Data in Vancouver in May. “We don’t have inventory like other ecommerce platforms, what we’re doing is connecting buyers and sellers, and data plays an integral role into how we go about doing this.”

About five years ago, eBay made the conscious choice to go all-in with open source software to build its big data platform and to contribute back to the projects as the platform took shape.

Read the full article about eBay and watch the presentation, below.

Share your knowledge and best practices on the technologies and projects driving the future of open source. Submit a speaking proposal for ApacheCon and Apache Big Data today!

Submit an ApacheCon Proposal      

Submit an Apache: Big Data Proposal

Not interested in speaking but want to attend? Linux.com readers can register now for ApacheCon or Apache: Big Data with the discount code, LINUXRD5, for 5% off the registration price.

Read More:

10 Great Moments from Linux Foundation 2016 Events