The Myth of Self-Organizing Teams

A hot topic in the agile world is "self-organization".  The reaction against tight command and control management structures has swayed the pendulum all the way over to chaos. 

First, I understand that every team is different, and my views are tainted by my personal experiences, which include heavy work in Austin, TX and with various companies sprinkled across the U.S.  I have seen (and made) arguments stating that self-organizing teams are much more productive and effective than tightly-controlled team.  I now believe that a balance is critical (surprise, surprise).

To understand my perspective, you should know that I have been in software management for just over two years, and before that I came up through the ranks as a software developer.  My views on self-organization have changed with my role, but in either position, they have not been extreme.

When I was an individual contributor, I was hardily in the self-organization crowd, mainly because I preferred not to be directed.  What I found was that I and so many of my very intelligent co-workers at several companies felt that we as the team knew the best way to proceed, and I somewhat resented management handing down decisions that seemed misinformed for the situation.  The fact was that many of management's decisions were misinformed.  There was no alternative since management swooped in for a weekly status meeting and then was gone again.  The lack of consistent, involved management on a day-to-day basis severely colored my opinion of managers (at least in I/T).  With this lack of engaged management, across several organizations, the team has no choice but to self-organize or remain in constant chaos.  The self-organizing did happen to a certain extent, but only after much chaos and posturing to determine which team member would surface as the "lead" since management had failed to identify such person.  Since most of the time the team was made up of peers, making decisions was slower than necessary because no individual had the authority to make the decision so that we could move on.  Consensus and needless discussion ruled.  Over time each team member learned to carefully select the battles in which to engage so that these discussions could be limited, but the time wasted was significant.  If any of my former co-workers or managers are reading this, you were the good group (the ones described above would not be reading this blog).

The above represents some of my experience as a software developer, and because of this experience, and reaction against managers who were not properly leading the team, I have been in favor of what the industry calls "self-organizing teams".  Some other writings on the topic (for and against) are below:

It is very important to emphasize that my experience managing and growing software teams has changed my thinking regarding self-organization.  My current situation as Chief Technology Officer of Headspring Systems finds me managing a company with 10 software developers, not including me or our Chief Architect, Kevin Hurwitz.  In our line of work, consulting projects, we cannot afford the time it takes for a project team to go through Forming, Storming, Norming, just to get to Performing.  While that has to happen, ultimately, we had to find a way to streamline the forming and storming phases.  This is where management comes in.

My approach to management has some roots in what not to do.  From some of my past experiences, I know that I will be more successful if I'm engaged closely with my teams in not only supporting them but also in directing their activities.  If we think about a continuum, dictatorship is as the opposite end of pure consensus.  If management has a tight command and control approach, the organization just will not scale and cannot grow.  A single manager can only control so much.  A manager must know how to delegate if the organization is to grow.

As with anything in life, self-organization is a balance.  My guys employ self-organization with certain things, but there are other things that are directed.  Point:  Self-Organization does not require Self-Directing.  A team that is self-directing will likely not accomplish anything significant.  A software team within a company cannot make every decision.  It can only make certain ones.  For instance, what market segment to compete in is a decision that has likely already been made; however, what unit testing framework to use might be up to them.  This is where the balance between direction and delegation comes in.

We have certain things that are mandatory for Headspring projects.  These things are considered competitive advantages, and no project team has the ability to deviate.  This may seem like command and control, and it is to some level.  This is the balance I am talking about.  Some of the things that are "dictated" to the team are (most other decisions are delegated to the team):

  • An automated, continuous build
  • Very high (near 100%) unit test coverage
  • O/R Mapper for data access
  • IoC container required
  • Onion Architecture as architectural approach
  • Extreme Programming practices
  • Some coding standards
  • Sitting together

The above are just some of the things that are "handed down from above with an iron fist", and one could argue that a team should have the authority to decide their own process framework or whether or not to do unit testing, but that is a business decision.  If I were running a Research and Development group, I would certainly have different levels of delegation, but the above are considered core to our practice and key to project effectiveness.  If I'm wrong, then it is clearly my fault, but that is one reason why management is hard:  anything that is wrong with the organization is the manager's fault, either by having the wrong people in place or sending them in the wrong direction.

The Self-Organizing Team is a myth because it communicates an absolute.  What is more real is an appropriate level of direction coupled with suitable delegation and trust.  In order to make our project teams effective, I must provide guidance and a behavioral framework for the team to operate in.  Within these somewhat strict guidelines, all other decisions are delegated, and I trust them to make the right choices.  This balance between directing and delegating has proved to be the appropriate balance between self-organization and dictated-organization.  The team members are very happy as well because they are pushed in the direction of success.

General management support is always key.  This approach to organizing would not work at all if I was disconnected and uninvolved with the teams.  I must stay on top of progress and current blocking issues in order to keep the team as effective as possible.  By staying engaged, I also can see problems coming from a distance, and I can take action to solve the problem before it becomes a blocking issue for the team.  Software tools are an easy example that hangs up so many organizations.  Any software tool for software development has an ROI story, and many sub-$1000 tools are available.  Compared to developer time, these tools are dirt-cheap.  Just yesterday, I purchased new licenses of Redgate SQL Compare for the team because they were incorporating it into the process for database migrations.  I've seen several managers try to negotiate away a small tool cost just because the procurement process for the company is difficult or it is politically unpopular to ask for special allocations.  This one small example is a symptom of the type of management that exists.  Supportive or tourist (comes to visit once in a while) management.

Conclusion

In order for a team to exist, it must have a mission, a purpose in life.  This mission must be directed, and is probably what caused the team to form in the first place.  Management must find the appropriate balance between directing and delegating.  Because of my background in software development I am able to competently direct my teams in the right direction while at the same time delegating tactical decisions that are contextual.  Furthermore the Principal in charge of a single project team decides what further decision to make singly and which to delegate to the individual developers.  The experiences in my career have shaped my views on self-organization, and I have found that neither self-organization nor command-control are appropriate for an effective software team.  A team might perform in spite of either, but a carefully balanced direction/delegation ratio will increase effectiveness exponentially.


Trackbacks

insurancesitesfind » Blog Archive » The Myth of Self-Organizing Teams Posted on 11.11.2008 at 9:30 AM

Pingback from insurancesitesfind » Blog Archive » The Myth of Self-Organizing Teams

Dew Drop - November 12, 2008 | Alvin Ashcraft's Morning Dew Posted on 11.12.2008 at 7:56 AM

Pingback from Dew Drop - November 12, 2008 | Alvin Ashcraft's Morning Dew

Arjan`s World » LINKBLOG for November 12, 2008 Posted on 11.12.2008 at 8:49 AM

Pingback from Arjan`s World » LINKBLOG for November 12, 2008

Comments

Shane said on 11.11.2008 at 9:03 AM

Good read. I'm a big believer in the Forming, Storming, Norming, Performing concept.

I especially like this line "The Self-Organizing Team is a myth because it communicates an absolute".

Yoda said it best "Only the Sith believe in absolutes".

Absolutes should be discouraged.

-Shane

Aaron Lerch said on 11.11.2008 at 9:08 AM

I enjoyed the book "The Five Dysfunctions of a Team: A Leadership Fable" as a good example of the balance between allowing a team to be self-organizing and yet directed at the same time, and also a good discussion on the types of issues that can arise. It's a somewhat cheesy read, but if you can get past that I definitely recommend it. www.amazon.com/.../0787960756

Context Context Context said on 11.11.2008 at 9:19 AM

@_@ why don't you just say it depends on the context. Suddenly 1000 words of blog are reduced to a sentence.

This isn't rocket science.

Karthik said on 11.11.2008 at 9:53 AM

Great post. I think its important for developers to stay transparent about whats worked for them and what hasn't when it comes to management decisions as well. Despite spending time on research and "the right way to do things" when we are in technical roles, those of us who answer the call to take leadership roles tend to fall back on instinct (which is often wrong) when it comes to managerial decisions.

Jeffrey Palermo said on 11.11.2008 at 10:27 AM

@Shane,

I don't really like the "no absolutes" mantra because the appropriate balance in some areas is heavy to one side with only a few, specific exceptions. This is still balance for effectiveness by getting the most value out of a practice without taking it to the extreme and encountering diminishing returns.

For instance, take the absolute of unit test coverage: 100% test coverage. I mandate this, BUT, there are exceptions and cases where there are diminishing returns. It's skewed heavily to an absolute, but stops short of futility. A 50% balance would not be appropriate here. Very close to an absolute is.

Jeffrey Palermo said on 11.11.2008 at 10:27 AM

@Aaron,

I'll have to check out that book, thanks.

Jeffrey Palermo said on 11.11.2008 at 10:28 AM

@Karthik,

I agree. I prefer to talk about concrete experiences as opposed to theory.

Jeremy Gailor said on 11.11.2008 at 11:30 AM

Interesting, because what you think of as self-organizing, I think of as just chaos. A development team without business providing it direction doesn't get much done. I think that most people who are touting things like self-organization and agile and XP forget what those things really are about, which is improving the relationships between teams within an organization to engender a sense of trust and reliability between them. Our engineering team make every decision regarding how engineering is done, and for us that is pair programming, TDD, and a host of practices, some that come from agile & XP, and some that don't. It interests me that you said a lot of time was wasted because someone needed to be a "leader". When hiring new engineers to the team, one of the things we make very clear is that the engineering team has no leader. Everyone is on even footing in decision making, and everyone is expected to be respectful of other peoples opinions (you'd be surprised how many people this dissuades from going any further in our hiring process). Despite this we never waste time arguing over the right way to do things. Someone makes a proposal on how to get it done, someone else adds some feedback, and everyone else either speaks up or agrees to move forward. I think the most time we've ever spent deciding on a way to proceed was 20 minutes.

I think the self organization that you speak of may fail when people on the team don't care about being a responsible member of the team. Our team as a group takes steps to ensure that everyone who has an opinion is given a measure of consideration and that they feel like they are responsible for the quality of our output at every step of the process, and part of that responsibility is to not let it disagreements get personal and to keep moving forward.

I have been a part of teams like you have described, but my sense looking back is that they didn't have the sense of responsibility that I was just referring to. But mandating the list of things you do to your engineering team just seems like putting your fingers in the pie of something that already has a full bevy of fingers poking. I don't see it as providing direction so much as saying "we don't consider you responsible enough to make these decisions, so here you go". That to me would generate resentment. If people are arguing about making a simple tool purchase, then maybe those people are not a right fit for your team, and should either be talked to one on one, or moved to a different team, or allowed to move on to a better fit for them.

What I've seen is that putting the responsibility of delivering accurate features that the business people are prioritizing highest and holding to the best estimates that can be made on the engineering team, and keeping frequent communications between the teams and individuals, the team itself will start to feel a sense of responsibility to being the best that they can be, and will becoming a functional responsible member of the organization.

Phew... sorry, long comment, but after nigh a decade in software development, I've got more horror stories than not, and I am so proud of my current team that it's hard to proselitize.

Shane said on 11.11.2008 at 12:41 PM

@Jeffrey - I'm not sure if you are disagreeing with me or not. I know you stated you don't like my "no absolutes" mantra, but what you go on to describe seems to me to fit exactly into my own concept of my no absolutes mantra. For example, you mandate 100% code coverage, but you mentioned you do make certain exceptions. Therefore 100% code coverage is not an absolute.

To qualify what I mean by no absolutes, 99.9% is not absolute in my mind. To me, absolute means 100%, no exceptions. Absolute is so finite that one can basically shut off their thinking process and blindly follow their "absolute" rule. And this to me is a danger.

I follow best practices the majority of the time for many things, but in the back of my mind I'm always asking "is this the right thing to do?". I always cringe whenever I ask someone why they are doing something a certain way and the answer comes back "because we've always done it this way"

Knowing the rules will make you good, but knowing when to break the "rules" will make you great (assuming of course one knows how to properly break the rules)

Jeffrey Palermo said on 11.11.2008 at 2:11 PM

@Jeremy,

I understand what you are saying and some of my experience is chaos masked as "self-organization", which does miss the intent.

I have also worked on an engineering team like your current one where the team performed superbly, and there was no tech lead explicitly. In that case, however, management stayed close and supported the team and removed any roadblocks as well as made directing decisions. The team did have a leader, and it was management.

Thanks for the comment!

Jeffrey Palermo said on 11.11.2008 at 2:13 PM

@Jeffrey,

It appears we agree. My guess is that if we sat face to face and discussed it, we would be of one mind.

I agree with your follow-up as well.

Paul Keeble said on 11.11.2008 at 3:13 PM

Not sure I agree with this definition of a self organising team. I would argue that a self organising team has members in it that provide business insight and direction and appropriate leadership and management. The difference is that the team makes decisions about scope, and they present the designs to each other for peer review. That way you get better balance in the development process so that key refactorings/tests/whatever get done as well as business goals. If the business, managers and developers all accept scope for an iteration they have joint responsibility for getting it done.

Developers should not be left on their own, the agile practice is all about having a customer on your team! A self organising team has to include wider expertise than just developers and indeed wider than the business and even the testers. The whole idea is to avoid 'drive by management' and random requirements. My favourite book which has some decent concrete examples of how it should work in practice is Agile estimating and planning: www.amazon.co.uk/.../ref=sr_1_1

Jeffrey Palermo said on 11.11.2008 at 4:40 PM

@Paul,

If I were to give a definition of Self-Organizing teams, I would likely have one very similar to yours.

The use of the "Self-Organizing" moniker has led to chaos or the lack of direction in some cases. For project managers who are looking into agile, some think that is means they have to stop directing, but a careful balance is what is called for. Regardless of the terms used, a healthy team with a competent leader will work out the appropriate balance between directing and consensus.

Craig Bowes said on 11.13.2008 at 1:49 PM

I agree wholeheartedly. Management has to tell development what to build. Since you're the CTO, you also "dictate" certain practices like the ones you mentioned, such as ORMs, TDD, CI, etc. This is being a technology leader/mentor and I don't think there's anything anti-agile about having tech leaders or architects. I'm in a similar role and make similar "pronouncements" at my job.

The only thing I would ask is, are you still actively developing code? If not, you may eventually start dictating technology choices that are antiquated. For example say SQL Server 2012 comes out and somehow Microsoft has magically made it an awesome OOP database, then an O/R Mapper is no longer the right choice. Your team may know how cool the OOP features of sql 2012 are before you do. In this case, you'd be forcing them to use NHibernate when its no longer needed.

I think to make the technology choices, you have to actually be writing applications and "eating your own dogfood."

Jeffrey Palermo said on 11.13.2008 at 2:30 PM

@Craig,

Yes, I am still actively writing code, and as soon as I'm removed from that, I have to completely delegate that responsibility. I have to admit that I don't make as many architectural "pronouncements" as our Chief Architect since he's living and breathing architecture more than I am.

I'm am wary of my area of competence and to delegate areas that fall outside it.

Yves Hanoulle said on 12.09.2008 at 10:22 AM

I agree with some parts and I disagree with others.

I want to react to "self-organizing is chaos". I think that a really self-organizing team might look like chaos, but they do achive things and much more than any other team.

I agree that going to the FSMP takes a long time the first time.

I say the first time, because whatever happens to the team (a new member added, one falling away) you are back at square one.

That is not a problem because teams that have been at performing will quickly go back to that state.

What I wrote in my blogpost you have linked to, is that for every state you need a different kind of leadership style.

In the forming phase, you will instruct a lot. And that is where you will say that the team has to do automatic builds, has to pair up and do CI. That is exactly what you do.

From the moment the team start to discuss about certain norms, it is time for you to hold back more and to use more a coaching style.

I had a small talk about this at BarCamp in Gent this year:

www.slideshare.net/.../bar-camp-gent-t