Friday’s Digest #4 [MS Project, Inventions, Crisis, Lean]

  • TIME’s posted Best Inventions of 2008. I wonder why LHC has 5th position…
  • Are you spending too much time planning? Traditional project management tools are really dangerous sometimes 🙂 Nice example how to not plan your project.
  • The End. The era that defined Wall Street is finally, officially over. “The way we thought about it was, ‘By shorting this market we’re creating the liquidity to keep the market going.’ ” “It was like feeding the monster,” Eisman says of the market for subprime bonds. “We fed the monster until it blew up.”
  • James Shore thinks that “Agile is failing all around us , and I’d hate for the failure of the fad to take down the truly useful ideas that we started with“. I tend to disagree. There were thousands failed software projects with traditional methodologies. And MS Project still the most popular PM tool over the world. Agile just should not be treated as a silver bullet.
  • How Kanban and time boxing may be applied for game development. Pretty detailed article. It shows that Kanban in fact may use a kind of iterations to line up work.

Software Development is Complex Adaptive System. No Doubt.

Complexity science is very young. Cybernetics, ecology, sociology, meteorology all study complexity. In general, a complex system consists of interacting components. The result of this interaction can’t be predicted by observing an individual component. For example, human brain consists of neurons. The brain has ‘memory’, while each neuron doesn’t. Ants’ colony behavior can’t be predicted from the behavior of an individual ant. And it is impossible to understand software development process looking at how just one developer works.

We are especially interested in a particular case of complex systems called Complex Adaptive Systems (CAS). The main difference is that CAS may learn and change (adapt) over time based on previous experience. CAS remember the history, and that is the main difference from chaotic systems.

There is no common definition of Complex Adaptive System (CAS). One of the most popular definition was offered by John H. Holland

A Complex Adaptive System (CAS) is a dynamic network of many agents (which may represent cells, species, individuals, firms, nations) acting in parallel, constantly acting and reacting to what the other agents are doing. The control of a CAS tends to be highly dispersed and decentralized. If there is to be any coherent behavior in the system, it has to arise from competition and cooperation among the agents themselves. The overall behavior of the system is the result of a huge number of decisions made every moment by many individual agents.

CAS has quite many common properties. However, hardly a system should have all of them to be CAS. Moreover, researches define different sets of CAS properties. Let’s try to review the most common properties and see how software development process can be described in terms of those properties.

CAS Property Software Development interpretation
System component.
People in development team and others involved (product owners, stakeholders).
Each agent in CAS reacts to information flows. In return, the information that this agent will receive in the future is influenced by his actions. This feedback may strengthen or weaken agent actions.
Development team relies on previous experience (previous project, previous iteration) for decision making. Decisions influence future actions.
Emergence is the way complex systems and patterns arise out of a multiplicity of relatively simple interactions. Small actions of agents lead to unexpected emergent system behavior and it is impossible to predict system behavior based on the behavior of an individual agent.
Small errors pile up and may cause huge problems, even stop project at all.
Each person in a development team does not have all info about the project.
People communication and collaboration leads to unexpected results.
Non linearity
Feedback and emergence cause non-linear system behavior.
Will a project success or fail? You never know from the start. Predictability in software development is hard (almost impossible in many cases). So, you can’t create a big plan upfront, you can’t create a big design upfront. Non linearity is a natural property of CAS, so all that BDUF, BPUF are just plain wrong attempts.
Sensitivity to initial conditions
Also known as Butterfly Effect. . Even one small change may lead to significant effect. Again it plays against predictability, you can’t predict weather for more than 2 weeks. Butterfly effect is a property of chaotic systems, so CAS is a chaotic system as well.
A small bug in the code may cause huge problems to the end user. Unavailability of a key project person on some particular day may cause major slippage because a wrong decision was made while this person was away. I bet you have such examples from real life.
High Adaptability
CAS is very flexible and adaptable. It can keep its basic structures and behavior, reacting to changes in external environment and taking advantages of these changes.
Development team should react to environment changes (new requirement, technology change, lead developer loss, etc). It should adapt to the new conditions and try to take advantage even from the worst events.
Information exchange
Agents exchange information.
People communicate in development team and outside development team (stakeholders, product owners, etc).
Agents in CAS cooperate to reach the goal.
People in development team cooperate to produce the software. BTW, game theory studies cooperation and may be applied to software development. That was already done by Alistair Cockburn.
Agents may have specialties. Strong CASs have a number of various agent types. This provides more opportunities to use environment features and to adapt to them.
Obviously, we have Developers, Testers, Scrum Masters, Product owners, System Administrators, etc. If a team consists of developers only, it will be less efficient than cross-functional team.
There’s no hierarchy of command and control in a complex adaptive system. There’s no planning or managing, but there’s a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback.
Traditionally, there is hierarchy in software development. There is a Project Manager who rules the project, there are Functional Managers, there is VP of Development, etc. In agile development there is no hierarchy. The goal is to have self-organizing team.
Edge of Chaos

A system in equilibrium does not have the internal dynamics that enables it to respond to the environment and will slowly (or quickly) die. A system in chaos stops functioning as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities.
It is hard to create software in Order. Development team will not have enough creativity to resolve unexpected and complex problems. That’s why heavy methodologies do not fit software development. It is impossible to create software in Chaos. We all know which results hack & fix brings. Ideally we should produce software using a relatively light methodology (agile?). A set of simple rules allows edge of chaos and powers creativity, flexibility and success.

This is just a quick overview of common CAS properties. Later we will go into details and draw more parallels between CAS and software development. But even in this short list we’ve got some nice phrases like self-organization, feedback, information exchange, cooperation, adaptation – does it remind you something?

Agile Software Development and Complex Adaptive Systems: Intro

There are quite many agile processes in the world already. We have Extreme Programming, Scrum, Crystal, DSDM, Feature Driven Development, Lean. Some of them are famous, some are maybe dying. The point is, we have plenty of agile processes. In general this is not too bad. We’ve got alternatives, they move agile forward and solve real problems. On the other hand, I feel the lack of information on the roots of agile processes. Most articles available are missing the point on my opinion.

Why iterative development works? Why agile manifesto is the truth? Why traditional project management techniques miserably fail on software development projects? Somehow many people feel the right direction. They have a background that guides their decisions. We feel that agile manifesto brings real value. We has proven this in practice after all! But I personally feel that there is something deep and cool behind most of agile practices. Something that will explain why this all works.

While reading Jeff’s post about Scrum roots, I found a reference to Complex Adaptive Systems. I went digging into more details. I read a lot about CAS and related things and there were many “a-ha” moments. There are so many parallels between CAS and Software Development. It is pretty obvious that Software development is a Complex Adaptive System itself. It has almost all basic properties of CAS and it is interesting to read, for example, about adaptation in various CAS and think about its application to the software development and agile principles.

I believe that a deep understanding of Complex Adaptive Systems will shed some light to the real roots of agile development and will answer important questions. With this knowledge in hand it will be easier to see why this works and why that failed. It will be easier to make decisions and create a better software faster.

However, I am not sure how interesting this will be to the people 🙂 In the next several posts I will write on why software development is CAS, describe important properties of CAS and apply them to software development (adaptation, feedback, self-organization, emergence, edge of chaos, non linearity, etc). Let’s see how it goes, maybe we’ll find something interesting 😉

Friday’s Digest #3 [Design, Web, Kanban, ExtJS]