Agile Data Science – A Prerequisite

A colleague recently came back from his MBA class with a useful insight: Agile practices work like a (financial) leverage in project management, magnifying good things on the way up and bad things on the way down.

There are already many good books and articles on the benefits of agile software engineering and agile data science so I won’t dwell on that. What I want to discuss instead is when agility doesn’t work.

The key principle behind agility is tightening the feedback loop so that the project team can converge to the desired goal in an efficient way. From that fairly obvious principle comes the first and most important prerequisite: a common goal for all project stakeholders, or at least a desire to have a common goal if that isn’t well-defined at the start of the project.

I have been involved in many different projects in my career so far and, knowingly or not, I have adopted agile practices (in one form or another) in most, if not all, of them. I have had my fair share of successes and the occasional disasters. Reflecting now on the failures, the one common denominator is lack of cohesion and a common goal among project stakeholders.

In all those cases, we have stakeholders whose incentives are misaligned, which leads to active politicking, which in turn leads to a breakdown in honest communication. Applying agile practices in such environments won’t work; in fact, it will exacerbate an already bad project environment because tightening the feedback loop with misaligned project stakeholders will lead to frequent changes in direction and a constantly shifting goal post, followed by project stress and fatigue and ultimately failure.

Here is one way to sum up this mental model: agile practices lead to fast convergence when a common goal is present; without a common goal, agile practices lead to (violent) oscillations!


One thought on “Agile Data Science – A Prerequisite

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s