"Just Enough" Planning - An Agile Concept

The Agile framework is all the rage. It aims to solve limitations introduced by waterfall. The framework is driven by value and priority, not fixed scope and heavy upfront planning. But, Agile and its Scrum variant is just a set of theories not intended to fully prescribe practice. There are many challenges that observing strict Agile/Scrum can present in practice, especially for agencies attempting to adopt it outside of a product development process.

Background

I believe the theory behind Agile and Scrum is sound. Where waterfall advocates for rigid planning first and foremost, Agile/Scrum advocates for just-in-time planning based on the highest priority and most valued items at that time. Changes are made ongoing based on learning and applied at that point in time, which is normally at the beginning or end of a sprint. Agile/Scrum is iterative and friendly to change. This is highly attractive, because no project can ever be fully planned upfront and everyone wants the flexibility to learn and evolve.

This approach may work great for product lifecycles, as this is the originating use case for Agile/Scrum principles. But, the theory of Agile and Scrum can vary a lot from the practice of Agile and Scrum (as the term framework implies). This is fitting, as Agile and Scrum definition assume the "ideal" scenario. But, ideal may only exist for product teams and doesn't match what I have seen in practice when applying Agile/Scrum concepts to project teams.

Jason Heaster, a coworker of mine, articulated this scenario in a different light. When Physics experiments are run, there are outside factors that influence the results of those experiments. Examples may include friction, wind resistance, temperature, etc. Running an experiment in a vacuum where none of these factors exist greatly simplifies the process of experimentation, as fewer external influences can affect the outcome. Yet when the results of the experiment are applied to the "real world", the outcome may be significantly different due to these externalities. In the same way, going from Scrum in a vacuum to Scrum in the real world has its own set of externalities.

What are some example externalities specific to projects (agency/professional services/contracting) that differ from products?

• Contracts often set expectations around scope and timelines; this complicates Agile concepts of prioritization and horse trading if not handled with care 
• Development team members may shift and change and tend to require specialists; Agile/Scrum calls for the same team and that team members are self-organizing and cross-functional
• Development team members should have some sense of high-level architectural guidance to help influence their design choices and refine the degrees of freedom; this is contradictory to the Scrum process which prescribes identifying/planning/discovering work at the time in which it is  
• Scrum-prescribed events requiring full teams, while they build competence, can cause quite a hindrance on development velocity
• Product owners are normally external and can have varying levels of experience serving formal Agile/Scrum responsibilities; this fundamental role could be compromised
• All project participants must follow and demonstrate maturity in Agile / Scrum; projects with many partners involved or with participants unfamiliar with the process can introduce unforeseen challenges, additional training needs, or improper expectations
• Product owners often mistake the flexibility of Agile as uncontrolled scope and open prioritization; this may de-prioritize contractual obligations and can complicate expectations around milestones like MVPs, completion of contractual obligations, fixed sprint goals, or the impact of re-prioritization
• Product owners may struggle to define "value" in a chaotic environment with a large number of stakeholders and ongoing learning; Agile / Scrum mandates this as a required principle for effective operations

The key takeaway, for me, is that those we serve may not quite understand how Agile helps or hurts their goals. But, they are purchasing it when they agree to work with agencies that are Agile. Their expectations may align more with waterfall and there needs to be a mechanism for keeping this tension in check. This article sheds some light into this.

Potential Solutions

In a lot of ways, Agile projects require some level of waterfall. A process ripe with change and flexibility should inherently have a set of checks and balances ensuring changes do not jeopardize expectations. Agile and Scrum promote transparency, but there seems to be a lack of analysis around the defined impact from change.

There are some Agile-like concepts that support change management. The notion of "horse trading" is a vital role in prioritization. If something gets moved into a sprint, something of equal or greater value must be moved out to maintain the timeboxed sprint.  But does true Agile promote effective horse trading? Transparency is a large part of Agile, but externalities like contracts change the definition of what needs to be made transparent. Traditional Scrum states that the analysis of items is intended to happen only during sprint planning and "just in time". But, any item in the backlog could be re-prioritized at any time. Change management needs to support an adequate analysis of impact, not just what is known at a point in time.

How can a product owner truly be empowered to prioritize if he/she has no predefined sense of impact? How can a developer ensure the work done is complementary to the goals of the entire project? Neither solution works, because there is no upfront planning. Where Agile is designed for a product and a product's ongoing learning, project contracts often have expectations and pre-defined timelines / dollar amounts. Product owners require a greater level of responsibility over where and how their hours are spent. They need broader knowledge of how a change impacts the entire project, because one item change may actually push a required item out of the established timelines. An item planned at a specific point in time may introduce a large amount of refactoring because it was never analyzed foundationally and countless hours were burned on developing a contradictory solution. And, probably most egregious, there is a lack of acknowledgement that if you speculate on the size of an item (small, medium, large, or extra large), this exercise assumes some predefined solution that is often not documented and the imprecision of these categories actually threaten fixed-sized sprints.

The solution is straight forward: you need planning. But, this is widely viewed as an anti-Agile pattern. Examples of planning events include Sprint 0, discoveries, and backlog grooming. Anurag Prakash summarizes a purist's perspective on exactly how these events are more waterfall than Agile. But, as my friend Jenn Sramek summarized:

"if you [have no upfront planning], it can be that you do not learn enough about what you are taking on. The scale may be too big for your budget and you don't learn until you are mid-stream and have expectations already set. It is all good until it's not."

Those supporting pure Agile / Scrum often don't realize they needed more upfront planning and expectation setting until it's too late. You have to establish shared understanding early and often. If not, a project will likely end up in crisis.

Thankfully, there is a middle ground I refer to as just enough planning. The goal is simple: plan enough to establish a baseline of understanding and empower product owners and developers in their decision making. You can't just go on good faith that you understand each other or will establish that "just in time". Just enough planning is not complete planning. And, the level of just enough planning should be flexible based on the specific needs of the project and the capabilities of the team. This concept can be cleanly applied to Agile / Scrum in two principle forms: just enough upfront planning and just enough ongoing planning.

Just enough upfront planning

This is a time to evaluate the broad needs of the project. The concept of a "Sprint 0" discovery can serve as an effective upfront planning tool. This can include the following concepts:

• User stories / story maps should be evaluated and a skeleton backlog should be established during this time. 
• Story points should be entered for every backlog item with an initial architectural recommendation complementary to the estimate. 
• Establish project milestones that measure progress against product owner expectations. Then, map backlog items to these milestones. Milestones may include MVP, beta testing, launch, phase 2, etc. 
• Define an upfront summary of the comprehensive sprint-goals for the project, prioritizing foundational components first. 

Just enough ongoing planning

Ongoing planning occurs throughout the project and is critical, as it is inevitable that new requirements will surface and existing expectations are refined. There is a need to evaluate the impact when a product owner re-prioritizes items in the pre-established sprints, set upfront by sprint planning.  The following list are concepts that facilitate effective change management and the transparency needed to understand the impact of change:

• Developers evaluate the architectural recommendation when assigned an item. This should help to restrict the degrees of freedom, but developers should maintain the freedom to pursue alternate solutions that are more ideal for architecture or achieve an equivalent goal more efficiently. 
• Sprint demos / retrospectives present and review release-ready work from the sprint, feedback is entered as new tickets.
• All remaining estimates should be refined based on a comparison of estimated versus actual hours worked within the last sprint during the sprint retrospective.
• Refinements are also made to the project roadmap and sprint goals, based on any re-prioritization. This can occur during the sprint retrospective.
• New requirements may be learned as items are tested during product owner sign off. New tickets should be created and new requirements defined. Product owners should avoid adding new requirements into existing items.
• Routine and frequent backlog grooming sessions facilitate the discovery of all new backlog items that could be re-prioritized effectively at any time.  
• Daily reports should be sent out and/or presented during the scrum meeting to articulate project-wide and milestone-specific progress with respect to the work completed and the work remaining. 

Summary

Pure Agile and Scrum advocates for flexibility, but lacks the structure needed to communicate and evaluate impact. This can negatively impact projects that have fixed expectations. Just enough planning doesn't add the same level of upfront rigor that waterfall maintains. But, it provides a baseline of understanding that can help better understand expectations and establish the tools needed to better facilitate the flexibility product owners demand.