The new Team Structure

Once upon a time, programming, like painting, was fun. I remembered how I wrote my first program using my 128 steps programmable calculator and how I went into a flow state quickly.

I am quite sure that many artists are having similar experiences - when painting pictures, writing articles, composing songs & etc. We fully engaged for a few hours to at most a few days, creating something we are proud of. When it comes to artistic creation, one thing is obviously in common – the required knowledge during the work-in-progress is mostly tacit in nature. Put it simply, you cannot speed up the job by arranging people working in shifts, asking them to pass on the tacit knowledge between shifts. Indeed, there is no such need because the work-in-progress is usually too short to warrant any sensible “project management”, which might jeopardize the fully engaged flow state and hampered the otherwise high creativity.

Scaling out a little bit, small group painting can still be fun. They first agree upon a common goal and then self-organize and paint together, sharing common tacit knowledge. There are many studies about human aspects of building an engaged and empowered team. Three keys from the book “The 3 keys to empowerment” (Ken et al).

²  Share information with everyone (whole team listen to user requirements, not just Architect, BA or SA)

²  Create autonomy through boundaries (e.g. Governance, tools, architecture and time-boxing development)

²  Replace the old hierarchy with self-managed teams (team members take up jobs themselves. Tasks are not assigned)

Moreover, employees’ biggest motivator is making consistent & meaningful progress (producing demonstrable small pieces), as concluded in the book “The Progress Principle” (Teresa et al).

But what happens if we scale further, from a picture to a mural that involve a hundred artists? What if the surface is unknown and need some research to decide on the priming? Behind arts and creations, we do have Engineering - tools, technologies and architectures. For writing, we need to learnt Grammar and for composing songs, we need to understand music theory. If we want to complete the mural as planned, we have no choice but to manage the Engineering portion of it.

Given the advancement in computing, computer applications nowadays are usually in the scale of huge murals. With more and more mega-projects around the corner, the way we used to program was never be the same again. To manage the sciences behind the scene, the arts & creation portion is partitioned in such a way that it is difficult for us to engage fully, enter the flow state and be highly creative.

Can arts be arts and science be science? That depends on how we “divide and conquer”. I observed that once there are risks arising from the sciences portion, we tend to divide in the same time slice across the board, including arts. We should be dividing the loaf of bread but we divided the chair altogether! In this awkward way of dividing, the artists must stop in the middle of his creation and pass his tacit knowledge between stages.

The very first divide and conquer method I have learnt is SSADM. Using tools like DFD and ERD, we analyze the current physical model to produce the current logical model. We then design the new logical model and produce the new physical model. We then build and deploy the new physical design. The next one I have learnt is RUP which provides us a whole wealth of object-oriented tools and document types. We transit from Business Modeling, Requirements, Analysis and Design, Implementation, Test and finally to Deployment. The major difficulty with these two most popular methodologies is that very often, we rely on the tacit knowledge in our mind to travel back and forth between analysis, design and construction in order to build the “right” product. These methodologies don’t allow us to work on an element from end-to-end but force us to deliver explicit knowledge out of tacit knowledge between stages. These are examples of dividing science and arts together with the same handsaw.

Back to the hypothetical mural painting problem, what we should be managing is the priming and scaling. The most common divide and conquer technique is to scale a small outline sketch into a large wall using grids (picture on the left). Stage I is a engineering stage in which we manage the R&D, priming, gridding and sketching. The skeleton contains the initial agreed contents of the mural and more importantly, the interfaces between adjacent grids. Stage 2 is an arts and creative stage in which all 24 painters paint, in their normal style, simultaneously, from end-to-end (analysis, design & code) on their assigned grid within a time-box of say, one day. When the painters start to add details, the owner may change mind. But everybody has to observe the one day time-box rule. For example, the mural owner has exchanged the table on the beach with a boat. Time’s up; pens down and we enter the engineering stage 3 where we have to manage the drying up of paint and application of protective coatings. If, in stage I, we also do all the design up front in transparencies, project them onto the wall and just apply paints, we will end up with a mural good to look at from a distance but without details when close up.

I have learnt a great deal from “The Frog and the Octopus” model (Philippe 2011). The frog may say that painting should be the same everywhere while the octopus may say that mural painting is different. My impression is that it is a single tracked project. The octopus may determine which track is the most appropriate and the whole project will change to that track. No matter which track to change to, very often, the frog is to be sacrificed. Ironically, the frog is the primary author of the codebase, on which the successfulness of the product depends!

My question is that why a project can’t have multiple tracks? If we divide and conquer properly, the frog can stay on its own track while each arm of the octopus can has its own track. In software projects, it is the team structure.

Below is the new team structure for the new V-model in my last post. The unusual part is the breaking up of the traditional development team into a Business Process team, build team & PMO team. The build team is artistic in nature especially for the frog. So, it is not hybrid-agile or Water-SCRUM-fall. Some teams are waterfall all the way through while the build team is purely agile. I believe that both extending agile by adding waterfall elements and forcing waterfall to adopt some agile practices are unnatural and unnecessarily confusing (sort of neither fish nor fowl).

Besides PMO, all teams are following the same pattern, Specification with measurements (just the measurements, focus in exploration with users), development work (focus in getting user feedback) and finally checking the measurements (either manual or automated).

(Refer to the new V-model)

Business Goal Team:

1.          Envisioning and determine measurable business goals, including the detail measuring methods.

2.          Ensure that “measuring points” are implemented (goal testable).

3.          Drill and accept the measuring process.

4.          Assign measuring resources and define the reporting process.

5.          Final review to determine success

Business Process Team:

1.          Define Business Processes using UML process activity diagrams and Identify Scenario Steps and decompose into user stories. Prioritize user stories and agree upon the initial iteration plan. This is the sketch outline in grids to kick start the Build team.

2.          With the feedbacks from the build team, manage the change to the Business Processes diagrams and Scenario steps. Perform “test detailing” to transform it to a detail UAT step-by-step plan with User Guide.

3.          Plan and conduct SIT, UAT and final acceptance.

Build Team:

1.          Paint on its own user story grid with relationships between user stories defined in Scenario Steps. Frog: all development works are the same.

2.          Explore with users and confirm user story specification as measurements.

3.          SCRUM iteration with immediate user feedback on UI mockup and working software.

4.          Make the measurements executable (automation) with green lights. Demonstrate to users and energize the team.

5.          Feedback to Business Process team changes to Business Processes, Scenario Steps and user stories (add / delete / changed / merged / split).

Technology Team:

1.          Define non-functional requirements (measurements) and testing steps.

2.          Define architecture, tooling, build prototypes and perform proof-of-concepts.

3.          Produce reference implementations, design patterns and developer training kits for the Build team.

4.          Specify, build and test common / critical / scientific parts, such as workflows, cross-cutting security API, those that require regulatory compliance and those that may endanger human life. Provide API and documents for the build teams to use.

5.          Hardware procurement, installation, testing, hardening & performance tuning.

6.          Deployment drills and non-functional test acceptance.

PMO Team:

1.          Overall resources planning. They can be virtual teams and very flexible. In the extreme, all five teams may be sharing just a few members. For examples, the build team members can join business process team and technical team initially and join the PMO team after the last iteration.

2.          Data Conversion planning: data analysis, mapping, confirmation, mocks & verification.

3.          Cutover planning: existing interfacing analysis, special arrangements, confirmation, cutover steps & dress rehearsals.

4.          Source code baseline versioning, built and deployment management.

5.          Deliver the living documentations composing of Business Processes diagrams, Scenario steps with User Guide and Build Team’s user story measurements.

6.          Operation, Incident management & Support: Produce required operating manuals and deliver trainings.

The new V-Model

The principle is the same in all 4 layers of the V-model. Specification is accompanied by accurate Measurements.

The idea of Measurement came from the incident of buying a dining table (http://www.novadeko.com.au/media/pdf/DIN-0301-0001.pdf). My wife was about to place order while I, as a peer and have my say in that consensus, stopped her. I took a copy of the table’s specification home and did a careful fit-for-purpose analysis based on the measurements (width, length and height). To my surprised, my wife’s eyeball judgment was so accurate. We had exactly 1803mm wide to fit the table between two cabinets. The table was ordered online and got delivered the following day. However, we failed to fit the table into the space. I measured and found that the width was 1805mm. When I demanded a replacement of the defective table, the sales person pointed out to me the last sentence in the specification -- tolerance of +/- 5mm. 1805 is still within specification! Although I had to deal with the unfit table, I admired the simplicity & explicitness of the specification.

In a more generic sense, Measurements can also be processing steps, such as steps for defrost, reheat, grilling & baking of a microwave oven. The act of measuring is to determine if the documented outcome can be produced by following the steps.

These measurements can be used for

1. Fit-for-purpose analysis (packaged) / Deliberate Discovery (custom build)
2. Specifications for developers
3. Acceptance criteria
4. Documentation & training

Like the specification of an electronic component. The tables and graphs of measurements can help you to determine "fit for your circuit". They are also used to determine if a particular component is defective.

After agreeing upon the measurements with customer, the next thing is NOT to produce an automated measuring robot (like TDD). Showing the unfinished product to customer and obtain feedback is more important. Lastly, we build the automated measuring robot for final acceptance. Because we have the measurements and we know that final acceptance is done by the robot, testability is by design. There is no need to write the test first.

"Testing" or "Verification" are too "IT" but Measurement is very common in many disciplines. Different levels has different Measuring methods, targeting different audiences.

	Business Goal	Business Process	User Story	Program
Specification	The business function	Credible scenarios and their business purposes	As a ..., I want... so that ...	Business rule definitions.
Measurement	The target. e.g. a number how to measure	Execution steps.	Given, When, Then, Examples	test cases
Fit for purpose determination	Do we need to meet this target?	Can we accommodate the workflow?	Are the features fit our work?	Are these our business rules?
Transition to Acceptance	Establish baseline, Multiple releases, Collect statistics	Test detailing after each iteration. Add UI references.	Prepare test data, Automate Given, When, Then.	Write the test code
Transition Timeline	Months to Years	In parallel with iterations	Within 1 iteration	In hours to 1 day.
Acceptance	Target met.	UAT Successfully executed manually	All examples pass with green lights	All test cases pass with green lights

Business Goal Measurement
Measuring method of the Business Goal should be specific. Is it a moving average? Should we eliminate maximum and minimum? Should we set a standard deviation? What is the baseline? Should we allow for +/- 5% tolerance? After fixing measurement methods, "measuring points" must be built into the application to make it "measurable". We will use these measuring points to measure after go-live for a period and even after several enhancement projects and determine if the Business Goal has been met.

Credible Scenarios as Specifications

Credible Scenarios are the scenarios you would encounter in real life. The measurement is the execution steps and UI guide. To illustrate the concept, I would like to borrow the publicly available SAP Best Practices Scenarios map for illustration. Click on the Scenario Map link and you can find all 21 credible scenarios related to the Business Process of "Sales", such as

1. Sale from stock
2. Using 3rd party with shipping
3. Free of charge delivery
4. Sales of planned service
5. etc

To view the details of a particular credible scenario, click on the credible scenario and click the "i" icon. For example, execution steps of "Sales Order Processing: Sale from Stock" are:-

1. Sales Order Entry
2. Delivery Creation
3. Post Goods Issue
4. Create Billing

The big picture is the UML Process Activity Diagram on the left. It shows the start, all credible scenario flows and the termination across different stakeholders' boundaries. The red line marks one of the credible scenario flow, which we are going to decompose into execution steps.

Execution  steps and UI guide as Measurements

Fig 1

We should decompose these steps into User Stories (Fig 1), which are prioritized to form the initial release plan (how many iterations and the user stories in each iterations). This is the measurement of Credible Scenarios. To avoid big-design-up-front, we should omit all UI details. The measurements up to this point are for high level "fit for purpose" analysis and scope of Acceptance.

Fig 2

I like the idea of "Test detailing" in Paul Gerrard's book "The Business Story Pocketbook". Starting with the Measurement (Credible Scenarios, execution steps and initial user stories), we progressively complete the detail UAT execution plan in parallel with the user stories development iterations.

However, I found that the traditional "waterfall" project management approach is more appropriate in managing "Test detailing", completing Fig 1 by producing Fig 2 & 3.

Fig 3

When user stories are "done", we can also dump the screen, describe input fields, define possible values and etc (Fig 2 & 3).

2011 to 2012 are two fruitful years full of innovations in the software specification arena. Gojko Adzic's Specification by Example; Paul Gerrard's The Business Story Pocketbook and my little attempt to add Measurements to my hybrid-Agile project. Surprisingly, SAP also published the whole set of SAP Best Practices Business Process Documents (BPD) -- Scenarios, test data, detail execution steps and UI operations. The only different from my version is the missing screen dumps! Below is extracted from SAP Best Practices' home page. Note (1) fit for purpose determination & (2) test execution. Because they are executable, they are also living documents.

SAP Best Practices include clear methodologies and step-by-step approaches. Every package has extensive reusable documentation for self-study, evaluation, as well as for project team and end-user training. SAP Best Practices provide complete pre-configuration settings that give you everything you need to run specific key processes with minimal installation effort.

User Story Specification & Measurement
These are the well known "As a ... I want ... so that" user story specification and the Gherkin "Given / When / Then / Examples" measurements.

Program (Business Rule) Specification & Measurement

Below is a business rule Specification example of converting a requisition into a purchase order:

A requisition item cannot be converted into a purchase order in the following cases:

• if the master data is faulty or incomplete
  (for example, the vendor is blocked or there is no current order price)

• if Customizing is faulty or incomplete

• if input data is missing (for example, no account assignment) 

Measurements are the unit test cases -- examples of when requisitions are not converted in PO and when requisitions are converted to PO. 

We are not doing TDD. We design by specifying the measurements before coding but not writing the test before coding. We write code and invite user feedback before writing automated testing for these measurements. Code quality is still guaranteed because first, measurement is the design and second, we code with testing in mind (i.e. write testable code).

UAT can also be Specifications, by applying Specification by Examples

Specification must contains measurements. It is too obvious. Like dining tables & electronic component. But it is not the case in the Software industry, at least in my past 20 years as software developer. Only recently did I learn how to write Software Specifications with measurements from Mr. Gojko Adzic's books -- Specification by Examples & Bridging the Communication Gap.

However, I found "Example" somehow too casual and not mean for serious use. Hence, I prefer calling it "Specification with Measurements". If you look at the specificaiton of an electronic component, there is a description in English as well as accurate measurements in tables and graphs for you to determine "fit for your circuit". First,  measurements determine "fit for purpose". Can the dining table fit your house? Second, measurements determine acceptance. The delivered dining table not according to the measurements should be returned.

Unfortunatly, I come from a company with a "Control" first and then "Competence" culture (ref. An Agile Adoption and Transformation Survival Guide by Michael Sahota). I have to adopt instead of transform. Hence, Water-Scrum-Fall is the maximum agility allowed. That is, we need to have an initial planning and scoping phase to produce something for the management to signoff. Then, we can execute the agreed number of Scrum iterations. Finally, users will accept the system when successfully executed a UAT.

Apart from "Water" and "Fall", the Scrum part is pretty stright forward. Specification by Examples is very effective and we pick it up quickly.

As shown on the left, unit test cases are our program specifications and SpecFlow story tests are our functional specifications.

How about the initial signoff document and the UAT plan?

I found that traditional user acceptance testing (UAT) is having a bad name in the Agile circle. 

"agile acceptance testing has very little to do with user acceptance testing. In fact, they are on completely opposite sides of a software project. User acceptance testing happens after development and it is often the final confirmation before money changes hands" -- Bridging the Communication Gap

The book Bridging the Communication Gap mentions that we can "have one or two acceptance tests for the whole larger workflow to verify key paths through the steps". Draw the workflow on the wall and keep an eye on the big picture when discussing user stories. However, "Agile acceptance testing is very different from user acceptance testing".

What a lonely path. I have to find my way.

The "Water" Phase and the signoff Document

From a high level, our business process oriented system is best described by a Busines Processes flow diagram and a set of Crediable Scenarios. Credible Scenarios can be "Procedure to handle complaints from our customer" or "Procedure to handle users applying for a refund". These Credible Scenarios are critical to the business and will form the core of the final UAT.

Hence, these Credible Scenarios, like user story examples, are also specification of the system in the highest level.

On the left is the Business Processes of the system and one of the Credible Scenarios marked in red. In our project, we have identified a total of 23 Credible Scenarios.

Horizontal segments are user parties responsible for a particular Business Process.

Next, we expand each Credible Scenerio into execution steps with description, purpose and expected result, as shown on the right.


In this initial scoping stage, the User Story and UI manual Reference column are left blank.

Next, we arrange meetings with working level stakeholders to go through each Credible Scenario, the execution steps and identified user stories. UI manual Reference is still blank for now.

These stories are prioritized, estimated and plugged into the initial release plan, say, of certain iterations. Besides prioritizing based on business value, users may choose to complete higher priority Business Scenarios first, such that they can execute them eariler.

The initial signoff documents are:

1. The Business Process Flow diagram
2. The Credible Scenarios with detail steps (Specification as well as UAT scripts)
3. A list of initial user stories identified
4. The initial release plan of a fixed number of iterations

The Implementation Phase

We establish a separate QA team to revise the affected Business Scenerios and execution steps after each iteration. 

They also develope the UI Guide with detail explaination of each step, screen dumps, fields and button definitions

On the left is an example of a revised Business Scenerio execution steps.

On the right is an example of a UI reference.

You can see that in parallel of Agile development iterations, the QA team gradually develope and complete the UAT plan with UI guide.

Actually, users will not wait for the completion of all iterations before executing UAT. Even when some Business Scenerios are half-baked, users are eager to partially execute them to gave feedbacks. Besides testing the system, they also test the execution steps and UI Guide. They also use this UAT document as their business process documents and training guide. When asked how to handle user complaints, they will refer to the execution steps and the UI guide.

The "Fall" Phase

During final UAT, end users will study the Business Scenarios and execute the scenario steps inside, like doing exercises in a text book.

Credible Scenarios and execution steps are similar to story test examples. It is the specification in the Business Process level.

The refinement of steps and the UI guide is like coding the SpecFlow test the make the UAT plan executable "manually".

On the left is the three levels of Specification by Examples

Addition Benefits

In additon to the UAT docuemnt, we also work out a Business Scenario & User Stories matrix for our future maintenance team. To change a Business Scenerio, you immediately know the  impact on user stories. To change a user story, you immediately know the impact on Business Scenerios.

Last but not the least, don't forget to measure the Business Goal. It is also a Specification with Measurement.