Nice Introductory Video on Scrum

Recently I came across a YouTube video titled "Scrum in Under 10 Minutes".  I watched it a couple of times and came away impressed.  The creator, Hamid Shojaee, did a very nice job with this overview.  I recommend that you view Scrum in Under 10 Minutes for yourself and see what you think.  The video is even supplied in HD format.

Adiabatic Changes

In thermodynamics, we learn that most changes to a system are irreversible.  The entropy genie is let out of the bottle, and once out cannot be put back in.  Certain types of change, however, are reversible.  We call these changes adiabatic.

In an adiabatic change, the system neither gains nor loses heats to its surroundings, but I think that the concept applies nicely to software systems too.  Let's dig in a little bit and see how it works.

If we make large-scale changes to a complex software system -- especially a system that's being developed or maintained by a team of more than a handful of software engineers -- and those changes go wrong (i.e. the system fails its regression test suite), we must diagnose what went wrong.  Yes, if you are using a source code control system, you can simply back out all the changes that have been made until you restore the system to its last known good state.  I suppose in that sense, all software changes can be viewed as adiabatic.

But if you have multiple engineers involved, it is costly to back out everyone's changes and simply start over.  You could try backing out each engineer's changes one at a time.  We back out one set of changes and rerun the regression tests.  Does the system now pass?  If so, we might have a clue as to which changes caused the problem.

We might.

But what if it wasn't any particular engineer's changes that were at fault?  What if it was instead the interaction between changes that, taken individually were benign, but when taken together had a pathological effect?  Backing changes out one at a time, might not lead you to the right answer and might even mislead you.

On the other hand, if each change to the system is applied incrementally, we might be able to back the changes out without misleading ourselves.  So we emphasize incremental changes to complex systems. We emphasize incremental changes to local data structures. We emphasize changes that can be verified to have local effects.  We emphasize changes that due their incremental nature are reversible, adiabatic.

So we see that incremental changes to complex systems are easier to manage.  This is why I advocate phased development efforts. This is why agile methods work best (or at least easiest) on smaller projects involving fewer than 10 people: the changes are more often verifiable with regards to the propagation of their effects.  The changes do not cause the system to depart from its equilibrium (i.e. passing all regression tests) state.

Maintaining reliability thus becomes mostly a matter of sequencing a series of small changes and testing in between application of changes. While this cannot completely shield us from undesired interactions between changes, it will at least reduce their likelihood.  It will make most changes adiabatic.

The Delphi Method and Project Management

Spend some time studying software development models.  Fashions change and so do the sizes of projects and the sizes of the teams tackling those projects, so try to avoid getting too enamored with any particular model. Instead try to see behind the individual vocabularies and preferences.  You'll find that whatever they're called, software development models are really risk management activities.  It matters little whether you're looking at Boehm's 1988 paper about the Spiral Model or a new video posted about an agile development model.  In the end, the project manager is using various tools and ideas to manage risk in a software development project.

The artifacts of the various models are good and interesting.  The "Burn Down Chart" from Scrum is appealing to some.  The feature set lists and "Progress Bars" from Feature Driven Development are also attractive.  All will probably work -- if you use the procedures wisely and have good work estimates to begin with.  If you  have good data, you can extrapolate to see likely end dates.  If you have good data, you can see which teams are struggling and try to move work around to make things better.  

To the extent that your work estimates aren't so good, your artifacts probably won't be either.  If you keep good records and manage to survive for a few years, you might be able to get some mean productivity measurements for your product development teams.  These could be useful (if your staff turnover is low) because project decomposition into functions, features, objects, or whatever building block your organization uses will probably be done by a slowly changing set of staff members.  If you keep records and count the number of building blocks completed per engineer per unit time, then for future projects you can use that data for rough schedule estimates.  

If you manage to survive, you'll be able to refine your building-blocks-per-unit-time estimates over multiple projects and multiple years until -- for your teams, anyway -- they might give you some useful estimating tools.

But what do you do in advance of that?  How do you obtain estimates that are credible and reasonable before you have historical data for your teams?  You might consider some variation on a Delphi Method.  (Note: if this link goes bad, please Google some variation on "Delphi Method" or "Delphi Process" to find another paper describing it.  There are also wikipedia articles on this method.)

Delphi Method Summarized:

In any viable development organization, you must have some subject matter experts.  If you don't, well, you're going to have trouble developing viable products, so let's assume that you have two or, better yet, three subject matter experts (SME's) on your team.  Here is a quick summary of the Method:

1. Initially, each SME, working independently, develops a work estimate -- a forecast.
2. Each SME expresses her/his forecast in a specified manner, e.g. via a form, and submits the forecast to the project manager or other Facilitator.
3. The Facilitator combines and summarizes the individual forecasts, preserving anonymity while identifying areas of disagreement and important details.
4. The SME's are convened as a panel to discuss the summary report, ask questions, etc.

The process then restarts at the top and goes through all the steps again.  Iterations continue until a consensus is reached or until the project manager feels that the process has converged as far as it can (in which case the project manager will have to make the final call).

The key to the Delphi Method is that each SME operates independently while developing his or her forecasts.  By combining the forecasts, we assume that we'll end up with something that at least represents some varying points of view.  By letting the Facilitator synthesize the results, we avoid the situation where a dominant personality exerts excessive influence on the process.

Once you have work estimates for your project, you can apply them in whatever development method you choose.  Again, keep a record of actual "burn rates" of features.  If you have several developers working on several projects, you will eventually develop some baseline productivity data that will be valuable in subsequent projects.  Even if you continue to use the Delphi Method to develop work estimates, your historical record will be extremely valuable as a sanity check.

Groups

TPW, a very intelligent friend of mine, has wide ranging interests. One area to which he often returns in his recreational reading could be termed "primate psycho-biology". Even though I've not read nearly as much on the subject as TPW, I have stumbled across a few concepts that apply to management. They explain so much about behaviors we see in the workplace. Humans are, after all, primates.

Now, primates form groups. This seems to be a basic behavior. Perhaps we are programmed at the DNA level to form groups. It is not difficult to visualize roles the group could have in the selection (or de-selection) of traits. After all, banishment from the tribe practically guarantees that one's genetic material is not passed to subsequent generations. Thus, the psychological drive towards membership in a tribe or group is very strong.

One important part of a manager's job is to direct the formation and maintenance of groups that benefit the enterprise: We, the employees of this company, are a special group -- The Tribe. We must prevail. Our competitors are other tribes. The herds, the land, the business environment cannot support us all. They must not prevail. Our survival depends on it.

A difficulty with all this is that the primate group formation drive is constantly active. Also, we tend to form hierarchies of groups: our family, our community, our region, our country... So we may subscribe to the notion of the enterprise as a group, but without careful guidance from our managers, we'll also form local, atomic groups that center around our individual work teams and closest co-workers. So the QA department has a group identity that is distinct from that of the Technical Writing Team. Both of these identities are distinct from Marketing or Development.

Through inattention, careless managers, may allow local group identities to impair cooperation and flows of information between teams. Ambitious managers may exploit group identities as part of a strategy of self-promotion. In either case, the enterprise is harmed.

Wise managers, managers who truly care about the health of the enterprise, will accept their teams' local identities while consistently channeling those identities in constructive directions. Wise managers will always refer to other teams respectfully. Wise managers will encourage cooperation and mutual support among teams and, most importantly, will visibly embody that cooperation in their public dealings with their own peers. Privately, managers may argue and negotiate. They should test assumptions and plans rigorously and strive to develop the strongest strategies. Publicly, however, managers must themselves be active members of their own group, their own team.

So we use the basic, group forming drive to establish a work identity that is comfortable for our employees and healthy for the enterprise. We discourage group behaviors that are unhelpful. This is not so hard, but it does require conscious behavior on our parts.

Using What We've Learned

Now that we're a bit clearer on energy, what it represents, and how we can measure its usage, we can begin to apply our knowledge to a real-world situation. Let's focus on something practical: the amount of energy we use in our home. According to the US DOE's Energy Information Administration, about 20% of the energy we use in the United States is consumed in residential settings. That's significant, and it's something over which we can exert some control.

It is not very hard to calculate our baseline energy use and then see how the seasons impact our energy use. To do this, we'll need a year's worth of utility bills. You can use any 12 consecutive months for this; January through December, or June through May, or whatever is convenient. It doesn't matter where you start your record as long as you have 12 consecutive months' worth of data.

If you haven't been saving your utility bills, you can call up your service providers and request a summary from each of them. If they'll give you multiple years, then you can do some averaging to get some more solid numbers, but even one year will give you a great start in doing residential energy analysis.

Adjusting Organizations

Reorganizations happen.  They come in many shapes and sizes.  They are justified in myriad ways.  All are costly and disruptive.  They defocus the staff and make people generally nervous about their jobs.  There are good reasons to do reorganizations, however, and if you undertake a reorganization for good reasons and articulate those reasons well, the people impacted by the changes will move forward.  The enterprise is made stronger, your organization's effectiveness improves, and your credibility as a leader is maintained.

Organizations evolve organically over time.  Team charters that started out crisp begin to get blurry, fuzzy around the edges.  Perhaps it has become difficult to decide where a new project should fit.  Perhaps, one team has had to grow to get a big project (or set of projects) done, but this is changing, and that team is now going to need to bring its focus in tighter.  Employees that were busy and productive find themselves underutilized or stymied in their efforts to contribute.  Perhaps their interests have changed.  Perhaps they have learned new skills and are eager to apply those skills.  Perhaps the needs of the encompassing organization have changed.

Realignment of charter with functional structure and redistribution of staff to improve productivity are both good reasons for a reorganization.  Key to the success and smooth transition is the complete buy-in of your staff of managers.  This won't necessarily happen immediately.  

Sometimes good managers may need to lose projects so that the functional organization makes sense and inter-team dependencies are managed more efficiently.  Sometimes you and your staff will need to work hard to find the right fit for good staff members who are ready to move into new roles.  These changes can be emotionally charged and require trust to pull off.  Your management team must have learned, over time, to trust you.  They must have learned, over time, to believe that you are telling them everything that you can.  They must have learned that if you say, "I'm sorry, but I can't discuss that right now.  I'll elaborate on that as soon as I'm able," that it is for good reason and that you will let them in on whatever-it-is as soon as you are able.

So, above all, you must work every day to develop trust.  There is really only one way to do this.  You have to be honest and forthright in your communications.  Always.  If you have developed trust, your staff will go with you during difficult times.  If you have not developed trust, they won't.  It's really just about that simple.

Okay, so you have a staff that trusts you, but you still need to make some changes.  The first step is to get agreement on what the overall charter for the department is (or has become).  You need to articulate that charter and get agreement -- usually in a managers' meeting -- about it.  Depending on personalities, you may need to meet one-on-one with your staff members so that they feel comfortable describing their concerns about the charter as articulated.  

Once the charter has settled, you and your staff need to partition the charter in such a way that the work is divided up in an intelligent way.  Ideally, this will result in approximately equal amounts of work for each of your teams and each of your managers.  You should work hard to set up an equitable distribution of good projects.  This will be tempered by the interests and specialties of your various teams, but the more balanced the work, consistent with the needs of the organization, the better.

Partitioning the charter intelligently basically forces an organizational structure for your department.  This is a good thing because it reestablishes a crisp charter for each team.  Once your managers have understood and accepted this organizational structure, you all need to work together to adjust the details.  You will work together to enhance communication lines and make sure that inter-team dependencies are clarified and formalized.  You will work with your managers to anticipate the questions that your lower level managers and individual contributors will have and you will develop forthright answers to those questions.  You will work with your managers to develop roll-out presentations at all levels so that the story behind the changes is articulated well.  This is a lot of work, but it is important to the enterprise.  If you do it well, this can make the message compelling and motivating to your entire department.  Take your time and get it right.

Finally (and this will be hard for you) you must be willing to approach your peers (and your common manager) to propose that team members or even entire teams move into your peers' departments from your own if that makes sense for the encompassing organization.  In other words, you must be willing to do voluntarily what you are asking your staff to do.  Of course, you need to discuss manager and team reassignments privately, with the impacted staff, before announcing them to your overall management team or your department as a whole.

If you work in a good organization, a healthy organization in which the senior managers are really looking out for the health of the enterprise, staff, managers, and teams will move across department boundaries as it makes sense.  If your organization doesn't work that way, it would probably be good for you to look for one that does.

Once you've got all this done, you can tackle the mechanical aspects of the reorganization: co-location, infrastructure changes, paperwork, and so on.

Be sure to make note in the subsequent performance reviews of your staff members the contributions and sacrifices each made during this challenging process. 

So, to summarize:

1. You should undertake a structural reorganization for sound, structural reasons. 
2. You must bring your staff along with your reasoning and gain their acceptance -- not merely their acquiescence. 
3. You and your staff must roll this out clearly and crisply.  Then pause and allow the individual teams to percolate questions up.  Answer the questions forthrightly.
4. Take care of interdepartmental changes.
5. Handle the mechanics
6. Formally acknowledge the work in your staff members' subsequent performance reviews.

Reorganizations done this way will enhance management's credibility and make people understand that everyone's place in the organization is secured first and foremost by performance. Reorganizations done this way will help staff members see that management is looking out for the health of the enterprise while putting real effort into staff development. 

Representing Energy

Now we're getting somewhere. We are beginning to develop a little bit of intuition around the idea of "amounts of energy" as represented by KWH. That's actually a pretty big deal. The concepts are a little abstract and certainly not trivial.

So now let's set up a table so that we can convert between various energy measurement schemes. It turns out that we can find conversion factors between these measurement schemes. They're published on the web. At the bottom of this post, I'll list some references that you can chase down if you want, but let's cut to the chase.

• If we heat our home with propane, we're billed for gallons of propane.
• If we heat our home with natural gas, we're (generally) billed for therms.
• If we heat our home with wood, we buy cords of wood.

Just like we multiply how long something is when it's measured in feet by 12 to get it's length expressed in inches, or we multiply pounds by 16 to represent the same amount of weight in ounces, we can look up multiplication factors for the energy contained in all these sources (and more).

Then, for example, we can convert the number of gallons of propane we used last January into an equivalent amount of energy expressed in our preferred units, kilowatt-hours (KWH).

Here's our table of conversion factors:

Converting to KWH

Fuel Type	Unit Type	Conversion Factor
Natural Gas	Therm	29.3
Natural Gas	Cubic Feet	0.3
Propane	gallon	26.5
Heating Oil	gallon	40.6
Firewood	cord	5900

So if we use 10 "therms" of natural gas, we multiply that by 29.3 and we see that we used an amount of energy equivalent to 293 KWH!

A couple of details:

1. Sometimes utility companies tell you the amount of natural gas that you used in terms of "cubic feet" instead of therms. That's okay. You just multiply the number of cubic feet by 0.3 to get KWH.
2. There are lots of different types of wood used for heating. I found a source that listed over 50 different types. There was a big range of values. I just averaged and rounded off to get 5900 KWH per cord. That's probably not all that accurate. If you know what kind of wood you use, you can see the references below and calculate a more accurate number.

In case you'd like to dig into this some more here are some references:

1. Wood Note that the numbers are given in million of BTU's per cord
2. Propane
3. Heating Oil
4. Natural Gas

Measuring Energy, Part II

Okay, in the previous post we explained why it's not easy to talk about how much energy we're using. Now let's settle on one way to represent amounts of energy.

Let's use "kilowatt hours". We'll abbreviate this as KWH. We see this unit of measurement on our electric bill, so it's at least slightly familiar.

So what is a kilowatt hour? That's the amount of energy used if you run a 1000 watt microwave oven for an hour. That's equivalent to the energy used if you burn a 100 watt light incandescent light bulb for 10 hours OR if you burn a 60 what bulb for 16 and 2/3 hours (16 hours and 40 minutes).

We intuitively grasp that there is a "flow" of electricity from the outlets in our home whenever we use an appliance or other piece of electrical equipment. The number of watts an appliance uses is a measure of how big a flow is needed to run that appliance. In fact, we actually express that flow in terms of a "current" of electricity. The number of watts that an appliance uses is calculated by multiplying the electrical current flowing through the appliance whenever it's switched on by the voltage supplied by the electrical company (voltage, continuing with the flow analogy, is roughly like the pressure of the flow). There is a simple mathematical equation that represents this relationship:

P = I x V

Here:

"P" means "Power"

"I" means "Current"

"V" means "Voltage"

So, Power equals the Current times the Voltage.

Now we can keep decomposing these relationships. We can represent Current ("I") in terms of electrical charge and some other things. We can take these relationships all they way down to some very fundamental definitions of things in the physical world, but let's just keep it simple.

We'll represent the energy that we use or save in terms of the units kilowatt-hours or KWH.

Next, we'll see how to represent all of our energy calculations in terms of KWH.

Measuring Energy

Everything in the physical world is energy. That's what Einstein's famous equation says, and most people with at least a high school education are familiar with the general idea of the equivalence between mass and energy. Yet energy remains a somewhat abstract concept and we often toss the word around in imprecise ways. Why is that?

I think it has to do with ease of measurement. If you ask, "How tall is Cindy?" Most everyone knows how to answer the question.  You have Cindy stand against a wall, make a mark on the wall with a pencil, and then use a tape measure to see, with pretty good precision, how tall Cindy is.

Other things are easy to measure. How much milk do I add to this pancake batter? Well, the recipe says "one cup", so I get out a measuring cup, fill the measuring cup up to the line that says "one cup" and I pour it in the batter.

How much does Joe weigh? Ask Joe to stand on a scale and read the result.

So in each of these cases, there is a simple instrument – a tape measure, a kitchen utensil, a scale – that we know how to use to answer our question.

It's not that way with energy, however. We don't "experience" energy directly the way we do someone's height and weight or a volume of liquid. Recall from our earlier post that energy is the potential to do work. If we direct some energy to something, say we turn on a light, we see the result of that application of energy (the light bulb glows). We don't, however, directly "see" the energy.

Not only that, but we are also faced with several units of measurement for energy, and these units don't have nice neat relationships between one another.  

If we're measuring length in the metric system, we know that 100 centimeters equals one meter. In the "English" system, there are 36 inches in a yard. There are 16 ounces in a pound. We see the subdivisions, the relationships.

For energy, we are faced with scales of measurement that are more abstract. Again, that's because we don't directly "see" energy. In addition, it is unclear how the various scales relate to each other. What are the relationships between, BTU's, Kilowatt-Hours, therms, and so on?  Not only that, but we often apply volumetric units (cubic feet of natural gas, gallons of propane or gasoline) to sources of energy.

So we need to set some ground rules for how we describe how much energy we're talking about. We'll settle on one primary way to represent the amount of energy that we're using (or saving or delivering) and then we'll show a way to convert other units of measurement – such as those we find on our utility bills – into that one way.

Sort of like comparing apples to apples.

Belaboring the Obvious

Two things:

1. The size of an organization and the scope of the projects it takes on are the main determining factors in deciding the type and amount of process that is likely to be useful.
2. In the end, all projects reduce to: figure out what you want to do, do it, make sure it does what you want, deliver it, and maintain it.

In a startup situation, you (had better) have a tightly-knit team of strong producers. They are highly motivated. They understand their goals. They understand not only what each must do, but how what they do impacts their team mates and contributes to the success of the enterprise.

At the other end of the spectrum, you have a giant R&D organization comprised of many specialties. There are technical marketing engineers, software development engineers, hardware engineers, technical writers, program managers, functional managers, software QA engineers, hardware QA engineers, system verification teams, application engineering groups, technical support staff, customer training specialists, internationalization & localization staff, outbound marketing professionals, administrative support staff, IT support... The list goes on and on, and that's just the R&D organization. While this specialization can accomplish great things, it often leads to a certain fragmentation of purpose. People can, for instance, become confused about who is a competitor and who is a potential resource. People can become confused about goals and impacts. A certain level of abstraction overlays everything.

It even takes more words to describe a big, mature organization than it does to describe a startup.

Yet each person in either organization does, at the meta level, the same thing: figure out what you want to do, do it, make sure it does what you want, deliver it, and maintain it.

The reasons cited for process are many, but really, the successful manager puts in place exactly enough process so that the people succeed at this: figure out what you want to do, do it, make sure it does what you want, deliver it, and maintain it.

(Yes, I realize I said that several times.)

At its best, process clarifies purpose and avoids wasted work. At its worst, heavy process -- and it nearly always seems heavy to those who must execute it -- just slows people down.

We start from the original HP mental model assuming that people want to do a good job. Building on that assumption, we try to define and implement the minimum amount of process to enable that.

Inaugural Post

I will post essays here that pertain to modern technical management.  We will discuss the important aspects of a software development process, what constitutes a good process, and what things to avoid.  We'll learn that different sizes and types of organizations benefit from different types and scales of process.

Without Hot Air

If you're interested in a reasoned, intelligent discussion of sustainable energy sources and energy conservation, then I suggest that you take a look at Sustainable Energy -- without the hot air , by David JC MacKay.  For you high-tech types, there is also a text-only, Kindle version available for 99 cents.  Not only that, MacKay says that he didn't write the book to make money, so if you'd prefer, you can download the whole text in PDF from the author's website.  If you  don't want to download the whole thing, you can read individual chapters on the website as well.

MacKay bases his arguments on quantitative analyses of known data.  This makes his arguments much stronger and harder to refute than arguments that are long on emotion and short on verifiable facts.  Since my opinion is that sustainability is too important to be saddled with weak arguments, I certainly welcome dMacKay's efforts.  Although his data focuses mostly on the situation in the UK (that's where he lives), the types of analyses he does may be extended to other developed nations, including the US.  

The book is accessible to readers at most levels.  MacKay puts all the challenging math in a section labeled "Technical Chapters".   

Thanks to PicardOut for the recommendation.

Energy

Energy is the capacity to do work. That work can involve moving things or animals or people around in a car, boat, train, or a plane. The same term, "work", could also refer to heating or cooling a building, lifting people in an elevator, powering a computer, and so on. Basically all of our activities, including life itself, require that capacity to do work -- energy.

Some things require a lot of work to get done. Imagine a large, drafty, uninsulated building on a cold night in northern Montana. Imagine heating that building to a temperature at which most people would feel comfortable sitting still, wearing sandals, shorts, and a t-shirt . That would take a lot of work, a lot of energy. Conversely, some actions require very little energy: a baby is strong enough to bat a beach ball and make it move.

These examples are simple and easy to visualize. Obviously, many situations can be much more complex. How do we deal with that complexity? Well, one way to manage complexity is to avoid qualitative statements such "a lot of work" or "very little energy" and instead measure the amount of energy consumed by a particular activity.

So instead of saying, "It takes a lot of energy to heat our house," we ask, "how much energy did we use to heat our house last winter?" and we try to measure that energy. Then we can compare things more quantitatively. We can determine, for example, how much energy was required to heat a particular house to a particular temperature during the waking hours between November 1st of 2008 and February 28th of 2009. Then we can do the same measurements for a different house and compare the measurements to see which house needed more energy.

Where do we go from here?

Welcome!

In this blog, we'll explore information - quantitative information - about what you can do to become more energy efficient. We will want to be confident that we're making sensible choices. Justified confidence is based on knowledge and understanding and thus involves learning, so we'll begin by learning how to measure and track our energy use.

I hope to make this process fun and interesting.