Hands-On Machine Learning Using the R Language

Hello, folks. This post is for those who plan to attend my “Hands-On Machine Learning Using the R Language” session at the Better Software / Agile Dev / DevOps conference on Wednesday, November 8th, 2017.

You will want to install R and the R-Studio IDE in advance. The session will assume you already have these installed.

Below are the data files and code I will be using during the presentation.

house_prices_in_19083

home_price_preds_BAD.R

Raw SMS ham / spam data

spam_analysis_BAD.R

 

Your Annual Plan Is Already Wrong

Your year-at-a-time plans and goals are already wrong, but if you know this about them, they are still worth making, provided you don’t take them too seriously.

Another January is upon us, bringing us into the season of annual planning exercises. Annual plans are the organizational equivalent of New Year’s resolutions – we feel obligated to make them, even though we are unlikely to keep them.

In most organizations, there is little practical difference between the end of December and the beginning of January, and so there are few objective grounds for over-weighting this part of the year as a time for course corrections and new ventures. Yet, the unblemished calendar invites us, like fresh snow, to create something remarkable upon its canvas. We aspire to high art, and this seemingly demands a plan that will keep us from wandering around in circles, doubling back over our tracks, and generally making a mess of the yard.

My friends, the mess is unavoidable. Success is rarely achieved in a straight line.

We should no longer be surprised by the annual objectives which seemed so central in January and were scarcely relevant by June. We can hardly doubt that our signature accomplishments in a given year were often unimagined at the start of that year. If we are attentive and realistic, we recognize that these discrepancies between planned and actual are as much the norm as the exception.

Humans are poor planners and feeble fortune tellers, and pretending otherwise amounts to damaging obstinacy. Let us instead embrace our obvious limitations and our well-documented fallibility, so that we can prepare and act accordingly. We are equipped with lanterns, not flashlights. We can see what is close at hand, but most of the path before us will remain dark until we are right on top of it.

The rise of agile software development methods is an outgrowth of this recognition. Agilists accept that they cannot see the future very well. They don’t try to look very far ahead because people simply aren’t very good at doing so and because changing circumstances often invalidate plans made on large time scales. Instead, agilists work in small iterations, experimenting and successively refining until they have made their way to their goal. With each iteration of their output, they evaluate what they have done so far and what the next best thing to do would be. They keep long range goals in mind as a guiding star, but they navigate in the moment by more immediate and accessible beacons. If they make a mistake, they won’t go very far before giving themselves a chance to correct it and get back on course.

The corollary lessons for annual planning exercises, including personal goals, are clear. If your planning cycle is annual, then your plan is already wrong, not because of its established content but because it doesn’t contain realistic mechanisms for error correction and adjusting to change. The winds of the world are abrupt and fickle, and we can’t keep sailing in the right direction by changing our tack on a merely annual basis.

Instead of planning annually, organizations should:

  • plan for multiple time horizons, such as “within three months,” “within six months,” “within a year,” and so on
  • make near-term plans the most detailed (because these cover the ground we can actually see well)
  • make long-term plans more directional and less defined
  • revisit plans regularly, multiple times within a year, for review and adjustment
  • accept that plans sometimes turn out to be wrong, and that there is no shame in adjusting to new information or realizations

The annual plan is a fossil, left over from the days in which things moved slower and we knew less about ourselves. It’s time for us to evolve into planning cycles that are more natural, realistic, and credible.

Defining a Strategy: A Useful Test

You don’t have a strategy until somebody thinks it isn’t a good idea.

Here’s a simple test for any strategy you’ve defined – can you think of an organization that wouldn’t be following the same strategy? If not, then you’ve only defined a goal, not a strategy.

“Cutting costs” is not a strategy. Everybody wants to do that. If you say something about how you will do that (what you will consciously sacrifice in order to cut costs), and not every organization would have the same answer (hint: “eliminate waste” and “streamline operations” are too generic / formulaic to count), then you are on your way to a strategy.

A strategy must give some indication of how you will be different from your competitors. You don’t have a strategy until somebody thinks it isn’t a good idea. Having the right strategy is a much more complicated matter, but the test above (i.e., would anybody do otherwise?) will keep you from settling on generic pablum that lends only the illusion of strategic thinking.

One-Time Raise or Multi-Year Bonus Increase?

Meet Victor – he currently makes $100k annually with a 10% annual bonus. He has learned that he is getting a promotion next year, and will be receiving additional compensation. He can choose between two options – a one-time 5% raise or having his annual bonus upped to 15% for next year and all subsequent years. Which should he choose?

At first glance, many would choose the bonus, based on the idea that the bonus is a recurring benefit. In the first year, some would say, the additional 5% bonus would be equivalent to the additional 5% salary, but in subsequent years, the recurring benefit from the increased bonus (15% versus 10%) would make the bonus the more attractive option.

Let’s do the math and examine this claim. Currently, his $100k salary and 10% bonus bring him $110k per year. If he takes the 5% salary increase for next year, he will make $105k, and his 10% bonus will tack on an additional $10,500, bringing his annual haul to $115,500. In contrast, if he takes the extra 5% on his bonus, he will continue to make $100k for his salary,  but his bonus will rise to $15,000, bringing his annual compensation to $115,000 – $500 less than what the salary increase will bring.

What about subsequent years? Will the recurring benefit of the increased bonus make it the more profitable option in year 2? For the sake of realism, we will assume a 3% annual merit increase on Victor’s salary, but we will assume this to hold true across both of the scenarios we are tracking. So, if his initial move was to go with the 5% salary increase, which brought him to $105k in salary, then a 3% merit increase in the next year will bring him to $108,150. Once his 10% bonus has been paid, he will have made $118,965 for the year.

What would have happened in year 2 if he had gone with the bonus? His 3% merit increase would have brought him to $103k in salary. Then, his big 15% bonus would kick in, giving him another $15,450. Yet, the two together come to only $118,450, or $515 less than he would have gotten in year 2 had he taken the initial salary bump instead of the bonus. The gap has now gotten wider, from $500 the first year to $515 in the second.

You can probably see how this will play out. The gap will just keep getting wider, because he is always 5% higher in salary than he otherwise would have been for a given year, and then his bonus is calculated on top of that. In year 3, assuming another 3% merit increase, the initial salary bump results in a salary of $111,395 and a bonus of $11,140, yielding $122, 535. In contrast, the initial bonus would bring a salary of $106,090, and a bonus of $15,914, yielding a total of $122,004. The gap is now about $530 in favor of the salary increase.

The math here is pretty simple, but maybe not terribly intuitive. Many folks seem to think that the recurring bonus is the thing to get excited about, but they don’t realize that when salary increases are calculated this way (and for many people, they are), that one-time salary bump has even greater recurring benefits. Food for thought the next time you have to negotiate compensation …