Freedom of choice – Three Ways DevOps is Revolutionizing Enterprise Software

“I’m free to do what I want, any old time”

- “I’m Free” – Rolling Stones

Here comes the Revolution

Anyone reading the tech news today these days can see that something profound is happening with enterprise software. Venture capital money is flowing into enterprise startups (including Sumo Logic). Old stalwarts like Dell and BMC are likely to be taken private in order to rework their business models. The stock of other software titans are being punished for clinging to old, proprietary models. Most of the old school crowd are trying to improve their image by buying younger, more attractive companies – like IBM with Urbancode. So, exactly what is happening?

Some of it is clearly not new. The big fish in the enterprise pond are always gobbling up Looking upthe small innovators, hopefully improving the lot of the larger company. Why then are some calling this the Golden Age of Enterprise Software? Many will point to BYOD, Cloud, Big Data, DevOps, etc. Personally, I think there is a more subtle trend here. More than ever before, software developers have tools at their fingertips that allow them to deliver software quickly and efficiently, and conversely they are also being held more responsible for the performance of that application.

In the best case scenarios, this has led to highly disruptive and innovative practices that shatter the enterprise software model (take Etsy and NetFlix as two prominent examples). Instead of an operations team passively receiving poorly tested code from unengaged developers, you get highly automated, constantly adapting architectures deftly driven forward by highly-skilled, and highly-motivated DevOps teams. I am not saying something new here. What I personally find interesting here is the disruptive effect this is having on enterprise software, in general. Here are three general trends I see:

 

1. The Rebirth of Automation

This is the DevOps trend that most point to, which is dominated by Puppet Labs and OpsCode (Chef). It seems 90%+ of DevOps discussions start here. The deft combination of flexibility, expandability, and community appeal to the development mindset already conditioned by the open source movement. The idea of “Infrastructure as Code” is a natural extension of first virtualization, then cloud computing. It is so easy to create new “servers” now, that there is no excuse not to completely automated the build and maintenance of those servers.

2. The “Re-discovery” of the Customer

The proven theories of lean manufacturing have long stalked IT in the form of concepts like lean software software development and six-sigma. And some of the DevOps community is trying hard to bring these concepts to IT Operations. Underlying this is the trend towards the importance of consumer and user satisfaction. The switching costs are so low, and the modes of feedback so verbose, that companies can no longer afford to ignore their users. This means that the lessons learned by the automotive industry – eliminate everything that doesn’t provide value to the customer – are now essential for the IT industry. This is not good news for legacy software companies associated with the image of uncaring, passive IT departments. It is also fueling the rise of cost-effective solutions like SaaS (are million dollar software solutions gathering dust providing customer value?).

3. Measure Everything

Example Etsy GraphAs the DevOps movement takes on more of Lean thinking, then the importance of measurement rises. In the seminal book on DevOps – “The Phoenix Project” – monitoring is a central theme. We see this in the real world with Etsy’s efforts. They are monitoring thousand of metrics with statsd, providing insight into every part of their application. So, what’s different here? In the old world, what you monitor is dictated to you by software vendors who deliver generic metrics to fit all customers. In the new world order, developers can add metrics directly into their logs, or through a tool like statsd, and monitor exactly what they want to. In the spirit of open-source, it is more important to get what you need (custom, relevant metrics), rather than get it in a pretty package. In essence, this means that the old Application Performance Monitoring (APM) tools may be headed for a rude awakening. Why do you even need an APM tool, if you can pump custom metrics to a generic graphing tool, or a log analysis tool? Well, I am not sure that you do…

These points are only one small part of what is changing, and I don’t claim to know exactly what the future bodes for IT software vendors. What is obvious, though, is that the barriers to entry for innovation are low, and the money willing to chase it is plentiful, so this is definitely a golden age – just not for the old school, perhaps…

* Picture of statsd graph from Etsy’s blog – Code as Craft

3 Reasons why DevOps isn’t changing IT faster

iStock_RobotRaceThe recent article from Luke Kanies (from Puppet Labs) on Wired.com really got me thinking. Similar to Luke, I have had an interesting vantage point to observe the changing nature of systems administration – spending time as one myself – over the last decade or so. From my graduate physics department, to Loudcloud, EDS, BladeLogic, BMC, and now Sumo Logic, I have seen the best and not-so-great IT shops and how they operate. Not all of those IT teams I saw in action were, or are now, adopting best practices like automation and DevOps. The trend that Luke points out means that operations teams that continue to languish in constant firefighting mode, relying on ad-hoc scripts and the sweat off the admin’s brow, are becoming more obvious for being clearly out of step with the direction of the industry.

So, why aren’t all operations teams, and the techies themselves, falling over each other to embrace tools like Puppet, SumoLogic, and other DevOps/Automation tools? Clearly some organizations are embracing them. Why not all of them? I have few of my own ideas here, and I would like to hear yours as well.

1. The move from “Artist” to “Manager” is not natural

Back when I started in IT, most IT admins “owned” a small collection of devices or applications. Server admins owned a handful of servers, database admins a few databases, network admins a few switches and firewalls, etc. They controlled access to their systems jealously, and took personal pride in their operation. They were artists, and their systems, and the way those systems were managed, were an art form. As IT budgets have shrunk, and the load on IT increased, this level of care is impossible.

Yet, you still find many admins jealously guarding their root access privileges, instead of moving to a shared responsibility model with other admins. Why? I think it is the same reason why I feel such satisfaction after cooking a meal, building new shelves in my garage, or fixing a leaky faucet. I did it myself, and it feels good to start and finish something. Participating in automated processes can be deeply unsatisfying. That is why admins need to learn new skills, and find new pride in the quality of their automation or find satisfaction in steadily improving quality.

2. Using Automation may seem like losing control

One conversation from my IT past sticks in my mind more than any other. I was on site with a customer trying to explain the benefits of automation to a group of systems administrators. One system admin floored me by insisting that she could more accurately, and more quickly, make changes to 20 UNIX servers than I could ever do with automation. It was like some modern version of John Henry calling me out to some man vs. machine contest, and subtly decrying the inhumanity of my automation tool. I can’t even remember my answer now, but this perspective is at the root of much of the push-back against automation and DevOps. Instead of looking at the business outcome – better experience and value for the customer – some frustrated system admins see these new ideas as a direct affront to the quality of their work. This is precisely why I think the fundamental shift in DevOps is from a internal IT focus to an external customer focus. That way admins can measure their success by customer impact. Not an easy change to make, but it is essential for IT’s continued relevance.

3. Change seems hard/bad/unnatural/unneeded

Isn’t the root of the resistance here really the natural tendency to resist change? On the other hand, how many times have operations teams been assured that the latest IT fad will reduce their workload and improve quality, only to see the opposite happen? So what’s different about DevOps? I think I could write a whole blog entry just on that, but a few things come to mind. First, the focus is on customer value, which greatly simplifies priorities. Second, it’s all about outcomes, not process for the sake of process. Finally, it is all about continuous improvement driven by the experience of the people on the frontline. This means that admins must be rewarded going forward for doing things that increase customer value, rather than putting out fires or pleasing angry executives.

So, it all comes back to culture – surprise, surprise. I think this will be the primary challenge of DevOps going forward. How do we overcome the IT culture so resistant to change, while providing an attractive way for all of those systems administrators to breathe easily in their new roles?

Absence Makes the Heart Grow Fonder – Right? …

Thanks to all of you who have read my blog in the past and have taken the time to come here again… Thank you! The last few months have been crazy. In short, I sold my house in Washington, DC, moved my family to Silicon Valley, and started working at a startup called Sumo Logic. I wrote a blog on the Sumo Logic site explaining my decision.

So, the last few months have been insanely busy, stressful, and exhilarating. But, I also neglected my new project – blogging. Bottom line, I plan on starting up again now and making up for lost time. And I hope that you will continue to indulge me and get some enjoyment out of my attempts to make sense of the world of technology.

Thanks again for your readership and I look forward to our future discussions.

DevOps needs a layered approach – Not only process or automation

With any new, emerging area the tendency is for advocates of each new approach to attempt to invalidate or minimize all earlier approaches. Sometimes this is appropriate, but rarely is progress so clear cut. In this vein, I would like to comment on Phil Cherry’s post on DevOps.com. First off, I appreciate Phil’s addition to the discussion. I think his delineation between automation approaches is very interesting. However, the devil is in the details. Here are the highlights of my views on this:

Package-based automation

As a former BladeLogic guy, I would be remiss if I didn’t correct a few points in Phil’s analysis. Phil may be confusing OS Vendor packages (RPMs, MSIs, etc.) with configuration management packages. Systems like BladeLogic build packages based on some sort of configuration object system. In other words, the server is represented as a set of configuration objects (a registry key, setting in a config file, etc.) in a particular state. The packages are usually represented as desired states for configurations based on that same object model. There is no reason that those packages have to be applied “all in one go”, since packages can be chained and included in larger jobs with decision conditions. That said, I agree that this type of automation is per-server based, for the most part.

Application Understanding

I do agree that Phil’s definition of automation models don’t understand multi-server dependencies or really know what an “application” is. Phil does ignore in this context that there are other automation approaches that do bridge this multi-system approach by building on the automation platforms. In particular, the trends within virtualization and cloud have pushed vendors to create multi-server, application-focused automation platforms. You can find solutions with established vendors like BMC or VMWare, with open-source platforms like Puppet with OpenStack, as well as with startups like ElasticBox. Bottom line, it is vast oversimplification to limit an overview of DevOps-capable automation to automation tools with a server-heritage only. This area of automation is clearly evolving and growing, and deserves a more holistic overview.

How does process fit in?

As John Willis, and others, have said many times before, culture and process are just as much a part of a devops approach as basic automation. So, it appropriate for Phil to end with a process-based approach. Clearly rolling out an application requires an understanding of the end-to-end process, how steps are related, and how services are dependent. I do feel that Phil left out a few key points

Process Management and Deployment Automation are not the same

I feel like Phil blurs the line between managing the process of release, which is a people-process issue, versus managing the deployment of an application. The latter involves pulling together disparate automation with a cross-server/application-focused view. Process management, on the other hand, deals with the more holistic problem of driving the release of an application from development all the way to production. They are both needed, but they aren’t the same thing.

What about coordination

One of the biggest drivers of DevOps is getting Dev and Ops to coordinate and collaborate on application releases. This means driving Dev visibility forward into Ops, and Ops visibility back into Dev. It isn’t just about creating well-aligned deployment processes, but also managing the entire release process from the code repository to production. This means we need encapsulate pre-prod and prod processes, as well as non-system activities (like opening change tickets, etc.).

What about planning

Releasing and managing applications is just about the here and now. It is also about planning for the future. Any process-oriented approach has to allow not only for the coordination of deployment processes, but also needs to allow for the establishment of clear and flexible release processes visible to all stakeholders. In particular, a process management system should provide visibility to the decision makers, as well as the executors. Applications clearly affect non-technical activities like marketing and executive planning, so it is important that business leaders be able to understand where releases are out, and when important features are likely to arrive.

What we need is a layered approach

Bottom line, we need to solve all of the application release issues – process, deployment, and automation. In the spirit of DevOps, those “layers” can be solved somewhat independently with loose coupling between them. We drive the people-process coordinate, encapsulate all of the complexities necessary to deploy an application, and then drive the low-level automation necessary to actually implement the application. All of these work together to create a full approach to Application Release Automation. Any solution that ignores a layer risks solving one problem, and then creating a whole new set of them.

IT is War : DevOps & ITIL through the lens of military history

Climbing into the DevOps vs. ITIL debate is like stepping into a minefield, as least from my vantage point. Both have serious-minded proponents, and engender the kind of passion that lesser methodologies only dream of.  But, after this very issue has come up multiple times recently, I really felt compelled to think more about it. Most of the attempts at reconciling the two haven’t resonated with me. So, I have tackled this issue the way that appeals the most to me – using military analogies.

Military history is probably my favorite part of the huge topic of history. I particularly enjoy understanding how advances in weapons, tactics, and organization have influenced the course of events. The evolution of IT over the years has been a lot about adjusting process and tactics to the available tools and competitors. The evolution of military strategy boils down to the same thing, so I think a quick look at the evolution of military tactics over time can shed some real light on the DevOps & ITIL debate.

Discipline and Organization beats Chaos

Success in war usually favors the bold – those who embrace new technology and changemacedonian pike phalanx their tactics to deal with new situations and threats. A few days ago I watched a documentary on the massive defeat of the invading Persian army by the Greek city states in 490 B.C.. What many people don’t know is that it was the technological advantage of iron lances and large shields, and the highly coordinated group maneuver called the Phalanx that gave the Greeks the edge. Moving as one unit, with shield interlocked, spears pointed front, the Greek hoplites were an unstoppable force cutting a swath through the Persians

Fast forward a thousand years. The Europeans, with their endless wars of the 17th, 18th, and 19th centuries made an art napoleon-army-salamanca-spainform of the highly trained foot soldier with a musket. Napoleon represents the pinnacle of that art. Coordinated volleys of musket fire from trained soldiers, supported by well-placed cannon, and fast moving light cavalry, could wipe out less well-equipped and trained troops. Napoleon also worked at a scale unheard before of his time, mastering logistics for hundreds of thousands of troops in the field. With his armies, Napoleon was able to dominate the Europe for 20 years, and his ideas lasted longer.

Tools can obsolete Tactics

The effectiveness of these large scale maneuvers was upset by the advances of the 19th century, particularly during the U.S. Civil War. The vastly increased accuracy of rifled muskets and cannon, and the high rate of fire afforded by caplock (percussion cap), meant that the magnificent infantry charges of the 18th century only made for easier targets and mind-blowing casualty rates. This reached a peak with the mindless violence of trench warfare in World War I.

MarineRaiders

And again, armies adapted. Along with the introduction of the tank, it was small group tactics that won the day and broke the stalemate in World War I France. Instead of ordering breathlessly stupid charges into the face of machine gun fire, small squads of soldiers could adapt to the circumstances and more rapidly advance.  World War II continued the refinement of those tactics. This didn’t mean that large scale coordination wasn’t still necessary. Artillery and air support still needed to be coordinated with the soldiers on the ground. Tanks and mobile infantry could move quickly and pack a powerful punch (which the Germans perfected with Blitzkrieg).

Very interesting, but what’s the point

Other than indulging my need to geek out with talk of weaponry, this rapid flyby of history does have a point. Successful armies over time have adapted their tactics and tools to meet the threats at hand. The ancient Greeks and Napoleon used organization and discipline to overwhelm their enemies. In the face of the devastating weapons of the 21st century, successful armies used more flexible and fast-moving tactics to dominate their slower moving enemies. All of these militaries adapted to their circumstances and made the most of what tools they had.

I don’t think IT is all that different. ITIL made a lot of sense when confronted with the chaos of IT operations, and the need to provide stable services for a business questioning the value being derived from their investment. With well-documented processes and coordination, IT departments could confront and conquer the chaos.

Conversely, DevOps has arisen in the wake of the pressures exerted by a hyper-competitive business environment and hard-to-please users with no end of choices. Just like the soldiers facing rifled carbines at Gettysburg and those facing machine guns nests in French trenches, IT operations teams trying to please the 21st century Internet user can’t march into battle with the highly coordinated, but rigid, maneuvers of ITIL. By the time they perform the service management equivalent of a pivot, the business has lost customers and revenue. In the word of U.S. General George S. Patton of World War II fame –

“A good plan violently executed now is better than a perfect plan next week”.

On the other hand, DevOps teams need a backdrop of coordinated services (e.g. cloud services or automation) to enable their agile methods, just like the U.S. Marines in World War II needed artillery and aerial support.

So, what’s the takeaway? We should never compare methodologies in a vacuum. Any methodology needs to solve today’s problems, not yesterday’s. The whole point of methodology is to provide a way to repeat the successes of the past. So, you need to find the Von Clausewitz that has succeeded where you want to succeed, and follow their lead. The methodology that best helps you meet your goals today is the right one every time.

Users want Conversations, not Monologues – How does that change DevOps and APM?

My blog last week about the new reign of the end-user has started some interesting discussions, both online and off. In particular, I had a Google Hangout with some of my colleagues, and we got on to topic of end-user monitoring and DevOps. Chris Dancy (@servicesphere) riffed off of my comment about the low switching costs of moving from one web app to another, and talked about empowering users. Users (That is ME, YOU, US) are no longer dependent on their business’ IT department, their ISP, or techie neighbor to get what they need from technology. If we don’t get what we want from our provider, we will do it ourselves, or got to a “competitor”. The “walls” to personal innovation just aren’t there anymore.

I made the comment that what we really need is “End User Activity Monitoring”, which my good friend Dave Williams has talked about extensively. The basic idea is that, to give the user the experience they really want, we need to be able to measure that experience no matter where they are or what device they are using. We can extend that “monitoring” in a more human way by asking “questions” as opposed to just monitoring bits and bytes. Jason Frye (@fryfrye) responded to this in the Google Hangout with the great observation that we can take it one step further and show them their own data. The provider can be proactive and let the user know that there are problems, and that a solution is being worked.

So, as I was thinking about this, it really came together with one idea. Humans want to converse, not be talked at. I was amazed to learn that my 18-month learns language not simply by hearing it, but by listening to, and watching, a human speak it. She want to talk WITH me, even before she can actually speak. And I think it goes to technology as well. I want my world to respond to me. My experience with certain brands (like Amazon.com and Apple) has become almost familial (which makes Kindle and iPad seem like a slightly awkward argument over dinner). When Amazon asks for my review, I feel included. Even though I am annoyed when a shipment is delayed – telling me somehow makes it less annoying. Even with consumer gadgets, it applies. I love my Garmin watch when I run, not because I really do anything with all that GPS data. I just love the second by second feedback on my performance.

So, what does this all have to do with DevOps and App Performance Management (APM)? Well, I think it is very easy to thinking of the relationship with the “user” as a one-way relationship. Even when trying to make the user happy, it is more a push, then persuasion. So, what is DevOps tools could take not only live APM data on the measurement of end-user performance, but also more subjective data from the user, into account. What if the application proactively warns you of trouble, and makes suggestions. What if the application actually suggests other things you might like (like other applications). I think this concept is pretty well understood in retail (like with Amazon.com or Zappos.com), but I am not sure that it has really made its way into the thought behind DevOps and APM. The Social IT revolution is no less powerful than the DevOps movement, and it is in our (IT professionals) best interest to bring them together. User not only need a relationship with the help desk – they need relationships with IT operations staff, Developers, and all the other powers behind the curtains. Clearly some internet companies have mastered parts of this, but I would really like to see this interactive empowering of the end-user to work its deeper in IT. We are ALL users after all, aren’t we? So we should understand how they feel.

The Mythical Application Owner – And Why They Matter to DevOps

I have been on both the customer and the vendor side of Application Management over the last decade. So, I was surprised how much trouble my team and I had really defining the Application Owner as a sales target. With our collective backgrounds, I expected it to be a breeze. Of course the CxOs, VP Operations, and others are well know entities. So why is the application owner so difficult to find? One reason is because the definitions are all over the place.

For example, one definition on the IT Law Wiki says:

An application owner is the individual or group with the responsibility to ensure that the program or programs, which make up the application, accomplish the specified objective or set of user requirements established for that application, including appropriate security safeguards.

Snore… I could be in charge of Microsoft Office for my company and fit that technically focused, but inherently logical, definition. NIST has a similarly technically oriented definition. Now in all seriousness, we can see that part of the problem is that I really mean a business application owner, not a technical owner. In that vein, a definition much more to my liking can be found on this blog entry from Nick Spanos on his Lean IT blog. In the spirit of lean methodology, he brings in business/customer outcomes, business processes, etc.

So, who cares? I think anyone who cares about DevOps should care. As DevOps expands from a grassroots movement to a business-changing phenomenon, it needs to continue to develop thought patterns that appeal to those footing the bill. This is where so much good thinking in IT falls down – the inability to articulate the value to the business. That said, I don’t think it has to be complicated. I see two over-riding characteristics:

They live, eat, and breathe application revenue

For every revenue-generating application out there, there is someone being measured on that revenue. For a small company, that might be the CEO. For a Fortune 500 company, there could be dozens of applications, each with an owner. Regardless of industry, somebody goes to sleep at night worrying about whether whatever.com is measuring up to expectations.

They care (or should care) about customer value

Customers pay the bills, so worrying about revenue means obsessing about customers. I think that one of the common themes for companies successfully implementing DevOps is the over-riding need to deliver customer value. This is coincidently very much in line with lean methodology – which is why lean and DevOps are so good together. An application that is always in danger of losing customers and competitiveness is fertile ground for DevOps. If your application isn’t in a competitive space, why would you even bother?!

Pretty much everything else is going to be different by industry. An application owner at SuperCoolStartup dot com will likely be very different than the app owner at BigRetail dot com or the MBA educated person running SomethingOrOtherFinancial dot com. What they should have in common in the commitment to increasing revenue by increasing customer value.

Now, one caveat. Not every application has revenue. I would argue that the lack of the profit imperative makes a wrenching change like DevOps much more difficult, but you can insert “mission” or “fund raising” for most of these points.

So, why does this matter? DevOps practitioners, and IT organizations in general, have an unprecedented opportunity to make IT matter to the business more than it ever has before. IT can become an enabler for revenue, rather than a pure cost center. However, to do that, they need to learn to express the value of DevOps in revenue and customer-focused terms, and begin to make the case directly to the application owner. Over time, I think the pendulum of power is swinging from the CIO and VP operations to the application owners. So, IT needs to make some new friends!

IT has been deposed – Long Live King User

Most of us (if you are reading this, you are a self-selected subset) are fascinated with trends in technology and IT, which is great. The problem is that, with all the activity and progress going, it is easy to miss the forest for the trees. I think it is really instructive to take a step back sometimes and ask the big questions. Like this one – Are some of the major technology trends of our day – Cloud, DevOps, App Performance Mgmt (APM), Smart Phone and Tablets, etc. – completely separate trends, or all part of a larger narrative? I don’t ask that to reduce the power of those trends – I just think that it is worth asking if something more seismic is going on.

So, to answer my own annoyingly rhetorical question – I think there is a bigger trend. The user, in whatever context in which he, or she, operates, wields more power today than ever before. Entire industries are being built for the sole purposes of making users happy – or least making them think they are happy. And for those verticals that aren’t so obviously pandering, any competitor that ignores the experience of their end-users is bound to crash and burn. And I am not just talking about reverse voyeurism of over-sharing Facebook users and the beauty of Apple products. Staid and complacent industries everywhere are being shaken to their core as users demand the same level of experience in the business world that they experience when interacting with applications in their personal lives. So, here are a few of the trends I follow, and how I think this “mega-trend” ties them all in.

DevOps and Agile Development

One of the main purposes of Agile Development is delivering customer value. The first of the twelve principles of the Agile Manifesto is “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software“. Agile development puts the customer on a pedestal, and works to make the customer satisfied. Why? Because the customer pays the bills, that’s why. And not only do they pay the bills, in the world of the Internet users can quickly take their business elsewhere when they are not satisfied. At the end of the day, on the major objectives DevOpsis to push this maniacal focus on delivering customer value all the way into IT operations – where it is has been sorely lacking.

Application Operations and Application Performance Management (APM)

As the less hip and more staid cousin of DevOps, APM has an even easier case to make about the primacy of the end-user. In the past, APM has been primarily about bubbling up complex application performance data from a complex application architecture, and trying to make sense of it. While that kind of deep-dive analysis is important, it can be misleading and distracting. Recent trends in APM, particularly around those same Internet applications driving the DevOps movement, are focusing more and more on the end user. With the “switching costs” so low today, users will leave for the competitor’s site if there is on average a 250 ms or more difference in response time. That is 1/4 of one second. In only 2009, a study Forrester put the acceptable wait time at 2 seconds. Today, in the case of high-performance sites like Google, even 400 ms is too long. Bottom line, if your customers are receiving sub-par response times, you are probably losing them to competitors. It really is that simple. And if an IT department isn’t measuring end-user performance, they are essentially allow their company to stumble blind into a pool of sharks/end-users. Good job IT.

The Business User’s long sojourn in the wilderness of crappy IT support is over

This trend is probably the most disturbing to an IT department and most exhilarating for any IT user. Business IT users are no longer satisfied with flimsy laptops made by do-it-cheaply hardware manufacturers and web applications with appalling user interfaces, seemingly built for Windows 3.1. Apple, Google, Facebook, and others have taught them that technology can beautiful, easy to use, secure, AND effective. The IT departments’ insistence that the need for advanced functionality and security necessitates technology that induces suicidal thoughts and dulls the mind into oblivion, no longer stands. IT users want to be free. They want to bring their iPad, Android phone, and Mac Book to work. Most IT departments don’t really like the stuff they peddle anyway. Now they just need vendors that actually deliver them products that don’t make their users scream in frustration.

Social Media is the tool of User Domination

And how does King or Queen User rule? Through social collaboration tools. Social media is not just about showing pictures of your cute toddler (guilty…) or trying to remove tags from embarrassing photos anymore. Social media is about capturing the best ideas from the people who stand to benefit the most from their realization. It is about encouraging healthy competition and ego-boosting for the sake of information sharing and content creation. And, most of all, it is the platform of user rule – Social IT. And it can’t be ignored. Countries creating “fake internets” and companies with draconian IT policies won’t stop the change. Users always find a way to flaunt the rules, like they have since the dawn of time.

So, is this a good thing? I believe that it is. I think the the essential lesson here is that business has always been about delivering value for the customer. The difference now is that customers actually know how much value companies, organizations, and countries are delivering. Keeping “the man”, the president, and the boss accountable. That is definitely a good thing.

Kaizen and the Art of DevOps Automation Maintenance

And now we come to the most “boring” part. Right? Maintenance. The death of joy for the innovator. Or is it? I don’t think so. Continuous innovation is at the core of DevOps and Lean methodology. Maintenance is essential to keeping the spirit of DevOps strong, and automation that isn’t improving will grow stale and useless.

So, let’s review the IT Automation Curator’s job description on last time:

  • Collect existing automation, and then Catalog it where others can find it (See Part 2)
  • Develop new automation based on requirements from IT (See Part 3)
  • Train others on how to use the automated processes (See Part 4)
  • Maintain the existing automation

Going back to Lean methodology, we can look to the idea of Continual Improvement or Kaizen. There are 3 main areas from Masaaki Imai‘s 1986 book Kaizen: The Key to Japan’s Competitive Success.

  • Reflection of processes. (Feedback)
  • Identification, reduction, and elimination of suboptimal processes. (Efficiency)
  • Incremental, continual steps rather than giant leaps. (Evolution)

Use Metrics and Reporting for Feedback

How can you improve if you don’t know how far you’ve come and how well you are doing? That’s like going on a diet without ever weighing yourself or even looking in the mirror. Healthy weight loss involves such small changes every week that it would discouraged if you didn’t look at changes over the long term (I speak from experience). So, why do so few companies and automation vendors include metrics and reporting on automation efficiency?! The whole point of implementing automation is to reduce waste, reduce costs, and increase velocity. But you have no context for understand if you have succeeded if you don’t have metrics and reporting (I talked about this in a previous post).

Ruthlessly eliminate sub-optimal processes

The whole point of this process is get rid of wasted effort and time – muda. The hardpart here is that once you agree to continually improve your automated processes, you have to be ruthless in your evaluation of their efficiency. That means there are no sacred cows. Just because somebody smart and dedicated invested hours of their life into creating something, doesn’t mean it can’t be improved or scrapped entirely. The whole team has to be dedicated to, and incentivized towards, efficiency and continuous improvement. This point is important – these kind of improvements come from the grassroots – not the top. If the people in the trenches aren’t bought into continual improvement, it won’t work.

Baby Steps, not Leaps of Faith

I know the hardest part of this approach for me is the gradualism. I like to grandiloquently solve grandiose problems with lofty and visionary solutions. The problem is that most of those involve large amounts of kool-aid, and they are never finished. The truly mature IT organization has to keep their eye on the goals of the business, and relentlessly reduce muda – step by tortuous step. We can refer back to the weight loss analogy. You lose weight through all of the small victories – Do I really need that donut? One serving is good enough. But bringing us back to our first point – small victories only show up as victories when you can measure your long term progress. Otherwise it looks like tentative, timid, risk-adverse behavior.

And so, we reach the last chapter of my IT Automation Curator series. It has been a lot of fun writing it, and I hope that you enjoyed it as well. I am looking forward to continuing to explore how the proven methods of lean and agile can be applied to DevOps and Operations overall.