Agile – Tom Cabanski

Interview on DevOps

From time to time, I get the opportunity to talk to industry reporters about agile and DevOps. Today, I was interviewed via email for the first time, which turned out pretty interesting. Here are the questions and answers from that interview.

Please briefly describe how the company is using DevOps, including when it began, which DevOps tools and for which types of projects.

We see DevOps as a culture that encompasses people, practices, tools and philosophy. In that sense, it has become central to everything we do to develop, maintain and operate our e-commerce sites for Blinds.com, JustBlinds.com, AmericanBlinds.com and, of course, Home Depot custom window coverings. Infrastructure is code that evolves in concert with our other software components. DevOps happens inside our agile development teams and often draws in specialized resources from our operations group. It also happens inside our infrastructure group and often draws in developers. It’s part of our DNA.

The tools aspect of it is pretty standard stuff. We use Git and GitHub for source control. All our application and infrastructure code is there. Puppet helps us with rolling out and managing servers. Our backends are mostly .NET so we use Octopus Deploy to help with rolling our code. TeamCity is in the middle of our development process and code there is used to expose deployments and tie them together with builds. Logs are mostly managed by Splunk though we’ve played with an ELK stack for this as well. Nagios is used for infrastructure monitoring. NewRelic is our app monitoring tool and we depend on it to alert us to problems with the user experience. All our alerts get fed into Pager Duty for escalation management. We’ve been experimenting with Consul for discovery and config. We’re also experimenting with Docker. What’s holding us back there is .NET on Windows. Of course, that story is changing with .NET Core and Windows 2016 on the horizon so we have high hopes for Docker as a next step.

What were the business drivers for deploying DevOps?

Agile drove our adoption of DevOps. Our adoption of agile was driven by our organization’s culture more than anything else. One of our key values is “experiment without fear of failure”. Another is “improve continuously”. Over the years, our whole IT process had gotten into that uncomfortable place where limited resources lead to a difficult relationship with the rest of the business. They saw us as standing in the way of all the cool experimentation and improvement they wanted to do. Agile helped us break down the walls that had developed and form a true partnership for innovation. DevOps is a necessary part of the agile process. How can you innovate constantly if deployment requires an over-the-wall handoff and lots of manual intervention to get done? If operations and infrastructure are not intimately involved in the process, how can you support and manage it once it gets into production?

What benefits has the company seen from DevOps?

DevOps enables agile, which allows us to continuously improve. It’s a big part of how we were able to deliver on all the promises of our new e-commerce platform, which lead directly to the acquisition by Home Depot. It has allowed us to continue to innovate and thrive inside a Fortune 50 corporation and take on new challenges to help drive innovation outside of the custom window coverings business. DevOps is like oxygen for the agile process. Without it, it’s very possible that we would have ended up with “agile in name only” where agile terminology is used but nothing really changes and the organization doesn’t see the kind of exponential increase in innovation that we’re benefited from here.

Any challenges of deploying and using DevOps, and how were they addressed?

Our biggest challenges revolve around security and compliance especially now that we are part of one of the largest retailers in the world. We’re still learning how to deal with all that when it comes to sharing responsibility for deployment and infrastructure between developers, infrastructure and operations engineers. We’re constantly tempted to solve these problems with handoffs and work hard to avoid that. Now that we have trust across all the impacted groups it’s much easier to work through them and come up with ways to address compliance without undermining the velocity of innovation.

Quickie Review of “The Evolution of Everything: How New Ideas Emerge”

“The Evolution of Everything: How New Ideas Emerge” by Matt Ridley is a very thought provoking book about how our world has been largely shaped by bottom-up trial and error evolution rather than top-down intelligent design. Along the way, the author challenges assumptions about everything from how religion created morality to the role and impact of government in education. In the end it is a bracingly optimistic look at how our civilization got here and what the future may hold.

Think of the incredible web of economic activity that makes the apparently simplest of products, the humble pencil, available as explained in the famous story, “I Pencil” (check it out on YouTube if you have not seen it before). Nobody designed the entire supply chain. No single person or small group of people direct it. No government requires it or builds it. Even so, tens of thousands of people somehow coordinate their activities to make it possible.

Think, too, of invention. Edison invented the light bulb, right? What do you think would have happened if he died the week before his discovery? Well, Humphrey Davy started the march towards an efficient electric light many years before and Edison was just the best funded of those pursuing the task. It is certain that if Edison had failed, someone else would have stepped forward likely mere months later. This pattern, the almost inevitability of invention, can be seen in just about every case. Steve Jobs did not direct the invention of the smart phone. Apple just got there a bit sooner and with a bit more innovation than countless others. If Steve had died a few years sooner, Android may have been first or someone else would have stepped up with an almost identical device. The Wrights flew one of the first powered aircraft but certainly not the best. If they had crashed on the beach before anyone noticed, Curtis or one of the dozens of others chasing the flight dream would have gotten the credit. In the end, the world would have had all these advancements around the same time no matter who died, failed or gave up because they were actually the result of many improvements and ideas contributed by thousands across many years of constant evolution.

New ideas, the author suggests, are almost always the result of natural selection between competing ideas. The pace of innovation and improvement continue to grow because human beings are more empowered than ever before to collaborate and breed new ideas. For example, a developer in Houston, TX can work on an open source project with contributors from all over the world. The other important ingredient, competition, is also much easier to achieve today. An entrepreneur in Africa can compete on equal footing with another in New York thanks to the Internet and easy access to rapid travel.

The optimism in the book comes from the fact that the improvements in our lives, the countless gadgets that have made communication easier, food cheaper, close cleaner, people healthier and leisure more accessible to billions, are evolving more quickly today than they have ever been able to in the past. Today’s poor have more access to knowledge, education and communications than the richest that lived in America at the beginning of the 20th century. A man that cannot attend the best university can pay pennies at an Internet cafe and get access to more books and more research than a tenured professor at Harvard could have hoped for 40 years ago. We are living in an age of accelerating progress and opportunity for all that will breed more of each.

Check out the book on Amazon.

Lissome Project

Just started working on the design for the project I discussed in my last post. I decided to call it Lissome, which means, among other things, “nimble”. Seems like a decent name for an agile task board. The Balsamiq mockup can be found in the Lissome repository.

I decided to license it under AGPL . I realize this license is fussier than many other open source licenses about making source changes freely available, but that is precisely the point.

Building Something Twice (Just for Fun)

I’ve been playing with Node.js a bit lately and have been looking for something fun and useful to do with it. Along the way, I’ve also developed a curiosity about how a well-written, non-trivial Node.js application might compare to an equally well-written .NET application in terms of features, quality and development time. Today, I am starting a spare time project to satisfy my curiosity.

The idea is to develop an agile task board application with a highly-interactive web front end and two different fully scalable backends: the first in Node.js and the second in .NET. My goal is to make the backends as interchangeable as possible. My initial guess is that the only difference in how the client will communicate with the backend is in the area of server notifications, which will probable use socket.io with the Node.js backend and SignalR with the .NET backend. For simplicity, both backends will use the same hosted NoSQL and search facilities. I’m leanings towards RavenDB hosted by RavenHQ simply because I am familiar with it and because it includes powerful search capabilities.

The whole thing will be open source and I’ll blog about it as I progress. I’ll start by sketching a basic UI wireframe so I can put together an initial product backlog. Wish me luck.

When Green Fields Become Killing Fields

If you ever consider taking on a greenfield project to replace a working production system that, though it is a big ball of mud, continues to improve and runs on a reasonably modern OS such as Linux or Windows, weigh the risks carefully. The legacy system has hundreds of useful features that will have to be recreated in the new software. Think about how much effort has gone into the development of the legacy system. Even if you believe the greenfield will do it faster, be realistic about how much faster. For example, if the legacy system was developed over ten years, can you really deliver a better replacement in under two years or more than three times faster? Consider also that your shiny new system will not be proven in production until it can get to production, which may add months to the end of the schedule or cost your business in lost productivity when the new system proves initially more unstable than the legacy system. It is often better to take the resources you planned to dedicate to the re-write and use them to gradually carve off and replace pieces of the legacy system instead so that you end up with what you wanted in the first place: something just like the existing system but more reliable and easier to maintain. It may take a little longer and cost a bit more, but it is actually far more likely to succeed because it delivers incremental business value along the way.

If you do decide to march across the greenfield anyway, please consider Pickett’s charge for an example of how deadly the combination of a beautiful greenfield and the illusion of quick victory can be.

On a sunny summer’s day on July 3, 1863, General George Pickett lead his men on an ill-fated march into history:

After the great guns fell silent, three divisions of resolute Confederate infantry, around 12,500 men, stepped out from the trees and formed battle ranks. Stretched out in a battle line over a mile long, with fixed bayonets and flags flying, the Southerners began their slow, determined march up towards Cemetery Ridge, three-quarters of a mile away. Waiting on top, the Union troops crouched behind stone walls, watching and holding their fire. When the Confederates were halfway across the intervening field, the Northern artillery opened up and began tearing great holes in the approaching lines. As the Southerners drew nearer, the Union infantry unleashed volley after volley of musket fire, mowing down the advancing enemy. Somehow the ragged lines kept reforming and on they came, despite the devastating carnage, quickening the pace and howling their “Rebel yell.”

— An account in the Philadelphia Enquirer

General Lee ordered the attack despite the reservations of his second in command, General Longstreet, because he felt the Union army would break under the assault, his army would march on to Washington and the war would be won. Longstreet, on the other hand, had long ago come to the conclusion that new technology made assaults against entrenched defenders on better ground sheer folly. He had watched as the Union had assaulted General Jackson’s strong defensive position at First Manassas back in 1861 only to be slaughtered by the thousands. He did not want to see the Army of Virgina suffer a similar fate assaulting the hills of Gettysburg. Instead, he suggested a flanking maneuver around the immobile Union defense so that the Confederates could take up a strong defensive position between the Union army and Washington thereby forcing what he believed would be a costly Union attack. Lee, likely tired and sick, would have none of it. He wanted to bring the war to an end and he thought his men would once again do the impossible.

History, in the form of the cannons of the Union army, ultimately proved Lee wrong. Thousands were slaughtered as they marched bravely onward. A contemporary Union newspaper account tells of the gruesome climax:

A few hundred Southerners, bravely led by General Lewis A. Armistead, breached the Union lines and briefly hoisted the Confederate flag on top of Cemetery Ridge before being overcome. The rest of the Southern troops could not even reach that point, and were forced to turn back.

Their battle-flag was planted boldly upon the crest and a shout went up from the demons who seemed to court death and destruction, but our lines swayed but for a moment; back they went, the ground was regained, and our lines again intact. At other points they succeeded in gaining temporary advantages, but ere they could realize their importance they were torn to pieces and hurled back upon their column, and so the column swayed until they could no longer get the troops to make a charge.

— An account in the Philadelphia Enquirer

Nobody knows what would have happened if Lee had listened to Longstreet that fateful day. Perhaps the war would have dragged on even longer before the Union’s economic and population advantages ended up carrying the day. Perhaps some European power would have stepped in to negotiate a peace with the Confederate states remaining independent. The only certain thing is that the Confederates never again threatened ultimate victory despite fighting on for nearly another two years.