Interview on DevOps

From time to time, I get the opportunity to talk to industry reporters about agile and DevOps. Today, I was interviewed via email for the first time, which turned out pretty interesting. Here are the questions and answers from that interview.

Please briefly describe how the company is using DevOps, including when it began, which DevOps tools and for which types of projects.

We see DevOps as a culture that encompasses people, practices, tools and philosophy. In that sense, it has become central to everything we do to develop, maintain and operate our e-commerce sites for Blinds.com, JustBlinds.com, AmericanBlinds.com and, of course, Home Depot custom window coverings. Infrastructure is code that evolves in concert with our other software components. DevOps happens inside our agile development teams and often draws in specialized resources from our operations group. It also happens inside our infrastructure group and often draws in developers. It’s part of our DNA.

The tools aspect of it is pretty standard stuff. We use Git and GitHub for source control. All our application and infrastructure code is there. Puppet helps us with rolling out and managing servers. Our backends are mostly .NET so we use Octopus Deploy to help with rolling our code. TeamCity is in the middle of our development process and code there is used to expose deployments and tie them together with builds. Logs are mostly managed by Splunk though we’ve played with an ELK stack for this as well. Nagios is used for infrastructure monitoring. NewRelic is our app monitoring tool and we depend on it to alert us to problems with the user experience. All our alerts get fed into Pager Duty for escalation management. We’ve been experimenting with Consul for discovery and config.  We’re also experimenting with Docker. What’s holding us back there is .NET on Windows. Of course, that story is changing with .NET Core and Windows 2016 on the horizon so we have high hopes for Docker as a next step.

What were the business drivers for deploying DevOps?

Agile drove our adoption of DevOps. Our adoption of agile was driven by our organization’s culture more than anything else. One of our key values is “experiment without fear of failure”. Another is “improve continuously”. Over the years, our whole IT process had gotten into that uncomfortable place where limited resources lead to a difficult relationship with the rest of the business. They saw us as standing in the way of all the cool experimentation and improvement they wanted to do. Agile helped us break down the walls that had developed and form a true partnership for innovation. DevOps is a necessary part of the agile process. How can you innovate constantly if deployment requires an over-the-wall handoff and lots of manual intervention to get done? If operations and infrastructure are not intimately involved in the process, how can you support and manage it once it gets into production?

What benefits has the company seen from DevOps? 

DevOps enables agile, which allows us to continuously improve. It’s a big part of how we were able to deliver on all the promises of our new e-commerce platform, which lead directly to the acquisition by Home Depot. It has allowed us to continue to innovate and thrive inside a Fortune 50 corporation and take on new challenges to help drive innovation outside of the custom window coverings business.  DevOps is like oxygen for the agile process. Without it, it’s very possible that we would have ended up with “agile in name only” where agile terminology is used but nothing really changes and the organization doesn’t see the kind of exponential increase in innovation that we’re benefited from here.

Any challenges of deploying and using DevOps, and how were they addressed?

Our biggest challenges revolve around security and compliance especially now that we are part of one of the largest retailers in the world. We’re still learning how to deal with all that when it comes to sharing responsibility for deployment and infrastructure between developers, infrastructure and operations engineers. We’re constantly tempted to solve these problems with handoffs and work hard to avoid that. Now that we have trust across all the impacted groups it’s much easier to work through them and come up with ways to address compliance without undermining the velocity of innovation.

Quickie Review of “The Evolution of Everything: How New Ideas Emerge”

“The Evolution of Everything: How New Ideas Emerge” by Matt Ridley is a very thought provoking book about how our world has been largely shaped by bottom-up trial and error evolution rather than top-down intelligent design. Along the way, the author challenges assumptions about everything from how religion created morality to the role and impact of government in education. In the end it is a bracingly optimistic look at how our civilization got here and what the future may hold.

Think of the incredible web of economic activity that makes the apparently simplest of products, the humble pencil, available as explained in the famous story, “I Pencil” (check it out on YouTube if you have not seen it before). Nobody designed the entire supply chain. No single person or small group of people direct it. No government requires it or builds it. Even so, tens of thousands of people somehow coordinate their activities to make it possible.

Think, too, of invention. Edison invented the light bulb, right? What do you think would have happened if he died the week before his discovery? Well, Humphrey Davy started the march towards an efficient electric light many years before and Edison was just the best funded of those pursuing the task. It is certain that if Edison had failed, someone else would have stepped forward likely mere months later. This pattern, the almost inevitability of invention, can be seen in just about every case. Steve Jobs did not direct the invention of the smart phone. Apple just got there a bit sooner and with a bit more innovation than countless others. If Steve had died a few years sooner, Android may have been first or someone else would have stepped up with an almost identical device. The Wrights flew one of the first powered aircraft but certainly not the best. If they had crashed on the beach before anyone noticed, Curtis or one of the dozens of others chasing the flight dream would have gotten the credit. In the end, the world would have had all these advancements around the same time no matter who died, failed or gave up because they were actually the result of many improvements and ideas contributed by thousands across many years of constant evolution.

New ideas, the author suggests, are almost always the result of natural selection between competing ideas. The pace of innovation and improvement continue to grow because human beings are more empowered than ever before to collaborate and breed new ideas. For example, a developer in Houston, TX can work on an open source project with contributors from all over the world. The other important ingredient, competition, is also much easier to achieve today. An entrepreneur in Africa can compete on equal footing with another in New York thanks to the Internet and easy access to rapid travel.

The optimism in the book comes from the fact that the improvements in our lives, the countless gadgets that have made communication easier, food cheaper, close cleaner, people healthier and leisure more accessible to billions, are evolving more quickly today than they have ever been able to in the past. Today’s poor have more access to knowledge, education and communications than the richest that lived in America at the beginning of the 20th century. A man that cannot attend the best university can pay pennies at an Internet cafe and get access to more books and more research than a tenured professor at Harvard could have hoped for 40 years ago. We are living in an age of accelerating progress and opportunity for all that will breed more of each.

Check out the book on Amazon.

Quickie Reviews of Two Books to Help With Mindfulness at Work (and Elsewhere)

When you are mindful…You become keenly aware of yourself and your surroundings, but you simply observe these things as they are. You are aware of your own thoughts and feelings, but you do not react to them in the way that you would if you were on “autopilot”…By not labeling or judging the events and circumstances taking place around you, you are freed from your normal tendency to react to them.

— A Guide to Practicing and Understanding Mindfulness

One of the things I regret most about my younger self is my extreme fear of failure and the way it drove many unhealthy behaviors. Sure, it had its uses. For example, I often used that fear to drive me to work harder, read more, learn more and generally stand out from the average. However, it also made me a very difficult boss in the early days of my company as I drove everyone as hard as I drove myself and rarely but too often yelled at my employees for simple mistakes. I’m ashamed to say that this made me a very poor leader at times. When the chips were down, when the going got rough, I worked harder than anyone and so did my team but they did so often in fear of the slightest stumble. My ugly behavior extended into my personal life too where my fear and doubt sometimes made me quick to anger and cost way too many personal relationships.

To this day, I can’t say for sure where my deep fears come from. Western therapy gave me some theories but never really helped much. Instead, I learned how to see my fears for what they were and took away their power to control me through lessons I learned from Buddhist practices and Eastern philosophies. The simple ability to recognize the blooming of an emotion, how it changes the way my body feels and being able to choose in that moment instead of reacting had a very powerful impact on my life. It helped me grow my business and build a far stronger team of happier employees. It lead me into a great marriage with an extradorinaiy woman. After I sold my company, it helped me find a home at Blinds.com along with a role I love. Although I still have a long way to go, I certainly feel like I’m better off today than I was those many years ago mainly because I better understand what’s going on inside my head and my heart at least most of the time.

Lately, I’ve noticed that mindfulness is making its way into the business mainstream.  Companies like Google are actively training employees in mindfulness practices because it improves the quality of communications, helps people get more done by enhancing their focus and generally improves the happiness and satisfaction of their employees. To that end, here are two books that can help get you and your company get started.

“Fail, Fail Again, Fail Better: Wise Advice for Leaning Into the Unknown” by Pema Chodron is nothing more than a commencement speech and an interview and nothing less than a brilliant and gentle lesson about the importance of facing up to your failures with compassion and understanding. It only runs about an hour and a half so I was able to listen to the Audible version while walking one afternoon. Something about the combination of walking 7 miles and the lessons in the book left me with tears my eyes. Often, it felt like the author was talking directly to me and my struggles.  Yes, I often took refuge in blaming others when I was involved in failures. Yes, even more often I judged myself unforgivably responsible for every failure. The secret, as she says, is to learn to sit quietly with the feeling instead of pushing it off quickly with blaming others or self. Use failure to become stronger, more compassionate and more able to make your way through life.  She tells a story of a man who walks through the surf to swim out to sea. A large wave comes and knock him down.  As he lays at the bottom with sand in his mouth and in his eyes he thinks of two choices: get up or, well, die. He gets up and walks further. He gets knocked down again, lays at the bottom and gets up again. Each time he gets knocked down, it becomes easier to get back up because of the habit. Easier said than done, of course, but there’s far more nuance to it and she does talk about it a bit more as the book goes along. The commencement address that opens the book is full of humor and good will even as it challenges the new graduates in the audience (and the reader) to go out into a world full of unknowns and many, many opportunities for failure.  Check it out at Amazon.

“Search Inside Yourself: The Unexpected Path to Achieving Success, Happiness (and World Peace)” by Chade-Meng Tan is a straight-forward guide to the mindfulness course the author helped to assemble and now brings to people throughout Google. Tan is one of Google’s earliest engineers and brings an engineer’s mindset to mindfulness. He provides plenty of scientific evidence for the benefits of mindfulness training, which makes the seemingly squishy ideas behind things like meditation much more approachable for highly analytical types. His enthusiasm for the subject and his infectious joy come through page after page. There’s no doubt that the title on his business card is accurate: “For he’s a jolly-good fellow, which nobody can deny”. I would highly recommend getting this one in print or Kindle form so you can easily go back to the practical exercises.

Quickie Review of “Soft Skills: The Software Developer’s Life Manual”

Soft Skills: The Software Developer’s Life Manual” by John Sonmez of Simple Programmer fame started out very slow and way too obvious for my tastes. After the first chapter or two I wondered what possessed me to pick up a book that bills itself as “a guide to a well-rounded, satisfying life as a technology professional”. I started to think of the author as Captain Obvious. I imagined him with a foam bat in his hand that he would earnestly use to punctuate each of his points with a gentle whack to my head. “You need people skills”, whack! “Be picky about where you work”, whack! “Have a specialty”, whack! I stuck with it, though, and ended up glad that I did. Yes, some of the advice is obvious and facile. After all, the book is covering an awful lot of territory. Some of it is aimed at people with far less than the 30 years of experience I have in my career. However, the book offers solid advice and a good starting point for deeper exploration on most of the topics it covers.

The book consists of 70 short chapters broken into seven sections: career, marketing yourself, learning, productivity, financial, fitness and spirit. The short chapters create a nice rhythm when listening via Audible since you can always finish a chapter before getting out of your car or otherwise turning off the audio. Later in the book I learned the short chapters were both a motivational technique for the author and an intentional choice to let readers digest the material in bite-sized chunks.

Personally, I found the sections on productivity, fitness and spirit to be most interesting as I am starting to delve deeper into those areas myself. The productivity section was particularly well done. The author’s personal experience of how disciplined use of time changed his life clearly gives him a passion for the topic. He ties planning, pomodoro time management and training techniques together to form an interesting and very practical way to get more done than you ever thought possible. He also takes on notorious time wasters like TV and video games based on his personal experience. It is very compelling advice. His advice on fitness is similarly personal, impassioned and effective.

His advice on investing is equally passionate but edges out into very dangerous territory advocating things like option trading and highly-leverage real estate investing. Although he briefly notes the risks involved, he down plays them quite a bit. He also advocates “good debt”, like the million dollars or so in mortgages he says he currently holds on his rental properties. On the upside, he references “Rich Dad, Poor Dad” author Robert Kiyosaki and spends a little time clearly explaining the key differences between things that make you money, like stocks and bonds, and things that waste your money, like buying cars with “easy monthly payments”. All in all, his financial advice is a strong net positive though I would advise people to take on less risk than the author advocates, avoid debt of all kinds and generally follow more of Dave Ramsey’s advice on money.

I would recommend this book to almost any technology professional. It’s likely that almost every reader will find several ideas in the book they can use to improve their life and career. I would especially recommend the book for someone just starting out in their career. You can think of the book almost like a very inexpensive mentor. At the very least, it can put you on a path of continual self-improvement.

Rabbit Operations 0.9.0 Released

This version is a minor maintenance release that includes the ability to set a different expiration time for error messages to give you more time to analyze them and possibly replay them. It also includes an upgrade to the latest stable release of RavenDB.  Check out the project website for more information.

The next release, 0.10.0, will include some major improvements including a new GUI and the ability to view and analyze statistics collected from all messages.  You can check out the plan on our Trello board.

Trying Out Serverless Applications at AWS — JAWS, Lambda, API Gateway

I just finished up a POC for an integration with a major retailer that allows our Customer Engagement Center to better assist customers buying custom blinds in store. It leverages AWS Lambda, AWS API Gateway and AWS SES to deliver a massively scalable, highly available web API with no servers to manage. I am really liking the tools provided by the JAWS Framework that made it far easier to manage and deploy the solution. I hope to do a full blog post on this in the near future.

Rabbit Operations Version 0.8.0 Released

This is a minor release but it contains one significant goodie: The details dialog for errors now has a section that displays a nicely formatted stack dump. Here are the complete release notes:

  • Nicely formatted stack dump shown on details view of error message
  • Ability to view queue stats as a gauge
  • Improve performance of search screen especially when bringing back large sets of large messages
  • BUG FIX: Small memory leak in poller due to RavenDB profiler stats
  • BUG FIX: Retry to overcome RavenDB transients under high loads when there are more than 40 active queue pollers

Check it out the Project Site for more details.