Advanced Distributed System Design with Udi Dahan — Part 4 — Go Forth and Make Things Simple

You can find part 1 of this series here.

Day three started with the group exercise on the hotel problem.  We broke up into groups of seven and gathered around white boards to discuss service boundaries and messages.  The experience was both enlightening and depressing.  Our group began listing various use cases and what data was involved in each.  We identified a couple of service boundaries and named them: Room, Guest, Pricing, Billing.  We then started arguing over the first couple of screens.  Service changed names.  Then more and more responsibility started to converge in the renamed Room Management Service.  It knew about room inventory, it knew about the guest, it knew about the reservation and it handled the front desk.  A couple of us started to point out that the service was getting too big about the time Udi walked over and started to talk to us.  He reminded us that hotels overbook and have processes to help guests that happen to show up when no room is actually available.  After all, guests cancel hotel rooms all the time.  He answered a couple of questions, glanced at the whiteboard and then wandered off to confuse the next group.

We continued running around in circles for the entire 45 minutes or so we were given for the exercise.  We’d break up things into smaller service and then start pulling things back into a giant uber-service.  We squabbled over the service names.  We’d look at the screen and think one thing; We’d look at the data and think another.  With 10 minutes left we scrambled to draw a sequence diagram to at least understand the message flow.  We accomplished absolutely nothing.  Seven smart, experienced system architects full of new Udi knowledge couldn’t come up with even the beginning of the service breakdown for a simple reservation, check in and check out system with about three screens in total.  It was frustrating in the extreme.

The mood and discussions during the break that followed were bleak.  Analyzing a system this way is hard!  Many of us murmured the unthinkable.  Maybe we just weren’t capable of it.  Others took refuge in what they knew worked before they came into the class.  They murmured heresy of their own.  Perhaps this SOA stuff doesn’t work.  After all, not everyone in the community agrees with Udi.  Maybe he’s not as smart as we thought.  Everyone knows you shouldn’t build a system without a central database and all-powerful RI.    Maybe Amazon does stuff this way, but we’re not Amazon.  I can build a great layered architecture that will work and scale just fine.  Maybe we need some message and services, but we can do that without breaking the system into tiny little services.  Besides, did you get his nutty suggestion for the project structure?  Heck, did you ever look at the NServiceBus source code?  I mean how are you going to manage hundreds of little projects, some with only one class.  Now that’s complex.  And on and on it went.

After the break, Udi explained that the point of the exercise was to help us understand what is involved in SOA analysis.  He pointed out that we had to talk to the business in a slightly different way to tease out real requirements. He then gave us ideas and tools we could use to help get us started.  Although there is no step by step guide to doing a good analysis, there are certain things to avoid.  Don’t name your services.  You probably will get the name wrong at first and that will lead you to group the wrong things into the service because the name will insist you do so.  Help your users get past their assumptions about computer systems because that impacts how they communicate requirements.   Help them focus on the underlying requirements by asking “why”.  Why do we need a preferred customer flag?  What do we do with preferred customers?  How do they become preferred?  If we give discounts to preferred customers, what are the rules?  Will they change in the future?  Have they ever changed in the past?

By the time we hit our next break I was certainly feeling better.  I had started to truly understand that SOA requires a different mode of thinking.  It was going to take time to master it.  I knew that Udi could not give me answers; Too many of them would be “it depends” anyway.   Instead, he was going to help me understand how to start asking the right questions.  It was almost like one of those old Kung Fu movies — wax on – wax off, cook blindfolded or snatch that stone; At first you don’t even understand where the lessons are leading and then one day the light dawns.  The master must empty the student of the old ways before starting to teach the new.  One idea and some practice leads to the next and the next and the next until the student realizes that mastery is nothing more or less than an understanding of how to practice and continually learn.

Udi spent the rest of the course setting us on the path to enlightenment.  He delved a little into details here and there, but for the most part the course was about ideas both big and small.  By day 4, every time Udi mentioned “best practice” there would be a ripple of giggles in the room.  On day 5 it dawned on me to start tweeting some of the funniest: “Best practices are like cargo cult” and “It’s a best practice, (pause) that’s how you know it is wrong.”  were two of my favorites.  By the time 4pm on Friday rolled around almost everyone was dreading the end.  There were so many questions.  There was so little time left.

The final discussion was ostensibly about how to make your web tier scale. In typical Udi fashion, he started by illustrating how wrong best practice can be.  He illustrated the fallacies that drive the move to in process and distributed cache.  He discussed real-world examples of how caching can actually reduce performance under load.   He talked about how one client he advised removed the cache and got a 30% increase in performance.  He did not suggest that caching never works.  He just made it clear that it is not the magic bullet it might appear to be.

He then turned the discussion towards all the caching available in the web itself — the local browser cache, the ISPs cache and CDNs.  He showed us how easy it is to take advantage of these things to lower your web traffic and improve scalability.  In the end, it was clear that he was really showing us how to stop building applications that simply use the web as a transport mechanism and instead build applications that live mostly inside the web.  He was once again challenging us to think in a new way.

I learned so much at the class it’s going to take me awhile longer to fully absorb the experience.  It was undoubtedly worth the price.  It’s the best professional training course I’ve ever attended.  I haven’t felt this excited about building software since I discovered punch cards and Fortran.

The flip side, the scary side, is there is so much to learn.  2/3 of our team really wants to bring true SOA to our greenfield project.  Even though coding does not start in earnest until March 1, we’ve gathered most of the high-level requirements and written a fair amount of code we thought was pretty close to done.  If we go SOA, we have to redo some of the analysis and some of the code.  We would also be taking a huge risk because none of us have ever actually built a system this way.  We’re just not sure it is a good idea.  We’re going to take advantage of Udi’s Architect Hotline to get specific advice about how we might proceed.  I have a feeling we are going to have to stick with the architectural approach we have at least for phase 1.

For me personally, it’s a little different story.  I have been working on some ideas for a hosted product that I plan to develop in my spare time.  There is no doubt in my mind that it will be built on an SOA.  There’s even a couple of areas in it that will benefit from CQRS.

I think transitioning myself to SOA is about as important as the transition I made from mainframe to PC-based client/server back in the 80s.  The mind shift reminds me of the mindshift required to go from procedural programming to OO.  I heard the same kind of resistance from some of the experienced  architects in the room in Austin that I had heard from some of the old hands in the mainframe COBOL shops that were my home back in the day:

  • It might work sometimes, but not in the general case.
  • Layered architectures are proven.  SOA is not.
  • I can find all kinds of programmers that understand layered architectures.  If I go SOA, I won’t be able to find help.
  • The business won’t understand SOA.
  • There’s no way the business will accept any kind of eventual consistency.
  • The business can’t live without a monolithic relational database.

I can list tons more items, but you get the point.  They all boil down to one simple argument: “It’s different and, therefore, scary.”  I would argue that what is really scary is how easily a beautiful layered architectures turn into a big ball of mud in a relatively short period of time.  What’s scary is how all the RI in the world throws away flawed data without any consideration for its business value.  What’s scary is how 20 years of OO, patterns, unit testing, dependency injection and the application of solid principles like DRY have left us with the same big balls of mud we were building back in 1990.

SOA is all about applying the single responsibility principle, which good OO was supposed to embody and so rarely does, to the business domain.  Once we do that, it might actually be possible to build flexible, scalable, highly maintainable systems.  It’s about enforcing business policies in the domain model instead of trying to model the business there.  It’s about letting the infrastructure do the heavy lifting.  It’s about building systems that are “of the web” instead of just on the web.  It’s about writing less code with more thought.  It’s about making the actual implementation simple. I guess that’s why Udi calls himself the “Software Simplist” .

If you get a chance, go to this course.  If your employer won’t pay for it, take vacation and pay for it yourself.  What more can I say?

Author: Tom Cabanski

Software Developer and Entrepreneur

%d bloggers like this: