Google AppEngine and Separation of Concerns
In the world of writing software there are 2 challenges every developer must face: how to keep your concerns separate, and how to survive the impending deadly boredom of writing all your data CRUD (Create, Update, Retrieve and Delete) functionality. There are a gazillion solutions out there for both. In the realm of functional isolation much has been written, studied, and practiced for separating your program’s concerns. You can adhere to various design patterns; you can establish a pattern and an engine to manage that pattern yourself; you can use one of the several freely available frameworks that already exist (such as Spring). This is good.
In the realm of CRUD management (and boy, is it a crappy job… ba-da-dum!) there’s a plethora of available frameworks for almost every language. These object persistence frameworks handle all the boring, lowdown, uninteresting CRUD functionality that developers loath (but which is a necessary evil). This is also good.
What’s bad, however, is the interoperation of these two challenges rarely seems “natural”. The reason is that object persistence frameworks can really mess with your separation of concerns. The domain classes you write can suddenly become the transport means in your data access layer, thus blurring the lines of what code goes where.
A week ago I started a new application that uses Google AppEngine. Google AppEngine provides an object persistence framework as part of its platform. It also provides a Model-View-Controller type of framework for writing all your web apps. From day one my colleague Kevin Pierce and I have struggled to determine how to separate domain logic from data logic. In AppEngine, your “onion” from the outermost layer to the innermost looks more or less like the following:
[Template] (html/js/css) -> [View] -> [Model] -> [DataStore]
The problem with the above onion is between the View and the Model. Without having something in between your views can (and will) become enormously bloated, full of miscellaneous helper modules, and will operate directly against the Model/Datastore, thus blurring the lines of responsibility in your application. This will eventually make your life a living hell. Kevin and I decided to throw in an intermediary layer between View and Model, called Domain, which houses all the business logic of our application.
[Template] -> [View] -> [Domain] -> [Model] -> [DataStore]
The idea is that the View layer will only ever directly interact with the Domain layer, and never directly against the Model layer. For example, if the View needs to retrieve a Person from the DataStore, instead of going directly using the Model, the View will ask the Domain for an object that’s responsible for retrieving a Person from the DataStore and interact with that object (a classic Factory). Because Python is a dynamic language, in which everything is an object (including classes, functions and modules) this becomes extremely easy. Take the following example:
# in myapp.models.person module
from google.appengine.ext import db
class Person(db.Model):
first_name = db.StringProperty()
last_name = db.StringProperty()
def get_person_by_id(person_id):
return Person.get_by_id(person_id)
# in myapp.domain.person
class PersonProxyFactory(object):
def create(self):
from myapp.models import person
return person
# some_view.py
def get_person(request, *args, **kwargs):
from myapp.domain.person import PersonProxyFactory
factory = PersonProxyFactory().create()
factory.get_person_by_id(request.GET.get('person_id'))
# return some template
In the above example the View has no direct knowledge of the Model; the only layer the View explicitly knows about is Domain. The Domain has knowledge of the Model, and because in python a module is just another object, the Domain’s factory can return the actual person module. The View need not be aware of what the returned object actually is, instead it need only understand what the interface of that object is.
This separates the concerns nicely while still allowing us to use the persistence framework, but it poses some difficult questions. To what extent must we wrap functionality? Is it appropriate for layers above the Model layer to assume that a Person class know how to save itself (via an instance method of save() or put())? Where do we draw the lines with this solution?
How have you tackled the dichotomy of object persistence frameworks and the separation of concerns?
Getting Software Development Right.
I’ve been doing a lot of thinking about “computers” and this whole “Internet” thingie, and I think it’s safe to say that software is The Next Big Thing, and has been for the past 20 years. When you consider all the software that exists in the world, and how all this software allows us to do the things that we do, it’s pretty mind boggling how much code has been written. I’m often amazed at cell phone technology and all the software that’s been written to support a simple phone call. The software on the phone, the software to support the various infrastructure, for routing calls, for determining GPS positioning, for billing, roaming, text messaging, etc. It’s enormous! I recently read that a new, fully equipped mobile phone can have an excess of over 5 million lines of code; half the R&D cost of a new car is invested in software. Heck, I have to surf the internet whenever I wear a suit so I can remember how to tie a tie. Software, evidently, is here to stay.
There are billions, if not trillions of lines of code that exists in the world today. Trillions of lines of code that have generated billions upon billions of dollars in revenue. Trillions of lines of code that affect almost every single thing that we do. And upon these trillions of lines of code, billions miss the point of what software is supposed to do – which is all the nitty gritty stuff that we hate to do. In a nutshell, software’s purpose is to automate, not digitize.
Perhaps I’m jumping to conclusions, but I find that a lot of software out there is focused too much on the whiz-bang trap, and not enough on making my life easier, more productive and better. A good example is a product I recently worked on. The product in question was a Point of Sales system which did a lot of great things. When it came to data mining, the tracking of key metrix, and real-time cross-continental financial reporting the system was second to none. It had fantastic features that cut away all the chaff and showed you exactly how well your business was doing up to the second. However, when it came to managing some key components of the business the software was not only unhelpful, it was down right painful to use. But it wasn’t painful to use because the user interface was complicated, it was painful to use because it didn’t actually help you accomplish those key components. The difficult manual process that existed before purchasing the system (which was performed on paper) was still a painful manual process in the system. The only difference was that instead of using a paper and pencile, the user would use a keyboard and a monitor (and a lot of mouse, but that’s another issue all together). These core components were simply digitized and not automated.
Now, don’t get me wrong, there were parts of the system that were automated (the reporting, metrix tracking, etc.) however they were the wrong parts. The parts that were automated were all secondary features that were the byproduct of using the primary features of the system (in which very little automation existed); or they were automated processes that required so much manual configuration up front that it was barely worth setting up. In other words, the system did a poor job of making the primary, day to day business processes and administrative activities easier because it failed to automate those processes and activities.
And this, I believe, is one of the key differences between poor software that makes you want to shutdown your computer and software that makes you want to fork over a thousand dollars! These little features that automate key components have huge impact. The reason why I use Google Mail isn’t because of the Gigabytes of available space (I don’t even know how much disk space I have with Google anymore), it’s the feature that orders my email threads into conversations instead of a dumb sort-by-date-received. I like using iTunes because all I do is click the song’s “purchase” button and 30 seconds later I have it on my hard drive and it’s playing on my computer speakers. I love using my iPod because when I want to get new music onto it I just plug it into my computer. Done.
These are all examples of products that have oodles of competition but they leave their competition in the dust. Why? Because the designers identified the painful processes that traditionally exist in these applications and they automated those processes so that the users don’t have to worry about them anymore.
Not only do I enjoy using software that automagically removes the painful parts of my life, I also enjoy creating the automagic software – and that’s exactly what I’m doing at VendAsta right now. We’ve identified some pain points in the business processes of our client and we’re automating the solution so they never have to worry about it ever again. Intrigued? Then send us a resume – we could use your creative input.
Never go Back for Seconds
I come from a small, blisteringly cold corner of Canada called Regina, Saskatchewan(Regina’s the city, Saskatchewan is the easy-to-draw-hard-to-spell province). The population hovers somewhere around 200,000 people, and the temperature fluctuates between -40 C in the winter and +35 C in the summer. That, coupled with the fact that we’re the capital city and thus the political center of the province, makes for a fairly interesting year-round climate. I’ve lived here for 2 years.
The first winter I lived here I experienced something remarkably annoying, something that flustered me so much I absolutely dread having to face it when it occurs. This something could literally ruin my day. This something is how the city handles heavy snow fall… or, rather, how they mismanage heavy snowfall. Essentially here’s what happens.
Day 1: Mother nature unleashes a fury of ill tempered wind and snow. You get cold feet and a runny nose, and then dread having to shovel your walk.
Day 2: You think about shoveling your walk, but get lazy. Coincidentally, the city feels the same way about plowing the streets.
Day 3: You shovel your walk and, if you’re responsible, you throw down either some salt or some sand so that the mail man doesn’t kill himself when he’s delivering the latest issue of the Sears Canada Catalog. The city sends out the plows.
Day 4: Your car slides uncontrollably out of control into oncoming traffic but, calling on the evasive maneuvering powers of Mr. Zulu you avoid catastrophe and regain control of your vessel.
Day 5: You repeat day 4 and thus decide it’s best to just walk everywhere.
Day 6: The city decides to throw down some salt or some sand so that its fine citizens don’t kill themselves when delivering themselves and their families to their destinations.
It’s Day 3 that drives me absolutely batty. The city always plows the streets on these big snow days, but in doing so they leave huge stretches of sheer ice. The roads will often be far worse afterwards than if they had simply left the loose snow. It usually takes the city a good couple more days before they unleash their army of dump trucks and spread sand on all the roads so that our cars actually have some traction and stop at red lights. What bothers me about this isn’t the fact that it’s so hard to drive, it’s that the city didn’t finish the job when they should have, and it makes my “user experience” one of pain, anxiety, and in the worst case scenario, fiscally expensive.
Why the city doesn’t dump sand immediately after plowing the streets is a mystery to me. They must be able to foresee the road conditions they’re creating when they plow, so why not do something about it? Why wait so long afterwards? It’s obvious that the people using the roads will require traction, so why not give them traction instead of making them wait in pain for it?
The software industry is chalk full of similar examples. For instance, at my previous job our flagship product was a retail Point-of-Sale system. One of my projects was to integrate the services provided by Flipswap directly into our product so that our customers (wireless retail stores) could increase their revenue for doing practically nothing. Sounds great (and it is – free money!), however, for our first release we didn’t include any reporting or ways of reconciling these types of transactions. This is an odd thing to do considering our flagship product’s claim to fame is the ability to track and reconcile practically everything in the world.
So we created a fantastic new revenue boosting feature that was counterproductive to the main philosophical objectives of the product – the same philosophical objectives that our customers bought into - with the intention that we would introduce the missing functionality “in the future”. Of course, when it comes to software, the words “we’ll do it later” are 4 of the deadliest words you can utter, as “later” can be months or even years down the road, if ever. For us it was about 6 months later, and like any new feature there were bugs and it was far from perfect. However we spent the time to get things fixed up, and the reporting features are now reliable and stable… 14 months after the original feature set was released.
The need for reconciliation and revenue reporting for this new feature was easily foreseeable, but we frustrated our customers with a complete lack of it for 6 months, and then frustrated them more by unleashing a somewhat mediocre implementation. It’s just like the city waiting 2 days to plow my street, then waiting 2 more days before sanding those streets such that the things are actually usable. We - the people responsible for developing software (and I’m not talking about just developers) need to perform our due diligence and examine what problems our new features will introduce to the user, and then make the appropriate changes such that those problems will not exist. We want to enable and empower our customers, not hinder them. We want to make a natural extension of their business that allows them to do their jobs faster and better. We want to automate a process, not digitize a process. So when we address the buffet table of options, lets not just load up our plates and choke back what we have and then go back for seconds. Lets take a good hard look at what we want and need right off the bat and get it all in one serving.
