Google AppEngine and Separation of Concerns
In the world of writing software there are 2 challenges every developer must face: how to keep your concerns separate, and how to survive the impending deadly boredom of writing all your data CRUD (Create, Update, Retrieve and Delete) functionality. There are a gazillion solutions out there for both. In the realm of functional isolation much has been written, studied, and practiced for separating your program’s concerns. You can adhere to various design patterns; you can establish a pattern and an engine to manage that pattern yourself; you can use one of the several freely available frameworks that already exist (such as Spring). This is good.
In the realm of CRUD management (and boy, is it a crappy job… ba-da-dum!) there’s a plethora of available frameworks for almost every language. These object persistence frameworks handle all the boring, lowdown, uninteresting CRUD functionality that developers loath (but which is a necessary evil). This is also good.
What’s bad, however, is the interoperation of these two challenges rarely seems “natural”. The reason is that object persistence frameworks can really mess with your separation of concerns. The domain classes you write can suddenly become the transport means in your data access layer, thus blurring the lines of what code goes where.
A week ago I started a new application that uses Google AppEngine. Google AppEngine provides an object persistence framework as part of its platform. It also provides a Model-View-Controller type of framework for writing all your web apps. From day one my colleague Kevin Pierce and I have struggled to determine how to separate domain logic from data logic. In AppEngine, your “onion” from the outermost layer to the innermost looks more or less like the following:
[Template] (html/js/css) -> [View] -> [Model] -> [DataStore]
The problem with the above onion is between the View and the Model. Without having something in between your views can (and will) become enormously bloated, full of miscellaneous helper modules, and will operate directly against the Model/Datastore, thus blurring the lines of responsibility in your application. This will eventually make your life a living hell. Kevin and I decided to throw in an intermediary layer between View and Model, called Domain, which houses all the business logic of our application.
[Template] -> [View] -> [Domain] -> [Model] -> [DataStore]
The idea is that the View layer will only ever directly interact with the Domain layer, and never directly against the Model layer. For example, if the View needs to retrieve a Person from the DataStore, instead of going directly using the Model, the View will ask the Domain for an object that’s responsible for retrieving a Person from the DataStore and interact with that object (a classic Factory). Because Python is a dynamic language, in which everything is an object (including classes, functions and modules) this becomes extremely easy. Take the following example:
# in myapp.models.person module
from google.appengine.ext import db
class Person(db.Model):
first_name = db.StringProperty()
last_name = db.StringProperty()
def get_person_by_id(person_id):
return Person.get_by_id(person_id)
# in myapp.domain.person
class PersonProxyFactory(object):
def create(self):
from myapp.models import person
return person
# some_view.py
def get_person(request, *args, **kwargs):
from myapp.domain.person import PersonProxyFactory
factory = PersonProxyFactory().create()
factory.get_person_by_id(request.GET.get('person_id'))
# return some template
In the above example the View has no direct knowledge of the Model; the only layer the View explicitly knows about is Domain. The Domain has knowledge of the Model, and because in python a module is just another object, the Domain’s factory can return the actual person module. The View need not be aware of what the returned object actually is, instead it need only understand what the interface of that object is.
This separates the concerns nicely while still allowing us to use the persistence framework, but it poses some difficult questions. To what extent must we wrap functionality? Is it appropriate for layers above the Model layer to assume that a Person class know how to save itself (via an instance method of save() or put())? Where do we draw the lines with this solution?
How have you tackled the dichotomy of object persistence frameworks and the separation of concerns?
Google AppEngine PolyModels and ModelForms
I recently created my first PolyModel class in Google AppEngine for a new feature. The feature was pretty straight forward - allow comments for some of the entities in our application Homebook. We wanted each comment to be tied back to its target entity – a room, house, or image – in a 1-to-many (each entity can have many comments). Using PolyModel we found this to be pretty easy without duplicating a bunch of code. Essentially the structure of Comment class hierarchy looked like the following:
class Comment(PolyModel):
# bunch of common fields
class RoomComment(Comment):
target = db.ReferenceProperty(reference_class=Room, collection_name=’comments’)
class HomeComment(Comment):
target = db.ReferenceProperty(reference_class=Home, collection_name=’comments’)
class ImageComment(Comment):
target = db.ReferenceProperty(reference_class=Image, collection_name=’comments’)
Really straight forward – simple class inheritance. I then created a simple factory method that would fetch the proper class type for a given instance of one of our commentable entities. Things were moving along nicely (complete with lots of beautiful unit tests – you do write unit tests, right?) and it was time to create the add_comment view.
For the most part we use Google AppEngine’s djangoforms ModelForms, a theoretically clean and easy way to create web forms based on data models. I created my standard ModelForm class (based on our parent Comment class) and the view against which the form would post, only to be blindsided by the oddest of validation errors:
- _class
- This field is required.
Huh? _class? What gives? Well, after a couple hours of soul searching, googling (which turned up nothing), and a few tears, my genius coworker Tony Arkles had an epiphany: The PolyModel class has a _class field which is used to keep track of the PolyModel instance’s class hierarchy. We added the _class field to our form’s exclude list and the world righted itself, cats and dogs learned to coexist, and Liz Taylor finally discovered a happy marriage.
Now the question is, why is the ModelForm rendering a field that’s not a google.appengine.ext.db model property? Can anybody from Google answer me that?
Traditional Skills Are Still Needed in Software Development
In one of my earlier posts I mentioned that for the past 7 years I primarily worked in the world of .NET. I enjoyed those 7 years. Microsoft’s .NET Framework is rich and fun to use. The C# language continues to impress me as it matures, the Framework’s classes and libraries are intuitive to use, and the documentation, despite being about 2.5 gigs, is thorough, easy to understand, and complete. As far as I’m concerned, Microsoft’s .NET Framework is a home run as far as frameworks go; it’s the big “Pro” when it comes to developing MS based applications.
That being said, there’s a lot of cons. You pretty much have no choice but to use Windows. You’re realistically stuck with having to use Microsoft’s flagship IDE, Visual Studio .NET. Despite having fantastic integrated debugging, the IDE itself is very… Microsofty. It’s very point and click. The keyboard shortcuts aren’t very powerful (mouseless navigation is nearly impossible). It’s expensive (if you’re not using one of the stripped down “free” versions). It’s bloated, slow, and buggy. It crashes all the time. It has very little support for Agile development, and what support there is completely misses the point (I’m looking at you, MSUnit). I also think that VS.NET teaches novice developers how to do everything wrong (code-behind is a wickedly evil thing). Overall, the entire thing really kinda sucks, and if it weren’t for ReSharper the thing would be almost useless.
About 2 months ago I started focusing hard on iPhone and Mac OS X software development. One of the things I liked right off the bat was that I didn’t have to buy any development tools. In fact, I didn’t even have to download anything because my Macbook Pro came with everything I needed to get going. IDE for development? Check. Performance monitoring tools? Check. App for creating interfaces? Check. Incredibly impressed with the tools that come out of the box, ready for development (including Subversion, python, ruby, IDE’s and graphical tools)? Check. As far as development tools and “out of the box” readyness for somebody to get their geek on, Apple hit a home run. This is Apple’s “Pro” entry.
Fast forward 2 months. Ask me if I’m still developing my iPhone application. Go on. Ask. I’m serious! Do it!
Are you still developing your iPhone application?
Hell no.
Uh… why not?
Because despite having great tools out of the box (+1), the developer documentation is so horrendous that it artificially inflates the learning curve by a factor of 10 (-1 gazillion!). Yes, there are oodles of example code at the Apple Developer site, and yes there are oodles of videos too. That’s great if you already have a decent 5,000 foot view of the technological components you want to work with, but it sucks if you don’t have that 5,000 foot view. And the only way to have that 5,000 foot view is if you’ve already been developing in Cocoa Framework for the past several years.
The problem is in the details, literally. Their code examples demonstrate the raw intricacies of the how, but not the conceptual understanding of the what and why. As a result I find it incredibly difficult to see the low level implementation details when I don’t have a decent understanding of the classes involved at a higher level. At the other end of the spectrum are their videos. These videos are 50,000 foot pie-in-the-sky views that give the audience a really casual understanding of general stuff. They’re essentially marketing videos for nerds. They’re great if you’re just getting started and you need a super-high level view of how the various frameworks piece together, but they don’t prepare you to actually do anything.
So what about the documentation that comes with the development tools?
A great question! Let me take the Socratic method and answer your question with a question. What documentation? Imagine, if you will, the horror of opening up the documentation tool for Xcode (an entirely separate application that helps you search and organize Apple’s documentation – good in theory, fails miserably in practice) and finding that the documentation for almost every single class and method is only 1 sentence long, and that 1 sentence is the most formal, computer-sciency idiomatic statement you can possibly imagine. That’s Cocoa’s documentation. The inline comments provided by the header files are better, but barely.
This has frustrated me on so many occasions that the euphoria I experienced while trying to develop my first iPhone application quickly turned into the searing pain of frustration and anger. It frustrates me doubly so because Apple is a fantastic company with fantastic products for which I desperately want to contribute, but I can’t. I have a limited amount of time every week that I reserve for Apple development, and their lack of documentation is completely counter productive. I can easily spend the entire time searching for a snippet of easy to read, easy to understand documentation about, for example, what methods you have to implement in order for your class to act as a delegate to some other class. The documentation just isn’t there, and if it is, it’s so horribly organized that I couldn’t find it if it jumped out and kicked me in the junk.
This brings me to the point which I eluded to in the title of this entry. It’s great to have lots of fun, good looking, easy to use tools. It’s great to sit down and hammer out oodles of code. It’s super duper fun to compile your application and watch it run, crash, and burn. Those are great skills to have, but the thing that really makes software development a joy is the same thing that makes almost any other traditional profession a joy: clear communication.
I once met a kid in grade 10 that was a computer whiz, but was failing out of school because he refused to apply any of his efforts towards his non-technical, non-sciency classes. At the top of this failure list was English. I asked him why he didn’t like English.
Because it’s stupid. I’m not going to use it! I want to be a programmer, not an author! I don’t care about that stuff!
Needless to say I had to inform him that a very large portion of my job is not writing code, but communicating complex ideas to other people. People that are both tech-savy and technically-inept. Communication is one of, if not the most important skill in software industry. He was shocked.
I once had a job in which I was told, on my first day, that my responsibilities would be “to port application X into application Y”. I asked where the requirements were and if I could have access because I will need to review those requirement in order to do the job. I was told that there weren’t any requirements, and that I should just “read the code.” Naturally the person who told me this was not a technical person, and I couldn’t get mad at them because as far as they knew reading code was just like reading a well written novel. In reality, reading code is more like reading a very poorly written novel, in which there are either far too many or far too few characters, and all the pages are mixed up, and there are no page numbers, and all of a sudden it just ends.
For whatever reason, Apple has forgotten the importance of clear communication. This is unfortunate because Apple is in a position in which they can take a significant dent out of Microsoft’s developer base if they wanted to. Apple has all the fun sexy toys. They have all the innovation. They have all the trend and momentum right now. Microsoft, conversely, has a failing strategy in almost every segment of the computer industry. If Apple could deliver a rich, thorough, and well organized set of documentation for their Cocoa framework, it could be a huge blow to Microsoft’s developer strategy. So far, though, Apple is completely dropping the ball.
Apple. If you’re listening – get busy and floor me with some excellent documentation.
Readers. If you’re willing – floor me with some excellent resources that tell me the what and why, and not the how of the Cocoa Framework.
VendAsta Technology is an All You Can Eat Buffet.
This week marked my first week on a new project at VendAsta. For the past 7 years I’ve spent my professional life awash a sea of .NET technologies. Web? ASP.NET. Desktop? Win32 & WPF. SOA? SOAP & WCF. And so on and so forth. That all came to an end this past Monday when I transitioned away from our .NET project and started work (or training, rather) on our own spin up application.
Excited? Indeed. But that overhwelming sense of “oh man, what have I gotten myself into” has been red-lining for 5 days. Reason? I have literally a dozen technologies that I need to learn… and they’re adding all the time. Currently, this is the list of technologies I need to learn:
- Google App Engine
- BigTable (or is it DataStore now?)
- Django
- jQuery
- Python
- YUI (Yahoo! User Interface)
- FBML (Facebook Markup Language)
- Facebook API
- CSS (Yea, it’s been years and I can’t remember jack)
- OpenID
- OpenSocial
- FriendConnect
These are all technologies we are currently planning on using in our new application. The list may (and likely will) change during as the requirements change, but so far this is it. As I said, I’m pumped… but man… can I do it? Can I manage this curve? You know it. Can you? Would you like to find out?
You and all your mashups… like I’m afraid of you, Web 2.0!
TV On The Radio’s “Dear Science,” Makes Me Pee My Pants.
My entire life I’ve been heavily into music. At the age of 4 I enrolled into the obligatory call-to-childhood by taking group organ lessons. These lessons displeased me, but it started me down the path of a life rich in musical tapestry. I took private piano lessons for 14 years following the group organ lessons. I took 10 years of private saxophone lessons. 2 years of private vocal lessons. I performed in several musicals, started my formal post-secondary education as a drama major, and have purchased my record collection twice thanks to a rather expensive break-in of my car when I was 20. I’ve often told people that, given the unfortunate choice, I’d rather be blind than deaf. I wouldn’t even hesitate. I can bear not golfing or driving again, never judging my own body again, and never having to see another Internet shock site again, but never hear music again? Never hear the rhythms of nature, the humming of a street light, or another TV On The Radio album again? I’d rather die.
I will admit, however, that I am no life time achievement award when it comes to music appreciation. At the two years leading up to the turn of the millenium, when boy bands were at their peak and rock music was debatably at its lowest point in probably 40 years (yes, even lower than the disco era), I gave up on music. I stopped listening. I stopped buying. I stopped reading and searching. I lost the will to seek. And then something interesting happened. An 18 year old kid from the US named Sean Parker, over the course of a few weeks, decided to make the international music industry as we know it completely, 100% obsolete. A kid, who didn’t know a thing about programming, taught himself C++ and wrote the infamous Napster – a product that allowed me to start finding music again.
However, while Napster was great, and while it introduced me to a lot of great bands, it wasn’t delivering anything completely earth shattering in the context of new bands. Napster allowed me to find the bands that I knew about, but it was no substitution for finding new bands I’d never, ever heard of before (well, this isn’t entirely true, I found a lot of new bands, but the crap-to-awesome ratio was way outta whack – the process was not very efficient). Only reading and fingering through various zines and papers could do that. But it was Napster that got me interested enough to dedicate time out of my week to finger the various zines and papers.
Young Liars EP
Come 2003. While fingering various zines and papers, I come across a review for an EP by a band called TV On The Radio. I sample them, and in 30 seconds flat I’m completely hooked. TOTR’s Young Liars EP is still, to this day, one of the best EP’s I’ve ever heard. For being only 5 songs it’s rich, deep, complex, compelling. It’s 22 minutes (!) of highly compelling and engaging music. It starts off with deep bass, harmonious vocals, and a shock of “Holy Crap! This is Awesome!”. It finishes with a 4 minute a cappella version of The Pixies’ Mr. Grieves. It was my favourite thing in my collection in 2003.
The interesting part of all this is that TV On The Radio didn’t have any albums out at the time. This EP was their first and only published work. When was their full length LP going to be released? Unfortunately for those that heard the EP (over, and over, and over, and over again), it would be at least a year until TOTR finally released their debut LP. For a year there was speculation, anxiety, hopefulness that they wouldn’t blunder and show the world that Young Liars was just a fluke.
Desperate Youth, Blood Thirsty Babes
The 2004 release of “Desperate Youth, Blood Thirsty Babes” was successful in proving that Young Liars was not a fluke, but 100% intentional. The album announces itself again with deep driving bass, distorted and muffled drum machines, and a “HERE WE ARE!” horn section. The lead off track also cements TOTR’s schtick of complex vocal arrangements. It quickly becomes clear to the listener that Young Liars was just a sample of what the band could do. Layers, layers, and more layers.
Not only does TOTR prove their creative and musical spirits on this album, they also show they’re masters of atmosphere. TOTR have no problem creating scenery in their songs. They’ll make you feel the song, not just hear it. From anxiety to rambunctiousness, loneliness to love, the album has a knack for making you feel a whole lot of your past.
Return to Cookie Mountain
In 2006, and in typical deep bass fashion, TOTR impaled the masses with their second LP, Return to Cookie Mountain. With enormous critical success the album built up from TOTR’s foundation of deep base and harmonious melodies by incorporating a lot more white noise and a lot more looping. But each song is almost in its own genre. Hours is like a bunch of Swirling Dirbies listening to neo-psych rock (at least that’s the image it conjures for me). Province is like the childhood song from your youth, with a blazing hot son setting over the lake while you, 3 years old, run around naked on the beach. Wolfs Like Me bring TOTR back to solid, driving fashion after the minature tour de senses. The album continues from there, exploring different sounds and rocking your ass.
Dear Science,
And like the precise clockwork you would expect from an expert watchsmith, in 2008 TOTR releases their third album, Dear Science, a surprisingly upbeat and, dare I say, happy album. Of course the album starts with a rich, deep bass, followed by some of the happiest (if not simple) scatting I’ve ever heard. We’re talking teenager scatting. The album explore’s hiphop (in the most frantic use of the word), soul, funk, and syphonic rock (without all the cliche symphony). This is, of course, all on top of the foundation built on the previous albums.
Remember the first time you got drunk, and you were like “Oh my Gosh, this is weird!”? And then remember the second time you got drunk you were too excited so you over did it and said “Oh my Gosh, this is terrible!”? And then the third time your were more refined and had a better idea on how to pace yourself? That’s TOTR’s third album. It’s definitely the most mature of their three and a half albums. The refined quality runs the risk of it having the least amount of replay value of their collection, but it’s still beautiful, atmospheric, and heavily textured. It’s still amazing. It’s still TV On The Radio, and it’s still going to make you sit up and take notice.
