Google AppEngine and Separation of Concerns
In the world of writing software there are 2 challenges every developer must face: how to keep your concerns separate, and how to survive the impending deadly boredom of writing all your data CRUD (Create, Update, Retrieve and Delete) functionality. There are a gazillion solutions out there for both. In the realm of functional isolation much has been written, studied, and practiced for separating your program’s concerns. You can adhere to various design patterns; you can establish a pattern and an engine to manage that pattern yourself; you can use one of the several freely available frameworks that already exist (such as Spring). This is good.
In the realm of CRUD management (and boy, is it a crappy job… ba-da-dum!) there’s a plethora of available frameworks for almost every language. These object persistence frameworks handle all the boring, lowdown, uninteresting CRUD functionality that developers loath (but which is a necessary evil). This is also good.
What’s bad, however, is the interoperation of these two challenges rarely seems “natural”. The reason is that object persistence frameworks can really mess with your separation of concerns. The domain classes you write can suddenly become the transport means in your data access layer, thus blurring the lines of what code goes where.
A week ago I started a new application that uses Google AppEngine. Google AppEngine provides an object persistence framework as part of its platform. It also provides a Model-View-Controller type of framework for writing all your web apps. From day one my colleague Kevin Pierce and I have struggled to determine how to separate domain logic from data logic. In AppEngine, your “onion” from the outermost layer to the innermost looks more or less like the following:
[Template] (html/js/css) -> [View] -> [Model] -> [DataStore]
The problem with the above onion is between the View and the Model. Without having something in between your views can (and will) become enormously bloated, full of miscellaneous helper modules, and will operate directly against the Model/Datastore, thus blurring the lines of responsibility in your application. This will eventually make your life a living hell. Kevin and I decided to throw in an intermediary layer between View and Model, called Domain, which houses all the business logic of our application.
[Template] -> [View] -> [Domain] -> [Model] -> [DataStore]
The idea is that the View layer will only ever directly interact with the Domain layer, and never directly against the Model layer. For example, if the View needs to retrieve a Person from the DataStore, instead of going directly using the Model, the View will ask the Domain for an object that’s responsible for retrieving a Person from the DataStore and interact with that object (a classic Factory). Because Python is a dynamic language, in which everything is an object (including classes, functions and modules) this becomes extremely easy. Take the following example:
# in myapp.models.person module
from google.appengine.ext import db
class Person(db.Model):
first_name = db.StringProperty()
last_name = db.StringProperty()
def get_person_by_id(person_id):
return Person.get_by_id(person_id)
# in myapp.domain.person
class PersonProxyFactory(object):
def create(self):
from myapp.models import person
return person
# some_view.py
def get_person(request, *args, **kwargs):
from myapp.domain.person import PersonProxyFactory
factory = PersonProxyFactory().create()
factory.get_person_by_id(request.GET.get('person_id'))
# return some template
In the above example the View has no direct knowledge of the Model; the only layer the View explicitly knows about is Domain. The Domain has knowledge of the Model, and because in python a module is just another object, the Domain’s factory can return the actual person module. The View need not be aware of what the returned object actually is, instead it need only understand what the interface of that object is.
This separates the concerns nicely while still allowing us to use the persistence framework, but it poses some difficult questions. To what extent must we wrap functionality? Is it appropriate for layers above the Model layer to assume that a Person class know how to save itself (via an instance method of save() or put())? Where do we draw the lines with this solution?
How have you tackled the dichotomy of object persistence frameworks and the separation of concerns?
Traditional Skills Are Still Needed in Software Development
In one of my earlier posts I mentioned that for the past 7 years I primarily worked in the world of .NET. I enjoyed those 7 years. Microsoft’s .NET Framework is rich and fun to use. The C# language continues to impress me as it matures, the Framework’s classes and libraries are intuitive to use, and the documentation, despite being about 2.5 gigs, is thorough, easy to understand, and complete. As far as I’m concerned, Microsoft’s .NET Framework is a home run as far as frameworks go; it’s the big “Pro” when it comes to developing MS based applications.
That being said, there’s a lot of cons. You pretty much have no choice but to use Windows. You’re realistically stuck with having to use Microsoft’s flagship IDE, Visual Studio .NET. Despite having fantastic integrated debugging, the IDE itself is very… Microsofty. It’s very point and click. The keyboard shortcuts aren’t very powerful (mouseless navigation is nearly impossible). It’s expensive (if you’re not using one of the stripped down “free” versions). It’s bloated, slow, and buggy. It crashes all the time. It has very little support for Agile development, and what support there is completely misses the point (I’m looking at you, MSUnit). I also think that VS.NET teaches novice developers how to do everything wrong (code-behind is a wickedly evil thing). Overall, the entire thing really kinda sucks, and if it weren’t for ReSharper the thing would be almost useless.
About 2 months ago I started focusing hard on iPhone and Mac OS X software development. One of the things I liked right off the bat was that I didn’t have to buy any development tools. In fact, I didn’t even have to download anything because my Macbook Pro came with everything I needed to get going. IDE for development? Check. Performance monitoring tools? Check. App for creating interfaces? Check. Incredibly impressed with the tools that come out of the box, ready for development (including Subversion, python, ruby, IDE’s and graphical tools)? Check. As far as development tools and “out of the box” readyness for somebody to get their geek on, Apple hit a home run. This is Apple’s “Pro” entry.
Fast forward 2 months. Ask me if I’m still developing my iPhone application. Go on. Ask. I’m serious! Do it!
Are you still developing your iPhone application?
Hell no.
Uh… why not?
Because despite having great tools out of the box (+1), the developer documentation is so horrendous that it artificially inflates the learning curve by a factor of 10 (-1 gazillion!). Yes, there are oodles of example code at the Apple Developer site, and yes there are oodles of videos too. That’s great if you already have a decent 5,000 foot view of the technological components you want to work with, but it sucks if you don’t have that 5,000 foot view. And the only way to have that 5,000 foot view is if you’ve already been developing in Cocoa Framework for the past several years.
The problem is in the details, literally. Their code examples demonstrate the raw intricacies of the how, but not the conceptual understanding of the what and why. As a result I find it incredibly difficult to see the low level implementation details when I don’t have a decent understanding of the classes involved at a higher level. At the other end of the spectrum are their videos. These videos are 50,000 foot pie-in-the-sky views that give the audience a really casual understanding of general stuff. They’re essentially marketing videos for nerds. They’re great if you’re just getting started and you need a super-high level view of how the various frameworks piece together, but they don’t prepare you to actually do anything.
So what about the documentation that comes with the development tools?
A great question! Let me take the Socratic method and answer your question with a question. What documentation? Imagine, if you will, the horror of opening up the documentation tool for Xcode (an entirely separate application that helps you search and organize Apple’s documentation – good in theory, fails miserably in practice) and finding that the documentation for almost every single class and method is only 1 sentence long, and that 1 sentence is the most formal, computer-sciency idiomatic statement you can possibly imagine. That’s Cocoa’s documentation. The inline comments provided by the header files are better, but barely.
This has frustrated me on so many occasions that the euphoria I experienced while trying to develop my first iPhone application quickly turned into the searing pain of frustration and anger. It frustrates me doubly so because Apple is a fantastic company with fantastic products for which I desperately want to contribute, but I can’t. I have a limited amount of time every week that I reserve for Apple development, and their lack of documentation is completely counter productive. I can easily spend the entire time searching for a snippet of easy to read, easy to understand documentation about, for example, what methods you have to implement in order for your class to act as a delegate to some other class. The documentation just isn’t there, and if it is, it’s so horribly organized that I couldn’t find it if it jumped out and kicked me in the junk.
This brings me to the point which I eluded to in the title of this entry. It’s great to have lots of fun, good looking, easy to use tools. It’s great to sit down and hammer out oodles of code. It’s super duper fun to compile your application and watch it run, crash, and burn. Those are great skills to have, but the thing that really makes software development a joy is the same thing that makes almost any other traditional profession a joy: clear communication.
I once met a kid in grade 10 that was a computer whiz, but was failing out of school because he refused to apply any of his efforts towards his non-technical, non-sciency classes. At the top of this failure list was English. I asked him why he didn’t like English.
Because it’s stupid. I’m not going to use it! I want to be a programmer, not an author! I don’t care about that stuff!
Needless to say I had to inform him that a very large portion of my job is not writing code, but communicating complex ideas to other people. People that are both tech-savy and technically-inept. Communication is one of, if not the most important skill in software industry. He was shocked.
I once had a job in which I was told, on my first day, that my responsibilities would be “to port application X into application Y”. I asked where the requirements were and if I could have access because I will need to review those requirement in order to do the job. I was told that there weren’t any requirements, and that I should just “read the code.” Naturally the person who told me this was not a technical person, and I couldn’t get mad at them because as far as they knew reading code was just like reading a well written novel. In reality, reading code is more like reading a very poorly written novel, in which there are either far too many or far too few characters, and all the pages are mixed up, and there are no page numbers, and all of a sudden it just ends.
For whatever reason, Apple has forgotten the importance of clear communication. This is unfortunate because Apple is in a position in which they can take a significant dent out of Microsoft’s developer base if they wanted to. Apple has all the fun sexy toys. They have all the innovation. They have all the trend and momentum right now. Microsoft, conversely, has a failing strategy in almost every segment of the computer industry. If Apple could deliver a rich, thorough, and well organized set of documentation for their Cocoa framework, it could be a huge blow to Microsoft’s developer strategy. So far, though, Apple is completely dropping the ball.
Apple. If you’re listening – get busy and floor me with some excellent documentation.
Readers. If you’re willing – floor me with some excellent resources that tell me the what and why, and not the how of the Cocoa Framework.
VendAsta Technology is an All You Can Eat Buffet.
This week marked my first week on a new project at VendAsta. For the past 7 years I’ve spent my professional life awash a sea of .NET technologies. Web? ASP.NET. Desktop? Win32 & WPF. SOA? SOAP & WCF. And so on and so forth. That all came to an end this past Monday when I transitioned away from our .NET project and started work (or training, rather) on our own spin up application.
Excited? Indeed. But that overhwelming sense of “oh man, what have I gotten myself into” has been red-lining for 5 days. Reason? I have literally a dozen technologies that I need to learn… and they’re adding all the time. Currently, this is the list of technologies I need to learn:
- Google App Engine
- BigTable (or is it DataStore now?)
- Django
- jQuery
- Python
- YUI (Yahoo! User Interface)
- FBML (Facebook Markup Language)
- Facebook API
- CSS (Yea, it’s been years and I can’t remember jack)
- OpenID
- OpenSocial
- FriendConnect
These are all technologies we are currently planning on using in our new application. The list may (and likely will) change during as the requirements change, but so far this is it. As I said, I’m pumped… but man… can I do it? Can I manage this curve? You know it. Can you? Would you like to find out?
You and all your mashups… like I’m afraid of you, Web 2.0!
Customer Service Is Your Cheapest and Most Potent Weapon
Companies compete to get your business. Go figure. They may compete on feature sets, trends, loyalty, any number of things. But the cheapest and most potent way to compete is on Customer Service. Customer service is King – it’s what will keep the customers loyal; it’s how your customers will give you the benefit of the doubt when you have a mediocre release; it helps your customers focus more on your brand. Customer Service is where it’s at, and I think we can take a cue from non-profit organizations.
I used to have a radio show with CFCR in Saskatoon, Canada. CFCR is a non-profit, community owned and operated radio station. It’s fan-freakin-tastic. Twice a year they have big fund raising drives that last anywhere from 2 – 4 weeks. The first drive is in September/October, and is a straight up “Please donate money” drive. The second drive is a membership drive they have every spring. The combined income between these two fund raising drives creates about 65% of the station’s income (the last 35% coming from ads for local businesses that take place at the top and bottom of every hour). By donating your hard earned cash you would receive a tiered swag bag of stuff depending on the size of your donation, and you’d also be put into grand prize draws, etc. It was enough to get you interested, keep you listening, and feel good about donating some bucks.
In LA, California there exists another non-profit community radio station, KCRW. Now I’ve never lived in LA, and I’ve never actually heard KCRW using an actual radio, but I do listen to them daily via their online streaming audio (you can also find them in iTunes Radio under Alternative -> KCRW Music). About a month ago I made a donation to KCRW and received a free (awesome) mug. Again, much like CFCR that was enough to keep me listening, interested, and thinking about my favourite radio station.
But a funny thing happened. Today I received an email from the staff of KCRW in reply to my previous post about my KCRW mug. The email was a huge thank you for writing some kind words about KCRW. An exchange of emails flowed over lunch. And it’s for this reason why I’m already starting to think about how I can not only donate to KCRW next year, but how I can pledge more.
KCRW went out of their way to contact me out of nowhere to say thank you. But not “thanks for your pledge”, not “thanks for your support” or “thank you for listening”, but “thank you for saying such kind words.” That’s enormous. That one, simple email transformed KCRW from some faceless (yet awesome) community radio station in another country into actual people that are affected by what I do and not do. That email gave me a connection to the insides of the business and it allowed me to get past the dollars and cents and product, and start focusing on the people and the relationship I have with those people.
That email cost KCRW approximately 20 seconds in time and nothing more. The result is that I’m actually thinking about putting money aside every month to become one of their KCRW Angels.
Now, I don’t think my response to their email is typical (in fact, it might be construed as borderline creepy), but it’s a reaction that plenty of businesses can have with their customer base by taking a small amount of time and connecting with their customer base on an honest, straight forward level. If you engage your customer base, the customer base will engage you, and the payback potential is enormous.
Today’s exercise in customer service motivates me to not just create bigger and better software products, but to be engaged with my customers as I do it. To be forthright, honest, and appreciative of their feedback in a true and genuine sense. How can we leverage today’s technologies such as Twitter and blogging to truly connect with our customer base – I mean really connect, not just publicize a product road map. I think the secret is to have really unpolished news and statements, interviews, free-flowing dialog from developers, etc. Let the customer in on the ground floor, let them see the nitty gritty (without giving away your trade secrets). Sure, this will give your competition insight as well, but who cares? Focus on your customer first, your product second, and you’ll make so much money you won’t even remember who your competition is.
Basic Components of a Unit Test
About 7 years ago (okay okay, it was ooooooooooooonly 3 months ago) I wrote part 1 of a hopeful multi-part Introduction to Unit Testing. In that inaugural post I gave a high level over view of what Unit Testing is, the benefits and costs of using Unit Tests as part of your development process, and that you should just believe me and start writing them because they are good and I am smart and you are also smart and great minds think alike. In Part 2 I will define the different components of a Unit Test and how they work together. The technology I will use to illustrate my examples is NUnit – a free, open source unit test framework available for .NET.
The 50,000 Foot View
A unit test can be broken down into 5 parts:
- The System Under Test (SUT) – the particular chunk of code we wish to validate.
- The Testcase Class – The class that contains all the tests we will execute against the SUT
- The Test Method – The actual test
- The Test Fixture – The class responsible for preparing the System Under Test for testing
- The Test Runner – The utility responsible for actually running the test
Each component listed above has specific responsibilities and is relatively self contained. Together, however, they’re like the Voltron of Coding. They come together to create something beautiful and extremely powerful; something that destroys bugs and keeps your universe (domain) a safe and happy place for all travellers (programmers, project managers, marketting peeps, customers, and pointy haired bosses alike).
The System Under Test
The SUT is the class, feature, or some small snippet of code that you want to test. It is the thing that the Test Methods will validate to ensure all is working (or not working) as expected. This is the Universe that Voltron is protecting.
When I first started practicing Test-Driven Development I used to have a pretty strict mindset that the SUT was always the entire class that I was developing. I have found that, over time, this can create Really Really Big Testcase Classes and, just like “normal” classes, Really Really Big Testcase Classes are difficult to maintain and create a lot of headaches and tears of joy pain.
What I do now is I start going down this road, and then I refactor my tests as needed. For example, if I want my class to have the ability to save data to file, then I’ll develop a Testcase Class that tests only the saving features of my class. If that class also has a Load or Parse feature, than I’ll create a Testcase Class that tests only those features. In both these cases it is very likely that I can use the same Test Fixture, so my code reuse goes up, along with my productivity.
Test Fixtures
The Test Fixture is a class responsibile for setting up the SUT and initializing its state in preperation for testing. Sounds pretty vague, doesn’t it? I’ll help illustrate this with an example, because examples solve everything.
Over the past 5 months I’ve been working on a system that’s heavily file based. As such, the unit testing has been a real pain because when it comes to file based systems, it’s hard to keep the results of the previous test from impacting the results of the next outcome without doing a lot of file copy/move/etc. work. In TDD lingo, file-based systems are hard to test because they are Persistent Fixtures by nature (more on that in a future article). To help make it easier, I created a Test Fixture that several of my Test Classes use. This Test Fixture creates, on the fly, the directory and file structure that’s required in order to interact with the SUT, as well as the data-containing files. At the end of my test (during the Teardown phase), this directory and file structure is removed. The Test Fixture creates the directory and file structure from a templated directory and file structure at the start of each test, and destroys that structure at the end of each test. If the Test Fixture didn’t do this, then my tests would manipulate the original files, and subsequent tests would become unpredictable (and you haven’t experienced pain until you’ve experienced the pain of randomly failing tests).
Sidenote: The downfall to the above strategy is that, depending on the size of the files/directory structure that needs to be created, the time it takes each test ot complete is blown up big time, resulting in Slow Tests. Slow Tests are bad because they discourage you from running them. Try to keep the total time to run all your tests to under 30 seconds, if possible, and your life will be full of happiness and beautiful people.
But not only did my Test Fixture create the physical environment required for me to successfully interact with the SUT, it also initialized the various objects that interact with the physical environment. By putting all this (tedious) logic in the Test Fixture, my Testcase Class and Test Methods can concentrate on doing what they’re supposed to do: ensuring the SUT does what it’s (not) supposed to do. Also, it allows each Test Method to have its own Test Fixture, thus isolating the that Test Method from any other Test Method. Hot.
Testcase Classes
The Testcase Class is just a normal everyday class. There’s nothing special about it – it doesn’t hold magical properties or have any powers that are beyond the powers of any other class. The only special thing about a Testcase Class is that it is denoted as a Testcase Class. Some testing frameworks use a strict naming convention to denote a class as a Testcase Class. NUnit denotes a class as a Testcase Class by decorating the class with an unfortunately named Custom Attribute called “TestFixture” (located in the NUnit.Framework namespace).
Wait – A Test Fixture Is A Test Class?!
The above confused me for a great deal of time. The developers of NUnit assumed that each Testcase Class would be written for a Test Fixture. That is, a 1-to-1 relationship of Testcase Classes to Test Fixtures. This assumption starts to break down if you decide to create a Testcase Class per Class, or Testcase Class per Feature (such as my example above). In the real world, I’ve found that “A Testcase Class is a Test Fixture” can create difficult to maintain test classes, and fixture code responsible for preparing the System Under Test tends to be duplicated across multiple Testcase Classes and test suites. For the rest of this article and related articles – in fact, for the rest of your career – define the Testcase Class and Test Fixture as two completely different, unrelated entities and I promise you that your job, your family, your loved ones and all those around you will be better for it.
Test Methods
The Test Method is a good ol’ standard method that validates (or invalidates) something in the SUT. Writing a Test Method is almost more an art than a science, but typically the Test Method has 4 phases:
- Setup the System Under Test (using your Test Fixture, of course)
- Interact with the SUT
- Determine whether the expected outcome has been obtained
- “Teardown” the test fixture and put the universe back into it’s original state (kinda like a soft reset)
Test Methods should be as concise and precise as possible. Writing a Test Method called “ItWorks()” that checks a dozen different things is not a good approach, because when “ItWorks()” fails, you’ll have no idea what is failing, or why it’s failing. A Test Method should be clear and concise. Ideally you should be able to look at a failed test and determine what failed (and why) by just looking at its name alone.
The above implies that you have really good names for Test Methods. Naming is hard. Really hard. I find naming to be one of the hardest things to do in programming. When writing a Test Method I always think “Fact: [State the Fact about the SUT]“. The Fact (minus the word Fact) becomes the name for my Test Method.
For example, lets say I’m writing a Stack class and I’m currently writing the Pop method. I could either write a Test Method called “TestPop()”, or I could write a Test Method called “Pop_Removes_Item_From_Stack()”. When the “Pop_Removes_Item_From_Stack()” test fails (and it will fail), you’ll know exactly what is wrong, where as “TestPop()” will force you to read through n-lines of code, and fire up a debugger, and step through line by line, and cry.
But it doesn’t stop there – because we’ve written test explicitly called “Pop_Removes_Item_From_Stack()”, we must write other explict tests, such as “Pop_Throws_Invalid_Operation_Exception_When_Stack_Is_Empty()”, and other such explicit tests. Sure, the method names are not what we’re used to, but they serve a different purpose. In a sense, the test methods are documented examples of what the System Under Test does, not clear, concise, clever chunks of API that you’re exposing to other developers. Fear not the long method name!
Test Runners
The Test Runner is the thing that executes all the tests. Typically this is some piece of software that reflects over your Test Classes, finding Test Methods, and executing them one by one. Out of the box, NUnit comes with a really basic Test Runner (called NUnit-Gui) which gets the job done, but isn’t the greatest experience. Alternatives are the free Test Driven.NET, which supports a vast array of xUnit frameworks and supporting frameworks (such as mock frameworks, etc.) and plugs right into Visual Studio. The best solution that I’ve used, however, is the pay-for Unit Test Runner that’s part of JetBrains’ ReSharper which, if you are programming in Visual Studio, you absolutely postively have to have. It ain’t free, but it’s worth every single penny, and it’ll pay for itself in productivity boosts within a matter of days.
A Short and Sweet Summary
So, in essence, we have the following. A Testcase Class has lots of Test Methods. Each Test Method is an actual test performed against the System Under Test. In an ideal world, each Test Method will create the Test Fixture, interact with the SUT, validate the SUT does what it’s supposed to do (or not supposed to do, depending on your test), and then restore the world back to it’s original form. The Test Runner ignites the whole thing. Pretty simple, really.
But like most simple things, the idea is simple – actually putting it into practice in a maintainable, sustainable, and useful manner is hard. Hopefully Part 3 – which at this rate will get written some time after the 2010 Vancouver Olympics – can shed some light on how we can approach this.
