Helpers, scaffolding, tradeoffs and other stuff
In one of the very, very few coherent things I’ve seen him say in comments posted here and elsewhere, one Lucas Carlson brought up the other perceived advantage of JavaScript helpers: they save time:
Sure it is possible to add javascript helper functions to Django, and yes that would speed up initial development times and reduce bugs since Python is usually more terse than JS… even for expert JS programmers.
I’m going to ignore the “fewer bugs” argument because it seems to me that writing code in one language which ends up outputting code in another language is something that, however terse the input may be, is inherently just begging for problems.
Ian Holsman touched on the idea of time-saving and development speed in a more coherent way, over a series of comments.
helpers will get the majority of the work done for me. If I hit a bug, or want a feature the helpers can’t do THEN I start coding javascript. the helpers just get me further in less time.
could you imagine telling your boss.. “i had this great idea last night, and here is a working prototype…
it looks a bit fugly.. but it is good enough to show to some marketing types, and if we can get support from them we can hire a designer to make it look nice and work on the UI a bit more.
(Aside: I’ve used Django to do pretty much that exact thing.)
right now django gets me going with generic views, and the admin stuff, but then leaves you to your own devices. if it could provide a default CSS, and a default ‘generic’ form/layout it would help people get something up and running faster. and you can prototype your app and how the pages would be structured a bit easier.
So let’s talk about development speed for a moment.
The key word here is “initial”
It’s nice that Lucas recognizes this, because it’s the great deep dark secret of “helpers” of all stripes: the only time they reduce is that which comes between the start of the project and the first demo.
To understand why, consider a somewhat related feature that several people have asked Django to provide: the “scaffolding” offered by Rails and a couple of its clones. Scaffolding is a set of automatically-generated views/templates (in Rails, using the default template engine, a “view” and a “template” are pretty much the same thing — in other frameworks or other situations, not necessarily) which provide basic CRUD functions for a given data model. They’re provided with the goal of getting you off the ground quickly, and do so in two ways that I can think of:
- They give you an immediate rough user interface which can be tweaked a bit and then shown to the client for feedback.
- They provide a place where content administrators can begin putting stuff into the database while development of the application proceeds.
But somewhere along the road to deployment, there will be a moment when scaffolding just doesn’t cut it anymore and you’ll have to write the real views and/or templates for the project. This is intentional; as the name implies, scaffolding is meant to give you a place to stand while you build the actual application, not to give you a finished structure.
Generally, that moment will come after you’ve had your first successful demo and gotten the feedback you need to determine what the client really wants (as opposed to what the client initially asked for, which is never what they’ll actually end up wanting). So scaffolding gives you an incredibly fast path to your first demo and the feedback you’ll need to continue, but after that it’s pretty much useless and needs to be ripped out and replaced.
Back to JavaScript
Now, I’m going to go out on a limb and assert that the same will be true of pretty much any system of “JavaScript helpers” anyone ever develops. The reason for this is simple: there’s no such thing as “one size fits all” in web development.
Yes, you could provide a flexible set of “helpers” which just drop arguments into basic JavaScript functions, to get “flash a yellow fade behind the div with this ID” or “submit this form via XMLHttpRequest” or whatever.
But that’s it; for more complex effects — and sooner or later you’re going to need more complex effects — you have to buckle down and write JavaScript. Otherwise your “helpers” are going to get more and more complicated and less and less useful as you try to accomodate more and more needs, and very quickly you’ll arrive at something that’s not recognizably less difficult than using a good JavaScript toolkit.
Or — and I know this is a crazy idea — you could start out differently, by picking up a good JavaScript toolkit and learning how to use its equivalents of “yellow fade” and “AJAX form submit”. Then as your needs get more complex, you don’t have to throw away your helpers and start over with custom code; instead you can take the JavaScript you’ve been using and keep building on it with, well, more JavaScript.
To help illustrate this, let me talk for a minute about code generation.
DO NOT EDIT THIS FILE MANUALLY
One of the perks of my job — at least to me — is that I get to poke around in the company’s code repository, which has a full revision history of the code which eventually coalesced into Django. A few thousand odd revisions back, the proto-Django had features which, like scaffolding and helpers, would generate code for you; remnants of that are still to be found here and there in files which begin with the ominous message, “DO NOT EDIT THIS FILE MANUALLY. IT WAS GENERATED BY A PROGRAM.”
To touch on the “one size fits all” idea again, one of the problems with this sort of code generation is that you inevitably pay a hefty price in flexibility; the system is only as smart as you make it, and if you ever need to add “just one little extra feature” you need to re-wire the whole thing.
After a while, the effort and time spent rewriting the helper every time you need something just a little bit different start to get wearying and costly, and a better solution presents itself: take common pieces of functionality, and encapsulate their logic in a reusable way. Django’s generic views do exactly that, distilling and genericizing the logic of common operations into functions which are callable from anywhere. Where before the process of tweaking a generic view might have looked like this:
- Find the program which generates the views for this operation.
- Edit that program.
- Test that it now outputs views which do what you want, without adversely affecting any other views it has to generate.
- Re-generate your views.
Now it looks like this:
- Write a wrapper function around the generic view.
Django’s admin application also used to do a lot of nasty on-the-fly work to programmatically generate its templates; people who’ve been following Django from the beginning will remember this, because it wasn’t fixed before the first public release. The “new-admin” branch ditched that and introduced a simpler system which avoided generating templates on the fly; now the admin is much easier to maintain, and is much easier to selectively extend and override by adding, removing or changing individual templates.
JavaScript “helpers” in frameworks are just “THIS FILE WAS GENERATED BY A PROGRAM” in slightly prettier clothing; they can only do what they’re programmed to do, and it doesn’t take long before the cost of re-programming them to suit your changing needs gets to be prohibitive.
Meanwhile, those pesky JavaScript toolkits that nobody on Earth seems to want to learn have gone and done the JavaScript equivalent of generic views: they’ve taken the common bits of logical functionality and encapsulated them in reusable ways. Sooner or later that’s what you’re going to have to do, so why not be doing it from the start?
Anyway, back to the real meat of things, which is what actually goes on when you use a helper.
It’s a trade
What’s really happening every time you use a helper, or scaffolding, or anything else of that nature, is that you’re making a trade: in return for decreased time until you can demo your application, you accept the penalty of having to go back later, rip out the automagical generated stuff and write real code later on. You were going to have to write real code no matter what, of course, but using the helper put that off until a little later on in the development process.
Hopefully by now you understand why that is: helpers, by their nature, are never going to be able to scale up to the complex effects people will need for production-quality applications, so whenever they’re used they’re eventually going to have to be replaced.
On “agility” and how helpers don’t help it
Of course, at this point plenty of people will be thinking that the process of starting out with scaffolding/helpers, then progressing toward real views/templates/JavaScript exemplifies “agile development”.
True agile development is a great thing; one of the biggest of its Big Ideas is that, once you start delivering, you’ll have the ability to keep on delivering, with consistent quality, as various aspects of the project change over time. Often this aspect of agility is also referred to as “sustainable development”.
In agile development circles you’ll often hear talk of a principle called “YAGNI”, which stands for “You Aren’t Gonna Need It”; much of the day-to-day work of sustainability consists in answering two questions:
- What are we actually going to need?
- When are we actually going to need it?
Answering these questions correctly is an art which can take a lifetime to master, and requires an acceptance of the fact that the answers will change as you go. The linchpin of the whole thing is finding an answer to a third, overarching question: how do we set things up from the start to make it as easy as possible to add new things as we need them?
Dissertations have been written and stupendously long flamewars have been fought over the answer to that third question, and there’s still no consistent right answer, just lots of general guidelines and a need for people who have smarts, experience and a finely-tuned sense for figuring out the specifics on a case-by-case basis. But I can tell you, here and now, that anything which involves the phrase “and somewhere we have to plan to pause, rip out what we’ve been doing and write the real application” is starting out with the odds stacked heavily against it.
Another of the Big Ideas of agile development is short timeframes for the various iterations of the project; rather than a monolithic eighteen-month development period followed by a gala product release, agile development focuses on quickly making a set of changes — with a timeline measured in weeks or even days — and rolling them out; lather, rinse, repeat.
But the time tradeoff involved in using helpers to kickstart your project seems to contradict the goal of agility; you could take a stab at incrementally pulling them out and replacing them with “the real code” over a few iterations, but that’s going to add additional planning complexity to your project as you try to figure out which bits need to get rewritten in which order. And it will feel as if, instead of being agile and incremental, you’re back in a long-term monolithic-rewrite frame of mind.
So, no. Starting with helpers and ripping them out when you need “real code” later probably doesn’t exemplify agile development.
On the other hand…
There is something you can do, however, that will promote agility: start out with real code. The sooner you ditch the training wheels and write real, reusable, extensible code, the better off you’re going to be in the long run.
That means that you, or someone on your team, will have to actually learn JavaScript, rather than asking the backend guy to write “helpers” you can use.
That means that you, or someone on your team, will have to actually mock up page templates at the very start, rather than asking the backend guy to generate “scaffolding” you can use.
That means it will take you a little bit longer to get to your first working demo, but if your people are good at their jobs the time difference isn’t going to be significant enough to warrant doing things any other way. And the long-term benefits in agility and sustainability will be more than worth it.
The end
This ended up being much longer, and taking much longer to write, than I’d initially planned, but now I think I’ve exhausted the topic of “helpers” and my feelings about them.
Any questions?