sonic screwdriver

A geekier blog

Since I'm starting to get more into technical and programming stuff; since I'd like to say some things on it discoverable by certain programming communities; and since posting too deep on Merb or Rails seems to make a lot of people here go "Whathuh?"... I've started a new blog. It's at:

http://extraneous.org

Right now it's just quick-and-dirty.  I may eventually try to figure out a way to do postbacks to here on personal bits, or just keep them separate, or...  Well, I dunno.  Something.  I'll worry about it later.  But go there if you want to see me geek out.


Invisible Man

Letter to Wycats

So I’ve been a Rails geek for well over a year now. And then I went to Merb Day Atlanta and lo, I was converted. I liked the cleanness of Merb. I liked the incredibly flexible routing engine. I liked that it had a much more “blank slate” feel to it.

Of course a lot of that is Shiny New Thing syndrome, and the difference doesn’t even matter any more because now Merb is going to be Rails, but whatever. That’s not the point of this post. The interesting moment came at the end of the day: Yehuda Katz (leader of the Merb project) gave the closing keynote, and one of the things he mentioned Merb needing was a good content management system that could integrate with applications. This was something I’d been kicking around in my head already – I like some of the ideas of Radiant in Rails but feel its architecture is a bit backwards in some ways – so I spoke to him afterwards about my needs and whether there was anything already going on. He offered a few opinions on the state of current CMS projects, then gave me his card and said “E-mail me.”

I did. The following is the text of the e-mail I sent to Yehuda on December 8th. He never responded. I take no umbrage at that; he’s a busy guy, and I’ve fallen down on most of my own e-mail at Escape Pod. But I’ve continued to toy with these ideas, and over the Christmas week I’ve begun committing some real code to the project. I’m sharing the e-mail in the hopes that others may have useful thoughts about this.

From: Stephen Eley seley@aarweb.org
Date: Mon, 08 Dec 2008 18:09:26 -0500
To: ykatz@engineyard.com
Subject: Merb Day - thoughts on a Merb CMS

Dear Yehuda,

It was a pleasure meeting you at Merb Day Atlanta. I’m the guy who spoke to you after your keynote, saying I had an academic society site to rebuild and a Merb CMS would be a very valuable component. You gave me your card and asked me to e-mail you. Hi.

I’ve done some serious thinking since then about the philosophy you expressed: that what’s needed isn’t just another Rails-like app, but a framework like MerbAuth built on strategies and “communication primitives.” I’m curious A.) whether you know of any work already being done in Merb toward that goal for content, and B.) if you have given any thought already to what that interface layer ought to include for basic CMS functions.

I’ll give you the basic outline of my own thinking, and if you think these ideas have merit, perhaps you may be able to point me to other people or resources that could help put this together:

  1. “Content” is any page on a Web site not generated by an application controller or slice. It’s the spackle filling in all the cracks. Any possible path might have a content page, if the information architect decides one should be there. (But controllers take precedence in the event of pathname collisions. This is a fundamental difference from the /public directory, and one reason we can’t just pipe into it as static output.)

  2. Ergo, the CMS system belongs at the default route, either after or in place of /:controller/:action/:id. I suspect this would be best done with a defer_to in routing, although an exception controller or Rack handler could be other possibilities. Wherever it goes, there’s an invisible (never in the path) controller that shows and manages stored content.

  3. The four basic features of a simple yet useful content system (using Radiant CMS as a model of simplicity) are rendering, editing, metadata, and authorization. Hierarchy may or may not be a fifth. These are your “communication primitives” that can be implemented by separate strategies.

  4. A strategy should support all of the CRUD methods for rendering/editing. The main CMS controller will be RESTful; it will pass its basic work to the different strategies via defined methods.

  5. Strategy slices can store and retrieve content however they want – from a database, flat files, another application, whatever. In addition to showing the content itself, they’ll be responsible for delivering new/edit forms, other pages to manage errors, workflow or other optional features, etc.

  6. On rendering, all strategies will be tried in order until one doesn’t return nil. If none do, it’s a 404. Edits and deletes will pick the same page that a render would. For adding new content pages, either rules can be defined or the user can be presented with a choice.

  7. Application layouts should be used by the main CMS library to wrap content, so that there’s no obvious difference between content and app output. Defaulting to the main layout is obvious, but eventually there ought to be some mechanism (I don’t know where) for specifying an alternate layout.

  8. The content system ought to be able to retrieve and search on page metadata (author, title, last updated, etc.) and provide helpers to the main application so that they can be used in layouts or other app views.

  9. Authorization feels like a different set of choices unto itself, and should not be bound to the content strategy. I know there’s work on a MerbAuthz plugin; the content library ought to leverage that, or whatever other standard rises up.

  10. Hierarchy is very common in the real world: sites are trees and pages have child pages. But I can’t decide whether children should be nested resources with a separate set of add/remove/list primitives, or whether it should just be a side effect of page path, with child listings just one more piece of metadata. I’d value your opinion if you’ve made it this far. >8->

  11. Page state, versioning, and approval workflow are common CMS features, but they’re not external-facing and don’t warrant communication primitives. The individual strategy can implement them if it feels like it.

  12. I’ve been speaking of ‘app output’ and ‘content’ as if they were exclusive, but in future iterations, it could be possible to have both on the same page: app views could invoke helpers that yield content from the CMS wherever the developer wants it. This could make inline help, etc. easier to edit.

…That’s what I’ve got so far. Any thoughts? Is there anything vaguely like this in progress that I could contribute to, or would you suggest I invent this wheel in Merb? Like everyone else in the world, I have too many projects in too little time, but if that’s what it takes, it’s what it takes. I just need to get this Web site off of the shitty ASP-and-Microsoft-Access (no kidding) base that it’s on right now, and I want to do it right the first time.

Thank you very much for your time and consideration.

Sincerely,
Stephen Eley
Director of Technology
American Academy of Religion
404-727-7972 (O) 404-727-7959 (F)
http://www.aarweb.org

Looking back on that e-mail now, some of it I’m already looking at and saying “Yeesh, I really worked to make that too complicated.” I have a slightly different vision now. But it’s where I started, so I submit it for your consideration. I’ll talk more about what I’m actually doing in a near-future post.

Professor

QOTD

"There are two ways of constructing a software design; one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult."
 - Sir Anthony Hoare, Professor of Computing, Oxford University (creator of the Quicksort algorithm)

headphones alex

Merb Day Atlanta, Part 7: Closing Keynote

Yehuda Katz.  The keynote is called "Merb 2.0: The Long March Into the Future."  Core values of Merb:

Performance:
  • Benchmarks show Rack's a lot faster on Thin than Mongrel, on a "Hello World" call.
  • The built-in Merb router (i.e. using defer_to) is almost as fast as Rack on a direct return.
  • Once controllers are involved, it gets a lot slower.  This is where speed could be improved -- not view templates, but simply getting to the controller.
  • "As close to the metal as possible; or as close to the metal as you want."
  • Performance testing with KCacheGrind. (Wow, that's a cool interface.)
  • To do profiling, declare use Merb::Rack::Profile
Concurrency:
  • "The concurrency curve" -- the mroe threads you have, milliseconds per request should flatten out.
  • MRI and JRuby are very similar in concurrency handling with Mongrel; Glassfish handles a much higher thread load.
  • Thread safety is a core value of Merb.  But some features of Ruby are simply not thread safe.  (E.g. Ruby autoload.  Thus it's being taken out of Merb.)
  • "Shared state hurts puppies."
  • To ensure thread safety, use Thread.current ("global variable that's local to the thread") instead of class attributes.
  • Better yet, pass your state around instead of putting it into variables.
  • Use mutexes -- it locks hash states to make non-atomic operations atomic.  But use them sparingly.  (E.g. for global cache.)
Modularity:
  • "Modularity is impossible to determine from a set of talks like today."  (Because the purpose is to show how to get up and running, not how to build complex applications that need separation of concerns.)
  • Merb's dependency structure is highly configurable.  You can remove all dependencies and still have a Merb app.
  • Overriding behavior is more structured than it is for Rails plugins: some methods are marked @overridable and some aren't.
  • Some API functions are marked as 'public' and some as 'private.'  Public methods have a strong commitment that they'll continue working.
  • Examples of overridable methods: _filter_params, _template_location.
  • Template loading itself is modular: by default it looks for files, but you could override to get templates from the DB, from a string at the bottom of the code, wherever.  (Use _load_template_io.)
Good Ruby Citizen:
  • Rubygems sucks in certain ways (dependency failures), but it's getting better.  (By pushing it and breaking things, they're fixing core Ruby features and making rubygems better.)
  • "Working with community == helping the community."
  • E.g. Rack middleware.
Where This Is Going:
  • Apps as a first-class concept.  "Make it so that slices are just regular applications."
  • Slices right now need special configuration; fix it so that all configurations are global but can be overridden.
  • Need a good CMS application.  (I say: "Yes yes YES.")
  • Resources as a first-class concept.  Models and controllers shouldn't need to be passed to each other; let it all be bundled together, and authorization, etc.  I.e., don't have to specify controllers that just do the default behavior.
  • Core principle: "Simple cases can't get harder."
  • Further improve the Merb server.  Monitoring facilities, etc. in the short term.
  • Dynamic worker pools in the long term.  Perhaps remove the need for Nginx or another Web server.  (E.g. Swiftiply.)
  • Phusion Passenger 'could one day be awesome,' but does a lot of guessing.
  • "We want to be able to go onto a server with nothing on it, spin up a Merb server and you're running a bunch of processes."
  • Self-managing cluster.
  • Internationalization.  (Make Ruby 1.9 and Merb work better.)
  • Localization is harder.  Need a few different schemes.
  • Feed syndication, flat pages, route directly to a view and skip the controller.
  • Even better resource() helper.
  • Authentication, authorization, user management -- "Communication primitives."
  • Tailored stacks.  Designers want a framework that's "a bunch of HTML slots"; Web shops want better drop-in slices; etc.
Whew.  Drinks or go home and eat with Anna?  I'll see what John and Mel want to do.

This was a fun day.

the paper

Merb Day Atlanta, Part 6: CouchDB

I'd looked into CouchDB a while ago, and think the idea is amazing.  I started to play with it a little bit for an experimental slush-submission app, but decided the integration with Rails was too primitive and losing all the ActiveRecord magic was too high a price.  I might think about it again.
  • CouchDB is not a SQL relational database.  It's document-oriented: every document can have totally different attributes.
  • It's not suited for every application, but really complex data structures work better with something like this where you can serialize them, instead of having a ton of null fields in a table.
  • Replication, version control, and a REST interface are all central to the way it's built.
  • Everything is stored as JSON hashes. 
  • And queries are Javascript functions.  Literally.  If you want to find anything in it, you write a new Javascript function to do it and store it in the views document.  (This, to me, was the intimidating part.)
  • CouchREST is a simple Ruby wrapper API to access CouchDB and handle simple key and view creation.  This is good; when I watched the Peepcode Screencast a while ago, having to make all the views and determine in advance how the data would be retrieved sounded like more work than just sticking with ActiveRecord.
  • CouchDB sorts all new data according to its views upon insert.  So writes are slow but reads are fast on defined views.  'Temporary' views, i.e. on-the-fly queries, tend to be slow.
  • There's no record locking in CouchDB.  Because it's designed to be distributed, it assumes conflict will happen and you can specify how merges will work.
  • Hmm.  The '_rev' revision number cannot be used for version control at the application level.  It's just for conflict resolution, and old revisions may get flushed.  That bursts one of my illusions.
  • Heh.  The admin utility that comes with it is called Futon.  It's built totally in HTML and Javascript.
  • They're trying to demo this stuff, but the guy's terminal font settings are practically invisible on the screen.  This is why it's good to walk through things before you present.
  • Right now there's no user authentication built into CouchDB.  Hmmm.
  • Because you can build your whole app in Javascript, you could have true peer-to-peer Web apps.
  • I asked if there was a drop-in ORM provider for Merb that would let you use CouchDB instead of DataMapper or ActiveRecord.  They misunderstood my question and though I was just asking "Can you use CouchDB with Merb?"  Yehuda Katz clarified that ORM providers in Merb have other requirements.
The nutshell I'd heard before is this:
  • SQL databases == static data structures, dynamic queries
  • CouchDB == dynamic data structures, static queries
Next up is Yehuda's closing keynote.

alex & daddy

Merb Day Atlanta, Part 5: Slices

Here's where Merb starts to get pretty different from Rails.
  • Slices are small apps that can be included in an app and distributed as a gem.
  • Easy to integrate if written properly. They can stand on their own, or be embedded.
  • You can add them in the dependency.rb file like any other Merb dependency.
  • Same directory structure as a standalone app, as well as a 'lib' directory that includes the gem configuration stuff.
  • The 'config' directory is generated, but doesn't get packaged in the gem.  That's how the standalone/slice stuff works.
  • Sample code at http://github.com/markpercival/quickadmin
  • Layouts can be specified with a default in the lib/slicename.rb file.
  • That's also where metadata and additional routes get prepared.
  • Integration issues: it's hard to call 'before' filters, etc. on stuff in slices because they aren't methods in the main app.  The solution proposed is to create a mixin and include it in the main app module at init time.
  • You can put stuff in a /slices directory of your app, and that takes precedence over gems or the app itself.
  • Build and test the base functionality first, then worry about integrating it into a gem.
The rest is on Merb-Auth:
  • Authentication is pretty much the classic use for slices. 
  • Merb-Auth is a strategy-based authentication mechanism based on a combination of a slice, several strategy mixins, and a library of core methods.
  • The library gives you the ensure_authenticated method for 'before' filters.
  • Not being logged in and trying to access a protected resource is an exception.  The exception object calls unauthenticated which handles it with strategies.
  • You can specify several strategies (OpenID, password, HTTP Basic, whatever) and you're authenticated as long as one of them returns success.
Oh, and Merb doesn't have plugins.  It's all gems, of which slices are a particular type.  In Rails terms, slices would be closest to Rails engines (which, I was amused to hear just yesterday, are being added to core in the next version of Rails.)
sonic screwdriver

Merb Day Atlanta, Part 4: Routing

The guy doing the presentation on Merb routing is a bit of a nervous speaker, which is too bad because the subject is one of the things I wanted to know about most. 
  • Route matching can include regular expressions.  Yay.
  • You can match on subdomains just by referencing the subdomains[] array.  This is great -- I've done some Rails stuff where the domain name is part of the identifier, and it's annoying to have to look it up in the controller.
  • REST resources include the Rails Standard Seven actions, but also an eighth "/delete" action which is supposed to pop up the "Are you sure you want to delete that?" yes/no question.  Hmm.
  • There's a resource() helper that you can call instead of url() that cuts things down a bit, but I'm not totally clear on the difference just from the talk.  I'll have to read more.
  • You can make pretty URLs by including an :identify => option in the route.  Then it'll use that attribute of the model instead of the ID in the generated paths.  (Although it's still params[:id] Seems cleaner than a lot of the 'pretty permalinks' plugins in Rails.
  • You can have a defer_to route that executes any code you like at runtime to determine where things should get routed. 
  • Yehuda Katz (one of the Merb core committers) is right in front of me and he's correcting the presenter on some deprecated routing techniques and things he suggests that are "accidents" of the way the request gets passed.
  • Merb's routing doesn't have any of the recently-controversial memory issues that Rails routes have, because they aren't a great big data stucture.  There's just a single method that does all the routing.

ep froth milk

Merb Day Atlanta, Part 3: Haml

The first post-lunch presentation was about Haml.  He had an hour scheduled, but took half an hour.  It would have been 15 minutes, except I raised my hand and suggested he talk about Sass too.

I've been using Haml for all of my views for ages now, so I didn't learn too much on this one, but it's still nice to see the way I've been doing things validated.  If you aren't familiar with it, Haml's essentially a dumb abstraction on top of ERB that replaces HTML-style opening and closing tags with indentation conventions.  Which doesn't sound like much, but it makes views (or any other Web page code) a lot more readable.  Sass does something similar for CSS, with a couple of other goodies like constants, math, and partials.

Readability matters.  Beautiful code matters, not just to feed programmers' egos but because beauty enhances productivity.  Haml is one small but effective contribution to that.

Not to mention I like the attitude of the subcommunity.  How many technical documentation sites include things like "wildfire of chicken" or "Let's take that fucker and make it a Haml haiku!?"




microphone

Merb Day Atlanta, Part 2

Last hour of the tutorial.
  • Web services have a cleaner syntax in Merb, but they're basically done the same way.
  • Merb picks its format based first on the file extension, then on HTTP content negotiation.  (I.e., the "Accepts:" header.)  But what I'm wondering is: do any clients of anything commonly set those HTTP headers?  When is this better or more likely than asking for product.xml, etc?
  • DataMapper has an 'automigrate' to automatically build your development tables from your model specs but everyone says you should still use migrations for hte production database.  Three people (incl. me) asked if there was a task that worked like automigrate but would spit out a migration file instead of doing it in the DB directly.  The answer was "No, but someone should ask for it and we'll see if it gets done by the end of the day."  I Twittered it.
  • The Merb core code is pretty readable.
  • I like the cleaner method names.  It's not url_for, it's url.  It's not render :partial => "blah", :locals => {:foo => @bar},  it's partial "blah" :foo => @bar
  • Views are turned into methods in the controller when first called at runtime.  So the V and C of MVC are basically the same block of code.  Makes things really fast.
  • Okay, so Merb is using a replacement for 'rake' called 'thor'.  Which is fine, but some tasks are in rake and some are in thor.  That's confusing.  Pick one and stick with it.
  • They're talking about Rack now.  I know it's cool, but apart from "Use Rack and don't worry about it," how do we use this knowledge?
  • --Oh.  By sticking in middleware layers so that you can do special things for special sorts of requests.
  • So the thing Merb was built for in the first place: render_then_call or run_later for true multithreading and actions to defer later on.
Lunchtime!

sepia hat

Merb Day Atlanta, Part 1

 Okay, so let's try this liveblogging thing.

I'm at Merb Day Atlanta -- a one-day workshop to teach Merb to folks who are already experienced with Rails.  I'd heard a good bit about Merb; it was designed to be a lighter, more modular framework that wouldn't have the size, oddities and historical cruft that Rails carries with it.  

I was pleasantly surprised: I'd expected not to know anybody, but my longtime friends John and Melinda are here.  I'd worked with John at the company I got fired from in February.  Seeing them again makes the day more fun.

This morning is the tutorial part.  Thoughts so far:
  • The two presenters are good techies but very amateur presenters. It's clear this is the first experiment at this.
  • They're dealing with the sample code by pushing to GitHub constantly and then expecting people to pull.  Works fine, I guess (as long as GitHub doesn't have performance issues) but they're not really making it clear when there's been a change.  Maybe I should just write a one-day cron job that pulls every couple of minutes?
  • ...Except that as much as possible I'm trying to write the code in my own directory and then typing it in.  Again, mostly works fine, but they need to slow down at one or two points.
  • Merb really is smarter at some of the simple stuff.  I set the config to use_template_engine :haml and suddenly the generators are spitting out HAML instead of ERB.  Lovely.
  • Action arguments?  Also lovely.
  • SLICES!  Yes!  This is exactly what I was thinking of for my site architecture at work: a lot of smaller apps that can interact with each other, separate things like single sign-on, and share layouts and a bit of routing.  Single sign-on was the example they used.  This alone could be reason enough to do more of the system in Merb than Rails.
More later.  I need to finish the authentication part of this. Should just take a few minutes.