Adamanteus 0.5 released, now with more PostgreSQL!
Jun 5th, 2010

I've now finally gotten around to adding PostgreSQL support to Adamanteus! It wasn't really difficult, just hard to find time to do with everything that's been going on lately (quite busy at work and with the new house). The usage of Adamanteus hasn't changed at all, now you can just specify 'postgres' as your backend rather than just 'mongodb' or 'mysql'.

There is one slight caveat, however: the pg_dump utility does not allow you to provide a password non-interactively. For the time being, at least, that means that if you're using Adamanteus to back up a PostgreSQL database you can't specify a password (it will throw an exception if you do). The solution to this is to either 1) set up a read-only passwordless user for running backups, or 2) set up a .pgpass file on the machine from which you intend to run your backups (documentation here). I recommend the .pgpass option, it's quick and easy.

Adamanteus 0.5 is available from both bitbucket and PyPi.

Adamanteus: versioned backups of databases using Python and Mercurial
Mar 26th, 2010

Backing up databases is one of those things that I've always felt could be done in a better way. Traditionally I've done it with a simple shell script that used mysqldump or pg_dump to dump my database to an SQL file named using a timestamp, compress it, and maybe scp it off to some remote server for redundancy. This approach works just fine, except that I recently took a look at my backup directory for a project using that setup only to discover that there were nearly 5000 backup files taking up 11 GB (and this is using bzip2 to compress them!). Obviously not an optimal situation, especially considering that really very little changes from backup to backup, and it's quite possible that nothing changes at all for some of them. It simply makes no sense to store an entire dump of your database every single time!

Fortunately, this is a very familiar situation that we've got advanced tools to handle: version control systems. So I decided to write a little program to replace my shell script that would use a modern, advanced version control system to provide a much more reasonable solution. What I came up with was Adamanteus, a command line program written in Python that allows you to back up your database into a mercurial repository. It currently supports MongoDB and MySQL, and I plan on adding PostgreSQL support this weekend.

Using Mercurial immediately solves basically all the problems with my original approach. It stores diffs rather than full files, meaning you aren't wasting space with a lot of duplicate information. It also handles compression transparently keeping the file sizes down even for the diffs. Plus, because Mercurial is a distributed version control system it's very easy to provide redundancy by pushing and pulling to and from remote repositories. (Pushing/pulling to/from remote repositories isn't currently implemented, but that's also in my plans for this weekend.)

The project is far from complete, but I think it's sufficiently far developed to release as 0.1. Plans for the 1.0 release include:

• PostgreSQL support
• The ability to restore your database from a particular revision in the repository
• automated cloning/pushing/pulling of the repository
• Integration with Django as a management command

I think this is actually pretty close and it probably won't take too long for me to implement all of those, so hopefully I'll be able to push out a 1.0 release very soon. The one other issue holding up 1.0 is that I'd like to wait for MongoDB 1.5 which will bring mongoxport functionality in line with mongodump which is what I'm currently using. The issue here is that mongodump produces binary data files which don't play quite as nice with version control and lose you the advantage of only storing diffs. Mongoexport will export JSON or CSV files, which will allow it to take full advantage of Mercurial, but until 1.5 there's no easy way to use mongoexport to dump all the collections in a database which is the default behavior for mongodump.

Anyway, I'm definitely looking forward to some feedback on this project, as I suspect it could be quite useful to many people. Contributions are always welcome as well!

More on the massive ObamaCare Loophole
Mar 26th, 2010

Since my last post I've seen quite a few other people writing about exactly the same issue with ObamaCare that I saw: it actually encourages people to not carry health insurance. The most recent is someone's whose blog I've been reading for a while and whose opinions I generally respect: Philip Greenspun an EECS professor at MIT. He actually did the math to reveal that a family in Massachusetts making $100,000 can expect to pay around $20,000 per year on insurance. The fine for not carrying insurance, however, would only be $2,000. And, of course, the insurance carriers would be forbidden from denying them of insurance if they showed up already sick and actually in need of health care.

But there's more! Brian Caplan at the Library of Economics and Liberty (which I don't actually know much about, but I would guess is biased in the libertarian direction from a cursory examination of the site) suggests another unexpected side-effect of ObamaCare will be that employers will stop offering insurance as a benefit. Part of the reason it's currently so common for insurance to be an employment benefit is that the government has been encouraging this behavior since around WWII by not counting health insurance as taxable income. This, apparently, is going to change with ObamaCare.

This, however, I think is actually a good thing. I mentioned in my last post that I had some ideas on a different sort of reform that I've posted about elsewhere in the past; below is an outline of some ideas that I put together a couple years ago (around the time that Massachusetts was debating the inspiration for ObamaCare):

• Decouple health insurance from employment
• current system economically stupid
• encourages un-educated consumers
• reduces consumer choice
• reduces competition in the market
• current system unnecessarily restricts consumers
• people with health problems can't change jobs for fear of losing insurance
• true cost of insurance is hidden from consumers
• Make medical costs tax deductible
• all costs should be up to 100% tax deductible (maybe variable by income)
• makes health care significantly more affordable
• provides the same functional assistance as a single-payer system
• doesn't encourage reliance on the system
• doesn't require increased bureaucracy
• encourages personal responsibility
• Provide low/no interest loads to cover medical costs
• interest rate based on need
• removes problem of non-payment
• allows even the uninsured to afford very expensive procedures
• doesn't encourage reliance on the system
• payments are tax deductible under point 2
• Loosen regulations on insurance industry
• regulations such as setting a maximum deductible hurt both sides
• consumers aren't allowed to decide for themselves what they need
• insurers aren't allowed to innovate and find new solutions
• regulations reduce competition
• not all of them!
• regulations provide a disincentive for new insurance companies to enter the market
• the more insurance companies there are, the better for consumers

Obviously this is still rather rough, and some of the points are clearly designed to address things that have now changed. However I think this is a decent outline of changes that could be made to healthcare that would meet the goals of both sides of the public/private argument. I'm sure there's tons of discussion that could be had on just about every one of my points, so perhaps I'll take the time to break each one out into a separate blog post where I can go into more detail in the future. Also, I'll try and write some more about the code I've been working on as there's some good stuff involving MongoDB, MongoEngine, and—of course—Django.

What does ObamaCare actually do?
Mar 23rd, 2010

I've been generally on the fence about ObamaCare—and really about the various proposals to reform healthcare in general. I do think that our system needs reform, it's just that I've been unconvinced about the specific proposals made for the form that reform should take. But now it's 'happened' (although what we have is not really healthcare reform so much as health insurance reform) so the question is now different: what are the implications of our new system, and where do we go from here?

The implications, I think, are both quite interesting and not at all what most people expect/want them to be. For example, I think the new bill will actually increase the number of uninsured people in America (at least in the short term), and I think it will do this via two separate mechanisms:

1. An active dumping of high-cost and/or risky customers by the insurance companies who currently cover them, and


2. People taking advantage of the new system to actually save money without sacrificing healthcare by foregoing insurance

But isn't the whole purpose of the reform to make both of those specific things impossible? It's certainly the stated purpose, at any rate. So why would I claim that the effects of this bill are going to be exactly the opposite of what we were promised they would be? Well, what does the bill actually do?

Let's look at the first mechanism I mentioned above: insurance companies actively dumping expensive patients (even more so than they already do, I mean). Starting in 2014 they will no longer be able to do this, they will have to cover everyone. But for the next four years they are free to continue doing so, and it makes a lot of sense that they would ramp up this practice so that they can increase revenues as much as possible in preparation for that future. In the meantime the government is offering some sort of high-risk pool of insurance for those people that can't currently get coverage, so it won't even seem like that bad of a deal: the insurance companies win by increasing revenue, Obama wins by being able to point to the vast numbers of people that are already being helped by the program, the patients win because they have health care, and the taxpayers win because... well, actually we don't so much... But that's really not so bad; it's hard to complain about people with health problems getting the help they need, even if you don't think they're getting it in the best possible way and you don't want to have to pay for it. It's the other mechanism listed above that I think is more insidious: gaming the system.

How and why would people game the system? Well, it's actually quite easy and straightforward: as things stand it's quite easy to get the best of both worlds: save money by not purchasing health insurance and still be able to rely on health insurance to pay for your medical expenses. The reason this work is that the 'individual mandate' of ObamaCare is toothless: just as in Massachusetts it's cheaper to pay the fine imposed for not purchasing health insurance than it is to actually purchase that insurance. $100 per year or 1% of your income (whichever is more). That's a pretty paltry fine (though it will increase in 2014 along with the activation of the various other parts of the program). And couple that with the high-risk pools that you'll be able to buy into to get government subsidized healthcare regardless of pre-existing conditions it renders this whole change meaningless (or worse?)! Why would I pay thousands of dollars per year for insurance when I can pay much less and then, if I get sick or injured, simply buy into a plan that I legally can't be turned away from? I'd end up paying less money than I would if I actually carried insurance, but I would still get all the benefits of that insurance! As long as the fine/tax for not carrying insurance is less than the cost of actual insurance, and as long as it is required by law that you be able to buy insurance at any point regardless of your current health conditions, there is absolutely no incentive for anyone to ever purchase insurance right up until the point that they actually need it!

The implications of this are quite staggering: they basically undo the whole fabric of our healthcare system. Our system (even now) is based upon the idea that it's relatively cheap to pay for routine healthcare costs, but relatively expensive to pay for the bigger and rarer procedures. So we amortize those costs over time and over a large pool of people by buying into insurance plans where everyone pays in a relatively small amount and then, on the rare occasion that they need it, they're able to draw a large amount to cover larger medical expenses. But ObamaCare has undone that! Now we don't need to pay into the system (except in a barely token amount) in order to draw from it! If and once people realize this and start dumping their insurance plans, this will lead to the collapse of the insurance industry; it will simply be impossible to operate under the insurance model.

The result of this will be greater and greater financial obligation on the government to support the insurance industry and pay for that healthcare and therefore a greater and greater tax burden on us to pay for it until, eventually, we are all simply paying taxes into a government system that funds all our healthcare needs. Sound familiar? That's because it's a single-payer system—a monstrously constructed single-payer system built with the rotting corpses of the various insurance companies.

This is, of course, a worst-case scenario. In order for this two actually happen at least one of two things has to be true:

1. Our government has to be incompetent enough to not recognize and fix these problems before they overwhelm us, or


2. The intended goal of this plan was, all along, to simply set us on the path to a single-payer system

Neither of these options would really be all that surprising to me, frankly. I'm pretty sure our government is that incompetent, and I'm pretty sure that the people who initially conceived and designed this reform would prefer a single-payer system. But we'll have to wait and see what happens. Regardless, I think it's safe to say that there are really only two possible systems for health care that would be stable: a fully private system, or a single-payer system. There are benefits and disadvantages to both, and there are all manner of compromises between them that I think could actually be good (which I've written about elsewhere in the past, and should really write a post here about as well). But—as much as I hate to sound like I'm parroting those who truly are on the lunatic fringe—we really are looking at the opening salvo in a battle for single-payer healthcare in the US. Whether you think this is a bad thing or not is up to you.

A zsh prompt for Mercurial users
Nov 16th, 2009

My friend Sebastian Celis recently posted on his blog about a zsh prompt for Git users. Basically, it's a set of scripts for ZSH that allow it to display the current status of the Git repo you're currently in. Very cool stuff, but unfortunately I don't use Git (very often), and instead use Mercurial for most of my projects. So I decided to modify it to work with Mercurial.

Very little has changed from his Git version (in fact, in most files it was a simple s/git/hg/), so I'm not going to go over how it all works. If you want to know that, you should read his original blog post. Instead, I'm just going to link to my bitbucket project for it: Mercurial for ZSH.

It is, at this point, a pretty half-assed port. There's still some work to do to fine-tune it for Mercurial, but it works. Another thing I'm interested in doing is seeing if I can get it to auto-detect what VCS is being used for the current directory and act accordingly so that it doesn't have to be limited to either Merurial or Git (which goes along nicely with another project that I'm working on and will hopefully be able to write about soon). But, half-assed or not, I think it may be useful to anyone out there using both ZSH and Mercurial (or any any VCS, if you want to fork the code again).

Django admin awesomeness
Oct 15th, 2009

Yes, I realize that this is now my third post not related to DVCSes since I said my next post would be about DVCSes. So sue me.

I recently encountered an interesting requirement for the Django admin: we wanted people to normally only see (and be able to edit) objects that either they created themselves, or that the creator had assigned them to. Fortunately this is insanely easy in Django 1.1, all you have to do is override the queryset() method of the appropriate ModelAdmin like so:

This snippet very easily allows you to apply essentially any filter you want to the QuerySet that gets passed on to the change_list and allows you to provide an exception for super users (always a good thing!). That part was really easy and something I've done before (even in 1.0.x where the functionality is there, just not exposed). Where it got trickier was where the client also wanted the functionality to be able to view all the objects regardless of who created them, but still only have editing capabilities for their own objects (and only see the information available on the change_list for others).

This part was much harder, but fortunately also made very possible by the updates to ModelAdmin in the 1.1.x branch. The first thing I wanted to do was just provide a new URL and view integrated seamlessly into the admin. Again this was very simple with the new admin, and required only overriding the get_urls() method on the ModelAdmin:

Getting that all_objects view to return something essentially identical to the normal change_list, but with a completely different filter on the queryset, and some different display options was the real problem I had to address. Wrapping my head around the problem took some time, but fortunately even this was pretty simple once I really started to get into the flexibility of the ModelAdmin class.

Broken down to it's most basic level, what I wanted to do was return the change_list view from a Model Admin. This in itself is very simple to do, and requires very little code:

Once I figure that out it was pretty obvious that what I needed to do was subclass my ModelAdmin and just re-overide the relevant functions. Turns out this is really easy to do, and gives you a whole lot of flexibility. So I created a special AllItemsAdmin subclass of ThisAdmin, and overrode that queryset() method to return all the objects. I then had to figure out a way to get it to only display but not link to edit pages for objects that the current user doesn't own. Since I needed them to be in the queryset, this was a little trickier.

Anyone who's been working with the Django admin should know about the list_display option on the ModelAdmin. What you might not know (I didn't until recently) is that no only can the list_display list contain field names and properties from the model (Actually, if you didn't know that you can provide callable properties from your model, you should check that out. It lets you do some very useful things.), you can also provide callables from within your model admin. So you could set your list_display to something like ['title', 'my_function'] where my_function is a method on your model admin with a definition along the lines of my_function(self, object) that can perform operations on the Django object for that row and return whatever value you want. The normal options for model properties (such as allow_tags and short_description) work here as well. So what I was able to do with this was create a custom column for my change_list that looked at the specific object, checked the ownership properties as above, and then returned either just the name of the object, or the name of the object as a link to the regular edit page for that object. By setting list_display_links to (None,) I was able to prevent any of the fields from automatically be turned into links. Of course doing this required that method to have access to the request which it usually wouldn't, but since I was already working with a hacked up subclass of my ModelAdmin I was able to just override the __init__() to take the request object and pass that in when I instantiated it. What I ended up with was this really awesome view (if I do say so myself):

As you can see, this is also using another new feature in the 1.1.x admin: reversing of admin URLs. After this all I needed was a few simple changes to the change_list.html template for this model let me add a 'View all' link to go to this new view, and then the 'all' context variable being passed in as extra context simply tells it to provide a link back to the standard view otherwise.

The result of all this was a seamless integration of my custom 'view, but don't edit all' view into the Django admin.

Sorry about the downtime
Oct 14th, 2009

I just happened to check my Google analytics today and saw that my hits had dropped essentially to zero since October 8th. Turns out that my wsgi file was a little screwed up after my initial attempts to migrate my blog to django-mingus. I have, obviously, fixed it, though I still want to move the blog to mingus and will hopefully get the chance to actually do that soon. Until then, however, at least my blog is up!

Storing IP addresses as integers in Python
Oct 7th, 2009

I just saw this very interesting (for a programmer) blog entry on Storing IP addresses as integers. As the article says, anyone who wants to store IP addresses in a database is generally going to do as a string. However if you're really concerned about memory efficiency, you might want to find a lighter data type to use. Since an IP address is really just a collection of integers it would make sense to store it as an integer, however doing so without losing important information (specifically, the placements of the dots) becomes an issue.

The blog post in question introduces the the pack() and unpack() functions, which 'allow you to create and extract data into and out of binary-packed strings' and provides the Ruby code necessary to encode an IP address as a simple integer and decode it back to a dotted quad. I thought this was pretty cool, so I decided to write the equivalent Python code. This is what I ended up with:

It seems to work as advertised, though the packed string the encode() function returns is different from what Ruby gives you, so they won't interoperate (it would probably only require changing the pack() and unpack() formats to do so, but I didn't feel like experimenting to figure out which ones).

Of course, being a Django guy, the immediate application of these functions that I see is creating a new IPAddressField that stores the address as an integer instead of a string transparently.

Building RESTful APIs with django-piston
Sep 26th, 2009

Discovery Creative has a lot of web sites and apps out there. And quite a few of them need to send email in one form or another. Previously that has meant that every single one of those sites/apps needed to implement it's own mechanism for sending email. This is obviously a bit of a pain, not to mention a potential security risk, and a blatant violation of the DRY principle. So I was tasked with building a better mousetrap, as it were.

One of the cool new things I had heard about at DjangoCon 2009 (which I really should have written about...) was Piston, a framework built in Django for building RESTful APIs by the bitbucket guys. I'd never actually built any sort of API before, but it seems to be the thing to do, so I decided to take django-piston for a whirl and how it works. As it turns out, it works extremely well. So well, in fact, that when coupled with some of the fun tools that Django provides (such as ModelForms) you can easily build a RESTful API in no time at all.

Thanks to django-piston I was able to create a simple API that will allow us, moving forward, to use a single, centralized email solution for all our web apps. In fact anything that can send an HTTP POST request (including curl, which is what I've been using for testing) can send email using the API I created so long as it can also handle HTTP authentication (which django-piston easily handles against django.contrib.auth). I'm hoping, after a little more work and refinement, to open-source our API so that other can benefit from it (and, hopefully, contribute back to it!), but for now you'll have to make do with a sample project that I threw together for the most recent django-district meeting this past Thursday: Django-Piston Presentation. It's a bitbucket repository that includes all the code for a simple RESTful API that allows you to create, fetch, and delete objects from a simple Django object. This particular project was designed to be extremely flexible, and all one needs to do is add or remove fields from the model (or point it at a different model) to adapt it to just about anything. Hopefully it will serve as a pretty good instructional example to anyone who wants to create their own API with django-piston. The project also contains my notes on how to build and test it in an Emacs org-mode file.

Next up: Git, Mercurial, and moving on from Subversion.

Comments are go!
Sep 18th, 2009

Just a quick note: this blog now has Disqus comments. Turns out it's insanely simple to implement, thanks to Arthur Koziel's awesome django-disqus app. Took me no more than five minutes during my lunch break to get everything to it's current state.

Blog has been migrated to the Rackspace Cloud
Sep 17th, 2009

One of the reasons that I haven't been blogging lately is that I have a lot of planned changes for the blog. Most importantly, I've been planning on moving it to my Cloud Server on The Rackspace Cloud (née Mosso). I've finally taken care of that, and this post is the first on the new server. If you can see it, that means the DNS changes have propagated.

Now that I've done this I have a number of other changes planned. I'm going to completely overhaul the backend to the blog. In the course of the migration I upgrade from Django 1.0.2 to Django 1.1. Nothing else has changed yet, but I intend to also revamp the templates, add some fancy new feature to better integrate with the rest of my online life, and start using django-mingus. One of the more urgent changes I want to make is to switch to Disqus powered comments. Previously I was using django.contrib.comments along with Akismet for spam filtering. Akismet really just hasn't worked at all for me for the past few months, which is why comments are current disabled. Fortunately, Disqus should cut down on the spam by requiring registration without imparting too large an onus on the readers by using a number of common authentication backends like OpenID, Facebook Connect, and Twitter. I'm hoping it will make for a good compromise.

The most important change, of course, is that I intend to actually start writing again. I've got a lot of ideas that I've wanted to write about and simply haven't because either I didn't have the time, I wanted to wait until this migration, or I was just too lazy. Hopefully, that will be changing now!

The Colony: Another Discovery Creative project goes live!
Jul 9th, 2009

I will get around to actually writing something soon, I swear! But for now you'll all just have to make do with another announcement of a project going live. This time it's actually up on Discovery Channel site. It's a Flash and Django based app promoting one of Discovery's new shows The Colony, about survival in a post-apocalyptic LA. I've actually been looking forward to this show, can't wait to watch it!

Silverdocs.com: My first Discovery Creative project goes live!
Jun 2nd, 2009

I'm actually a bit late on this one, but I've been so busy with my next Discovery Creative project that I just haven't had time for much else. Anyway, from June 15 through 22 AFI, in partnership with the Discovery Channel will be putting on their Silverdocs Documentary Film Festival. Check out the website for more details on the festival and the films they'll be showing, not to mention an example of how awesome my (and the rest of Discovery Creative, I guess) work is.

No more dydxtech.com!
Jun 2nd, 2009

This is just a quick update. I'm phasing out the dydxtech.com. As a result, dydxtech.com now redirects to this page. None of the content from the original site is sticking around other than my portfolio, which is now available on this site at joshourisman.com/portfolio. I've done a half-assed job at re-skinning the portfolio page to look like the blog, and I'll put in some sort of link to it eventually.

Life in Maryland
May 10th, 2009

I've now been living in Maryland and working at Discovery Creative for a whole month, so I think it's about time I started writing again! My first couple weeks I was here alone while Jessi finished things up in Boston I had little motivation to do much outside of work, so I found myself going in to work early and home (I was staying with my aunt and uncle who graciously offered me a place to stay at their home in Bethesda during the limbo time between moving out of the condo in Somerville and into the apartment in Silver Spring) late. That left me pretty exhausted at the end of the day so relaxing, eating, and sleeping were much higher on my to do list than writing. Since Jessi got down here my time outside work has been dedicated to unpacking boxes and transmitting what little I know of the area to her and her sister Becky who flew out to help with the move. Now, however, we're unpacked—if not completely then at least enough to be comfortable—with most of our furniture in place and awaiting delivery on the few remaining items that we've purchased. So time to start looking towards the future again.

My experience here thus far suggests that there's a lot of interesting work ahead. I've got a few Django projects already going on, including working with some new stuff like OpenID, OpenCalais, Clickpass, Twitter, and many other things. I'll probably be starting work on a project using Google App Engine in the near future, and I've already begun learning about and starting to work on iPhone apps as well. So I should have lots of good fodder for technical posts in the coming months!

On top of that there's a whole new city to explore, and nearly the entirety of my family spend time with. Before we knew we were going to be moving, Jessi and I had been planning on getting kayaks this summer so we could spend some time on the Charles and Mystic rivers, maybe the Harbor Islands, and hopefully take them up to the Adirondacks to explore the lakes up there. That plan certainly hasn't changed as now we've got the Potomac and other bodies of water to play with. Plus we're now so close to Shenandoah that it would be a crime not to get in some camping and backpacking. (And after the missed opportunity last winter, I'm definitely planning on some winter backpacking in the Senandoah back country this year!)

So with any luck I should be doing a lot of writing on a lot of different topics in the future. At the very least I need to do some work on this site as I want to integrate my portfolio into the personal site and phase out the business one. And working on this site always seems to give my something to write about.