Andrew’s Blog

One of the first projects I inherited at Live Oak was a rather large custom CMS implementation with several modules for managing users and organizations in the nonprofit sector. The first iteration had some issues (I won’t get into them in this post, but they will most certainly be the topic of future posts on this blog), and for the last few months, I have been performing a massive overhaul of the data model and several of the frontend processes. It has been great to rip out some of the more obscure code and refactor the rough parts of the application.

However, there were some complications as well. New users have been registering and entering all kinds of valuable information for the last 7 months, and much of this data needed to be moved around to new tables and models. This presented some interesting challenges during development because the changes had to all be implemented in a one time application upgrade, eliminating the possibility of gradually moving the information into the new data model.

Thankfully, the upgrade went very smoothly, so I thought I would take a bit and summarize some of the principles that I utilized to make this happen. This should generalize pretty well to any web application that is heavily data-centric and needs to undergo significant model/schema changes with minimal downtime.

• Automate as much as possible - For small changes to an application, it is acceptable to just keep a running list of steps you have completed in your development environment and then perform these in your live environment when you roll out your changes. However, this is hugely prone to human error, and cannot be tested with great reliability. Maintain a sql file with all of your schema changes, and begin generating a script to perform all data migration operations as early as possible in your development cycle. Ideally, these will encompass all of your data migration (with the possible exception of filesystem changes). You will be most in-tune with the changes that need to happen at the time of implementation - not right before you are ready to go live!
• Use your data models in your migration script - When I originally started working on my migration script for this project I thought I would write an old-school procedural “quick and dirty” script to just shuffle the data around. But as I thought about it, I realized that I was ignoring the dependencies that existed within the database. For example, whenever a new user is created, it is not as simple as just inserting a row in the users table. I needed to create a user group, the associated group memberships, assign permissions, etc… Of course, all of this already happens inside of the data models in the application. So instead of going in the back door, have your migration script utilize your application’s data models so you can rely on code that you have already verified (hopefully!) to follow your business rules.
• Simulate your migration, and then do it again! - Creating a comprehensive migration script during the development phase of your application affords you the awesome opportunity to be able to do some simulations before you actually run your script in the live environment. This is pretty simple. Do a dump of the database from your live environment and import it into your testing database (if your database is too large for this, just choose a representative sample). Now you can run your migration script until you feel comfortable that you are covering all of the different scenarios that exist in your live database. Once you have fixed an issue in the script, just re-import the data from your database dump and try again. I had tested my migration script like crazy before I started simulations, and I still found quite a few issues that only turned up when I encountered the live data that had been entered by real users and not programmers. Once you have run the simulation several times with success, you are ready to think about deploying your application.
• Stage your application in the live environment - I know that it is not possible for everybody to do this. However, I have encountered several ugly MySQL bugs that affected either my development server or my live server (but rarely both). It is not possible to anticipate environment variables such as these - so if it is at all possible, get your application running on a new virtual host in your live environment as soon as possible. This is the perfect place for your co-workers, testers, or even your clients to interact with the application. After a while, you can be sure that you will not encounter issues with your live environment when you go live with your changes, allowing you to focus on more important things.

That should give you a good start on lowering the stress associated with large application changes in a production environment. Of course, I still ran into a couple of issues with the migration process - so don’t get down on yourself. The bottom line is that migrations are really ugly and almost always come accompanied with unexpected issues. The first goal of developers should be to minimize the chances that the affects of the resulting bugs will significantly affect the end user. Automation is a significant step in that direction.

So what have you learned about upgrades and migrations? Please share any stories and lessons that you have learned in the comments.

I have been spending some time today mentally preparing for the upcoming week. We are getting ready to begin planning and architecture on a big project at Live Oak, and it is bringing many new technical challenges that I have not directly dealt with before. I have been stressing for the last couple of weeks to figure these things out: which libraries and frameworks to use, issues with scalability and extensibility, the database layer and ORM, the list goes on. I want it to be utilizing only the newest and greatest techniques, but there are so many options out there that I don’t know where to begin.

As I think about it today though, I am feeling  better about these details. It is not that I have figured everything out in some kind of master plan. Rather, I have begun to think about this project from the other side of the screen: the end user. Like a novel, every line of code that we developers write translates into a unique experience for the end user. Applications, like philosophy or a work of poetry, can either take the end user to a place that is more clear and understandable or can lead to frustration or disconnect. The important thing to remember though is that no technology in and of itself necessarily creates this dynamic experience for it’s users.

Think of your favorite application - what does it convey to you? Gmail is one of my favorites. Gmail tells me that I can make sense of a barrage of information through filtering and tagging and stringing together threads of communication. Moreover, it is an inspiration in the sense that it is as fast and efficient as a desktop application because of the awesome architecture under the hood. I have been using Gmail since very early in it’s life in 2005, and have seen many different technical iterations that were inspired not just by new cutting-edge browser technology, but rather by painstaking attention to what would create the best user experience. That is to say, the client experience necessitated the cutting edge technology and not vice versa.

I think it is easy to forget this as a developer. We spend so much time working on the nitty-gritty details that make things work - which is essential, but often uninspiring. We want to see cutting edge techniques and elegance under the hood. But when this becomes frustrating to do, it is easy to become disillusioned with the whole process. In short, we often put the cart in front of the horse. I think this is less of a danger when the technology that we are utilizing has been carefully chosen to create a particular user experience. It is no longer an issue of technology as an end in and of itself, but rather as a means to something greater.

So I am going to try this week to remember the human side of the software that we are envisioning and architecting. The important thing to think about at this stage of the game is not the inevitably complex architecture of what we are about to do. Rather, we need to develop a clear understanding of the experience that we will be creating for the end user. Once we have paid adequate ettention to these big elements, the rest will be easier to conceptualize and architect.

As I was writing in my journal this morning, I started thinking about the many events in my life that led me to my current career. I’ll spare you the gory details of the typical “college grad meets the real world” story, but there is one factor that cannot be underrated in my journey to become a software developer: mentoring.

Coming from a nontraditional background, I never learned programming in any kind of academic environment. I picked it up while working customer service at a small software company, and played around for months before I tried writing some demo apps and then later started working on some of the internal applications that were used by only a few people. I found programming syntax to come pretty easily since it is just applied logic. However, it was the application architecture that started to get the best of me when I realized that I was running into common problems that had to have simpler solutions than the ones from which I was approaching them.

At this point there were several key people who I turned to when I got really stuck, but one in particular really took the time to actually mentor me and began to patiently sharpen my programming mind by teaching me about design patterns (or more importantly, how to use them to solve real life problems), showing me examples of code that he had written, and describing lessons that he had learned. I am not sure that I understood how important this was at the time, but looking back it is clear to say that I would not be where I am today were it not for his involvement at this pivotal point in my life.

This brings me to my reflection in my journal from this morning (and maybe a little call to action). I am no expert and most certainly have a long way to go still. However, I have learned a few lessons and am beginning to wonder if I may may be able to also play an important part in an upcoming programmers life. The most important thing about mentoring is not knowing every little detail, but rather, encouraging someone to stick with it and to either develop their love for it, or to realize that there is something better on the horizon. As such, I think there are a few key characteristics of successful mentoring in the world of software:

• Take risks - there is no reason to protect a new programmer from your code (that is what version control is for), assign real tasks that will stretch his or her abilities. It is this kind of real world trust that builds confidence.
• Don’t give too much preemptive advice - Allow him or her to take a stab at a problem and then help when asked.
• Frequent code reviews - Once the code is working, walk through it and spend time identifying patterns, pointing out optimizations, and talking about alternative approaches. This helps to frame all of the explanations in relation to problems that have been recently addressed by the programmer
• Warn against pre-optimization - Teach that the first priority is to get something that works, second to that you figure out if there are any bottlenecks that need to be optimized. This one took me a long time to learn, as it was made apparent to me that I spent way too much time analyzing problems because I was so worried that I would slow down the application. I became much more efficient when I learned this simple lesson.

I am sure that this list is not complete - so please add on in the comments if you have any other suggestions. In the meantime - get out there and see who you can encourage someone in their quest to become a good programmer!