programming

Tue, 04 Feb 2014

Computing for Data Analysis
Posted: Tue, 04 Feb 2014
Two mathematical shames haunt me from my college days. I never really understood calculus as much as I understood how to make Mathematica solve calculus problems, and I never took a class in probability and statistics.

I’ve been working on that second shame source over the last few weeks. In addition to self study through Statistics, I stumbled by chance upon Computing for Data Analyis shortly before the last session began. From there I also picked up S Programming, but let’s talk about the course rather than the books.

The course does not explicitly cover statistics as some had hoped. It provides a solid introduction to many aspects of the R programming language that is often used for statistical analysis. Even without any statistics experience, I kept up without any trouble, and my R experience will help as I dive deeper into stats in the coming weeks.

The course is short, and they estimate an average of 3-5 hours per week. Their estimate was spot on for me, but I already have a couple of programming languages under my belt, so this was incremental learning. People with limited software development experience struggled considerably with the programming assignments and spent more than the estimated time on the course. The last assignment involved extracting data from an ugly HTML file with regular expressions: it was like an easy day at the office for me, but only because I’ve extracted data from uglier files with. Your mileage may vary.

As I started to learn R during the class and thus started talking about it trying to recruit friends to join, Tal Yarkoni published The homogenization of scientific computing, or why Python is steadily eating other languages’ lunch

A Python guy learning R while an R guy blogged about Python taking over his programming toolbox amused my friends. I preach frequently about picking tools and sticking with them to increase the velocity with which interesting solutions find their way into users’ lives, but this was study time, not work time. R transitions quite easily from an interactive language to a “traditional” programming language, and the language design aids that continuous transition. For instance, most programmers spend a considerable amount of time working within the confines of a “for” loop, but “for” loops get ugly in an interactive environment. The vectorized nature of data and functions in R mean that, although occasionally useful, writing “for” loops is usually unnecessary and stylistically improper.

If you’d like to learn more along these lines, there’s a new Data Science Specialization offered by Johns Hopkins through Coursera. The course that I just finished is replaced with the “R Programming” course and you can mix and match classes if you aren’t interested in the certificate at the end.

If instead you’d just like to click a link to a new tool, check out knitr. It’s sort of like the IPython Notebook, but it feels like document creation may be a little more natural, and knitr supports multiple languages. I’ve only casually used both, so I’d welcome more enlightened opinions on them.

Computing for Data Analysis
- programming
Sat, 23 Nov 2013

Customize Your Customer Service
Posted: Sat, 23 Nov 2013
Have you considered purchasing over seven hundred customer service numbers? Twilio rents toll numbers for a single dollar a month, and toll-free numbers only cost a single dollar more. Twilio’s great, but they aren’t even the lowest cost provider. At those prices, can you afford to not own seven hundred customer service numbers?

That’s what the developers at Hotels.com did to help customer service agents quickly identify why a customer was calling and what they could do to help:

The company even created separate phone numbers (more than seven hundred in total) that dynamically appeared based on the page and how the customer got to it, so that when customers called a certain number, it would be obvious where they encountered problems.
Analytics at Work, chapter 8, page 142

A casual inspection of the site shows that this program was successful, because it’s still in use:

You can check it out yourself by watching the number in the upper left corner of the screen change as you first visit the site, search for a hotel, and click on a hotel for additional details.

How can you use data to customize customer service?

Anonymously collecting and aggregating it is a good first step. If you aren’t tracking usage of your sites and applications, you’re ignoring valuable data. There is a major US based website that was localized for countries across the globe, and their Japanese site had conversion rates significantly lower than any other location. Were there cultural differences at play? Analytics reports for their multi-page sales form showed that 90% of their customers bounced between two specific steps. Analysts found that the bounce inducing page was never translated from English to Japanese.

You may know about the DRY principle. Some programmers live and die by this one single acronym: don’t repeat yourself. Don’t make your customers repeat themselves, either. Ask for confirmation that the number on their account is the number their calling from, ending in 1234, rather than making them punch in the whole thing.

Allow users to opt-in to deeper data collection. If something goes wrong, don’t just show them an error page and apologize.Allow them to file a support request from the error page and automatically associate it with their account and bundle a log of the activities they were trying to accomplish with the ticket.

Just don’t be creepy. You probably remember Target taking all of this a little too far when it announced a teen daughter’s pregnancy before she did.

Customize Your Customer Service
- books
- customer-service
- programming
Sun, 03 Nov 2013

Tracking Habits is a Habit worth Having
Posted: Sun, 03 Nov 2013
Tracking habits is a habit worth having. My favorite tool is HabitRPG, but it doesn’t provide an easy way to view all of my progress (yet). To cement the knowledge in Interactive Data Visualization for the Web I created HabitRPG Chains.

Through the magic of cross domain AJAX and HabitRPG’s API, the page renders all of your recent daily habit activity. Green is good, red is bad, light gray is a day off, and dark gray is missing from the database.

There’s a demo account if you’re not a HabitRPG user. Check it out and let me know what you think.

Tracking Habits is a Habit worth Having
- programming
- habits
Sun, 27 Oct 2013
Interactive Data Visualization for the Web
Posted: Sun, 27 Oct 2013
A very concise explanation of callbacks and scope in JavaScript care of Interactive Data Visualization for The Web:
d3.json("someFile.json", function(json) {
//Put things here that depend on the JSON loading
});

//Only put things here that can operate independently of the JSON
console.log("I like cats.");
JavaScript
I just finished the book, and I took my time, working through all of the examples by hand.

Modern JavaScript isn’t anything like the stuff people of my generation churned out in the days of IE and Netscape Navigator 4. I think my previous attempts at grokking D3 were sidelined by my rush to skim through to the neat stuff. If you’ve had the same experience, please slow down and do the homework. I absolve you from chapters 1 through 3, but start paying attention in chapter four even when you don’t think it’s necessary.

O’Reilly gives you the option to read it right now for free (ctrl+f “Read Online Now!”), so no excuses if you want to learn D3.

Scott provides a full list of additional resources, both tutorials and examples. One of the experts is Jim Vallandingham, and Jim also produces VisKC. That site should interest about 100% of my regular readers.
Interactive Data Visualization for the Web
- javascript
- programming
Tue, 15 Oct 2013

Succumbing to NIH is a Risk
Posted: Tue, 15 Oct 2013
I’m evaluating OpenStack commercial partners, and I’m interested in their ability to blend with our current infrastructure. One vendor touted their automatic provisioning and configuration management system, and I wondered if it was just a re-brand of something like Puppet or Chef under the covers. Puppet would be fantastic since we’re invested in that technology already.

Google pointed me to a clarifying article:

… [PRODUCT] doesn’t rely on third-party products such as Puppet or Chef to handle deployments. He claimed there was risk involved with using such tools, especially with Puppet’s ties to VMWare, and suggested the learning curve was higher using them than with his company’s package.

Risk happens. We dance with risk all day every day. With no more specific information regarding the risks inherent in using an existing provisioning and configuration management system, I must assume those same risks are present using their home grown system. Nobody from Puppet Labs will stop by the home office of this vendor and demand exclusive use of VMWare hypervisors. The vendor is still free to write their own Puppet modules for managing their OpenStack implementations. The vendor is free to ignore Puppet all together while still choosing a completely competent and capable configuration management tool already in use by the community.

Concerning the learning curve problem, there’s only one explanation that makes sense to me. I think he left out some words. I think he meant something like “… the learning curve [for him and his PRODUCT team] was higher using them than with his company’s package.” That makes much more sense to me. It’s a tale as old as software development. I’ve seen it in my own lifetime more than I care to remember. It’s called NIH Syndrome, and it goes a little something like this: “we don’t have time to learn [INDUSTRY STANDARD PACKAGE], so we’re going to write our own instead.”

Because ignoring thousands of person hours worked by experts in the field of that tiny portion of the domain will surely pay off in the long run when your small team of generalists attacks the same problem for two weeks.

I’ve also heard trust arguments against existing components. “We didn’t write it ourselves, so how can we trust it?” Right. A package in use all over the world is obviously untrustworthy. That’s why I write most of my software on a custom home built computer that I soldered together myself. We’re going to need to make a bulk transistor order from Radio Shack so I can build enough equipment to run at scale. It’s the only way to be certain.

Succumbing to NIH is a Risk
- devops
- programming
Sun, 22 Sep 2013

Graceful Upgrades
Posted: Sun, 22 Sep 2013
Dear reader, I’m going to make some educated guesses about you. You are a software developer, maybe with a bit of operations experience. You buy computer gear, gadgets, and assorted electronic appliances as much for fun as for work and utility. You know how much RAM is in your computer and how fast it is. You enjoy most upgrades because upgrades bring new features and fix old bugs.

You are not like your users. Most of your users tolerate computers and software because managing life without them is tough today. They buy computers only when they must, and they upgrade software only when the upgrade is automatic, or when their accountant mentions that the old version of Quick Books Pro won’t automatically calculate payroll tax correctly after the first of the month. They are writing, calculating, and analyzing under deadlines. They may have learned about skeuomorphic design in the New York Times, but the concept did little more than raise their eyebrow and elicit a shrug.

Your users organize their lives and work with our software. They learned to use our systems cautiously, but they’ve grown confident over time. They trust our software and us by extension. Don’t betray that trust by radically redesigning a user interface without including a way to guide users through the changes. If your software includes a “new user orientation” feature, you’re already ahead of the crowd. Consider showing that to all of your user after a major upgrade. You, I, and our friends will quickly swat it away without a second thought, but our more cautious and careful users will appreciate a hand held tour of the new interface they’ll command.

Graceful Upgrades
- programming
Fri, 16 Aug 2013

Outsourced Password Hashing
Posted: Fri, 16 Aug 2013
The PassLib New Application Quickstart Guide reveals more information about password hashing than I knew at all before reading the page. Hashing and cryptography are two different things, but, as with crypto code, it’s best to leave password hashing to someone who knows the subject front and back.

I spent some time reading through the hashing routine in an old application to build an administrative password reset tool. We salt, and we use a decent hashing algorithm, but the code isn’t nearly as sophisticated as what we’d get with the default context provided by PassLib.

Here’s a preferential order of password storage algorithms:

-2. Passwords stored in plain text
-1. Passwords reversibly encrypted
0. Passwords naively hashed with a weak algorithm
1. Passwords salted and hashed
2. Passwords salted and hashed with a carefully chosen algorithm and procedure

Users will reuse passwords in your application. Storing weakly hashed or encrypted passwords opens your users’ email, social media, shopping, and banking accounts to fraud and abuse should your application ever be compromised. Users hate that. Don’t let your users hate you over something so easy to do well with the help of open source libraries.

Outsourced Password Hashing
- devops
- programming
- python
Mon, 22 Jul 2013
Daemonizing while dropping privileges
Posted: Mon, 22 Jul 2013
We talked about chroot jails the last time daemonization came up. This time we’re talking about privileges and what it means to drop them.

All users are equal, but some users are more equal than others. If something can be done, root can do it, and some actions can only be done by root. Root is the only user that can bind to any port under 1024 on a unix or linux host. Apache, or Nginx, or AOLServer can only bind to port 80 and 443 for the sake of serving web pages if the processes start as the root user.

Running a webserver as root is fine until someone finds a remote exploit in the ancient version of Apache that serves your site because patching is lame and it hurts your uptime. At that point a remote exploit will take over the root owned web server process and have practically unlimited access to the system.

That’s why dropping privileges is a thing we can do. A process can start with root privileges, do just the things that require root privileges, then immediately give up those privileges and switch to an unprivileged account. Now a remote exploiter will only have control of a process running as an unprivileged account, and they’ll need to put time into finding a root escalation attack rather than jumping right in to pilfering data from the system.

One note: users can belong to as many groups as the system administrator sees fit. Run the groups command right now to see what groups you belong to::
```
$ groups
tim adm audio video libvirtd admin
```
A process that starts life as a root owned process and drops privileges may not make it through to the other side in all of those groups. For instance, processes daemonized with the Python-LUST toolkit will only belong to the group explicitly specified in the daemon configuration file. If your process uses resources belonging to multiple groups in development, you may find that it stops working when deployed to production and daemonized as a process that drops privileges during start up.

And just for the record, there’s no excuse for not patching.
Daemonizing while dropping privileges
- programming
- unix
Tue, 09 Jul 2013
Daemonizing with chroot jails
Posted: Tue, 09 Jul 2013
Diffenbach runs in two modes: debug mode, where it stays attached to the current session and terminal, and daemon mode, where it detaches and runs silently in the background. Now it does, at least.

Until a week ago it only ever ran in debug mode in a tmux session. That’s pretty lame. The daemonization with Python-LUST was “done” in that hand wavy sense that some developers use. I had written the code, but I hadn’t tested it and worked out all of the kinks. Julython came to the rescue, and the Python-LUST integration started working five commits later.

Python-LUST defaults to dropping processes into a chroot jail. This restricts the running process to a tiny slice of the entire file system defined in the process’s configuration file, and it limits several attack vectors including path traversal and exploitation of local binaries.

File descriptors that are opened before the chroot call are carried into the chroot environment. Any files or commands that are needed at any time during the execution of the program must either be opened before the chroot call or be available within the limited chroot environment.

This includes log files, python modules, output directories, and external commands called through the os.system interface, and anything that isn’t opened ahead of time or available within the chroot will result in a thrown exception.

I decided to forgo the chroot capabilities of the Python-LUST library for two reasons:
1. Diffenbach makes explicit use of external binaries that require additional shared libraries. The footprint of the chroot environment gets big enough to negate many of the benefits of running within a chroot environment. I could compile a static binary of gphoto2 if I really wanted to run Diffenbach in a chroot environment.
2. Diffenbach is the only custom daemon running on its server, and it doesn’t run arbitrary code or commands fed to it via remote interfaces. That significantly lowers the threat level.
I also learned a bit about how dropping privileges works, but I’ll save that for my next post. Subscribe if you’re interested in learning more.
Daemonizing with chroot jails
- programming
- unix
Fri, 31 May 2013

Steak and Lobster
Posted: Fri, 31 May 2013
I’ve never cruised before. I don’t get excited about beaches, boats, or the ocean, so cruises cost a lot of money for 24x7 access to food and quiet time to read. I caught a thread on reddit concerning the death of nights and weekends for tech workers. A few tech workers commented that cruises were the only way they found peace on vacation, because most problems can be solved by the crew in the office if it costs $10/minute to talk to the guy vacationing on a boat. Concerning food, “either/or” was not a part of the lexicon, but “and” was:

I’m tempted to catch an off season deal and embark while wearing old man style pants with elastic waistbands just for the sport of it all.

We have our own steak and lobster moments. Python or Ruby? Both. Emacs or vim? Both. Java or C#? Both. Debian or Red Hat? Both. MySQL or Postgres? Postgres.

If there are zero additional constraints or prejudices coloring such a question, the best options are to flip a coin and get busy working or run a small experiment with both to form a stronger foundation for a real decision. We aren’t getting married to any technology, and each choice is both excellent and terrible depending on the problem at hand.

Should I learn Python or Ruby for freelancing?

Both. Or neither. Clients either have a technology in mind already or they will be confused and scared that specific technologies come up in the proposal process at all. The Rails shop that needs an extra hand wants a coder that knows Rails. The dentist that needs an appointment reminder system does not care about programming and programming languages as long as his missed appointment ratio looks better by the end of the quarter.

Most importantly, recall that my opinion in this matter is worth very little.

My opinion about barbell rows is as follows: **** barbell rows. Really. **** them. Stop wasting time worrying about barbell rows and get your deadlift up to 500. By then you’ll have your own opinion and you won’t have to worry about mine.

— Mark Rippetoe

Steak and Lobster
- programming
- career
Tue, 12 Feb 2013
Three Sets of Chisels Followup
Posted: Tue, 12 Feb 2013
I had a great followup to yesterday’s post that happened over IM today.

I was wondering what command you’d use to grep through .xls file on linux?

That’s a moment where my knowledge of my favorite tools helped me to eliminate those tools quickly from consideration. The text processing tools of the unix shell are terrible for searching through binary files like Excel spreadsheets.

If I had a single Excel spreadsheet with a ton of data to extract, I might convert it to a CSV file to then manipulate it further in my shell, but even then I’d probably just open it with Libre Office or Gnumeric and call it a day.

What if there were tens or hundreds of spreadsheets to sift through?
1. I’d talk to one of my Windows + .Net friends to see if PowerShell would work.
2. or ask them about C# APIs to do the same if PowerShell failed us.
3. or as a very last resort look to use an open source tool to do the necessary ETL work.
It’s strange writing open source and last resort in the same sentence. I’m the weirdo that kept my music collection encoded with Ogg Vorbis for most of a decade, and I haven’t played a commercial computer game since Unreal Tournament ran on my old Gentoo desktop. Maybe it’s old age that’s moving me from zealous to pragmatic, but I think I’ve realized that work isn’t about software for the sake of software. It’s about software for the sake of getting shit done.
Three Sets of Chisels Followup
- programming
- tools
Sun, 10 Feb 2013

Three Sets of Chisels
Posted: Sun, 10 Feb 2013

I was pissed (with myself) that I had three sets of chisels to maintain. Three sets to store. Three sets that would set me back in my efforts to master one set of chisels. —Christopher Schwarz from Tools are a Burden

We moved from a fairly modern 2000 square foot suburban home to a 900 square foot 1928 bungalow in 2008, and the drastic reduction in square footage quickly cured me of my hardware pack rat habit. My new office didn’t allow for my DEC Alphastation, my SGI Indigo Iris, or my Sun IPX, nor my stack of obsolete x86 carcasses.

The silliest part? I bought those classics long after they were past their prime. I used the IPX as a router at my parents’ house for a while, but for the most part the systems were there for the sake of coolness or nostalgia for unix years past.

Another move and easier access to virtualized hardware at home, work, and “in the cloud” further cured me of my desire to hoard hardware. Hardware today is so powerful that I’ve even removed myself from the 6-12 month upgrade treadmill.

I wish I was equally discerning with my software tools, but I’m prone to excitement over the latest and greatest. The professional cabinet maker can bet that one trunk of quality tools will last a lifetime, but we don’t have that luxury in the quickly changing field of software. Despite this, there are some tools that get called into action more often than others, and they tend to be long lasting cross platform tools that only get faster and more capable with years of practice: unix, emacs, python.

With a small set of default tools that are nearly ubiquitous, I can solve most problems fairly quickly, and I’ll quickly know if I need to use different tools all together for a given problem.

If construction (not collecting) is your goal, find some way to limit yourself - a tool chest is one way - so you can be a better builder than you are a consumer. —Christopher Schwarz from Tools are a Burden

Three Sets of Chisels
- programming
- tools
Tue, 01 Jan 2013

Fact Driven Development
Posted: Tue, 01 Jan 2013
Six groups of four students huddled around their respective lab tables, waiting for their dissection subject. A cat. A kitty. A dead, dyed, and preserved Felix catus domestica. Some kids hid their anxiety behind smiles, others gaped in silence. Cat dissection was the big deal in Biology II at Bishop Miege High School. Everyone experienced it, even if they never took the class. Formaldehyde wafted through the halls, following Bio II students through the rest of their day.

(The photo is from Flickr. Not my class, or my school, or even my decade…)

I wielded the scalpel about half of the time in my group. I’m not sure if they were conscientious objectors or just grossed out by the smell, but two of our four did as little cutting as possible. As the blade dug through tissue while we traced the route of blood vessels heading to the kidneys, a thought flashed through my mind: “Holy shit! We’re hacking the crap out of this cat, and we have the benefit of dyed blood vessels. Doctors do this to real people? And they live?!” I never had a strong desire to become a doctor, but that cut crossed it off my list for good. I liked my science exact, thank you very much.

Little surprise that I ended up slinging bytes for fun and money. Boolean logic is so orderly, and everything always lines up along powers of 2 or groups of 8. Unambiguous facts lead us through problems and root causes as long as we watch closely. Even the deepest and most difficult problems can be reasoned through with facts derived from direct observation, although we may retreat from the hunt with a sigh and a reboot at times.

There’s a downside to all this exactitude for people who play fast and loose while interacting with tech workers. “All of the processes restarted” is a verifiable statement, trivially so. Don’t think the sysadmin won’t check. “Every transaction is failing” will win no friends within the support ranks once they see that 2 out of 3 transactions are successful. The ISP support line operator knows when you’ve been bad or good, so reboot your modem for goodness sake.

Facts drive the growth and maintenance of technical projects and infrastructure. Gather data, analyze it, and act upon it to move forward. Running on gut feel works only as long as your gut is right.

Fact Driven Development
- devops
- programming
Wed, 19 Dec 2012
Signs and Travel Advisories
Posted: Wed, 19 Dec 2012
Blinders, blinkers, and winkers are three names for the same class of horse gear. Coming in various shapes and sizes, they all keep horses focused on the road ahead by blocking out distractions.

Humans, though, are different. We’re so smart that we don’t need to put blinders on to completely ignore the obvious information at our periphery, even when it’d do us some serious good. Our intent and determination can blind us to even the most obvious signs of disaster, and more subtle clues can be missed even when we’re carefully searching.

I use the Oracle Instant Client on a number of systems, and I’ve always used a URL format similar to that of Oracle’s Java thin client connections. From within Python’s SQLAlchemy toolkit, the URL format looks like:
```
oracle://user:password@hostname:port/database_sid
```
Systems with TNS configurations can alternatively use the following format:
```
oracle://user:password@tnsname
```
I was feeling a bit sorry for myself, because I was using the Instant Client rather than the full blown Oracle client complete with a tnsnames.ora definition file. This meant manually updating connection strings for a number of schemas that had migrated to another data center. While doing the updates, I tried to use the Oracle service name rather than database SID, as that would be one less thing to change after future migrations, but I couldn’t see a way to use a service name rather than SID name through SQLAlchemy’s configuration mechanism.

SQLAlchemy and cx_Oracle are both open source. This could be a fun thing to fix! With a fresh virtual environment and dependencies downloading, I filled the download time by reading the Instant Client FAQ. Right in the middle was this gem::
```
Naming methods that require TNS_ADMIN to locate configuration files
continue to work if the TNS_ADMIN environment variable is set.
```
Oh. So I can set a TNS_ADMIN environment variable just like with a full client install? So I’m done? No code change needed. I could have avoided all of this trouble in the first place had I read through the documentation more carefully when I started using the Instant Client.

There’s a happy balance between immediately jumping into action and hunkering down with documentation and research. I struggle with finding that middle path. I live at both ends of the spectrum depending on the subject.

For instance, technology and programming challenges ahead? Let’s get to work! We’ll figure it out as we go!

Meanwhile, when I should write a landing page to gather interest in a course I’d like to offer, I stall. “Not yet”, I say, “I need to read these 27 books on marketing and adult learning.”

Are you blowing past helpful signs and travel advisories on your way to a goal, or have you spent so much time studying the atlas that you’re still right where you started?
Signs and Travel Advisories
- programming
Sun, 16 Dec 2012

First Impressions of Mezzanine
Posted: Sun, 16 Dec 2012
If you personally know me, you probably know that Kate and I moved to Lancaster, PA this past summer. We’ve got a lot of renovation and exploration ahead of us, and that means we need some new web infrastructure. For the impatient or non-technical among us, our new site is called Bates and Metzger as a tip of the hat to the original two families that owned our place.

I run this blog with Hyde. It’s great, and I get a static HTML site out of it with each commit to my repository, but I wanted something with a friendly web interface to encourage Kate’s participation in writing posts.

I knew I wanted something written in Python. After checking out many of the available options, it looked like something based on Django would be the way to go, and Mezzanine looked both nice and active.

Installation was easy, the admin interface is friendly, and customization has been surprisingly quick. It definitely assumes that site admins and developers will have Django experience, but I’m faking that part well enough.

First Impressions of Mezzanine
- programming
- webdesign
Sun, 25 Nov 2012
ChronoFlow, My Custom Work Timer
Posted: Sun, 25 Nov 2012
I’m a huge fan of the Pomodoro Technique, but I kinda hate every single timer that I’ve ever used to facilitate it, and I’ve tried quite a lot. Web timers, OSX timers that hang out in the menu bar, fat client applications, Android apps. They’re all uniquely irritating.

This is classic yak shaving, and if I weren’t a programmer by trade I’d just pick the most decent existing solution and get on with life.

Here’s my simple but specific requirement list:
- Scalable. I have a lot of screen real estate, and it’s nice to have a big timer as a constant reminder when space allows it.
- Colorful. Not beautiful, necessarily, just green and red.
- Nice Sounds. I’m a musician, and ugly alerts and alarms upset me more than they rationally should.
I don’t need task tracking capabilities. I have emacs for that.

Easy, right? Actually, yes. 161 lines of code later, and it times my work iterations on half of monitor #4. The following tools made it remarkably simple:
- PyGTK for windows and widgets
- PyCairo for drawing
- pygame for sound support
Right now this is a “works on my machine” project, but you can still find it on Bitbucket. I’m going to live with it as-is for a while to see how I like it, and then I’ll investigate the following options:
- Replicate the same code with Quickly
- Hook into Pidgin with DBus to control my status when starting and stopping work iterations.
- Use Hydrogen and QSynth to make some nice alarm sounds.
ChronoFlow, My Custom Work Timer
- programming
- pomodoro
Thu, 15 Nov 2012

Before Your Morning Coffee
Posted: Thu, 15 Nov 2012
Folks are talking about Raymond Chen patching the MS Money binary because Money crashes during import of account transactions.

There’s a lot of love in the air for that old, end-of-life, product. There’s also a lot of love for Raymond walking through the debugging and patching process. It takes a lot of knowledge and skill to patch a compiled software program without any special access to the source code, and the article is worth a read if you’re into that sort of activity.

But I’m not really interested in all that. At least not today.

I want to talk about what you do before your morning coffee. See, down at the bottom of Raymond’s post there’s a little section called “Bonus chatter”, and at the bottom of “Bonus chatter” we get this gem:

Specifically, I said, “I feel like Jeff, who does this sort of thing before his morning coffee.”

Jeff corrected me. “If this was something I used to do before coffee, that probably meant I was up all night. Persistence >= talent.”

Persistence >= talent. That is the passage to note. Nobody is born with an innate talent for software development. Or writing. Or painting. Or auto body restoration. It takes persistence.

I’m going to admit something about myself that I’m not sure I’m happy about. I’m quick to judge IT folks who say they want to be programmers.

A quick tangent: I love the idea of running and finishing a marathon. Sounds exhilarating.

I hate the idea of training for a marathon. I’d much rather code, or cook, or build than run 5+ days a week for months at a time. I’m at peace with this. I don’t tend to divulge my love of the idea of marathon running much since I know I won’t put forth the effort to actually do it.

Point an IT guy or gal at well written and tested material, and they should be programming in very little time. Basic stuff at first, sure, but everyone starts there. Most often, though, their true desire to be a programmer without learning to program shines through.

Programming. It’s not hard. But it’s not easy, either. It’s easy to sit down one day to learn how to program, but you won’t get far. It’s hard to do that every single day for months at a time.

If you need help learning how to program, or clean a carburetor, or ice a cake, or anything else, find a potential mentor and bring your work with you. It gives the mentor something to work with, and it shows your dedication to the topic. Partial credit isn’t some relic of math classes long forgotten. It still works to your benefit. Show your work and get partial credit. Show no work for no credit. Be persistent and succeed.

Before Your Morning Coffee
- programming
- habits
Sun, 21 Oct 2012
Hill Work
Posted: Sun, 21 Oct 2012
I ran cross country for a couple of years in high school. Not well, but I did it. I’ve since learned that diet and consistent training outside of the season would have greatly improved my chances of success, but that’s water under the bridge.

Every Monday afternoon we’d warm up, jog past the school entrance, down Reinhardt Drive, and assemble at the corner of Howe and 52nd Place. For a group of kids that normally measured progress in one mile increments, we were in for a particularly rough work out measured not in miles, but in hills.

Hill work breaks down pretty neatly:
1. Sprint up the biggest hill available to you
2. Jog back down at an easy pace. Don’t run, don’t walk.
3. If not dead, go to 1
I exited that loop before dying, but we’ve already established that I wasn’t a fantastic team member.

There was a bit of good cop/bad cop dynamic going on with our coaches. The head coach would scowl his scowl of disapproval (your mom was right when she said your face can freeze that way) and occasionally yell that we needed to push up the hill hard. Coach Anderson was much more likely to tell us to take it easy on the way back down.

Why the emphasis on an easy jog back down the hill? Wasn’t this a work day? Sure, but we were pegging our ability to move oxygen through our body and convert it to energy, and the run up the hill is where we would get the most benefit. Better to really rest on the way down and go all out on the way up than to do both poorly.

Recent studies concerning interval and Tabata training claim that regular short sessions of intense work punctuated by even shorter rest periods are, minute for minute, far more effective at keeping us healthy and growing our capability for converting oxygen into work than equally regular long sessions of distance and endurance training.

I’ve long felt the same way about work. Work hard when you’re working, and rest when you’re not. Don’t fool yourself into thinking that a 12 hour day 5 days a week with “easy” 4 to 8 hour days on the weekend are doing you and your company a comparatively amazing amount of good. The burden of proof is certainly on you if you insist that hours equal productivity. Track how much time you’re really working, how many bugs you’re writing. I’d love a real time widget that would pop up when my typos per minute rose significantly above average, as that’s a pretty good indicator of fatigue for me.

The 40 hour work week came into fashion for a scientifically studied reason, and it wasn’t for your comfort.

Sprint

Work hard when you’re working.

I recently had a self imposed 12 hour work day spent developing some lab material for a workshop. I’d been making decent progress in fits and starts, but the workshop was coming up quick, so I applied caffeine liberally and cranked through a ton of work in short order. The work was good, and I felt fantastic about my progress. I had grand plans of an early bedtime so I’d be effective the next day.

A bit of misfortune changed those plans, and I ended up working through most of that next night. It was intense. It was focused. It was fruitful. It got a little stupid toward the end. Typos shot through the roof, as did hardcoding values, as did copy and paste coding. A younger me would have plowed through, bestowing upon the team a pile of code that would be trouble.

I was done. Caffeine had extracted my maximum sustained effort. I sprinted up the hill as long as I could, and it was time for a gentle descent.

Jog. Don’t Run, Don’t Walk

Work hard when you’re working. Rest when you’re not. Rest well, rest actively. Plowing through the better half of a bag of Reese Peanut Butter Cups and vegging out in front of Arrested Development is probably not the rest you deserve, even if it is the rest you want, and even if I’ve done this very thing and other things like it several times in my life.

Sleep well and make time for activities away from the keyboard to give your subconscious time to do what it does best: work on tricky problems without you. Solitary real world activities seem to work best for me: painting a room, plumbing, building stuff like desks and shelves, and even cooking. (It’s the eating of things cooked where I’m at risk of overdoing it…)

You’re probably different. Maybe it’ll be yoga, meditation, wrenching on cars, or practicing music. Find your non-programming activity that keeps your brain active while giving your programming muscles a rest.

I went home after my two day sprint, and I landed in bed for a long time. I didn’t play video games, and I didn’t surf reddit. I slept. It was gloriously comfortable, and I felt recharged when I woke up. After a day of cutting in paint around the house I was ready to get back into emacs and create.

Acknowledgments

I’m very happy to work in an organization that values effectiveness over hours clocked.
It’s almost like the lessons of Peopleware have made their way into the subconscious of managers so many years after its publication.

I’d also like to thank Dave Haygarth for posting his photos under a CC license. The photo I used in this post is available under the title Knowle Hill fell race.
Hill Work
- programming
- career
Sat, 13 Oct 2012
Scheduled Job Anti-Patterns - We Don't Need Version Control
Posted: Sat, 13 Oct 2012
Scheduled jobs tend to suck a bit. They’re usually written after they’re needed and dropped into place with little testing and no plans for fixing them when things go pear-shaped.

This is the third in the series. Here’s the full list that we’ll cover:
- Everything is important, email all results
- Nothing will break, so why worry
- Workflow orchestration with cron
- It’s just a script, we don’t need version control
It’s just a script, we don’t need version control

If you ever say these words to me as I’m cleaning up a mess caused by you making a “non-impacting change” to a scheduled job that isn’t under version control, then you’re going to see a look come across my face. Behind that look, my brain is calmly keeping my hands from reaching for the nearest blunt object while mentally filing you away as “someone who’s lucky to have a job in IT” and trying to figure out how to best extract ourselves from the mess we’re in. My brain’s multitasking just isn’t good enough to do those things and maintain a poker face. Sorry.

This is what happens when you say that version control isn’t necessary.
```
-rwxr-xr-x   1 jobsuser jobsuser    6805 Mar  7  2012 sales_prod_export.pl
-rwxr-xr-x   1 jobsuser jobsuser    7.6K Jul 10  2010 sales_prod_export_maint2_2.pl
-rwxr-xr-x   1 jobsuser jobsuser    7.7K Jun 16  2010 sales_prod_export_maint2.pl
-rwxr-xr-x   1 jobsuser jobsuser    7.7K Jun  1  2010 sales_prod_export_maint.pl
-rwxr-xr-x   1 jobsuser jobsuser    2270 Mar 10  2010 sales_prod_export_work.pl
-rwxr-xr-x   1 jobsuser jobsuser   15043 Mar 24  2012 sales_prod_force_send.pl
-rwxr-xr-x   1 jobsuser jobsuser   12564 Aug 26 04:24 sales_prod_verify.pl
-rwxr-xr-x   1 root     root       15659 Jul 19 19:52 sales_prod_verify.pl.bkp
-rwxr-xr-x   1 jobsuser jobsuser   12568 Sep 12 20:32 sales_prod_verify_maint.pl
-rwxr-xr-x   1 jobsuser jobsuser     16K Sep  3  2010 sales_prod_verify_maint2.pl
-rwxr-xr-x   1 jobsuser jobsuser     14K Jun 30 22:02 sales_prod_verify_maint_new_20120629.pl
-rwxr-xr-x   1 jobsuser jobsuser     16K Jun 30 21:55 sales_prod_verify_maint_new.pl
-rwxr-xr-x   1 root     root         16K Mar  1  2012 sales_prod_verify_maint_tim_just_this_once_20110809.pl
-rwxr-xr-x   1 jobsuser jobsuser     16K Aug 10  2011 sales_prod_verify_maint_tim_since_that_is_the_fashionable_thing_to_do.pl
-rwxr-xr-x   1 root     root         16K Jan 29  2011 sales_prod_verify_maint_variable.pl
-rwxr-xr-x   1 jobsuser jobsuser     13K Jun 29  2011 sales_prod_verify_maint_vsmith.pl
-rwxr-xr-x   1 jobsuser jobsuser   16207 Aug 13 15:34 sales_prod_verify-yesterday.pl
```
There are 17 copies of this one script. That’s dumb. This is a fictionalized real world example, so I’m going to stand up straight and take credit for these two here:
- sales_prod_verify_maint_tim_just_this_once_20110809.pl
- sales_prod_verify_maint_tim_since_that_is_the_fashionable_thing_to_do.pl
The sarcasm was lost on this system’s maintainer. Either that, or his brain’s multitasking managed to keep a straight face while he pondered the quickest way to punch me in the nuts and get away with it.

Version Control is too Easy

Version control is too easy not to use for the scripts and configurations that keep your shop humming. You may not trust that statement if your last encounter with version control involved CVS or, to a lesser extent, Subversion, but I swear to you that it is true.

Distributed version control systems let you set up repositories on a local system without ever interacting with a central server. You don’t need to request a new repository from the version control gatekeepers, you just need to init a new repository in place. For bonus points you’re going to back this repository up to a remote location, but that’s outside of our scope here today.

Right now you just need to pick one of the two following tools:
- Git (learn more about git)
- Mercurial (learn more about mercurial)
They both work just about everywhere that matters today, and they behave nearly identically for the stuff we’ll be doing. If you have developers on staff or as friends, ask them what they use. Otherwise, flip a coin.

Once you have the binaries installed, you’re only a few commands away from version controlled bliss. We’ll walk through creating a repository, seeing the status of the repository, committing unsaved changes, and reviewing the change log below.

With mercurial:
```
$ hg init 
$ hg status
? really_important_script.sh
$ hg add
adding really_important_script.sh
$ hg commit -m "Committing really_important_script.sh so we don't lose changes like last time." 
$ hg log 
changeset:   0:0d058a3f5c18
tag:         tip
user:        Tim Freund <tim@freunds.net>
date:        Sat Oct 13 16:09:06 2012 -0500
summary:     Committing really_important_script.sh so we don't lose changes like last time.
```
And with git:
```
$ git init
Initialized empty Git repository in /Users/tim/src/dvcsdemo/git/.git/
$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#   really_important_script.sh
nothing added to commit but untracked files present (use "git add" to track)
$ git add really_important_script.sh 
$ git commit -m "Committing really_important_script.sh so we don't lose changes like last time."
[master (root-commit) 9d9f3d1] Committing really_important_script.sh so we don't lose changes like last time.
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 really_important_script.sh
$ git log
commit 9d9f3d12eaa3d56a9bf0e12364c411baafed2753
Author: Tim Freund <tim@freunds.net>
Date:   Sat Oct 13 16:10:02 2012 -0500

    Committing really_important_script.sh so we don't lose changes like last time.
```
And that’s really all there is to it. Every time you make a change to your scripts, simply commit the changes before you move on to your next task. You’ll eventually want to learn how to revert to previous versions and share your repositories with others, but just doing what we’ve worked through today will set you up for success when making changes to scripts.
Scheduled Job Anti-Patterns - We Don't Need Version Control
- programming
- devops
Thu, 04 Oct 2012
Scheduled Job Anti-Patterns - Workflow Orchestration with Cron
Posted: Thu, 04 Oct 2012
Scheduled jobs tend to suck a bit. They’re usually written after they’re needed and dropped into place with little testing and no plans for fixing them when things go pear-shaped.

This is the third in the series. Here’s the full list that we’ll cover:
- Everything is important, email all results
- Nothing will break, so why worry
- Workflow orchestration with cron
- It’s just a script, we don’t need version control
Workflow Orchestration with Cron

I’m going to show you a small file built primarily for generating support tickets. There’s a crontab file on appserver01 that looks like this:
```
05 0 * * 1-5 /home/vrichards/salesdb/make_the_ach_file.pl 31 > /dev/null 2>&1
25 0 * * 1-5 /home/vrichards/salesdb/edit_the_ach_file.pl 31 > /dev/null 2>&1
27 0 * * 1-5 /home/vrichards/salesdb/copy_the_ach_file.pl 31 > /dev/null 2>&1
```
Sad yet? No? What if I told you there were 30 other sets of such jobs? Still not sad? What about when you log in to appserver02 and find:
```
30 0 * * 1-5 /home/vrichards/salesdb/send_the_ach_file.pl 31 > /dev/null 2>&1
```
Gah!!! On the surface these crontab files are built to generate a file and transmit it to a bank, but each of the following made me die a little bit on the inside:
- Scripts running out of a developer’s home directory
- Magic number parameters (“Tim!”, you yell, “the command has a help function and it’s up to date. That parameter is obvious and self documenting!” You’re a dirty liar, and I hate you)
- Redirecting all output to /dev/null
- Running one process across multiple hosts without justification when all steps could run on one host
Gross. All of it. We aren’t even here to talk about any of the above, but you should get in touch if you’d like to know why any of the above are terrible ideas.

We’re here to talk about trying to orchestrate a complex work flow with individually scheduled cron jobs. We are not here to talk about orchestrating a complex work flow with individually scheduled cron jobs, because that’s not really possible in any realistic way. We are here to discuss trying (and failing) to do the same.

The road to Hell is paved with good intentions

Our completely fictionalized developer, Victor Richards, means well. He really wants to be helpful and write things in a way that makes problem solving easy. Much like breaking up a monster function that spans pages of editor space into multiple easily composed functions, he’s broken up the creation and transmission of ACH files into discrete steps. That’s legitimately thoughtful. It allows the ops team to retransmit a file if the bank’s FTP server was down without regenerating it from scratch.

He’s built all the individual components required to make a fairly robust system that will churn out bank files each business day, but gluing them together with cron is a recipe for pain and an active support queue.

Turning up the heat

Victor’s good intentions turn Hellish toward the end of the month. Website traffic and orders go through the roof. It’s not unusual to double typical order volume at month end, and quarter and year end gets even crazier.

That first job in the chain, make_the_ach_file.pl, gets a full 20 minutes to run in the cron schedule. It’s a pretty beastly script that pulls all of yesterday’s orders out of the database to generate a bank file, and the queries used aren’t optimized all that well. The 20 minute window usually provides a 5 minute cushion before the next job, edit_the_ach_file.pl starts to run, at least until the calendar creeps up to the end of a month. The larger month end order volume means that the file generation either just barely completes in time or is late, and the multi-car pile up begins.

Traffic Control

We have three jobs that are tightly related, with each subsequent job being completely dependent on the job that runs before it. What we need is some way to ensure that the jobs run serially instead of achieving accidental parallelism when one of the jobs runs longer than expected. That turns out to be so easy that you’ll be forgiven for missing the obvious. It just takes one script:
```
#!/usr/bin/env bash
customer_id=$1
/home/vrichards/salesdb/make_the_ach_file.pl $customer_id
/home/vrichards/salesdb/edit_the_ach_file.pl $customer_id
/home/vrichards/salesdb/copy_the_ach_file.pl $customer_id
```
Then our crontab file on appserver01 looks like:
```
05 0 * * 1-5 /home/vrichards/salesdb/ach_file_suite.sh 31 > /dev/null 2>&1
```
As a bonus, we still have the individual scripts available for use during troubleshooting exercises should they be necessary.

This is still the poor man’s form of process orchestration, but we’ve eliminated an entire class of support issues by ensuring that all the individual jobs run in the correct sequence, regardless of how long any of those individual jobs takes to complete.

But wait! There’s more!

This is third in the series. Subscribe or come back tomorrow to see more.
Scheduled Job Anti-Patterns - Workflow Orchestration with Cron
- programming
- devops
Mon, 01 Oct 2012
Scheduled Job Anti-Patterns - Nothing Will Break, so Why Worry
Posted: Mon, 01 Oct 2012
Scheduled jobs tend to suck a bit. They’re usually written after they’re needed and dropped into place with little testing and no plans for fixing them when things go pear-shaped.

This is the second in the series. Here’s the full list that we’ll cover:
- Everything is important, email all results
- Nothing will break, so why worry
- Workflow orchestration with cron
- It’s just a script, we don’t need version control
Nothing Will Break, so Why Worry?

If you’ve spent any time at all running unix systems, you’ve seen the following pattern:
```
26 * * * * /usr/local/bin/fixStaleJobStatus.pl 2>&1 > /dev/null
```
Regardless of the return code or output of that script, you’ll never see a peep from it. All the output is dumped squarely into the bitbucket.

How do jobs end up configured like this? They’re usually written, tested, and watched closely for some period of time: a week, a month, or more. Once the author is convinced that the job is invincible, she gets tired of seeing the output in her inbox each day, so she adds the redirection to make things easy for herself.

Only marginally better are the admins that set up an elaborate maze of mail rules to filter any and all output from scheduled jobs into some rarely read folder. Sure, the output is there, but it may as well not be there if it’s getting ignored completely. Yes, it’s nice to be able to confirm the user error reports that will start to trickle in, sometimes weeks after the problem started, but it’s not exactly the professional way to handle things.

I suppose one step beyond that is the admin that rolls into the office at 10:45, opens Outlook, hits CTRL+A then Delete, and then gets up to retrieve his morning cup of joe. Cron output problems AND user problems solved all in one quick key chord. Shameful, but that dude is so far off the reservation that I’m sure he’s not here among us. Right?

So how do we dodge this?
1. Find a cron wrapper script that only notifies the team on errors. Cronic makes this super easy to pull off. It’s a shell script, so it should run just about anywhere that matters without compiling anything.
2. Edit jobs to write their results to a centralized location for intelligent monitoring and notifications. This has its merits, but you’re going to need some additional infrastructure and development time. A big enough shop will justify this. Email can only scale so far.
3. Wait for the users to call. They know when the important stuff isn’t happening. If a job fails in the server room and no one complains, does it really need to be fixed?
The astute among you will notice that we’re reusing the same list from our last installment. Two serious suggestions and one suggestion that is unfortunately in use in far too many places. These two anti-patterns really are identical twins that were separated at birth. Some might even combine the two into one, but let’s consider them both separately lest we over-correct from one straight into the other.

But wait! There’s more!

This is the second in the series. Subscribe or come back tomorrow to see more.
Scheduled Job Anti-Patterns - Nothing Will Break, so Why Worry
- programming
- devops
Mon, 24 Sep 2012
Scheduled Job Anti-Patterns - Everything is Important
Posted: Mon, 24 Sep 2012
Scheduled jobs tend to suck a bit. They’re usually written after they’re needed and dropped into place with little testing and no plans for fixing them when things go pear-shaped.

This is the first in the series. Here’s the full list that we’ll cover:
- Everything is important, email all results
- Nothing will break, so why worry
- Workflow orchestration with cron
- It’s just a script, we don’t need version control
Everything is important, email all the results

The tricky thing about configuring a scheduled job for the first time is making sure it actually works. The job is going to run in the dark recesses of the machine, and you’re going to wonder if it’s really working at first. Email’s a super easy solution to that problem.

Or the sales and refunds processing jobs didn’t run for three days last week because a disk was full. Customers are angry, and all of that bile and vitriol is landing right on top of your boss. These broken jobs are the most important problem in the world right now on the second floor of Spacely Sprockets, and the sternly worded interoffice memo makes that painfully clear through exuberant use of exclamation points and terrible grammar.

Marching orders in hand, every single scheduled job in the environment is configured to send an email upon completion or failure.

The only thing that progresses faster than the email notification configuration is a set of corresponding email rules blazing through the systems and operations group to filter all of that crap out of their inboxes.

Oh, look, we’re back to square one! As my manager likes to say: if everything is important, then nothing is important.

So how do we dodge this?
1. Find a cron wrapper script that only notifies the team on errors. Cronic makes this super easy to pull off. It’s a shell script, so it should run just about anywhere that matters without compiling anything.
2. Edit jobs to write their results to a centralized location for intelligent monitoring and notifications. This has its merits, but you’re going to need some additional infrastructure and development time. A big enough shop will justify this. Email can only scale so far.
3. Dedicate full time staff to sift through email boxes full of cron notifications and manually re-trigger jobs. There are people that actually do this. And if you think it’s a sound solution, then you’re probably in the wrong job. There’s probably a bank someplace that would love to hire you as a computer operator.
What if the notification systems break? Or the scheduler just stops?

Oh, they’ll break. Give it time. They’ll break spectacularly.

The big problem with any job monitoring system is watching the watchmen. Your people or machines have the ability to completely go off the rails and fail at their jobs for an innumerable number of reasons. That’s what sent us down this path of notifying on everything in the first place, remember? People take sick days, and machines have hard drives that fill up. OK, the hard drives shouldn’t fill up, but if you’re so smart, then why are you sending emails from every single one of your scheduled jobs?

Whatever system we use to notify the team of job failures, we also need to ensure that the scheduled job runner and notifications systems have a snug and warm place carved out in the NOC’s monitoring system.

Any decent method for sorting the wheat from chaff plus solid monitoring of the servers that do the grinding should set the course for a sane scheduled job environment.

But wait! There’s more!

This is just the first in the series. Subscribe or come back tomorrow to see more.
Scheduled Job Anti-Patterns - Everything is Important
- programming
- devops
Sun, 16 Sep 2012
Iron Blogger Update
Posted: Sun, 16 Sep 2012
This one is slightly meta. Blogging about software about blogging.

A few of us read about this Iron Blogger thing, and we decided we’d give it a go. I downloaded the provided software, poked around a bit, and thought I’d try to make something that non-technical folks could use just as easily.

I didn’t even link to the work in progress last week, as it was pretty terrible. I was able to manually configure our group, only to have multiple timezone issues bite us when processing posts.

In the past week I was able to steal away enough time to do the following:
- Fix the scoring issues caused by timezone differences
- Squash a number of other silly bugs
- Automate the deployment with Jenkins
- Publish the IronBlogger repository on BitBucket.
During the upcoming week I’d like to tackle the following:
- Back fill some decent testing. The commit log is downright shameful in this regard.
- Clean up the default template a bit - lots of default links that don’t work.
- Provide clear links to Atom feeds and OPML downloads.
- Allow people to sign up. Right now it’s a read only web app, and thus kinda lame.
Once it’s respectably working, I’d also like to reach out to Benjamin Mako Hill to see if he has any input on licensing. I’m heavily borrowing concepts from his Iron Blogger repo, so I’d like to respect his wishes with regard to licensing. Judging from his other work, I’m guessing it’ll land somewhere in the GPL family.

With all of that preamble out of the way, feel free to check out our progress so far:
- Our first official week
- The last week to be scored
Iron Blogger Update
- programming
Mon, 10 Sep 2012

A Quick Note About Timezones
Posted: Mon, 10 Sep 2012
Some of you lead programming lives that involve 3D graphics, or physics simulations, or DNA sequencing. You’re so deep into these domains that you’ve never had to deal with such mundane things as dates and times in your code. Bravo. I’m jealous. Skip the rest of this post and go make something cool.

Two people stopped reading, and the rest of us poor saps are left to talk a bit about time zones. Don’t worry, this’ll be quick.

Make sure your code is aware of timezones. Do you have a timestamp/date/datetime field in your database anywhere? Is your database smart enough to note the timezone of its clients? Is your software aware enough to know that it may be running code on behalf of users that are in a different timezone than itself? Are your servers running in the same timezone as your development machine?

I can’t answer yes to all of those questions, so I’ve gotta go work on some code…

A Quick Note About Timezones
- programming
Thu, 08 Jul 2010

The 7:00 AM Phone Call
Posted: Thu, 08 Jul 2010

I much prefer the infamous 3:00 AM phone call to any 7:00 AM phone call. I don’t particularly enjoy either, but I appreciate the gravity of any situation that warrants a phone call at 3:00 AM.

7:00 AM phone calls are often prompted by something strange that the early guy spots after getting in to the office long before anyone else. Although the situation usually warrants attention, it most often does not warrant waking anyone.

Three people on my team were called at 7:00 AM today because a scheduled report didn’t run at the normal time because of heavy load on our reporting server. Transactions were still processing. and the report was eventually delivered. My team was up late last night doing scheduled maintenance long after normal business hours to reduce impact and provide a high level of customer service. We were all a little perturbed to get wake up calls over a relatively minor issue after a long night.

Some folks are night owls because they get their best work done at night. Others are night owls because their work can only be done at night. The early birds can keep their worms, I only ask that they consider the night owl’s schedule before picking up the phone at 7:00 AM.

The 7:00 AM Phone Call
- programming
- life

programming

Sprint

Jog. Don’t Run, Don’t Walk

Acknowledgments

It’s just a script, we don’t need version control

Version Control is too Easy

Workflow Orchestration with Cron

The road to Hell is paved with good intentions

Turning up the heat

Traffic Control

But wait! There’s more!

Nothing Will Break, so Why Worry?

So how do we dodge this?

But wait! There’s more!

Everything is important, email all the results

So how do we dodge this?

What if the notification systems break? Or the scheduler just stops?

But wait! There’s more!