Given the tendency of Plus users to do the most illogical things possible while refusing to read or apply any kind of instructions given and then blame the software for behaving “unexpectedly”, it seemed only fitting to provide them with the one thing they couldn’t screw up: the Do Not Click button.

A lot of thought went into the Do Not Click button. Its placement was particularly important – it had to be accessible yet unobtrusive enough to go unnoticed for a while. The button itself had to be recognizable as capable of performing an action, yet simple enough to be overlooked by a casual glance. It had to display enough of a warning to be clear that clicking was ill-advised, yet low-key enough to beckon seductively. The final product is thus:

Fetching, no? It was placed at the very bottom of the About dialog, which I doubt anyone ever accesses unless by accident. If any one of the 150 users happens upon it, however, he’s all but guaranteed to email the other 149 to tell them about it. And that’s when the fun starts.

“What happens when the Do Not Click button is clicked?” one might ask curiously. Let me give you the most likely scenario:

User: “What’s this thing that looks like a stop sign with a white X in it? That looks weird. Wait, there are some hieroglyphs beside it…they say ‘d’…no, ‘D’…I’ve got it! It says ‘Do’! Phew, that was hard work. I’d better take a break so as not to strain myself.”

<- four hours later ->

User: “Why is there a window that says ‘About’ open? This stupid software, always doing things I don’t tell it to! I’m going to create a helpdesk ticket for IT to fix it! No, I should compose a poison-filled email about how crappy Plus is and how it’s always broken and send it to all of the other field technicians so they can join me in my uninformed griping!”

<- two hours later ->

User: “All that typing made me hungry. I’m going to go to the expensive steakhouse down the road and charge a hefty three-course to my company credit card and then take a nap.”

<- the next day ->

User: “Hmm, what’s this window that says ‘About’? There’s a button that has a stop sign with a white X in it. It doesn’t say what it does…<click> Nothing happened. <click> <click> <CLICK> Stupid Plus! How can I be expected to perform any work if this trashy program won’t do anything? Those idiots in IT are just trying to keep me from doing my job!”

Sadly, this is probably a realistic depiction of events. However, the Do Not Click button does do something – it records the number of times it’s been clicked. Yes, that’s right – there’s a persisted click count. When the user submits work (which is supposed to happen once every few days), the cumulative click count is stored to a server-side database along with the user’s username. There it waits until an intranet-accessible web page is requested, wherein it’s included in a report showing the users clicking the Do Not Click button the most.

And here’s the best part: Any “bug” report about the Do Not Click button being broken can be closed with “The button works as intended. Do not click the Do Not Click button.” When the question of why the button exists comes up, the answer is “The button is there to keep an issue from occurring. When the button is clicked, the issue occurs. If the button is not clicked, the issue does not occur. This is why it says ‘Do Not Click’ on the button.”

/**
 * Added 10/7/2009 by specific request of sjohnson. Of course there's no HD
 * ticket or other documentation for the request because everyone knows
 * normal rules don't apply to the field staff.
 */

Code comment written today:

/**
 * Removed 11/5/2009 by specific request of sjohnson. Of course there's no HD ticket
 * or other documentation for the request because everyone knows normal rules don't apply
 * to the field staff. sjohnson swears this is a showstopper and must be removed because
 * now that he actually has to work in Plus on a daily basis he understands how
 * ridiculously foolish it is to clear out the time entry fields after every entry is
 * added to the table.
 */

Mere days before a version of Plus was to be released, the manager in question demanded that the functionality be changed as it was “not the way the staff work”. He was made aware of the reason the product functioned the way it did and warned of the problems that would occur should his demand be met, but these logical beseechings fell upon deaf ears. He was given an opportunity (two full weeks) to test this change and he signed off on the final product. Almost exactly a month later, the manager had to take on the role of a field technician. As part of that, he had to use Plus, and blamed me for “making changes without [his] knowledge”.

This manager demanded that this and a second change be immediately coded and pushed into production as an “emergency” release…without any testing. He was warned of the expected problems of the second change and how testing helps ensure quality, but again he exhibited Deaf Man’s Vendetta. We’ll see how that turns out.

Update: A week later (11/13/2009) the same manager went into another panic and demanded the functionality be reverted again, emailing “can you please verify and correct this issue immediately? This is the same issue we had in October!”

As relayed in previous posts in this series, the Plus application was meant to facilitate the transmission of data from field technicians to the home office. However, at some point it was decided that Plus should also be responsible for transmitting and displaying data from the home office to the field technicians. This information ended up being things like equipment-specific metadata, customer contact numbers, location of each piece of equipment, a directory of everyone employed at the company that developed Plus (no, I’m not kidding), and schedule data. All of this came from a poorly-standardized, non-normalized database used by the company’s CRM application, Nexterna Clearview, which uses a batch-and-post system.

The schedule data was an important piece because it gave the field technicians a way to see scheduled work without having to use any brainpower. After adding this information to the Plus application the powers that be decided Plus should use it to automatically determine and display the appropriate data forms for the technician to fill out for each site visit. It was further decided that the forms themselves should dynamically change based on the equipment information from the CRM application.

Eventually this was all made ready and deployed to the field technicians. As usual, there were exclamations of “Wow, this is great!” and “This will make us so much more efficient!” at first. But soon the users started reporting problems. They’d enter data into a form and it would disappear the next time they downloaded schedule updates from the home office. It turned out Jim built Plus and the Plus server to use auto-generated database record IDs as unique identifiers for schedule data. When a field technician entered form data, it was tied to the schedule’s database record ID. When the form data was sent to the home office, the Plus server displayed it to home office users by establishing a link between the schedule’s database record ID and the record ID in the returned data. In a normal world, this wouldn’t be a problem – using database keys guaranteed to be unique is good practice, right?

The issue came into play when one of the schedulers at the home office altered a schedule after a field technician had already used Plus to enter form data. The scheduler, called a ‘CSAM’, would change the schedule’s location, equipment, date, assigned technician, and even go so far as to delete the schedule altogether and create a new one in its stead. The next time the field technician used Plus, he saw a set of blank data entry forms assigned to the new schedule instead of all the data he’d already entered under the old schedule. In essence, data entered into Plus was “lost” due to the normal day-to-day operation of the CSAMs.

Later I found that there had been many a heated argument betwixt Jim and benson about the use of auto-generated database record IDs as data keys. “That’s not how the users work!” benson would exclaim. “It’s the right way to do things!” Jim would shoot back. The end result was always the same – Jim would simply continue the way he wanted.

For quite some time people just blamed Plus for everything. “This stupid tool loses my data!” they’d gripe. “It eats my work!” Eventually I was able to show that a very small change to the existing CSAM scheduling process could avoid the majority of these situations, and the complaints changed to…well, they didn’t change at all. Everyone still made Plus the scapegoat, and the CSAM group refused to believe they could possibly have anything to do with the issues. Their leader wasn’t willing to even admit it was a possibility.

Finally I rewrote enormous chunks of the Plus application to use less volatile keys that mapped more directly to the way the users worked (vindication for you, benson). Of course, at the same time the CSAM group finally conceded to make that one tiny change (it took the head of the company to get even that), and they’re taking credit for reducing the number of issues with Plus data. Oh well, only 11 more days…

One of the prevailing requirements for the Plus application was that it allow field staff to enter data offline and “sync” it over the Interwebs at their leisure. In addition, the application had to get updated schedule information and previously-synced data and show it to the user while offline. As you know, the most logical design that meets this requirement is a disconnected client-server model. Therefore, a server had to be built to accept connections from the client, process incoming data, and send back schedule  information and previously-synced data.

This by itself isn’t such a big deal, right? Apache Tomcat exists just for tasks of this nature. It’s an entire server container that’s mature, been heavily tested, well-maintained, and trusted by enormous companies that could easily afford to choose any server product on Earth. It includes all sorts of advanced configuration options, logging facilities, exception handling, and standards adherence. There’s even a Windows installer package available! Installation, configuration, and basic Java implementation is fast and easy.

But since this was Jim’s project, it wasn’t fast, easy, simple, logical, standards-based, or maintainable. Jim rolled his own Tomcat server instead. Yes, that’s right – he ignored the ready-built server software available and wrote his own software to do the same thing. However, his version didn’t have the benefit of testing…or configuration options…or exception handling…or standards compliance…or maintainability…or – well, you get the idea. As a result, Jim’s Tomcat didn’t really work.

As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

Jim split the server implementation into two completely separate areas of the server itself. The actual application itself was only responsible for receiving client requests for schedule information – but not for sending the data itself back to the client. Instead, the application just zipped the schedule data into a temp directory and sent the file path back to the client. The client then connected to an SFTP server and downloaded the specified file. When the client needed to download previously-submitted data files, it connected to the same SFTP server, navigated to a hard-coded directory, and downloaded everything in it. Yes, that’s right – each user ended up downloading all data ever submitted by any of the 150 users every time the client updated. As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

The submission of data, however, was handled by the client. The client connected to that same SFTP server, navigated to a hard-coded directory, and uploaded individual files to it. Then a cron job ran every 60 seconds, swept the directory, determined the type of each data file by matching the beginning of its filename, loaded the data into a database, and moved the files to the “submitted data” folder that every client downloaded. If a file couldn’t be read, the cron job died and any file not already loaded was left in the folder. The next time the cron job ran the same problem would occur…without any notification to anyone.

I inherited all of this when I was stuck with the Plus project. Had I known what I was getting into, I would’ve said “You’ve gotta pay me more.”

On the first day, one of the contractors, Alicia,  was assigned to train me.   I should have known something was up when Alicia checked the code out and could not get it to compile.  She needed to go find Shaun, because ‘He had the working version”.  I asked, “Doesn’t everyone have a working version?”.  She replied, “No, Shaun does.”  It appeared that every time new code was added and changes were made, it broke everyone’s local copies, and all the developers would play around with the code until one of them got it to compile.  Then everyone would get that person to ‘recreate’ what he/she did on their computers.

Doesn’t everyone have a working version?

Boy THAT should have been a red flag. Apparently, Shaun was the only person who could compile and deploy the code.  This week. Eventually, with Shaun’s help, the application was checked out of source control and compiled on my computer.  But for the next week, I saw PPT presentations of the ’system design’, and code snippets on screen without actually having any interaction with the code.  When I finally pressed the point that I wanted a ‘walk through’, Alicia reluctantly agreed.  The application, being a J2EE implementation, was launched and the login page displayed.  Alicia then logged in using a test account and walked through the application, which bombed every other screen because “it can’t hit the service, you need to change the settings to point to a test service.”  Well, let’s do that, then….

Fast forward ANOTHER week.  The application can finally run on my local machine, but there have been a half-dozen configuration changes to the local server, MANY changes to the local code, which, I am instructed to never check in, and any attempt to run the application in ‘debug’ mode causes my laptop to freeze and crash…

Oh, and I saved the best part for last.  At some point during my ‘training’, I actually did look at the code.  I found the start of execution.  It was a Servlet.  It had over 2000 LOC in the doPost method.  It used reflection to dynamically instantiate dozens of other servlets who’s names and methods were contained in a Database.  It was the slowest running application on the planet.

I lasted 1 year exactly.  In that time dozens of developers were hired. Most left before me.

They really should have paid me more….

Next, a glimpse into how this was coded.