/**
 * Added 10/7/2009 by specific request of sjohnson. Of course there's no HD
 * ticket or other documentation for the request because everyone knows
 * normal rules don't apply to the field staff.
 */

Code comment written today:

/**
 * Removed 11/5/2009 by specific request of sjohnson. Of course there's no HD ticket
 * or other documentation for the request because everyone knows normal rules don't apply
 * to the field staff. sjohnson swears this is a showstopper and must be removed because
 * now that he actually has to work in Plus on a daily basis he understands how
 * ridiculously foolish it is to clear out the time entry fields after every entry is
 * added to the table.
 */

Mere days before a version of Plus was to be released, the manager in question demanded that the functionality be changed as it was “not the way the staff work”. He was made aware of the reason the product functioned the way it did and warned of the problems that would occur should his demand be met, but these logical beseechings fell upon deaf ears. He was given an opportunity (two full weeks) to test this change and he signed off on the final product. Almost exactly a month later, the manager had to take on the role of a field technician. As part of that, he had to use Plus, and blamed me for “making changes without [his] knowledge”.

This manager demanded that this and a second change be immediately coded and pushed into production as an “emergency” release…without any testing. He was warned of the expected problems of the second change and how testing helps ensure quality, but again he exhibited Deaf Man’s Vendetta. We’ll see how that turns out.

Update: A week later (11/13/2009) the same manager went into another panic and demanded the functionality be reverted again, emailing “can you please verify and correct this issue immediately? This is the same issue we had in October!”

As relayed in previous posts in this series, the Plus application was meant to facilitate the transmission of data from field technicians to the home office. However, at some point it was decided that Plus should also be responsible for transmitting and displaying data from the home office to the field technicians. This information ended up being things like equipment-specific metadata, customer contact numbers, location of each piece of equipment, a directory of everyone employed at the company that developed Plus (no, I’m not kidding), and schedule data. All of this came from a poorly-standardized, non-normalized database used by the company’s CRM application, Nexterna Clearview, which uses a batch-and-post system.

The schedule data was an important piece because it gave the field technicians a way to see scheduled work without having to use any brainpower. After adding this information to the Plus application the powers that be decided Plus should use it to automatically determine and display the appropriate data forms for the technician to fill out for each site visit. It was further decided that the forms themselves should dynamically change based on the equipment information from the CRM application.

Eventually this was all made ready and deployed to the field technicians. As usual, there were exclamations of “Wow, this is great!” and “This will make us so much more efficient!” at first. But soon the users started reporting problems. They’d enter data into a form and it would disappear the next time they downloaded schedule updates from the home office. It turned out Jim built Plus and the Plus server to use auto-generated database record IDs as unique identifiers for schedule data. When a field technician entered form data, it was tied to the schedule’s database record ID. When the form data was sent to the home office, the Plus server displayed it to home office users by establishing a link between the schedule’s database record ID and the record ID in the returned data. In a normal world, this wouldn’t be a problem – using database keys guaranteed to be unique is good practice, right?

The issue came into play when one of the schedulers at the home office altered a schedule after a field technician had already used Plus to enter form data. The scheduler, called a ‘CSAM’, would change the schedule’s location, equipment, date, assigned technician, and even go so far as to delete the schedule altogether and create a new one in its stead. The next time the field technician used Plus, he saw a set of blank data entry forms assigned to the new schedule instead of all the data he’d already entered under the old schedule. In essence, data entered into Plus was “lost” due to the normal day-to-day operation of the CSAMs.

Later I found that there had been many a heated argument betwixt Jim and benson about the use of auto-generated database record IDs as data keys. “That’s not how the users work!” benson would exclaim. “It’s the right way to do things!” Jim would shoot back. The end result was always the same – Jim would simply continue the way he wanted.

For quite some time people just blamed Plus for everything. “This stupid tool loses my data!” they’d gripe. “It eats my work!” Eventually I was able to show that a very small change to the existing CSAM scheduling process could avoid the majority of these situations, and the complaints changed to…well, they didn’t change at all. Everyone still made Plus the scapegoat, and the CSAM group refused to believe they could possibly have anything to do with the issues. Their leader wasn’t willing to even admit it was a possibility.

Finally I rewrote enormous chunks of the Plus application to use less volatile keys that mapped more directly to the way the users worked (vindication for you, benson). Of course, at the same time the CSAM group finally conceded to make that one tiny change (it took the head of the company to get even that), and they’re taking credit for reducing the number of issues with Plus data. Oh well, only 11 more days…

As demonstrated in Part I, Part II, Part III, and Part IV of this series, the Plus application was a mess. It never really worked and when the desired functionality was simulated, it caused other issues. To be fair, the department that requested Plus in the first place was also a mess – its manager not only couldn’t manage the department’s 150 employees, but further had no understanding of what they did or what they were supposed to be doing (and those two things were definitely not the same). As a result, change requests for the Plus application had no order, logic, or sanity, and often contradicted other change requests made by the same department slightly earlier. It was a constant cycle of “Make this change” “Why did you change that? Change it back!” “Make this change”.

Culpability of the project requester aside, the Plus application was awfully, horribly coded. When eight new complex Swing forms are cached in memory every time the mouse is clicked, you know there are some performance optimization opportunities ripe for consumption. The code’s terribleness was truly stunning in the same way a 450-pound person is – you’re disgusted and feel sorry for their plight but oddly intrigued at the same time because you wonder how they can possibly perform even the most basic of daily tasks.

As noted in previous parts of this series, after being charged with managing communication about the Plus application’s bugs I could hold back any longer – I had to dive in and fix some of the problems. It was like coming home every day to a towering stack of food-encrusted dishes – eventually you just have to clean them up. I fixed a few of the smaller bugs in the codebase, tested the fixes, and checked the changes in to the Subversion repository.

Later that week I sat down to lunch with the rest of the IT squad. The boss was there as usual and had some interesting news. “I talked to Jim last night,” he led off (Jim and the boss worked at the same company for years – when the boss was hired at our company, he brought Jim with him from their old employer). “He was furious that you changed the Plus code.”

“What’s the problem?” I asked warily.
“He was extremely upset because you changed his code,” the boss said.
“…but I fixed it…”
“Yes, I know. But Jim and I talked for a long time tonight. He sees the changes you made as an intrusion. He acted very protective of his code.”
“…but I fixed it…”
“Jim said it was like defacing a masterpiece. He called Plus his opus.”
“His opus? That’s normally used to describe an accomplishment. He considers Plus his greatest achievement?! Those saps who hired him are in for a surprise!”

Ahh, they definitely should have paid me more to put up with this.

One of the prevailing requirements for the Plus application was that it allow field staff to enter data offline and “sync” it over the Interwebs at their leisure. In addition, the application had to get updated schedule information and previously-synced data and show it to the user while offline. As you know, the most logical design that meets this requirement is a disconnected client-server model. Therefore, a server had to be built to accept connections from the client, process incoming data, and send back schedule  information and previously-synced data.

This by itself isn’t such a big deal, right? Apache Tomcat exists just for tasks of this nature. It’s an entire server container that’s mature, been heavily tested, well-maintained, and trusted by enormous companies that could easily afford to choose any server product on Earth. It includes all sorts of advanced configuration options, logging facilities, exception handling, and standards adherence. There’s even a Windows installer package available! Installation, configuration, and basic Java implementation is fast and easy.

But since this was Jim’s project, it wasn’t fast, easy, simple, logical, standards-based, or maintainable. Jim rolled his own Tomcat server instead. Yes, that’s right – he ignored the ready-built server software available and wrote his own software to do the same thing. However, his version didn’t have the benefit of testing…or configuration options…or exception handling…or standards compliance…or maintainability…or – well, you get the idea. As a result, Jim’s Tomcat didn’t really work.

As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

Jim split the server implementation into two completely separate areas of the server itself. The actual application itself was only responsible for receiving client requests for schedule information – but not for sending the data itself back to the client. Instead, the application just zipped the schedule data into a temp directory and sent the file path back to the client. The client then connected to an SFTP server and downloaded the specified file. When the client needed to download previously-submitted data files, it connected to the same SFTP server, navigated to a hard-coded directory, and downloaded everything in it. Yes, that’s right – each user ended up downloading all data ever submitted by any of the 150 users every time the client updated. As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

The submission of data, however, was handled by the client. The client connected to that same SFTP server, navigated to a hard-coded directory, and uploaded individual files to it. Then a cron job ran every 60 seconds, swept the directory, determined the type of each data file by matching the beginning of its filename, loaded the data into a database, and moved the files to the “submitted data” folder that every client downloaded. If a file couldn’t be read, the cron job died and any file not already loaded was left in the folder. The next time the cron job ran the same problem would occur…without any notification to anyone.

I inherited all of this when I was stuck with the Plus project. Had I known what I was getting into, I would’ve said “You’ve gotta pay me more.”

I enjoy using tables. They clearly demarcate the boundaries between datum, creating an easily-understood convention for interpreting information while retaining amazing flexibilities. They’re dead straight simple to program – an HTML layout table be created using as little as six tags with very logical names, a JSPWiki table simplistically requires only a few pipes, and Microsoft created an entire program designed around tables. They call it Excel.

According to Joel Spolsky, Microsoft  initially developed Excel as an accounting tool until some user interaction testing for v5.0 demonstrated how people gravitated toward its clean tabular interface in order to format data lists! In my workplace everyone seems to have a dominant Excel fetish. Requirements meetings for  report development requests usually have “I want it to be exported to Excel so I can work with the results” (and also, “This can be emailed to me automatically as an attachment, right?” and sometimes “Unless there are pink dragons on every line except those in which the total number of characters multiplied by the fractional representation of the current lunar phase is prime, this report will be useless”)  in there somewhere. One request from May 09 specified that the Excel document produced by a Crystal Reports report must contain a dynamic number of inverted columns based on arbitrary, inconsistent numbers in a database! People think Excel is better than Batman, the Time Lords, and Bill Watterson rolled into one powerful little green package because it has such great tables!

However, not every table implementation is done well. Fortunately for us programmers, the necessity of tables is ubiquitous enough to inspire table support in most GUI libraries. The Java Swing libraries, for example, include a number of interfaces and concrete and abstract classes for creating tables, displaying data in implementation-specific ways, handling user input, validating data, refreshing data, sorting by column, handling events, etc. Much of the table functionality is automatic and requires zero additional coding. Simple tables are quick ‘n’ easy to generate and they can get as complex as the programmer needs. JTables were built with user interface best practices in mind, and there’s an entire industry of highly-educated professionals determining what those should be.

Some of the data entry forms in Jim’s Plus application needed to display tabular data with well-known constraints (known max number of columns – 21 – and common datatypes, known maximum row count of 240) and handle user input. Any high school student with an Introduction to Java course under his belt would immediately think “JTable!” when faced with this requirement, but Jim’s first thought was more like “I can write better table code than those highly-educated nerds at Sun!” (this was a common theme with him; a future story will elaborate on how Jim rolled his own Tomcat server).

Jim's array-based table implementation

So Jim thought for a bit and came up with a grand idea. “I’ll put 5040 total textboxes (21 columns * 240 rows) into 21 separate arrays and just display them as if they were part of a table!” At first glance, his finished JTextField array-based table implementation  didn’t seem so bad. It had row numbering down the left side, you could enter data, and the textboxes creatd a decent approximation of a tabular format.

Unfortunately for the life-span of Jim’s “tables”, there were problems. Users saw an Excel-like interface and assumed Jim had rewritten Excel just for them. They complained that the arrow keys didn’t let them jump between fields and they had to use Tab instead. They complained that there weren’t averages at the bottom for numerical fields. They complained that there wasn’t any real-time input validation. They complained that the forms were excessively slow and difficult to work with. They complained that the fields were way to big and only eight rows fit on the screen at one time. They complained that the tab order went left-to-right when it should really hop betwixt the first two columns until the last row, and then it should revert back to “normal” left-to-right order for the rest of the columns (sadly, that is not hyperbole).

My personal favorite was when they complained that the cells didn’t turn yellow when their value was more than 20% away from the column’s overall average value.

Jim listened to all of these complaints about his pseudo-JTable. He bared his teeth in a pseudo-smile, like a Mafia don feigning benevolence, and started slinging code like a mentally unstable bull with diarrhea spreads fertilizer. Adding to the abysmal performance of his pseudo-JTable, Jim wrote a background thread to run every 30 seconds and calculate the overall average of numerical columns. He threw up another that iterated over every textbox and changed its background color to yellow if it was 20% away from its column average. He spurted out a FocusTraversalPolicy implementation to meet the wacky tab order requirements.

The user’s would have rejoiced if they weren’t such losers. “It’s just like Excel!” they would have cheered, then linked arms and skipped off toward a black hole.

Oops, ignore that last part. Somehow my daydream got mixed in.

Anyway, most of the users’ complaints about the fake JTable were addressed, but there were still issues. The weird 20% background color thread would falsely Crayola™* a field if it ran before the average-calculation thread did. The averages themselves were frequently incorrect. The textboxes were still enormous and took up far too much space, both vertically and horizontally. Jim had used BoxLayouts that automatically determined the appropriate size for GUI components, and there wasn’t any great way to reduce their size across the board (all of Jim’s form layouts sucked horribly and used up far too many resources, but I digress). The complaints about the table’s performance (now even worse) trailed off, but only because no one was willing to address them.

Then one day a new complaint came in. “I entered 240 rows of data and only 80 were saved!” sniveled the user. An investigation found that a very specific sequence of events could indeed cause the pseudo-JTable to display 240 rows for data entry but only save some of them. An outcry was launched, fingers were pointed, blame assigned, and demands made. The forms had to be revamped, and immediately.

New and improved: The JTable-based form

So yours truly analyzed the problem, determined that a GUI table is not core business functionality, and took Joel Spolsky’s advice. The basic pseudo-table interface had to be followed, of course, but Swing’s great JTable support was harnessed to quickly create an efficient, componentized table that addressed the users’ requirements and still performed like a greased gecko on methamphetamines.

Since it implemented the JTable components as intended by The Maker (Sun), the new table even added some features. Each column had a specific datatype and prevented inappropriate entries, and user input was validated in real-time. A cell’s border turned red when it contained an unacceptable value. Its background turned yellow (sigh) when the value was more than 20% away from the column average. The column average itself was recalculated when one of the column’s cells was changed. The columns were sortable, and users could use the arrow keys to navigate to any cell they desired.

It was a nice change. So nice, in fact, that another form was changed from yet another array-based pseudo-tabular GUI to use the more appropriate JTable (it’s going to take some time to get them to understand in-place edits for that one). When I first got to this form, none of the controls lined up or sized appropriately. The buttons on the right would randomly disappear, making it impossible to enter data. The labels had misspellings. Like the other tabular form, this one held all of its controls (an astonishing 4500 of them) in arrays and made each row visible as the user finished with the previous one. The JTable-based form eliminated the label repetition, corrected the spelling, appropriately sized the input fields, and introduced sorting. The buttons stay put as well.

The old array-based form after a bit of sprucing up

New and improved: The JTable-based form

So what did the users’ representative say when he saw the much-improved JTable-based forms? “The tab order’s wrong”.

* You know how annoying people say a word has been “verbed” if it was previously used only as a subject but they’ve incorrectly wedged it into a sentence where the verb should have been? Has anyone addressed the infinite recursive loop that comes from the act of “verbing” “verb”, considering “verb” is a subject until used incorrectly as a verb? Dare I utter it more clearly, lest a the loop creates a black hole? I do! Those are verbed verbs (and spelling errors)!

When first assigned to the Plus project I was told to play the role of “bug manager”. “Jim (see Part I) wants to do the right thing,” the boss remarked. “so he’s informed his new company that he’ll be providing bugfixes and support on the Plus project during his first month. We can contact him about bugs at any time during work hours and he’ll try to have a fix coded and checked into SVN before the next day.”

Given Jim’s apparent charity, my role in the project was simply to aggregate bug reports from the 150 users, de-duplicate and prioritize them, and ship the result off to Jim. This seemed decently straightforward; after all, I didn’t know anything about the 500-class codebase or the difference between intended functionality and broken features, and there wasn’t any requirements document, project documentation, or decent in-code documentation available. As the voluminous bug reports crashed in like a tidal wave in monsoon season during an earthquake, I went to work.

This system worked out okay for a couple of days, which is about as long as it took to get the first response from Jim regarding the bug reports (which represented 49 serious issues over a period of five months). Problems like “Application memory usage climbs in direct relation to the duration it is open” got solutions like “Upgrade RAM for all users”. Problems like “Autosave creates 500+ MB of identical files – runs continuously” resulted in (I’m not joking) “coded” (it wasn’t). And the rest, like “Data form randomly saves blank files and won’t allow user to enter data” (complete with concrete evidence and steps to reproduce), didn’t receive any response at all.

During this period I was the sole point of contact for the angry users of this product. Finally I became so frustrated that I started investigating the Plus codebase. Part I chronicles some of the unhelpful comments found there, but more interestingly I discovered that Jim had used a portion of his 9-month development time to add themes. Yes, that’s right – the Plus application was failing all sorts of tests for critical functionality, but it had themes.

More than twenty of them.

In addition, it had a feature that created a preview of each tab as the user hovered the cursor over the tab name. Also, the application’s splash screen was a 6MB animated GIF of a weird-looking spinning globe (almost-kind of-sort of like the company’s logo, except horribly distorted and using non-company colors). As if that weren’t enough, easter eggs littered the code landscape, activated by lucky keystroke combinations. Ctrl-Alt-F7 launched a Tetris mini-game complete with a high-score board. Alt-Shift-T popped something called “Tea Timer”, which turned out to be a dialog that allowed the user to choose between tea, hot chocolate, and coffee and see Jim’s estimated preparation time (no, I’m not joking). This appeared to be Jim’s introduction to Java Swing components, his experimental form in which to try those weird GUI things like listening for user-initiated events and laying out form controls. I hope this is so – the alternative is that Jim’s knowledge of Swing programming was around the average high schooler’s level in 2000.

As soon as the option came I ripped out every one of these “features”, giggling with glee and smiling devilishly. What bastard puts a Tetris game in a productivity application designed for people who were stereotyped as lazy, unmotivated, dishonest, unsupervised fools? What non-managerial sniveling turd decides pretty GUI themes is more important than core functionality? What lying coward assures everyone he will stand behind his work, is given the support to do so, runs away and then has the nerve to complain loudly because someone else has touched his sacrosanct code (to be explained in a future story)? And let’s face it, what loser thinks a Swing form acceptable as a first-week assignment for an Introduction to Programming course is anywhere nearly as cool as an easter egg is meant to be?

I ask these probing questions because after systematically exterminating every ridiculous non-essential programming perversion Jim wedged in, I added dancing monkeys for rector to find after I’ve been fired and he’s forced to take over the codebase

Plus was intended to be a company-changing desktop application giving remote field technicians the ability to record site visit information offline and later “sync” it with the company’s servers. The product was released straight from Jim’s development machine to production on 150 technician laptops just two weeks before Jim’s last day at the company. Being the only available developer, I was assigned to the project in his stead.

“Jim used to be at IBM,” the boss said. “I worked with him at my last job and brought him over when I was hired here. He’s a top-notch programmer. You shouldn’t have any problems with his code.”

And that’s when I found out what “you could be a god here” really means. When people are happy with a god people have sort of a vague “yeah, he’s cool, no smiting lately” attitude” when you bring him up. But when they’re pissed about something, anything , all you hear is blasphemy and threats of atheism.

On one hand, I did get a god status. On the other hand, there weren’t any bonus points for lack of smiting. As the bug reports and shouts of anger began pouring in, I started looking at the Plus code and found more and more reasons for god-to-god smiting action. Following are some of Jim’s code comments verbatim:

/* ---------------------------------------------------------------
 @author [Jim's real name here], [his side company's name here] LLC
 Date: 2008-4-18
 Project: Departed Friend
 --------------------------------------------------------------- */

Hmmm…seems like a bit of egomania here. The project’s name was definitely not “Departed Friend”. And…putting your one-man “consulting” company down as the author? Chintzy.

// Give it some time to complete the save.... This is just a guess
// and is not good code, but owell when you read this I'm sure I
// wont be working here anyway

This one really toasted my tootsies. At the time I was naive and thought good programmers felt a responsibility to write maintainable code. Boy, was I dumb. The code below this comment added a five-second delay to the application’s closing to prevent an exception in case the autosave mechanism happened to be running. The delay had no actual relationship to the autosave mechanism – it could have kicked off at the last second and still caused the exception, so the it was absolutely  pointless.

int n = JOptionPane.showConfirmDialog(god, "Are you sure you want to cleanup files older then 40 days that we have a more recient copy of?  Why I ask is that this step will tie up your computer for some time, so is a great item to run when you have a coffee break or before you go to bed.", "Are you sure you want to cleanup old useless files? ", JOptionPane.YES_NO_OPTION, JOptionPane.WARNING_MESSAGE);

Holy piquant pineapples, Batman! Definitely one of the longer (and more ridiculous) modal pop-up dialog messages I’ve encountered in an internal product’s code. Kavya Shivashankar (world spelling bee champion), you can rest easy.

//  Yeah this is the cheap mans way to use a lot of memory for something that most likely will not be visible.

This gem was above some code that instantiated up to 5040 JTextFields and added them to a Swing container – and immediately made them invisible. Jim was trying to present tabular data to the user and, instead of using a JTable, used 7 JTextField arrays to represent 7 columns of data. Is anyone surprised that one of the loudest complaints about the application was its abysmal performance?

// Its not returning the number I thought it would so we will just
// waste CPU time
for (int i = 0; i < 2000; i++)
{
}

Hurts…so bad…can’t…comment…

//TODO verify that this is all of them.  IF they added for more people then this is not good enouf but its 11:00 PM and im on #4 of the IPA here so this is as good as it gets for a friday night at a brewery. :-)

Jim worked remotely from another state 668 miles away. He obviously did some of his work at the bar. This does not surprise me.

As I found, they should have paid me more.