“How does the utilization report arrive at its numbers?”

It seemed like a simple, straightforward and innocuous question at the time. Unfortunately, it opened an institution-sized metallic food storage container of blind, soil-ingesting invertebrates.

You see, the person who’d commissioned the development of the utilization report didn’t really understand what he was building. He had vague amorphous ideas about what he wanted – perhaps he’d seen a report at some other company that was used to determine how well the employees were being utilized and decided it was a good idea. Based on second-hand knowledge and investigation into the reports themselves, here’s how I picture the utilization report’s progression in my head:

Project Requester: “I want a report that tells me if there’s enough work for my employees.” –note: He’s saying “show number of hours worked divided by number of hours that could have been worked”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to take holidays into account.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to take PTO into account.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It  should use the number of hours the employee has been scheduled to work instead of the hours actually worked.”
Programmer: “What?”
Project Requester: “It has to.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. If an employee had a job that ran late the night before, the Scheduling department schedules an employee for a block of time the next day in order to keep him from having to work. The report has to take that into account when it happens.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to show that an employee worked the whole weekend if he was scheduled for a Saturday.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to assume every employee spent two hours to get to the job site and two hours to get home.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong.”
Programmer: “How is it wrong?”
Project Requester: “The numbers can’t possibly be that low. The report is wrong.” –note: It showed an average of 64% utilization (i.e. employees jacked off 36% of the time for which they were paid). It was later found that actual utilization was 52% if they were afforded every possible benefit of the doubt.

The utilization report was used to determine year-end monetary bonuses for managers. However, every time the utilization numbers were too low to allow the maximum bonus, the managers would say “The report’s always been inaccurate. The numbers are really much higher.” And the bonus would be awarded.

It turned out “the utilization report” at my company actually meant six distinct reports. Here’s how the system worked:

• At the beginning of every month, a user would submit an IT ticket as a reminder to “run” the report on the 15th of the next month
• Somewhere near the 15th of the month, a specific developer would navigate to a directory on his machine and double-click an Access database
• The developer would then locate each of the six reports in the Access database and run them all in turn, exporting them as RTF files when they finished
• Each report was extremely resource-intensive and completely took over the IT guy’s computer, rendering him useless for hours – and even longer if the Access database crashed
• When all reports had been exported, they were attached to an email and sent to the user from Step 1

This process was repeated every month and caused much frustration for the user (because they couldn’t get the report immediately) and the developer (because it consumed his computer and he couldn’t get anything done). After hearing the developer curse and complain about the utilization report a few times, I set to work on the Access database. The process was changed to:

• On the 15th of every month, Windows Task Scheduler opens the Access database, triggering a VBA subroutine
• Each report is exported to RTF
• All exported reports are programmatically attached to an email message and sent to the user

The new process is entirely automated and takes about three minutes to complete. It turns out Access was running all of the report queries (a bunch of UNION ALLs in a number of views) twice – once to display the report to the developer, and again to export the data to RTF. In addition, the formatting process required to display the report inside Access took an enormous amount of time, whereas the formatting of the RTF to be displayed in an external viewer was far quicker.

Given the tendency of Plus users to do the most illogical things possible while refusing to read or apply any kind of instructions given and then blame the software for behaving “unexpectedly”, it seemed only fitting to provide them with the one thing they couldn’t screw up: the Do Not Click button.

A lot of thought went into the Do Not Click button. Its placement was particularly important – it had to be accessible yet unobtrusive enough to go unnoticed for a while. The button itself had to be recognizable as capable of performing an action, yet simple enough to be overlooked by a casual glance. It had to display enough of a warning to be clear that clicking was ill-advised, yet low-key enough to beckon seductively. The final product is thus:

Fetching, no? It was placed at the very bottom of the About dialog, which I doubt anyone ever accesses unless by accident. If any one of the 150 users happens upon it, however, he’s all but guaranteed to email the other 149 to tell them about it. And that’s when the fun starts.

“What happens when the Do Not Click button is clicked?” one might ask curiously. Let me give you the most likely scenario:

User: “What’s this thing that looks like a stop sign with a white X in it? That looks weird. Wait, there are some hieroglyphs beside it…they say ‘d’…no, ‘D’…I’ve got it! It says ‘Do’! Phew, that was hard work. I’d better take a break so as not to strain myself.”

<- four hours later ->

User: “Why is there a window that says ‘About’ open? This stupid software, always doing things I don’t tell it to! I’m going to create a helpdesk ticket for IT to fix it! No, I should compose a poison-filled email about how crappy Plus is and how it’s always broken and send it to all of the other field technicians so they can join me in my uninformed griping!”

<- two hours later ->

User: “All that typing made me hungry. I’m going to go to the expensive steakhouse down the road and charge a hefty three-course to my company credit card and then take a nap.”

<- the next day ->

User: “Hmm, what’s this window that says ‘About’? There’s a button that has a stop sign with a white X in it. It doesn’t say what it does…<click> Nothing happened. <click> <click> <CLICK> Stupid Plus! How can I be expected to perform any work if this trashy program won’t do anything? Those idiots in IT are just trying to keep me from doing my job!”

Sadly, this is probably a realistic depiction of events. However, the Do Not Click button does do something – it records the number of times it’s been clicked. Yes, that’s right – there’s a persisted click count. When the user submits work (which is supposed to happen once every few days), the cumulative click count is stored to a server-side database along with the user’s username. There it waits until an intranet-accessible web page is requested, wherein it’s included in a report showing the users clicking the Do Not Click button the most.

And here’s the best part: Any “bug” report about the Do Not Click button being broken can be closed with “The button works as intended. Do not click the Do Not Click button.” When the question of why the button exists comes up, the answer is “The button is there to keep an issue from occurring. When the button is clicked, the issue occurs. If the button is not clicked, the issue does not occur. This is why it says ‘Do Not Click’ on the button.”

/**
 * Added 10/7/2009 by specific request of sjohnson. Of course there's no HD
 * ticket or other documentation for the request because everyone knows
 * normal rules don't apply to the field staff.
 */

Code comment written today:

/**
 * Removed 11/5/2009 by specific request of sjohnson. Of course there's no HD ticket
 * or other documentation for the request because everyone knows normal rules don't apply
 * to the field staff. sjohnson swears this is a showstopper and must be removed because
 * now that he actually has to work in Plus on a daily basis he understands how
 * ridiculously foolish it is to clear out the time entry fields after every entry is
 * added to the table.
 */

Mere days before a version of Plus was to be released, the manager in question demanded that the functionality be changed as it was “not the way the staff work”. He was made aware of the reason the product functioned the way it did and warned of the problems that would occur should his demand be met, but these logical beseechings fell upon deaf ears. He was given an opportunity (two full weeks) to test this change and he signed off on the final product. Almost exactly a month later, the manager had to take on the role of a field technician. As part of that, he had to use Plus, and blamed me for “making changes without [his] knowledge”.

This manager demanded that this and a second change be immediately coded and pushed into production as an “emergency” release…without any testing. He was warned of the expected problems of the second change and how testing helps ensure quality, but again he exhibited Deaf Man’s Vendetta. We’ll see how that turns out.

Update: A week later (11/13/2009) the same manager went into another panic and demanded the functionality be reverted again, emailing “can you please verify and correct this issue immediately? This is the same issue we had in October!”

As relayed in previous posts in this series, the Plus application was meant to facilitate the transmission of data from field technicians to the home office. However, at some point it was decided that Plus should also be responsible for transmitting and displaying data from the home office to the field technicians. This information ended up being things like equipment-specific metadata, customer contact numbers, location of each piece of equipment, a directory of everyone employed at the company that developed Plus (no, I’m not kidding), and schedule data. All of this came from a poorly-standardized, non-normalized database used by the company’s CRM application, Nexterna Clearview, which uses a batch-and-post system.

The schedule data was an important piece because it gave the field technicians a way to see scheduled work without having to use any brainpower. After adding this information to the Plus application the powers that be decided Plus should use it to automatically determine and display the appropriate data forms for the technician to fill out for each site visit. It was further decided that the forms themselves should dynamically change based on the equipment information from the CRM application.

Eventually this was all made ready and deployed to the field technicians. As usual, there were exclamations of “Wow, this is great!” and “This will make us so much more efficient!” at first. But soon the users started reporting problems. They’d enter data into a form and it would disappear the next time they downloaded schedule updates from the home office. It turned out Jim built Plus and the Plus server to use auto-generated database record IDs as unique identifiers for schedule data. When a field technician entered form data, it was tied to the schedule’s database record ID. When the form data was sent to the home office, the Plus server displayed it to home office users by establishing a link between the schedule’s database record ID and the record ID in the returned data. In a normal world, this wouldn’t be a problem – using database keys guaranteed to be unique is good practice, right?

The issue came into play when one of the schedulers at the home office altered a schedule after a field technician had already used Plus to enter form data. The scheduler, called a ‘CSAM’, would change the schedule’s location, equipment, date, assigned technician, and even go so far as to delete the schedule altogether and create a new one in its stead. The next time the field technician used Plus, he saw a set of blank data entry forms assigned to the new schedule instead of all the data he’d already entered under the old schedule. In essence, data entered into Plus was “lost” due to the normal day-to-day operation of the CSAMs.

Later I found that there had been many a heated argument betwixt Jim and benson about the use of auto-generated database record IDs as data keys. “That’s not how the users work!” benson would exclaim. “It’s the right way to do things!” Jim would shoot back. The end result was always the same – Jim would simply continue the way he wanted.

For quite some time people just blamed Plus for everything. “This stupid tool loses my data!” they’d gripe. “It eats my work!” Eventually I was able to show that a very small change to the existing CSAM scheduling process could avoid the majority of these situations, and the complaints changed to…well, they didn’t change at all. Everyone still made Plus the scapegoat, and the CSAM group refused to believe they could possibly have anything to do with the issues. Their leader wasn’t willing to even admit it was a possibility.

Finally I rewrote enormous chunks of the Plus application to use less volatile keys that mapped more directly to the way the users worked (vindication for you, benson). Of course, at the same time the CSAM group finally conceded to make that one tiny change (it took the head of the company to get even that), and they’re taking credit for reducing the number of issues with Plus data. Oh well, only 11 more days…

Dim intTabCounter As Integer = 0

For intTabIndex As Integer = tcNewsDayTabs.Tabs.Count - 1 To 0 Step -1
tabDate = Date.Today.AddDays((intTabCounter * -1))
tabDate = NewsUtil.FindBusinessDayDate(tabDate.AddDays(1), ldTabPanelDates)
ldTabPanelDates.Add(intTabIndex, tabDate)
Next intTabIndex


This is an example of a time something looks good on the surface but has no functional value once you look a little deeper.  In a bad economy like we have these days I look at this as job security.

As demonstrated in Part I, Part II, Part III, and Part IV of this series, the Plus application was a mess. It never really worked and when the desired functionality was simulated, it caused other issues. To be fair, the department that requested Plus in the first place was also a mess – its manager not only couldn’t manage the department’s 150 employees, but further had no understanding of what they did or what they were supposed to be doing (and those two things were definitely not the same). As a result, change requests for the Plus application had no order, logic, or sanity, and often contradicted other change requests made by the same department slightly earlier. It was a constant cycle of “Make this change” “Why did you change that? Change it back!” “Make this change”.

Culpability of the project requester aside, the Plus application was awfully, horribly coded. When eight new complex Swing forms are cached in memory every time the mouse is clicked, you know there are some performance optimization opportunities ripe for consumption. The code’s terribleness was truly stunning in the same way a 450-pound person is – you’re disgusted and feel sorry for their plight but oddly intrigued at the same time because you wonder how they can possibly perform even the most basic of daily tasks.

As noted in previous parts of this series, after being charged with managing communication about the Plus application’s bugs I could hold back any longer – I had to dive in and fix some of the problems. It was like coming home every day to a towering stack of food-encrusted dishes – eventually you just have to clean them up. I fixed a few of the smaller bugs in the codebase, tested the fixes, and checked the changes in to the Subversion repository.

Later that week I sat down to lunch with the rest of the IT squad. The boss was there as usual and had some interesting news. “I talked to Jim last night,” he led off (Jim and the boss worked at the same company for years – when the boss was hired at our company, he brought Jim with him from their old employer). “He was furious that you changed the Plus code.”

“What’s the problem?” I asked warily.
“He was extremely upset because you changed his code,” the boss said.
“…but I fixed it…”
“Yes, I know. But Jim and I talked for a long time tonight. He sees the changes you made as an intrusion. He acted very protective of his code.”
“…but I fixed it…”
“Jim said it was like defacing a masterpiece. He called Plus his opus.”
“His opus? That’s normally used to describe an accomplishment. He considers Plus his greatest achievement?! Those saps who hired him are in for a surprise!”

Ahh, they definitely should have paid me more to put up with this.

One of the prevailing requirements for the Plus application was that it allow field staff to enter data offline and “sync” it over the Interwebs at their leisure. In addition, the application had to get updated schedule information and previously-synced data and show it to the user while offline. As you know, the most logical design that meets this requirement is a disconnected client-server model. Therefore, a server had to be built to accept connections from the client, process incoming data, and send back schedule  information and previously-synced data.

This by itself isn’t such a big deal, right? Apache Tomcat exists just for tasks of this nature. It’s an entire server container that’s mature, been heavily tested, well-maintained, and trusted by enormous companies that could easily afford to choose any server product on Earth. It includes all sorts of advanced configuration options, logging facilities, exception handling, and standards adherence. There’s even a Windows installer package available! Installation, configuration, and basic Java implementation is fast and easy.

But since this was Jim’s project, it wasn’t fast, easy, simple, logical, standards-based, or maintainable. Jim rolled his own Tomcat server instead. Yes, that’s right – he ignored the ready-built server software available and wrote his own software to do the same thing. However, his version didn’t have the benefit of testing…or configuration options…or exception handling…or standards compliance…or maintainability…or – well, you get the idea. As a result, Jim’s Tomcat didn’t really work.

As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

Jim split the server implementation into two completely separate areas of the server itself. The actual application itself was only responsible for receiving client requests for schedule information – but not for sending the data itself back to the client. Instead, the application just zipped the schedule data into a temp directory and sent the file path back to the client. The client then connected to an SFTP server and downloaded the specified file. When the client needed to download previously-submitted data files, it connected to the same SFTP server, navigated to a hard-coded directory, and downloaded everything in it. Yes, that’s right – each user ended up downloading all data ever submitted by any of the 150 users every time the client updated. As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

The submission of data, however, was handled by the client. The client connected to that same SFTP server, navigated to a hard-coded directory, and uploaded individual files to it. Then a cron job ran every 60 seconds, swept the directory, determined the type of each data file by matching the beginning of its filename, loaded the data into a database, and moved the files to the “submitted data” folder that every client downloaded. If a file couldn’t be read, the cron job died and any file not already loaded was left in the folder. The next time the cron job ran the same problem would occur…without any notification to anyone.

I inherited all of this when I was stuck with the Plus project. Had I known what I was getting into, I would’ve said “You’ve gotta pay me more.”

So one day I decided that I may have better opportunities somewhere else.  So I got a new job at an insurance company and was now exposed to a very pure .Net atmosphere.  The main role I would have would be developing and maintaining the portal that they used for their agents.  Seems relatively simple.  Certainly there would be some quirkiness to this portal but the whole thing would be able to reuse a few abstract patterns right?

No not right.  The author of this application wouldn’t recognize a pattern even if it hit him in the jaw.  Oddly enough he was a consultant making multiple hundreds of dollars per hour.  What I learned about the .Net world is they encourage database heavy applications.  You’re probably thinking, oh they must have had ten or twenty stored procedures in their database if that were the case.  No, not 10 or 20, not even 100 or 200.  This database had 551 stored procedures in it.  In addition there are 172 tables and 104 functions.  Now at the time I had used some SPs for some complex needs and what not but I hadn’t seen anything like this.  I wondered if I was in over my head and wouldn’t understand the deep complexities of all of this data and perhaps I made a big mistake coming to this company.  After working on this for 2 years I realized that the man who developed this portal was sent from satan to destroy my hopes and dreams.  I will give you an example to help illustrate of 2 tables.  One called DO_DOCUMENT and another called FS_DOCUMENTS:

Look familiar?  Well isn’t that special?  Why use one table when you can use 2 that are almost identical?  This must be some methodology that I am not familiar with.  Maybe the were using one table to store inactive data to increase performance.  Not likely, both tables had different types of documents in them based on business type.  I looked in the table on one contained 20,000 rows the other 50.  Seems like an opportunity for re-use to me.  This is one of many scenarios where logic or data was duplicated.  What is odd that in this dark place there were some point of brilliance.  But the .Net world was a bit different than the Java world.  The developers that I knew that were working in Java didn’t want anything in the database except data.  So this gave me an opportunity to discover the pros and cons of both.

The truth is we all write bad code at some time or another.  Somewhere out there someone is writing a message just like this about my code.  But one thing I will say about myself is I work as hard as I can to write as little code as I can.  Mainly because it is less work and more stable.  But stored procedures seem like extra work to me and that isn’t what I am all about.

On the first day, one of the contractors, Alicia,  was assigned to train me.   I should have known something was up when Alicia checked the code out and could not get it to compile.  She needed to go find Shaun, because ‘He had the working version”.  I asked, “Doesn’t everyone have a working version?”.  She replied, “No, Shaun does.”  It appeared that every time new code was added and changes were made, it broke everyone’s local copies, and all the developers would play around with the code until one of them got it to compile.  Then everyone would get that person to ‘recreate’ what he/she did on their computers.

Doesn’t everyone have a working version?

Boy THAT should have been a red flag. Apparently, Shaun was the only person who could compile and deploy the code.  This week. Eventually, with Shaun’s help, the application was checked out of source control and compiled on my computer.  But for the next week, I saw PPT presentations of the ’system design’, and code snippets on screen without actually having any interaction with the code.  When I finally pressed the point that I wanted a ‘walk through’, Alicia reluctantly agreed.  The application, being a J2EE implementation, was launched and the login page displayed.  Alicia then logged in using a test account and walked through the application, which bombed every other screen because “it can’t hit the service, you need to change the settings to point to a test service.”  Well, let’s do that, then….

Fast forward ANOTHER week.  The application can finally run on my local machine, but there have been a half-dozen configuration changes to the local server, MANY changes to the local code, which, I am instructed to never check in, and any attempt to run the application in ‘debug’ mode causes my laptop to freeze and crash…

Oh, and I saved the best part for last.  At some point during my ‘training’, I actually did look at the code.  I found the start of execution.  It was a Servlet.  It had over 2000 LOC in the doPost method.  It used reflection to dynamically instantiate dozens of other servlets who’s names and methods were contained in a Database.  It was the slowest running application on the planet.

I lasted 1 year exactly.  In that time dozens of developers were hired. Most left before me.

They really should have paid me more….

Next, a glimpse into how this was coded.