“How does the utilization report arrive at its numbers?”

It seemed like a simple, straightforward and innocuous question at the time. Unfortunately, it opened an institution-sized metallic food storage container of blind, soil-ingesting invertebrates.

You see, the person who’d commissioned the development of the utilization report didn’t really understand what he was building. He had vague amorphous ideas about what he wanted – perhaps he’d seen a report at some other company that was used to determine how well the employees were being utilized and decided it was a good idea. Based on second-hand knowledge and investigation into the reports themselves, here’s how I picture the utilization report’s progression in my head:

Project Requester: “I want a report that tells me if there’s enough work for my employees.” –note: He’s saying “show number of hours worked divided by number of hours that could have been worked”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to take holidays into account.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to take PTO into account.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It  should use the number of hours the employee has been scheduled to work instead of the hours actually worked.”
Programmer: “What?”
Project Requester: “It has to.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. If an employee had a job that ran late the night before, the Scheduling department schedules an employee for a block of time the next day in order to keep him from having to work. The report has to take that into account when it happens.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to show that an employee worked the whole weekend if he was scheduled for a Saturday.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong. It needs to assume every employee spent two hours to get to the job site and two hours to get home.”
– time passes –
Programmer: “OK, here it is.”
Project Requester: “It’s wrong.”
Programmer: “How is it wrong?”
Project Requester: “The numbers can’t possibly be that low. The report is wrong.” –note: It showed an average of 64% utilization (i.e. employees jacked off 36% of the time for which they were paid). It was later found that actual utilization was 52% if they were afforded every possible benefit of the doubt.

The utilization report was used to determine year-end monetary bonuses for managers. However, every time the utilization numbers were too low to allow the maximum bonus, the managers would say “The report’s always been inaccurate. The numbers are really much higher.” And the bonus would be awarded.

It turned out “the utilization report” at my company actually meant six distinct reports. Here’s how the system worked:

• At the beginning of every month, a user would submit an IT ticket as a reminder to “run” the report on the 15th of the next month
• Somewhere near the 15th of the month, a specific developer would navigate to a directory on his machine and double-click an Access database
• The developer would then locate each of the six reports in the Access database and run them all in turn, exporting them as RTF files when they finished
• Each report was extremely resource-intensive and completely took over the IT guy’s computer, rendering him useless for hours – and even longer if the Access database crashed
• When all reports had been exported, they were attached to an email and sent to the user from Step 1

This process was repeated every month and caused much frustration for the user (because they couldn’t get the report immediately) and the developer (because it consumed his computer and he couldn’t get anything done). After hearing the developer curse and complain about the utilization report a few times, I set to work on the Access database. The process was changed to:

• On the 15th of every month, Windows Task Scheduler opens the Access database, triggering a VBA subroutine
• Each report is exported to RTF
• All exported reports are programmatically attached to an email message and sent to the user

The new process is entirely automated and takes about three minutes to complete. It turns out Access was running all of the report queries (a bunch of UNION ALLs in a number of views) twice – once to display the report to the developer, and again to export the data to RTF. In addition, the formatting process required to display the report inside Access took an enormous amount of time, whereas the formatting of the RTF to be displayed in an external viewer was far quicker.

Given the tendency of Plus users to do the most illogical things possible while refusing to read or apply any kind of instructions given and then blame the software for behaving “unexpectedly”, it seemed only fitting to provide them with the one thing they couldn’t screw up: the Do Not Click button.

A lot of thought went into the Do Not Click button. Its placement was particularly important – it had to be accessible yet unobtrusive enough to go unnoticed for a while. The button itself had to be recognizable as capable of performing an action, yet simple enough to be overlooked by a casual glance. It had to display enough of a warning to be clear that clicking was ill-advised, yet low-key enough to beckon seductively. The final product is thus:

Fetching, no? It was placed at the very bottom of the About dialog, which I doubt anyone ever accesses unless by accident. If any one of the 150 users happens upon it, however, he’s all but guaranteed to email the other 149 to tell them about it. And that’s when the fun starts.

“What happens when the Do Not Click button is clicked?” one might ask curiously. Let me give you the most likely scenario:

User: “What’s this thing that looks like a stop sign with a white X in it? That looks weird. Wait, there are some hieroglyphs beside it…they say ‘d’…no, ‘D’…I’ve got it! It says ‘Do’! Phew, that was hard work. I’d better take a break so as not to strain myself.”

<- four hours later ->

User: “Why is there a window that says ‘About’ open? This stupid software, always doing things I don’t tell it to! I’m going to create a helpdesk ticket for IT to fix it! No, I should compose a poison-filled email about how crappy Plus is and how it’s always broken and send it to all of the other field technicians so they can join me in my uninformed griping!”

<- two hours later ->

User: “All that typing made me hungry. I’m going to go to the expensive steakhouse down the road and charge a hefty three-course to my company credit card and then take a nap.”

<- the next day ->

User: “Hmm, what’s this window that says ‘About’? There’s a button that has a stop sign with a white X in it. It doesn’t say what it does…<click> Nothing happened. <click> <click> <CLICK> Stupid Plus! How can I be expected to perform any work if this trashy program won’t do anything? Those idiots in IT are just trying to keep me from doing my job!”

Sadly, this is probably a realistic depiction of events. However, the Do Not Click button does do something – it records the number of times it’s been clicked. Yes, that’s right – there’s a persisted click count. When the user submits work (which is supposed to happen once every few days), the cumulative click count is stored to a server-side database along with the user’s username. There it waits until an intranet-accessible web page is requested, wherein it’s included in a report showing the users clicking the Do Not Click button the most.

And here’s the best part: Any “bug” report about the Do Not Click button being broken can be closed with “The button works as intended. Do not click the Do Not Click button.” When the question of why the button exists comes up, the answer is “The button is there to keep an issue from occurring. When the button is clicked, the issue occurs. If the button is not clicked, the issue does not occur. This is why it says ‘Do Not Click’ on the button.”

/**
 * Added 10/7/2009 by specific request of sjohnson. Of course there's no HD
 * ticket or other documentation for the request because everyone knows
 * normal rules don't apply to the field staff.
 */

Code comment written today:

/**
 * Removed 11/5/2009 by specific request of sjohnson. Of course there's no HD ticket
 * or other documentation for the request because everyone knows normal rules don't apply
 * to the field staff. sjohnson swears this is a showstopper and must be removed because
 * now that he actually has to work in Plus on a daily basis he understands how
 * ridiculously foolish it is to clear out the time entry fields after every entry is
 * added to the table.
 */

Mere days before a version of Plus was to be released, the manager in question demanded that the functionality be changed as it was “not the way the staff work”. He was made aware of the reason the product functioned the way it did and warned of the problems that would occur should his demand be met, but these logical beseechings fell upon deaf ears. He was given an opportunity (two full weeks) to test this change and he signed off on the final product. Almost exactly a month later, the manager had to take on the role of a field technician. As part of that, he had to use Plus, and blamed me for “making changes without [his] knowledge”.

This manager demanded that this and a second change be immediately coded and pushed into production as an “emergency” release…without any testing. He was warned of the expected problems of the second change and how testing helps ensure quality, but again he exhibited Deaf Man’s Vendetta. We’ll see how that turns out.

Update: A week later (11/13/2009) the same manager went into another panic and demanded the functionality be reverted again, emailing “can you please verify and correct this issue immediately? This is the same issue we had in October!”

When I started working for this firm, they had an existing DBA who fancied himself a pretty smart guy.  Shy of the countless double-negatives and mis-information he would spew on a daily basis, he was tolerable and he kept to his world of existing “black magic” processes.  He liked to refer to himself as “NULL Man” and act like a super hero when he was “fixing bad data”.  For the first 9 months of my tenure, I really had no idea what he did on a day-to-day basis, but had a feeling his primary goal was to figure out how to interject himself into as many business critical processes as possible in order to assure job security.

I was hired to bring up new apps and databases and do some .NET development as well.  After about 9 months on the job, the Director of IT called me into his office and asked if I could rework a financial report he needed to give our CFO for the upcoming year-end.  In previous years, NULL Man would require a dedicated week’s worth of work to compile this report and massage the data to get the “proper numbers”.  This seemed like an awful amount of time to compile a 12 page financial report,  so our Director asked if I could take a look at it.

Leaving the Director’s office, I decided I would just go over to NULL Man’s desk and let him know that I was required to get cross-trained on how to create this report.  Under this guise, my plan was to basically dissect what in-fact was going on and automate it as much as possible so I didn’t have to deal with this again in the future, as developing reports never really excited me.

Thankfully NULL Man had a “busy” month-end coming up and welcomed the opportunity to “finally unload some of this work to someone else”.  We started out going over the purpose of this report and eventually how it all worked behind the scenes.  It turns out that this report wasn’t 12 pages by chance.  It was “designed” that way; each page having it’s own dedicated query to retrieve, cleanse, and format the data with Excel basically just acting as a container app.  The queries I were given were actually all stored procedures (to help with performance, of course) and were on average about 500 – 800 lines of code.  At first glance, this report looked very complex until I began to dig in and see what was actually involved.

It didn’t take more than 5 minutes to determine the real reason for so many lines of code.  NULL Man in his infinite wisdom wasn’t really fond of set operations, or really any advantages a database gives you when storing data in it, and he was even more clueless when it came to Excel.  Sadly, the query for each page would look something like this:

-- Get Information for Page 1
SELECT Product, SubProduct, Cost, 0 AS NewCol01
INTO #tempTable1
FROM SourceTable
WHERE Product LIKE '%ProductLine1%'

-- Pull out information for Sub Product Line 1
UPDATE #tempTable1
SET NewCol01 = Cost
WHERE SubProduct LIKE '%SubProductLine1%'

-- Create a new Column
SELECT *, 0 AS NewCol02
INTO #tempTable2
FROM #tempTable1

-- Pull out information for Sub Product Line 2
UPDATE #tempTable2
SET NewCol02 = Cost
WHERE SubProduct LIKE '%SubProductLine2%'

...

-- Create a new Column
SELECT *, 0 AS NewColX
INTO #tempTableX
FROM #tempTableX-1

-- Pull out information for Sub Product Line X
UPDATE #tempTableX
SET NewColX = Cost
WHERE SubProduct LIKE '%SubProductLineX%'

-- Return Page 1 Information
SELECT Product, SubProduct, Cost,
    NewCol01 AS ProductLine1Cost,
    NewCol02 AS ProductLine2Cost,
    ...
    NewColX AS ProductLineXCost
FROM SourceTable
WHERE Product LIKE '%ProductLine1%'

UNION ALL

SELECT Product, SubProduct, SUM(Cost) AS TotalCost,
    SUM(NewCol01) AS TotalProductLine1Cost,
    SUM(NewCol02) AS ProductLine2Cost,
    ...
    SUM(NewColX) AS ProductLineXCost
FROM #tempTableX
GROUP BY Product, SubProduct

This is, of course, a generalized version of the SQL I had just been exposed to, but this was the basic methodology of every “page” in this report, and to my dismay just about every bit of SQL NULL Man ever created.  I asked him why he decided to take this approach, and to my utter bemusement, his response was, “It’s more efficient!”.  Ironic really, because any work being performed in the database at this point was a waste.

After verifying that every “page” of the report followed the pattern outlined above, I “optimized” the entire report into a simple query:

SELECT Product, SubProduct, Cost
FROM SourceTable

The bulk of the work was done in Excel with Computed Columns, Filters, and Grouping.  This whole process, from evaluation of the existing report to creation of the new one took a grand total of 4 hours, start to finish.  I sent the report onto our Director and the CFO and told them they could refresh the report at any time so they no longer needed to wait on us for up-to-date numbers.  NULL Man voiced his concern that the CFO might not fully understand how to use Excel, but eventually shrugged and said, “I told ‘em it’d be easier if we let Excel handle most of the work, but they didn’t listen.”  It was at this point I knew my future with this company would comprise of fixing and optimizing NULL Man’s work.

As relayed in previous posts in this series, the Plus application was meant to facilitate the transmission of data from field technicians to the home office. However, at some point it was decided that Plus should also be responsible for transmitting and displaying data from the home office to the field technicians. This information ended up being things like equipment-specific metadata, customer contact numbers, location of each piece of equipment, a directory of everyone employed at the company that developed Plus (no, I’m not kidding), and schedule data. All of this came from a poorly-standardized, non-normalized database used by the company’s CRM application, Nexterna Clearview, which uses a batch-and-post system.

The schedule data was an important piece because it gave the field technicians a way to see scheduled work without having to use any brainpower. After adding this information to the Plus application the powers that be decided Plus should use it to automatically determine and display the appropriate data forms for the technician to fill out for each site visit. It was further decided that the forms themselves should dynamically change based on the equipment information from the CRM application.

Eventually this was all made ready and deployed to the field technicians. As usual, there were exclamations of “Wow, this is great!” and “This will make us so much more efficient!” at first. But soon the users started reporting problems. They’d enter data into a form and it would disappear the next time they downloaded schedule updates from the home office. It turned out Jim built Plus and the Plus server to use auto-generated database record IDs as unique identifiers for schedule data. When a field technician entered form data, it was tied to the schedule’s database record ID. When the form data was sent to the home office, the Plus server displayed it to home office users by establishing a link between the schedule’s database record ID and the record ID in the returned data. In a normal world, this wouldn’t be a problem – using database keys guaranteed to be unique is good practice, right?

The issue came into play when one of the schedulers at the home office altered a schedule after a field technician had already used Plus to enter form data. The scheduler, called a ‘CSAM’, would change the schedule’s location, equipment, date, assigned technician, and even go so far as to delete the schedule altogether and create a new one in its stead. The next time the field technician used Plus, he saw a set of blank data entry forms assigned to the new schedule instead of all the data he’d already entered under the old schedule. In essence, data entered into Plus was “lost” due to the normal day-to-day operation of the CSAMs.

Later I found that there had been many a heated argument betwixt Jim and benson about the use of auto-generated database record IDs as data keys. “That’s not how the users work!” benson would exclaim. “It’s the right way to do things!” Jim would shoot back. The end result was always the same – Jim would simply continue the way he wanted.

For quite some time people just blamed Plus for everything. “This stupid tool loses my data!” they’d gripe. “It eats my work!” Eventually I was able to show that a very small change to the existing CSAM scheduling process could avoid the majority of these situations, and the complaints changed to…well, they didn’t change at all. Everyone still made Plus the scapegoat, and the CSAM group refused to believe they could possibly have anything to do with the issues. Their leader wasn’t willing to even admit it was a possibility.

Finally I rewrote enormous chunks of the Plus application to use less volatile keys that mapped more directly to the way the users worked (vindication for you, benson). Of course, at the same time the CSAM group finally conceded to make that one tiny change (it took the head of the company to get even that), and they’re taking credit for reducing the number of issues with Plus data. Oh well, only 11 more days…

Dim intTabCounter As Integer = 0

For intTabIndex As Integer = tcNewsDayTabs.Tabs.Count - 1 To 0 Step -1
tabDate = Date.Today.AddDays((intTabCounter * -1))
tabDate = NewsUtil.FindBusinessDayDate(tabDate.AddDays(1), ldTabPanelDates)
ldTabPanelDates.Add(intTabIndex, tabDate)
Next intTabIndex


This is an example of a time something looks good on the surface but has no functional value once you look a little deeper.  In a bad economy like we have these days I look at this as job security.

As demonstrated in Part I, Part II, Part III, and Part IV of this series, the Plus application was a mess. It never really worked and when the desired functionality was simulated, it caused other issues. To be fair, the department that requested Plus in the first place was also a mess – its manager not only couldn’t manage the department’s 150 employees, but further had no understanding of what they did or what they were supposed to be doing (and those two things were definitely not the same). As a result, change requests for the Plus application had no order, logic, or sanity, and often contradicted other change requests made by the same department slightly earlier. It was a constant cycle of “Make this change” “Why did you change that? Change it back!” “Make this change”.

Culpability of the project requester aside, the Plus application was awfully, horribly coded. When eight new complex Swing forms are cached in memory every time the mouse is clicked, you know there are some performance optimization opportunities ripe for consumption. The code’s terribleness was truly stunning in the same way a 450-pound person is – you’re disgusted and feel sorry for their plight but oddly intrigued at the same time because you wonder how they can possibly perform even the most basic of daily tasks.

As noted in previous parts of this series, after being charged with managing communication about the Plus application’s bugs I could hold back any longer – I had to dive in and fix some of the problems. It was like coming home every day to a towering stack of food-encrusted dishes – eventually you just have to clean them up. I fixed a few of the smaller bugs in the codebase, tested the fixes, and checked the changes in to the Subversion repository.

Later that week I sat down to lunch with the rest of the IT squad. The boss was there as usual and had some interesting news. “I talked to Jim last night,” he led off (Jim and the boss worked at the same company for years – when the boss was hired at our company, he brought Jim with him from their old employer). “He was furious that you changed the Plus code.”

“What’s the problem?” I asked warily.
“He was extremely upset because you changed his code,” the boss said.
“…but I fixed it…”
“Yes, I know. But Jim and I talked for a long time tonight. He sees the changes you made as an intrusion. He acted very protective of his code.”
“…but I fixed it…”
“Jim said it was like defacing a masterpiece. He called Plus his opus.”
“His opus? That’s normally used to describe an accomplishment. He considers Plus his greatest achievement?! Those saps who hired him are in for a surprise!”

Ahh, they definitely should have paid me more to put up with this.

One of the prevailing requirements for the Plus application was that it allow field staff to enter data offline and “sync” it over the Interwebs at their leisure. In addition, the application had to get updated schedule information and previously-synced data and show it to the user while offline. As you know, the most logical design that meets this requirement is a disconnected client-server model. Therefore, a server had to be built to accept connections from the client, process incoming data, and send back schedule  information and previously-synced data.

This by itself isn’t such a big deal, right? Apache Tomcat exists just for tasks of this nature. It’s an entire server container that’s mature, been heavily tested, well-maintained, and trusted by enormous companies that could easily afford to choose any server product on Earth. It includes all sorts of advanced configuration options, logging facilities, exception handling, and standards adherence. There’s even a Windows installer package available! Installation, configuration, and basic Java implementation is fast and easy.

But since this was Jim’s project, it wasn’t fast, easy, simple, logical, standards-based, or maintainable. Jim rolled his own Tomcat server instead. Yes, that’s right – he ignored the ready-built server software available and wrote his own software to do the same thing. However, his version didn’t have the benefit of testing…or configuration options…or exception handling…or standards compliance…or maintainability…or – well, you get the idea. As a result, Jim’s Tomcat didn’t really work.

As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

Jim split the server implementation into two completely separate areas of the server itself. The actual application itself was only responsible for receiving client requests for schedule information – but not for sending the data itself back to the client. Instead, the application just zipped the schedule data into a temp directory and sent the file path back to the client. The client then connected to an SFTP server and downloaded the specified file. When the client needed to download previously-submitted data files, it connected to the same SFTP server, navigated to a hard-coded directory, and downloaded everything in it. Yes, that’s right – each user ended up downloading all data ever submitted by any of the 150 users every time the client updated. As anyone might imagine, the directory quickly grew and eventually consumed all disk space on the server, crashing it.

The submission of data, however, was handled by the client. The client connected to that same SFTP server, navigated to a hard-coded directory, and uploaded individual files to it. Then a cron job ran every 60 seconds, swept the directory, determined the type of each data file by matching the beginning of its filename, loaded the data into a database, and moved the files to the “submitted data” folder that every client downloaded. If a file couldn’t be read, the cron job died and any file not already loaded was left in the folder. The next time the cron job ran the same problem would occur…without any notification to anyone.

I inherited all of this when I was stuck with the Plus project. Had I known what I was getting into, I would’ve said “You’ve gotta pay me more.”