Obscure Problems and Gotchas

I have been wanting to write about this for a long time now. I have seventeen years of professional software development experience at the time of writing this. Add all of my personal experience to my professional experience and I am well over twenty years easily. There is something I have noticed over the years that has always bothered me and there is no solving it. However, I have noticed that it is far worse in non-software development shops.

Code camp

Every year I make it a point to go to Code Camp, that’s what it has always historically be called, but most recently it’s been rebranded to SoFlo DevCon. I’m sure this link will go dead eventually, but here it is if you are interested https://linktr.ee/TechHubSouthFlorida. Honestly, not the best convention. It was far better back in 2004 and it has dwindled in terms of actual tech talk. Regardless, I go because you never know, you might find a few good learnings and I did this time. Very pleased with some of the things I saw. You can’t complain about free.

Why now?

I am bringing this up because one of the classes I sat in on had an interesting title: “There is no such thing as technical debt” presented by “Ennis Lynch”. I thought to myself, “Okay sure… let’s see what this guy has to say.”

Ennis’s presentation summary

• Stop whining about tech debt no one cares.
• Instead you have to do the leg work to prove to the business that what you are proposing will make the business money.
• Rewriting software from the ground up is not a good idea typically. You should instead use the strangler fig pattern.

Buddy boy here said everything except the one important business concept that summarized his entire presentation into one abbreviation: ROI – Return on Investment.

The older more experienced developers in the audience asked him valid questions and he deflected everything by calling us “unprofessional”. He said it numerous times. He then asked somewhere in the middle of his presentation if we could take a picture of him and put it on Linked-In. Nice… Well Ennis, this is going on Linked In – you asked for it – literally. At this point I was getting irritated because Ennis shut down someone’s question by performing a false dilemma fallacy argument. I wish I could reconstruct the stupid discussion from memory, but I cannot this was several weeks ago now. All I remember is the outcome:

• A developer asked a question about the necessity of performing unit testing of a system based on a principal of some kind.
• Ennis then shut it down by saying some like, “Well then there is no point in doing anything, we should all lie down and die then.”
• To which I turned around to the developer who asked the question and said loud enough for Ennis to hear me, “Well that wasn’t much of an answer was it?”
• The point is Ennis didn’t answer his question and instead deflected. The developer was making a good point.

Ad hominem

I asked a question at this point to Ennis and made it a point to inform him that he didn’t answer the other audience member’s question so I reiterated the point. Again not remembering exactly what I said so it was something like this:

“What do you do when there is a critical problem that if the business does not address it will harm the business?”

Ennis, instead of answer the question, then said something to the effect of “Did you put in the two weeks of time to prove to them it’s a problem?” I told him, “Yes I did and it didn’t matter. In fact, I don’t even bother anymore because I have learned that the only way to get business to do the right thing is to wait for the problem to get so bad that they start to hemorrhage money. Then they pay attention.” People in the audience laughed. Not my intention, but it happened.

Ennis at this point was red faced and said very sternly to me, “THAT’S VERY UNPROFESSIONAL”. I just stared at him. It was very obvious from this interaction that this clown hates developers and only cares about things that make money. You can throw a rock in South Florida and hit at least ten people like this and I am tired of them. It’s why his introduction was full of red flags for me. Failed software developer, who turned to a very fickle and baseless profession and flexing that he has made million dollar companies. You aren’t fooling me.

Ok cool story bro, what’s your point?

The reason I am bringing all this up is because it just furthers my belief that there is an inherent disrespect of software developers in the modern software development industry. It’s very insulting and it angers me greatly. However, like I already stated initially, there is no fixing this. The reason it cannot be fixed is because software developers are skilled professionals who are hired by people who know nothing at all about software in some cases (non-software shop). Even when those people do know about software (software shop), the business just gets in the way of making the right software decisions. This isn’t a hard concept to understand because there is a balance to what is asked of software developers and especially Principal Engineers (Architects).

How companies with software components basically operate

This is an oversimplification for argument’s sake otherwise this article will never end.

• For a company to stay alive and cash positive, then features or entire products must be delivered to customers in order to generate revenue.
• The largest cost to a company usually is personnel, also known as human capital or to put it simply salaries. That’s just a fact, not anything to be upset by. Morality aside, it’s why some companies will do yearly layoffs so that they can save money.
• Software developers work on tasks assigned by their managers.
• Managers are given projects to complete by dates set by the business or the Project Management Office (PMO).
• THEREFORE, what you are working on is rigid because there are goals to complete. No one wants to hear about non-goal oriented work because it means you aren’t delivering product.
• What Ennis failed to say was that the company wants a Return on their Investment (ROI).
• The work that software developers do must in most cases yield a ROI and if it does not then it is seen as a nasty cost.
  • Anything that yields a ROI is a Capital Expense (CapEx) usually. It all depends on how the accountants are instructed to categorize costs (salaries) that were used to complete a project.
  • Anything that does not yield ROI is considered an Operational Expense (OpEx) and this is not good for revenue. It means the business is spending money on things they are not going to get revenue on. It makes them angry for a lack of better words. Your salary has now cost the company money! How dare you!

Therefore, as you can see the point is that when you go to the business and you say, “Man we really need to rewrite a part of product X because those idiot contractors you hired did a horrific job.” all the business is hearing you say is “I want to spend money and you get nothing in return.” They honestly don’t care why you want to do this work, even though it can harm their bottom line horribly. That’s the nature of the beast.

So Ennis’s message is not wrong, he’s right – if you care so deeply about a problem you should waste two weeks of your personal time putting together a presentation and documentation that could be completely ignored by your manager. However, there is a small chance they will listen if you tell them that you can make them profit.

What if we let software developers do whatever they wanted?

Let’s suspend reality just for a moment and say that we had a company that was run by a whole slew of software developers with the best of intentions. They were going to write the best goddamned software ever, they were going to fix all bugs in the system, they were going to add every quality of life feature you could think of. That company would cease to exist because it would run out of money immediately. We all know that it’s hard to stop trying to perfect software, now let a bunch of well intentioned software developers work with no boundaries the cash burn rate would be too high and I would be impressed if there was a profit at all.

Striking a balance

The problem here should be obvious at this point:

• If you lean too much towards the business’s needs, they will inevitably pay dearly for a problem that was pointed out by a software developer. Businesses only understand money. So when they start losing money they will do whatever they can to make the hemorrhaging stop. This is called being reactive which is a BAD HABIT.
• If you lean too much towards the developer’s needs, they will run the company into non-existence. Profit is required in order to pay those salaries and if you aren’t delivering anything then no one can get paid.

What pissed me off about Ennis’s presentation was that he, like so many business oriented people, had this disdain in his voice when speaking about software developers. The fact that he was calling us unprofessional repeatedly made me very upset. What’s worse is that he made a younger developer sitting next to me think he would get a raise or promotion out of performing such a stupid amount of work to make the company do what’s obviously good for them. I made sure to say loud enough while staring at Ennis, “Don’t hold your breath.” The reason I can say this confidently is because I tried Ennis’s proposal long before he suggested it and no it doesn’t work.

Furthermore, it’s insulting to me, as a developer, that I have to convince a company to do what’s good for them. If I am raising the alarm to highlight a potentially threatening software issue and you ignore me because I cannot tell you how it will turn you a profit, then I guess that software problem will present itself in production and then we will fix it when the company screams in agony. It’s not my job to convince you there is a problem. Also, if the company goes under I will find a new job and I won’t lose sleep over it. I am not going to kill myself to prove to anyone that my professional opinion is correct.

Bad examples

Ennis had the gall to tell us a story about how PayPal got rid of their patch team because it was costing the company too much money and instead they looked at the cost/risk analysis of not patching versus the cost of what happens every time the patch team did their job. Apparently the cost of patching their systems was too high because enter some dumb executive reasoning here. Apparently security flaws don’t matter to PayPal. I could not find evidence to corroborate this story, but it does sound dumb enough to be real. The point is that PayPal has made a morally stupid decision and they will pay for it in the future I am sure.

Let’s review some other stupid business decisions that were made because “profit”:

• In 1980 Bayer sold HIV tainted medication knowingly https://www.cbsnews.com/news/bayer-sold-hiv-risky-meds/
• In 1996 Firestone’s tires were causing Ford Explorers to flip over. Firestone made the financial decision to do nothing about this because it was cheaper not to. https://en.wikipedia.org/wiki/Firestone_and_Ford_tire_controversy
• In 2019 Boeing’s 737 Max software was outsourced to $9-an-hour engineers. It was cheaper to use offshore than it was to use competent developers and people died as a result. https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-737-max-software-outsourced-to-9-an-hour-engineers I have this article blown up and hung in my office.

I guarantee you, without a shadow of a doubt, in each of those cases there was an engineer who said, “Hey, don’t do this because of these reasons.” and the executive team just looked at them and said “Shut up” and did it anyhow.

My point with bringing all of these examples up is there has to be a balance. Companies should listen to their engineers when there is truly a problem OpEx be damned. Engineers need to understand that not every problem is a big problem because CapEx is very important. In order for a company to continue to thrive it’s important to strike a balance. That’s how employee happiness is gained. If employees are happy, then there will be higher retention. This is not rocket science.

Disrespect

The reason I think disrespect is so high against software developers is simple. It’s hard to understand what software developers do, therefore resentment builds and we are seen as a nuisance. I, for one, refuse to work at a company that says any of the following things anymore:

1. You get paid too much.
2. You guys are just a cost to the company.
3. You guys are too expensive to justify.
4. We are going to offshore your jobs.
5. I am going to offshore all of your jobs.
6. I don’t know what you do all day.
7. This is not a software development shop.
8. This is not a software development shop and if you don’t like it there’s the door.

This list can go on, but I think you get the idea. In each of these cases it’s clear that this is not a software developer friendly environment. I think it’s completely unfair that these companies hire software developers and then are upset to learn that they have hired software developers. It’s a textbook case of a bait and switch.

Insulting actions

Here are some insulting actions that companies I worked at have performed ad-nauseum. It’s comical at this point.

Getting an expensive second opinion

• Developer: “Hey manager, we need to do X, Y and Z.”
• Manager: “I don’t believe you, so I am going to hire a consultant for too much money.”
• Consultant: “Hey manager, we need to do X, Y and Z. Also I spoke to your developer and they told me, that they told you this already why didn’t you listen to them?”
• Developer: * stares at manager *
• Manager: * silence *
• * Manager gets fired *

Getting urgent projects done using contract labor

• Manager: “We have an urgent project that needs to get done in six months, therefore we are hiring contractors and you are going to help interview them and ramp them up.”
• Developer: “That’s a bad idea because they don’t understand anything about our system or how it works. Ramping them up is like ramping up a fulltime hire (FTE) which usually takes like three months so this doesn’t make sense.”
• Manager: “Too bad, we don’t have enough developers and we can’t hire more FTE.”
• Developer: “Okay, well I am warning you that you are wasting your money and that these contractors are more than likely going to just botch the whole project and we will have to do it all over again which means you are going to pay twice as much.”
• Manager: “Shut up you don’t know what you are talking about.”
• * Twelve months later *
• Manager: “Why aren’t we finished with this project yet? We are six months over due.”
• Developer: “Well the contracting company you insisted on using only has incompetent and desperate people from 3rd world countries who hardly qualify as software developers.”
• Manager: “That’s not an excuse, you are just not managing them properly.”
• Developer: “I am not a manager, I am a software developer.”
• Manager: “You are not being a team player.”
• * Manager gets fired *
• Interim-Manager: “Okay how do we fix this?”

This one always baffles me, because it always has the same outcome. It’s always a giant failure. It doesn’t matter how many times I have warned the business that this is a bad idea, they choose at that exact moment to totally forget what the management triangle is. It’s a fact that nine women cannot make a baby in one month, this applies to software too. Let’s say you do hire decent contractors, then there is a possibility that they can produce quality software, but they still won’t have tribal knowledge because that is gained by working a place for years not months. That’s a fact and that can harm the outcome of a project too.

Warnings of impending doom

• Developer: “Hey manager, I found a serious problem we need to address otherwise there could be serious implications. Here is an email documenting me telling you this so that when you completely ignore me and it happens I have irrefutable proof I warned you.”
• Manager: “There’s no time.”
• * serious implications occur *
• CIO: “How could this happen?”
• Developer: “I warned about this exact thing happening, but I was told there was no time to fix it. I know how to fix it.”
• Manager: * Sweating *
• CIO: “How long will it take you to fix this developer? Manager! Did you know about this?!”
• Developer: “I told Manager it would take two sprints to fix.”
• Manager: * Wide eyed * “No this is the first time I am hearing about it.”
• CIO: * Angry eyes *
• * Manager get fired *

I’ve been through this exact situation a few times, the manager doesn’t always get fired. Depends on how much they pissed me off, you might want to go to bat for them if it was an honest mistake. The biggest take away here is it’s not my job to convince the business how serious a problem is for them to fix. When I have tried in the past, all it did was hurt my career so I have learned to bring it up once and if they are not receptive drop it. According to Ennis I am unprofessional, but really it’s the business who is unprofessional here and I am willing to argue about this. I am right.

There’s no money in the budget

• Manager: “Let’s do your yearly review.”
• Developer1: “Okay, well I put down meets expectations for most of what I did and exceeds on some of these other things. I would like a raise or to be promoted.”
• Manager: “Ooooo sorry, there’s just no money in the budget for promotions. Here’s a 2% increase instead.”
• Developer2: “Hey did you hear that the new guy got brought in for 40K above our salaries?”
• Developer1: * Anger *
• Developer1: “Here’s my letter of resignation.”
• Manager: “Will you take a counter offer?”
• Developer1: “I thought there was no money in the budget?”
• Manager: “Well if you go get me an offer I can see what I can do for you.”
• Developer1: “If you made me upset enough to find a new job, I am not staying.”

I can say confidently throughout my entire career that the phrase, “There is no money in the budget” is a blatant lie. I have sat in on townhall meetings where the CEO is clamoring about record profits, but no one on my team had good health insurance or fair wages. There’s no fixing this, you just find a new job. Good riddance.

Conclusion

There has to be a balance between the business and software developers. You hired software developers to do a job, but if you resent their existence then you should have just outsourced your software to a software company. Ennis is absolutely wrong in suggesting that it’s the job of the developer to convince the business to do what’s good for them. It is entirely up to the business to decide what they do. If the business tanks as a result, the developer will find a new job. It’s unprofessional for the business to ignore their professionals.