Jedi Confession: The time I fired half the company

A long time ago in a galaxy far, far away…

I was on a quick weekend vacation in Miami. The sun was shining, the water was cool, and Ocean avenue was lit. My boys had just landed, and it was setting up to be a great weekend. It was Friday evening, and we had just successfully done our production release at my current customer, a well-known Fortune 500 company. It hadn’t been too challenging of a rollout, the use cases were reasonably straightforward, and we had an excellent phased approach to rolling out features. We had a full two weeks of testing, and everything was working great. It was at this point I should have known shit would hit the fan.

6 Months Earlier

I walked into the meeting room and took the seat in the middle of the table. It was a relatively large conference room with a great view of the company campus. The customer’s lead architect, program manager, and HR representative were all in the room, and my team was making their way in. The best thing about this customer was that everyone was amiable and excited about this project. It really showed the company culture, very loose and easy-going. As my team settled in and we began the meeting, I couldn’t help but feel excited about this project. It was starting out great, and the requirements were simple. Something had to go wrong soon.

Halfway through, the conversation with the HR representative and the customer architect starts to ask questions about the onboarding process. My team chimes in, and we ask the typical questions. When do you first create the HR record? How much data do you collect? When does that record hit the other systems? Every question is met with a straightforward answer. And then…..the question.

“What do you do when you terminate an employee?”

If this were a movie, this is the point where the dramatic music would come in, the camera would slow zoom onto the HR representative, and they would pause for effect. But it wasn’t; however, what the representative said next would foreshadow the disaster that would come later.

The HR rep took a pause before answering and gathered herself slightly in her chair. The customer architect leaned back in his chair with a slight smirk, and I knew this was not going to be good. She begins to explain the process of the old system that they used years ago.

“Oh shit.” I thought

It was a complex system, and back then, they had a different process for handling employees and contractors. When they switched to the current system, they also changed how they handled employees and contractors. Specifically because they had a high percentage of contractors that would become employees and vice versa. Shameka leaves work on Friday as a contractor and comes back in two weeks as a full-time employee on the same project she left. So they wanted to make sure that those employees had a smooth transition.

“Shiiiiiiit…” I thought

So they would let the employee keep the same record, put in their termination date, but keep the record active. Which means they had HR records with an active status but also a current termination date.

“FUUUUUUUUUUUCCCCC…” I screamed in my head

I looked around the room and noticed the lack of shock on my customer’s faces and the absolute horror on my team’s faces. We had seen this before, and we knew it didn’t end well. I asked the obvious question: “How do you determine who’s actually active and who isn’t?” I wish I could tell you I remembered what she said, but I honestly don’t, because at that moment, I knew that whatever she said would be bullshit and mean, we would be coding some fuzzy logic and manual process to determine if a user was indeed terminated. Spoiler alert: I was wrong.

When we finished the meeting with HR, the customer architect began to tell us a story of an employee that had worked at the company for several years. They retired but stayed on for a short amount of time afterward as a contractor. Their process was so bad that said employees continued to get paid for six months after they officially stopped working for the company.

Six.Fucking.Months

It stopped because the employee was also a saint and decided to contact the company to inform them that they were still paying him. Seriously, the guy was a saint.

Over the next couple of weeks, we worked diligently with the customer architect and HR to devise a plan for dealing with this issue. Luckily, HR was already working on the problem and finishing up a data cleanup initiative that would clearly delineate who was an active employee and who wasn’t. The issue, however, was that it wouldn’t be done in time for our go-live deadline. After a couple of weeks of negotiation, we decided to push back our release to await the clean data, and we pushed to get a test environment with production-like data to build our rules and test. Because you know, test data is always accurate.

2 Weeks before Go-Live

At this point, I made my concern known to my leadership team and the customers. We counted on accurate data from HR and had built rules that expected data in a particular state. If the state did not exist when we went live, the system would do what we programmed it to do: disable accounts of terminated employees. We decided that the risk was acceptable as we had assurances from HR that the data would be there. I, on the other hand, have trust issues. I informed the team to build a secondary workflow that marked the users as terminated in our system but had the actual process manual action, thus ensuring that nothing automated would take place. I also made sure we had a way of reversing any accidental terminations in case shit went wrong. It added some extra tasks to all of us and caused some late-night coding sessions, but thankfully we got those changes in because you just never know.

Back to Miami

It was a beautiful Saturday morning in Miami Beach. My head was pounding from way too many drinks the previous night, and I couldn’t wait to do it all again that night. It was roughly 11am eastern time, and I hadn’t yet turned my phone on, so I decided to plug it up and head downstairs to get some breakfast. I met up with one of my boys, and we grabbed some food and made plans for the rest of the day. First, the beach, then some paella, and the nightclubs. 

A perfect day.

I made it back up to my room and walked over to my phone. 5 missed calls, 15 text messages and counting, and a shit ton of emails. Instantly my stomach sank. The last missed call was from one of the devs on my team. I called him back to ask what was going on.

“Dude, have you not checked your email!” he screamed.

“No, it’s Saturday. Why would I?” I responded.

“The customer is flipping out; shit is not working, man, this is bad,” He said.

‘Fuck”. I responded

“Okay, fill me in. I’m checking emails right now”. He gets halfway through the explanation when I get an incoming call for a number I didn’t recognize. I told him to hold on, and I answered. It’s the architect from our customer. The following 3 minutes are an expletive-filled rant of threats, demands, and general hysteria. As you can imagine, the HR data wasn’t accurate, and the process we ran overnight had thus far fired about half of the accounts we had brought in. People were sending help desk tickets left and right, wondering why their accounts were no longer working.

Shit show center stage.

I tried my best to calm him down and let him know that we could fix the issue and get everyone’s access back. We first needed to stop the synchronization process and run the workflow we had created to repair the accounts. Unfortunately, this wasn’t enough for him; he wanted blood. Didn’t care if it was mine, my boss’s, or our CEO, but someone had to “pay” for this mistake. Now granted, he was in the meetings when we talked about the risk; he was fully aware of the issue that may arise and had full access to the results we saw during our testing, but the customer is always right….right?

I spent the next 6 hours discussing with my boss, and the customer, and their admin team fixing the situation. I informed my developers to turn off their phones and enjoy the weekend. I had the final say from our side to go live, and I would be responsible for getting it fixed. We quickly fixed the access problem; the rest of the time was spent just listening to the customer yell at us. Which I somewhat understand, but not really.

The moral of this story is young padawans; never trust HR.

No, just kidding. Something will always go wrong. No matter how well you code or how much you plan, shit just likes to hit the fan. Do your best to think ahead and give yourself options to navigate. Raise concerns early to your customers and leads, and leaders protect your team members. Above all, take a deep breath. No one is perfect, and we will all make mistakes; it’s how we respond to those mistakes that matter.

Until the next time. May the force be with you.