BACKGROUND
Working in IT is one of those things: you love it, but sometimes it can annoy the hell out of you, and that is extremely frustrating. You wake up in the morning, come to work, have plans of what you’re going to do, and then. All hell breaks loose. It can be exciting, and it’s funny when you remember it after the fact, especially if you fixed it, but in that moment, it really is not. Let’s face it, in all honesty, in this line of work, boring is good! Boring means things are running well, users are happy, nothing is breaking, and you can work on your projects/tasks as planned or develop new solutions (the kind of excitement you want), finding ways to make things better or run faster, etc. Things like that are the baseline standard.
Now, another thing about working in IT(and perhaps any other discipline), especially working in a team with people whom you’ve gotten to know well, you get a sense of their competencies their abilities what they can and cannot do, what they are capable and incapable of; Most importantly, there is the tendency to give people TOO MUCH CREDIT! The kind of Credit that can make a very simple, obvious issue like this, with a very, very, very straightforward solution, pass you by and cause you a very exciting morning.
In this blog, I will share a story about a recent experience I had diagnosing (yes, diagnosing) an ORA-01017 “invalid username/password: logon denied” error.
Please read that line again!
THE DETAIL
databases
I am a DBA and manage Oracle databases. One of the things we have to take care of is Oracle networking, configuring files like the tnsnames.Ora, listener.Ora, sqlnet.ora, ldap.ora, and maybe cman.ora on the database server and on the client (web servers or PCs, etc.). These files (in whichever combination you use) must be set up and working properly for client machines to establish database connections. If they have been working well for years (including the day before), why would you go over verifying that they all work well when an ORA-01017 error occurs?
Anyhoo, without giving too much detail, there is one database that most custom apps are written on, and another that houses commercial-off-the-shelf (COTS) software. My team (Application development + DBas) was tasked with extending the functionality of this COTs software, so for the first time, we had to write an app that connects not to the popular custom app database but to the COTs one. Hopefully,y you get the point about verifying the network setup, but still… too much Credit!
The application development
As a DBA, I am responsible for setting up the app environment: tables, users, roles, privileges, synonyms, etc. We had gone through meetings to review the app’s design and how it would work, and in this case, a database link was required because the COTs system needed to interact with data from the main custom app dev database! During the development and testing phase of the ap,p everything had gone smoothly, no ORA-01017 error yet on the day when it was supposed to go live in production (mind you, this functionally was being supported previously through some other means so it needed to work and on this “go live day” the previous ability was taken away) it started happening. What in the world could we have missed??
Too much damn Credit!
Password Encryption
The devs on my team use a mechanism to encrypt the database passwords for their applications. You give them the database password, and they “copy it” and paste it into this encryption tool. It spits out an encrypted password, which is what gets passed back to the database. It works! (but surely you can see what can go wrong!)
The analysis.
Is the password really wrong?
Immediately after I got this error, I attempted to connect to that database as that user using the database password, and… guys, it worked! So what on earth could be wrong??? The password is not wrong, so surely the reason the developer (and damn near everyone in the building 😔) is getting this error is that… (say it with me now). Still, nobody ever thought to arrive at that conclusion.
Like I said, TOO MUCH CREDIT!
Surely it must be the network files!
For some reason, my mind went (and it’s fairly valid, or I might just be straight up dumb!) We always only use the custom app dev database for applications. The client files on the custom app webserver are configured only to talk to that database! Have we missed setting it up so it can talk to the COTs database??? Maybe it’s trying to log in to the popular database we use, passing in credentials that don’t exist, which is why there’s an error??? So we connect with the sysadmins, review everything, attempt connections from that server using the SQLPlus client, and… everything works perfectly. So surely at this point we know what’s wrong, I mean it’s pretty obvious, right? right?
Wrong!
At this point everyone is going berserk, the manager is actually handling it pretty well not even messaging us at all and we spend; I don’t know the best part of an hour looking at all different sorts of things (I have lost track at this point everything went so fast and it was such a blur) trying to find out what could be happening. Still, the message popping up on everyone’s screen when they try to launch this new application is (drum roll please):
ORA-01017 invalid username/password: logon denied error!
THE RESOLUTION
At some point, something hits me, you know, one of those “Eureka” moments, and I just say to the developer.
Can you copy the password I sent you again, re-run it through your encryption thing and re-launch the application?
Lo and behold: The app launched! Wow!!! An absolute miracle! Who on earth would’ve thunk it???
It’s always these damn mice, I tell ya.
Where can I start? I was speechless; everyone was happy (more like relieved), and I just sat there at my desk. I was relieved as well but by all means not happy I was almost mad I didn’t know how I felt because it was right there. The error message clearly said, “Hey, the login info I’m getting doesn’t work.” Tell me, dear reader, have you never made a mistake/missed the last character when copying with a mouse? Do things like that not happen? Do you get immune to errors after 500 years of being a programmer?
We ain’t nothin but human.
The developer who wrote this custom app, I know and I know well. They are darn good at what they do, and they often save the day. There’s nothing wrong with them; it’s everything wrong with me because I couldn’t fathom that they would make a mistake copying a password. Perhaps, if they were a new hire or an intern, I would have felt more comfortable asking right at the beginning, “Are you sure you copied the password correctly?”
Discrimination based on talent and experience! guilty as charged! But hey, it’s like the character Roland Pryzbylewski says in The Wire, “How do you even know if that kind of thing is in your head or not?”
When I thought about this day after the fact, I remembered that there is absolutely no relationship between a person’s knowledge and/or experience and their ability to make mistakes. Mistakes are the most human thing ever. I gave this developer too much Credit, telling myself, “Surely they couldn’t have copied a password wrong,” when, in fact, that is a natural error anyone can make!
What did we learn, kids?
This blog was not necessarily about the invalid username/password error itself or its details, but rather about an experience with it. The main takeaway from this story is to check oneself and fall back on the basics. Sometimes we can be good at doing sophisticated things while being very negligent about basic errors.
We are all but human. Falling back to the basics of the job, concentrating on what you’re doing (instead of checking your cell phone), knowing your job well, and just maybe taking a problem at face value, can save you a very eventful and exciting morning. Cover your bases.
Remember, boredom in this line of work means good. If you want excitement, join the Baltimore “poh-leece” department. I guarantee your life will be a movie!
Cheers 🎊