BACKGROUND
Working in IT is one of those things, you love it but sometimes it can annoy the hell out of you and this is extremely frustrating. You wake up in the morning, come to work, have plans of what you’re going to do and then… All hell breaks loose. It can be exciting and it’s funny when you remember it after the fact; especially if you fixed it, but in that moment it really is not. Let’s face it, in all honesty in this line of work boring is good! Boring means things are running well, users are happy, nothing is breaking and you can work on your projects/tasks as planned or developing new solutions (the kind of excitement you want) finding ways to make things better or run faster etc. Things like that are the baseline standard.
Now another thing about working in IT(and perhaps any other discipline), especially working in a team with people whom you’ve gotten to know well you kind of get a sense of their competencies their abilities what they can and cannot do, what they are capable and incapable of ; Most importantly, there is the tendency to give people TOO MUCH CREDIT! The kind of credit that can make a very simple and obvious issue like this with a very very very straightforward solution pass you by and cause you a very exciting morning.
In this blog I will explain a story of a recent experience I had diagnosing(yes diagnosing) an ORA-01017 invalid username/password: logon denied error.
Please read that line again!
THE DETAIL
databases
I am a dba and manage oracle databases. One of the things we have to take care of is oracle networking; configuring files like the tnsnames.ora, listener.ora, sqlnet.ora, ldap.ora and maybe cman.ora on the database server and client (web servers or PC etc). These files (whichever combination of them you use) generally have to be set up and working properly for client machines to establish database connections. If they have been working well for years (including the day before) why would you go over verifying these all work well upon an ORA-01017 error!
Anyhoo without giving too much detail ; there is one database which most all custom apps ae written on and another that houses a commercial off the shelf (COTS) software. My team(Application development + DBas) was tasked with extending the functionality of this COTs software so for the first time we have had to write an app that will connect not to the popular custom app database but to the COTs one. Hopefully you get the point of verifying the setup of the network files, but still… too much credit!
The application develoment
As a DBA I am responsible for setting up the app environment; tables, users, roles, privileges, synonyms, etc. We had gone through meetings to review the design of the app and how it would work and in this particular case a database link was required as this extended functionality to the COTs system needed to interact with some data from the main custom app dev database! During the development and testing phase of the app everything had gone smoothly, no ORA-01017 error yet on the day when it was supposed to go live in production (mind you this functionally was being supported previously through some other means so it was quite important that it works and on this “go live day” the previous ability was taken away) it started happening. What in the world could we have missed??
Too much damn Credit!
Password Encryption
The devs in my team use some mechanism to encrypt the database passwords used for their applications so you give them the database password and they “copy it” and paste into this encryption tool and it spits out an encrypted password that is what gets passed back to the database. It works! (but surely you can see what can go wrong!)
THE ANALYSIS
is the password really wrong?
Immediately I got this error I attempted to connect to that database as that user using the database password and… guys it worked! So what on earth could be wrong??? the password is not wrong so surely the reason the developer (and damn near everyone in the building 😔) is getting this error is because… (say it with me now). Still nobody ever thought to arrive at that conclusion.
Like I said TOO MUCH CREDIT!
Surely It must be the network files!
For some reason, my mind went (and it’s fairly valid or I might just be straight up dumb!) We always only use the custom app dev database for applications. The client files on the custom app webserver are only set up to talk to that database! Have we missed setting it up so it can talk to the COTs database??? Maybe it’s trying to log on to the popular database we use, passing in credentials that do not exist and hence the error??? So we connect with the sysadmins review everything attempt connections from that server using the sqlplus client and… everything works perfectly. So surely at this point we know what’s wrong, I mean it’s pretty obvious right! right?
Wrong!
At this point everyone is going berserk, the manager is actually handling it pretty well not even messaging us at all and we spend; I don’t know the best part of an hour looking at all different sorts of things (I have lost track at this point everything went so fast and it was such a blur) trying to find out what could be happening. Still the message popping up on everyone’s screen when they try to launch this new application is (drum roll please):
ORA-01017 invalid username/password: logon denied error!
THE RESOLUTION
At some point, something just hits me you know one of those “Eureka” moments and I just say to the developer
Can you copy the password I sent you again, re-run it through your encryption thing and re-launch the application?
Lo and behold : The app launched! Wow!!! An absolute miracle! Who on earth would’ve thunk it???
It’s always these damn mice I tell ya
Where can I start I was speechless everyone was happy (more like relieved) and I just sat there at my desk. I was relieved as well but by all means not happy I was almost mad I didn’t know how I felt because it was right there. The error message clearly told us “hey the login info I’m getting don’t work”. Tell me dear reader, have you never made a mistake/missed the last character when copying with a mouse? Do things like that not happen? Do you get immune to errors after 500 years of being a programmer?
We ain’t nothin but human
The developer who wrote this custom app I know and I know well. They are darn good at what they do, they often save the day. It’s nothing wrong with them, it’s everything wrong with me because I could not fathom that they would make a mistake copying a password. perhaps if they were a new hire or an intern I would have felt more comfortable asking right at the beginning “Are you sure you copied the password correctly?”.
Discrimination based on talent and experience! guilty as charged! But hey, it’s like the character Roland Pryzbylewski says in the wire “How do you even know if that kind of thing is in your head or not?”.
When I thought about this day after the fact I remembered that there is absolutely no relationship between a person’s knowledge and/or experience and they’re ability to make mistakes. Mistakes are the most human thing ever. I gave too much credit to this developer as I told myself “surely they couldn’t have copied a password wrong” when actually in fact that is a natural error that anyone can commit!
What did we learn Kids?
This blog was not necessarily about the invalid username/password error itself and the details of it, but rather an experience with the error. The main takeaway from this story is just being able to check one’s self and falling back to the basics. Sometimes we can be good at doing sophisticated things and be very negligent of basic errors.
We are all but human. Falling back to the basics of the job, Concentrating on what you’re doing (instead of checking your cell phone), knowing your job well, and just maybe take a problem at face value, can save you a very eventful and exciting morning. Cover your bases.
Remember, Boring in this line of work means good. If you want excitement, join the Baltimore “poh-leece” department. I guarantee your life will be a movie !
Cheers 🎊