FreeIPA with AD integration inside an AWS VPC took what was going to be
a 60-minute downtime on a production server and let me solve the issue
in about 60 seconds ...
Story below:
One of the great things about moving a company to AWS is that I can
usually delegate far more administrative powers to the end-users and
developers than they are used to having on-premise or coming out of a
managed services environment.
The downside to this power is that sometimes people accidentally do
inconvenient things like break /etc/sudoers file on an EC2 node
And since the usual best practice for secure AWS operations is:
- SSH with keys only (no passwords
- Sudo rules that allow passwordless power
But all of this hinges upon a sudo system that works.
If your sudoers file is broken then your AWS admin key is useless
because you can login but you can't use sudo to elevate yourself. Total
catch-22 problem.
Getting admin access locked out on an AWS EC2 server is really nasty.
Not all of us can run immutable infrastructure and not every server can
be built from start to finish using automation. The server in this case
was a massive beast with dozens of disk drives and LVM volumes in use.
Nightmare to rebuild even with good backups and automation.
The fix for this is operationally nasty and time consuming:
- Shut down server
- Detach root EBS volume
- Start new server in same subnet
- Mount the broken root disk onto the other server
- Manually fix the things that are broken
- Detach EBS disk and move back to original server
- Start original server and hope you fixed all the broken things
This can take 60 min or more if your automation is not prepared for
things like this.
This is where FreeIPA with RBAC rules saved the day.
I was able to SSH into the server using my corporate AD username and
password
And thanks to a good RBAC rule that lets me run sudo on all AWS hosts I
was able to "sudo su" and enter my AD password to gain root . The OS
interaction between PAM and IPA does not depend on /etc/sudoers being
syntactically correct!
... and this was the pivot I needed to fix the broken /etc/sudoers file
and allow regular admin ops to occur
Just wanted to shout out IPA for this. It totally took a 60min
production issue and turned it into a 2 minute issue
-Chris
Show replies by date