Inspecting Introspection

Have you ever had a time where introspection was “running” and you were thinking “I really wish I could login to that machine and see what is going on….”?

We have all been there and we have all speculated if there was a default password in there or SOMETHING that would give us access, however, from the looks of things, there is not. So, what if we wanted to set the root password? Let’s take a quick look at what happens when a node is introspected…

Click to enlarge

So it pulls the inspector.ipxe which is located in /httpboot on the Undercloud node. The contents of this file look like this:

Click to enlarge

We can see here that inspector.ipxe has the node download the agent.kernel and agent.ramdisk which corresponds to what we see on the console above. These files are also located in /httpboot on the Undercloud node. It passes some additional parameters to the kernel which essentially kick off the inspection. At the end of inspection, the node powers down.

Knowing this, we know we can extract the ramdisk, make whatever changes we want to it, rebuild it and put it back in place for the introspection. Attached to this blog post is a script that does exactly that. The password you want to set for the root user can either be put in the script in the CLEARTEXT variable or you can pass it as an argument to the script. After you run the script, you will have a modified agent.ramdisk that has the following changes:

  • Root password is set to whatever you want
  • SELinux is disabled in the image SSHD changes:
    • Disabled the use of DNS (UseDNS no)
    • Root login is permitted (PermitRootLogin yes)
    • Password entry is enabled (ChallengeResponseAuthentication yes)

So I have tested this script a couple of times by killing off the openstack-ironic-inspector.service once I saw the the collector start to run on the console and I only ran into one issue: I was getting a connection refused when trying to SSH to the machine after introspection started, however I was able to console into the VM and tried restarting SSHD. I received a message saying something about the ownership of /var/empty/sshd needed to be owned by root and not world writable. I checked and that file was owned by UID 1000. I did a chown on it and was able to restart sshd and SSH into the machine.

Once you’re in, you can take a look at /var/log/messages and maybe get a better idea on what is going on since that is where openstack-ironic-python-agent.service writes all of it’s messages.

Hope this helps some people.

Back to Top