Episode: 1331
Title: HPR1331: A Tale of Chroot
Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr1331/hpr1331.mp3
Transcribed: 2025-10-17 23:39:41

---

Number Five
Gü
U
N
U
N
U
N
U
N
U
N
U
Hello, this is NY Bill, and I thought I'd talk to everyone today about TRUTE and recent
adventure, or well, it was almost a disaster, but TRUTE came through in the end and saved
my backside.
I'll go into that in a minute.
For those who don't know, TRUTE is a Linux command to be able to mount another system
and then TRUTE, change root, drop down to that system, and then the changes you make
are to that remote system.
You're no longer making changes on the system you're booted into.
So this is quite handy, let's say someone lost their password, which I've done this in
the past.
You can take a live CD, boot into the live CD, mount the drive that's in the system,
TRUTE to it, and then you are root on that hard drive.
You can just run the password command, set up a new password, then just back out, boot
up the system, and you can log in.
This is another one of those scary physical accesses, security type things.
If someone can get a hold of your system, they can do whatever they want.
So where I recently used TRUTE was, I almost killed my whole entire VPS, which is, I mean,
it's not much, but it's my little presence on the interwebs, it's my status net instance,
my media goblin instance, and well, there is a web page, but there's nothing on it really.
However, I had some false starts with status net, and it's kind of a, I have the hang of
it now, but it was, sometimes I would just do something and mess up the database, and
I'm not very good with going into databases and fixing things, so sometimes it was one
of those situations where it was easier to start over.
However, I'm glad people were patient with me, because every time you restart, everyone
has to resub you, and you have to resub everyone, and I think if anybody listens to like
previous HPRs I've done, at the end, I'm usually going, oh, my status net, you are changed.
These really issues.
So recently, I had it all nailed down.
I knew exactly how I wanted to set up status net.
I had my media goblin going fine, my instance, and I realized, I don't want to change this
status net again, so I paid linoid, and my service, my VPS is on linoid, so I paid them
an extra $5 a month for backup services, which are quite handy.
You get a day back up, a week back up, and a two week back up, and then a custom snapshot.
So seeming that I had status net up and running, I had my whole VPS exactly how I wanted it,
I did the custom snapshot, and I figured, okay, I can always come back to this if everything
goes wrong, which we're going to get up to in a minute, and I thought I was safe.
So here is how things went wrong.
My VPS is an arch system, and the arch recently did an upgrade or update where they're moving
everything from slash bin, slash S bin up to user bin, trying to like unify where all
your bin files are.
So this was a three step process.
You wanted to Pac-Man, that's why you dash, dash, ignore file system, and bash.
This would get the whole thing prepared and ready to go.
Then you do Pac-Man capital S bash, which is going to put in Simlinx, and get everything
pointing to this new location where all your bin files are.
And then there's just Pac-Man SYU, I believe, was the third one, just, oh, that one probably
brought in the file system.
I'm not really sure, but it was a three step process.
It's critical that you do all three steps.
I didn't.
So this was a couple of lugs back, and my friend Aesphere came into the lug, and he goes,
we both run arch, and he has an archlin node as well.
And he said to me, did you do this archup update?
And I said, yeah, I did it on my laptop, but I haven't done it on a little node yet,
because all the services are running, and I don't want to bring it down.
And at the lug, I was SSH tunneling through my linoid, so I didn't want to bring it down.
I said, I'll tell you what, you can be the guinea pig.
You go first.
And if it works for you, I'll do the upgrade as well.
So he did the upgrade, you know, I'm chatting with everybody, and then he like hits me
in the elbow, and he goes, it just, it's all came back up.
Everything's fine.
I go, all right, all right, so I'll do it too.
So I did step one, which is Pac-Manus, where you dash, dash, ignore file system, ignore
bash.
Let the thing grind away there, and the lug is quite a distracting place, so I always tell
people not to get into anything too deep.
One time, when to go, was into his MySQL database, and I'm like, is this really the place
for that?
This is what happened to me.
Like somebody came in, and I got up to say hello to him, or somebody started talking
at the table, and I walked down and got into that conversation.
For whatever reason, I didn't do the update, step two, and step three.
Totally slipped my mind, and I sat back down in my chair and checked media goblin.
It was still running, and status net, it was still running, and I figured I completed
the update.
Now, three weeks goes by, and in the past, Linode just for free, doubled everyone's RAM.
So that was great.
You click a button, you get a queue, when you're time, when you get to the number one slot,
they did the upgrade, you could watch it all happen, and boom, they doubled my RAM.
That was awesome.
This 20 or so days after the lug, Linode said, guess what?
For free, we're going to double everyone's storage space.
So again, I was like, oh, wow, it's just free double storage space.
I keep putting more pictures on media goblin, I'm thinking to torrent a few ISOs of distros
that I like, so why not free space?
So I clicked the button to upgrade, and I watch it, I watch the progress bar go by, and
it says upgrade complete, and then I've doubled your storage space.
However, to activate this storage space, you need to reboot your Linode.
So I think the uptime there was like 98 days on my Linode, and they're going to take
my uptime away, but anyways, all right, so reboot, nothing.
And I reboot it again, and nothing.
So there I thought, well, I got my backups, so I'll just go use the backups, and well,
some of you are probably realizing as I slowly did that it was 20 days ago that I partially
upgraded.
So my one day backup had this error, my one week backup had this error, my two week backup
had this error, and then the snapshot I took was from right after I got status net and
media goblin up and running, so it would have no pictures in it.
It would have no, yeah, I was in a pinch.
The other pressure was that if anybody recalls, like a few HPRs back, it was John Culp and
I where we met up, and we mentioned that if you want to see pictures from our meetup,
you can go see them, and they were on my media goblin instance.
So if I didn't get the current media goblin instance up, and I went back to this really
old backup, none of those links, no, no links to a media goblin would work.
So I think I'm getting a little bit ahead of myself, though, because when the system
didn't reboot, when the system didn't boot back up, I didn't know why it didn't.
I didn't realize it, I didn't even think about this update and not completing the update.
So I was just at a loss, I couldn't see any errors, I couldn't check any logs, I couldn't,
I didn't know how to fix it.
I had no feedback, I didn't know what was wrong, and I figured that Leno's upgrade broke
it.
So this was the first time I submitted a ticket to Leno and I said, look, I think your
storage update just hosed my instance, you know, where can we go from here?
So here's one thing about Leno, they have awesome documentation, and it turns out the
support is awesome as well, because I submit this ticket and I think within like 12 minutes,
some guy got back to me and said, I checked out your instance, here is the error lock.
So that's good support right there, not calling up some data center in India or something
or I don't know, but I was surprised at that and I was, because I thought I ruined my
whole VPS, it was nice to get something, some, some like, well, a shred of hope that,
okay, maybe moving in the right direction to get this VPS restored.
So what he suggested to me was that I go use the recovery tools.
So on the Leno dashboard, if you will, what you use to control your instance, there is
a recovery tab, and in that recovery tab, it gives you information about where, what
instances you have stored on what drives, you can keep multiple instances, let's say,
for instance, you could have an arch distro, and then another instance is a mint distro,
and then as many as you want until you fill up your allotted space.
So there was a couple of things I needed to do here, well, I read up on Leno first,
but, or, uh, Lish, Lish is the Leno shell, and what it basically does is you click on
this button in recovery, and it opens a GUI window, and it's bait that you put in your,
you know, your master password and all that, and you're basically, I guess you're SSHing
into the front end of their whole entire network, and then you can access your system and
your drives, whatever is not running to try and fix them.
I guess, basically, it's, it's analogous to starting up a live CD, and then you can look
at a drive that's on a system, and so here's where truth comes in.
So I was able to, in the recovery tab, take the instance that was broken and make a copy
of it.
So I wanted to work off a copy, and then I was able to see where that copy was on the
virtual file system.
So it turns out that this one was dev XVDF.
So now I know which virtual drive I want to true into to try and repair.
So the first thing you need to do is mount the root disk you're going to work on with
execute permissions.
So that's mount-o, exc, barrier equals zero, slash dev, and in my case, slash XVDF.
Then I could change to slash media, XVDF, and then mount the individual parts of that,
of the file system I needed, such as proc, insist, dev, and you're probably going to have
to look up what you need to mount for your own particular OS, because different versions
of Linux put different things in different spots, and I'm sure a quick Google search would
show you exactly which parts of the system you want to mount.
Once you've done that, you can trute that media XVDF, or whatever you want to call it,
folder.
So the command would be like, trute, backslash media, backslash, XVDF, slash bin bash.
And that will send you down into the hard drive, if you will, that you're trying to repair.
Okay, I didn't have bin, so now I got a whole other, like, it just this whole weekend
just was like compound problems.
So what I did for like a quick nasty hack, I did have everything that was in bin, but
it was just up in user S bin now.
So I was able to create, you know, backslash bin and backslash S bin and put simlinks in
there myself pointing up to user bin and then try all my mounts and my trute again and
boom.
Okay, finally, I got in.
I thought at this point, okay, I've truted a bunch of times, I know all I got to do is
finish the upgrade.
So I just did the Pac-Man minus S bash, this should complete the upgrade process and
get me rolling again.
It said user bin is on the system, okay, this is just an arch thing and now it's seeing
that I've put slash bin slash I've been the first part of the upgrade was going to remove
these.
So now I had to remove them again, but at least now I'm truted in and I finished the upgrade
and I go back to boot up my instance and it's still dead.
Try it again, still dead.
At this point, I think, always lost, but this was one of those things that, like, right
before you're falling asleep, boom, you remember the answer to a question, this is another
thing about truting.
If you're going to trute into a 32-bit system, you need to use a 32-bit live CD, same thing
for a 64-bit system, you need a 64-bit live CD.
You need the environments the same because you're using packages from the first system,
from the live CD to affect the second system.
So when I would go in there, it turns out, so what it turns out is that Lysh, Linoid's
Lysh is 64-bit and my arch is 32-bit and when I trute into the 32-bit and do an upgrade,
it wants to pull in 64-bit libraries and everything 64-bit, then you end up with a mixed
match of 32-bit and 64-bit stuff and, well, you can see how that all goes wrong.
So now I realize what was going wrong trying to repair through Lysh, but how can I, I can't
drive over to Jersey and sit in front of Linoid's racks and boot up with a 32-bit live
CD.
So, thinking this over, I realize this might take two trute sessions.
So here was my new plan of attack.
I used Linoid's Lysh to get in, create bin, create Espin, create the SimLinks to user Espin,
so this will allow the system to boot up.
I went to the dashboard and booted the system up, then SSH in via, this is getting complicated,
but this is the only way I could think to do it.
Now I used Lysh to go in once again to the running system and bring up networking, and
then I was able to use my 32-bit laptop to SSH into the running system, network was working,
I was finally able to trute from a 32-bit environment to a 32-bit environment.
Now it was just a matter of finishing that upgrade, well, first the complaint that bin
was still there and Espin was still there, which should have been taken care of in part
one of the upgrades.
So I removed those, ran the second part of the arch update, and finally boom, I rebooted,
and everything came back up, but I was able to turn on status net and media goblin, and
the web page was working, and it was a big sigh of relief and a lesson learned, quite
a few lessons there.
For instance, as soon as you get everything back up and running, why don't you take that
snapshot back up again, so you have something to go back to that's not ancient.
So that was my tale of trute, and hopefully it'll get people thinking about trute and
thinking that when they have a system that looks so host, that it's easier to start over,
you still have, it's like a surgeons tool, you can still get in there and repair that
drives.
And if nothing else, maybe you found my tale of well entertaining, it was okay to look
at an hindsight, but that was a rough weekend, so thankfully my contact information is still
the same.
I'm not a whole new VPS rebuilt thing, I'm still NY Bill at gunmonkinet.net, and status
net is SN dot gunmonkinet.net, so thanks for listening, and I'll talk to you guys later.
You have been listening to Hacker Public Radio, or Hacker Public Radio does art.
We are a community podcast network that releases shows every weekday Monday through Friday.
Today's show, like all our shows, was contributed by a HPR listener like yourself.
If you ever considered recording a podcast, then visit our website to find out how easy
it really is.
Hacker Public Radio was founded by the Digital Dark Pound and the Infonomicom Computer
Club.
HPR is funded by the binary revolution at binref.com, all binref projects are proudly sponsored
by linear pages.
From shared hosting to custom private clouds, go to lunarpages.com for all your hosting
needs.
Unless otherwise stasis, today's show is released under a creative commons, attribution, share
a line, free those own license.