Episode: 3423 Title: HPR3423: "upg.sh" my "dump.txt" to "note.md" Source: https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr3423/hpr3423.mp3 Transcribed: 2025-10-24 23:09:23 --- This is Hacker Public Radio Episode 34234 Wednesday, 15 September 2021. Today's show is entitled Up.sh, MyDump.txt, ToNote.md. It is hosted by some guy on the internet and is about 38 minutes long and carries a clean flag. The summary is, I upgraded my scripts. This episode of HPR is brought to you by archive.org. Support universal access to all knowledge by heading over to archive.org forward slash Donate. Hello and welcome to another episode of Hacker Public Radio. I'm your host, some guy on the internet. Today's show title is called Upgrade MyDumpToNote. Now that's a play on words, those are actually three different scripts. I created the Upgrade script and the dump script. And they were very simple, small scripts and I turned the dump one into something called Note. So I just made a show name out of those scripts to have like a little play on words. So if you guys would remember, the Upgrade script was very short, simple and it did the job. It also just, you know, stored the Upgrade in plain text in a text file. And that was pretty good. But after listening to, I believe it was Dave Morris and Andrew McMillan, they did the making books on Linux show, shout out to Dave Morris and Andrew McMillan. By the way, that was a great episode. There was just one problem I had with that episode. I'm a complete new when it comes to programming languages. So a lot of that went over my head. I did a lot of research after listening to that show to try and, you know, figure out things because they were saying things like compiling a book using make, both the first show and the second show. They were, you know, going into details about the process. And some other wonderful things. Great show, by the way, is just that you got a low level user like myself. I need to level up to better understand these shows. But they're still great. Keep them coming. I mean, I have no problem when I get free time to just, you know, bring myself up the power. Also real quick. Another reason why I did this as well is I heard a Clat 2 show as well. That's why I actually created the note script separately. But Clat 2 did a show about using XML. And it was another thing in there called Doc Books or something like that. Wonderful show. Lots of good information. I didn't even know that these technologies existed. So, you know, it's introducing me to new software and new technology. And, you know, I got another way to make documentation if I wanted to use that. So that's fantastic. But, you know, I just wanted to say that I understand what he was doing at the end of the show, sort of giving people a bouncing off point to create another show. And I'm kind of doing that with this show as well. So I'm going to be using a lot of markdown here, improving Clat 2 wrong, that markdown is fantastic. And it's not just for people who don't want to write. It's, you got to think about it. A lot of us new guys here. We need tools as well. So something like markdown is simple and easy for us to be able to contribute. I know the rest of you guys can use Python and everything else and be just fine, but we need the tools more appropriate for our skill level. And we can still do nice things with them. I'm going to demonstrate in this show. So let's get right into it. First up, we got our upgrade script. Now, what you'll see here in the show notes is I'll have the full upgrade script just pasted right into the show notes there. And you can copy that, put it into a file, name the file upgrade.sh, you know, give it execution permission and, you know, you just change my granted execution and run it. And it'll be good to go. Now, obviously that's not all we're going to talk about here. I'm going to go through a breakdown of this script. For most of you guys, I got a feeling that you guys can look at this and instantly know what's going on and just kind of tell very easily what's happening here. But for the new users, let me explain as best as I can. And it took me a long time to get these, this show out, because I had to do a lot of research on how to use all this crap. I had never used variables or anything like that. So I had to learn a lot to be able to bring this script together. And then at the last minute, I kept trying to get something squeezed in at the back end. There was a script there. And I finally got it working, but not the way that I wanted. But we'll get into that a little bit later. Right now, let's go ahead and do the breakdown from the top. So from the top of the script, you'll see your shabang that's the pound symbol or hashtag what they call it today. Follow by the exclamation mark that's called a shabang. Then you tell the script where the shell is that you're going to be running it with and that's the slash band slash bash. Now that's specifically for the back shell. If you're using ZSH or a fish or whatever else, you know, if you're already using that, you know what to do. For new guys, you're probably already using bash. If you're using an Ubuntu based system, so you're good to go with that. Now I included some information in there about switching it to shabang slash band slash SH for Shell from the research that I found that's like the default unique shell or whatever. So it's more portable, but I just keep it as bash because it works. A lot of my research came from going online, finding different things over tons of sites, as well as the books that I have. I have tons of bash shell scripting books. And you know, I'm just going through them, just finding information and using it to create this. One of the things I learned through one of these resources, rather it was online or in the book, you can just, you know, store the name of your script in your script using the comment. And that was helpful for me because when I was creating more, you know, I was bouncing in between scripts, trying to make things work and figuring out, how did I do that one thing in the other script? And then just referencing one and the other, and I could just see the name of the script inside of it. I was using them to do all of this. I have gotten really good at using them. By the way, oh my god, I love it. It's, uh, it's really good. I had some issues with the copy and paste from the clipboard, but that's that we'll get into that later. Now following the name of the script, then we got our variable section, where I just put a couple of variables in there. And I like them right at the top, because if you ever want to change the location of where the script is going to be stored, or the file name of the script, it's nice to have your variables right at the top. Now, when I was making the script originally, before I started going through the books and everything, I used to just type out the actual path and file name right out into the script. And then I learned, well, you know, hey, if I just did it in a variable, I can put the variables everywhere, and you can still change the values of them from the very top of the script, where the variables are defined. And that's a much better way of doing it. It cuts down on the syntax, makes it a little bit more readable. Let's go ahead and jump down into the script here, step one, where I have the formatting of the actual markdown file, where the output of the upgrade will be placed. Now, now there's a lot of words there. Basically, we have echo, paste different words into the script, using a markdown formatting. So you see that very first echo right there? It'll have the hashtag, followed by the system upgrade, surrounded by asterisk. You know, that's going to give us that header with bold text, and it's followed by date. That's just to let you know the date that you actually performed the upgrade. One of the things I used to do when I was in my earlier scripts, sorry, if you can hear the police ironing background or fire truck, whatever that is, but one of the things I used to do is I used to have these long lines of commands going all the way out. And what I read was, it's a good idea not to go past, you know, in the books it said line 76, but on line it said line 80, you know, different places have a column, not line, excuse me, a column where your script should not go past. And I settled on 80. So in them, I learned to set up in my MRC, a nice little column margin that gives me visual representation of column 80 so that I can try to limit my text from going beyond that. Also, I learned not to die on that hill. And you know, they were, that's part of the thing that took the most time because I didn't even know how to do that. I had to search around for examples on how do you actually break code, you know, how you just stop in the middle and make it go on the next line and keep working so that that was new to me. I had no idea how that works. And I get with bash is not exactly code, especially not the way I'm using it. But I mean, you know, still it's the same thing. I had to learn it. And what I learned was the backslash at the end of a command would, if you did a space at the end of the command and put a backslash there, you can then continue on the next line with the code. One of the other things I learned, this is from a book, I believe. One of those either was a bash cookbook or a shell scripting or something. One of these books I've got, they were saying to make sure you keep the important bits to the left. So when I first did it, I had like the pipe at the very end of the line. And then T would continue on the next line or whatever. I was like, no, it's actually a good idea. I actually put the pipe down there too. So you can see the redirect from the left versus try to figure out why you get what I'm saying. So I also took the time to add in some notes there on the right of these commands down here and a step-by-step breakdown to try and help new users who want to look at this and get a better visual representation. I try not to repeat it too much where it'll just become just a blast of just text on the right. But in the beginning, I labeled everything and then as we go further down, there will be less and less labels of the things that are already covered. Let's go ahead and move right on down in a step two. Now step two of our upgrade command is where we actually run our commands and then we add more formatting in where our commands are ran, but they're placed into marked down code block formats. So that way, you know, you separate the explanations, you know, the labels that are placed in the pile, you separate that from the actual standard output of the command when they're run. Now the commands we're talking about here are just pseudo-app, get update, and pseudo-app, get upgrade with the yes option added to upgrade. Now again, for the new users, you're looking to write over there and de-step-by-step breakdowns. That stuff on the right there with the little arrow and the two dashes followed by that text. That's not actual code over there. That's just a little bit of visual representation. And I figured in the actual script itself, one of the things I read is you don't want to put too many comments in there to where it's just a dot on the comments in your code. That's why these show notes, I'm treating the show notes like documentation. You'll have the script separate with just some basic, you know, shallow comments to help you out. But the show notes are very valuable because they're going to give you just a little bit more to go off of. All right, now let's move on into step three. This is the part that took me the longest because I had to actually go do a ton more research to get this working properly. So first off, we get a label going to let you know that the packages that are going to be upgraded, we're going to provide you some details of those packages. Now in the previous show, I discussed to you guys about how to find out which packages on your system needed to be upgraded, as well as how to, you know, sort of reveal that in different ways. I thought if I'm creating an upgrade script, it'd be a good idea just to kind of provide you some more of that in the script itself using the markdown format. So let's go ahead and discuss how I've done it. So as you can see there, we got a couple of echoes there for a label and the markdown formatting. Then let's move into the first variable there, the pkg list. That's where we used said to filter from the file that we're working on. We had to filter that file looking for a specific string and we used said to do that. Now we found the string that we wanted and then we had to find a second position in the file to stop the filtering. So a beginning point and an end point. That's what that said script that said command does inside of pkg list. After we found our beginning and end point, we wanted to keep all the information in between. So points one and two with all lines in between, we pulled that out of the file. We like printed a copy of that and we stored that copy inside of the variable pkg list. Now I got a thing I wanted, but I didn't want the first and last line, which were used to sort of govern the filter. So that's where we had to pipe it into the second said script, which is the one there was said, 1D, a semicolon, $ind, what that script does is the 1D is to delete the first line in that variable. So you're treating the variable like a file as though it's just a text file. You're deleting the first line, which is the part that says the following packages will be upgraded. The dollar sign D is to delete the last line, which is the dot dot upgraded. Now let me talk to you about that dot dot. That in bash represents any character. And I use only two of them because I upgrade regularly. If you're one of the people that wait a long time and upgrade and say you might have a hundred or a thousand packages that need to be upgraded, you want to add a couple extra dots there. Again, because I update regularly, it's never any more than about 30 or 40 packages. That's why I only use two dots there. And I put the carrot that upward arrow at the beginning of the dots, because I wanted the said script the first one that filters the beginning and end point. I wanted it to start at the beginning of that line, looking for the two digits followed by the word upgraded. So now immediately after that second use of said, which deletes the first with the one D and the last line with the dollar sign D, we piped it into exorges, which is going to then take that all those lines, pump them through another use of said, which is going to filter out a few duplicate packages. Each line in that file represents a package. That's why it's called package list. Now when you get those packages, some of them will be duplicated except they'll have the tag at the end of them was a colon I386. So it'll be like Firefox and then Firefox colon I386 is the same thing, but I guess coded for I386 or whatever. So I wanted to just remove those duplicate packages. So once all those uses of said and exorges are done, then you got the package list variable, which holds just a single list of packages within it. In the beginning, when I first made this, I dumped that into a file. I later learned that I can actually just store and use them from a variable. So I didn't have to constantly keep creating all these different files to get the value and use the value the way that I wanted to. Next variable, pkg, which is package cache. Now that's going to be the cache information. So what we did is we echoed the list that we just created using all those said scripts we discussed, we echoed the package list with a new line in between each item of the list into exorges one at a time. That's what the, when you use exorges with that hyphen N and then the one after it, it's going to take each item in that list and pump it individually into, and you see we have app cache search. So we pumped them one at a time into app cache search. Now in exorges, if you use the hyphen capital I, it'll give you sort of like a temporary variable. And you'll see here I'm using the underscore. I'm taking that underscore, storing those items that are being pumped through one at a time, each of those packages from the package list variable, putting it into app cache search. And then the search is being done for each and every package. Now one of the things you'll notice whenever you're doing an app cache search, it's going to pull up other related packages to the one you search. I didn't want all those additional packages that appeared. I only wanted the ones that are relevant to this upgrade, not the other related packages that are available into cache. Now, underneath this variable, I had to create a file of the package cache variable. So you see I echo the variable into a file. It's a text file called delete, because I don't want to keep it around, so I actually name it delete. We'll talk more about that in just a little bit. Let's go ahead and jump right down here. You see I'm using the package list variable once more. I'm putting it into exorgs. And remember, exorgs using the end that gives me the ability to pipe an item from that list one at a time using the end option. Then with the capital I, that gives me that temporary variable, that I'm going to put into echo to create multiple said commands. So if you're looking side that echo string there, you'll see I have in double quotation marks, I have the said using the end option, which basically means it's a print said script to begin with or said command. That's what the P is at the end of that said command for print. Now the end option with said is basically, you know, only print the things selected by said, the things that are going to be selected by said. If you look at the temporary variable there, the underscore, which is given by exorgs. And remember, exorgs is getting its information from the package list. So just an overview here, we're taking each item from the package list and making a said command out of it, a print said command. Now, whenever you're using said, you need a file to use said with. And I tried dozens of times to try to work with the variable package cache. And I could not get it to work, not with exorgs. So I had no choice. I had to create that file, the delete.txt file. And I stored the app cache search results, which is all the different results, including the packages that are going to be upgraded. Now, remember, we don't want all those different things from the cache. We only want the packages. That's why we use the package list to create a said print filter, which will print only the package list items from that massive list of items from the cache. That's why at the end of the said print command, you'll see in another set of double quotation marks, the directory variable followed by the actual file, which is the delete.txt file that we recently created with echo using the package cache variable. Now, we pipe all of that, all those said commands. Through that file, we run it into bash that way we can activate the said commands. Otherwise, it would just print said commands. So we want to actually activate them, run them as though they were in the command line being ran in bash. So we piped it into bash, and then right after we piped it in bash, bash, runs them, we piped that result into the markdown file. Yeah, that was a lot to say there. Then we just wrap it up with the rest of those markdown brackets there, and then go ahead and we use rm with the verbose option. That's the hyphen v verbose, and we remove the delete.txt file. Now, I added verbose there just so if you want to run it, you can see that it's actually going to do that. So we remove the delete.txt file, because we don't actually need the file. We just wanted our package list variable, not the variable. We just wanted our packages that are being upgraded, pulled from that list and displayed to us in a way that we wanted. That's all we wanted there. So we can delete that file. It's no longer of any use to us. If you don't want to use an rm in the script, you can actually comment that line out that rm hyphen v directory delete.txt. You can comment that out by putting a hashtag at the beginning of it, or just delete that line altogether. Following that, we can see where I have the variables, pkg list, package list, and pkg cache, which is package cache. Those two variables that we recently used, I empty the variables. I don't fully understand why I did that. I just did it just to empty all the value that was in them out. So I think that I did it correctly. It would just be blank variables now. I could have unset them as well. But as again, I don't want to go all that far. I just empty them. Now last but not least, at the bottom of the script, I read somewhere that putting eof or control d like the up arrow d, or the carrot d, whatever it's called, at the end of a file, tells whatever command that's running the file that is the end of the file. Like if you just put eof, capital eof, or whatever, certain commands that read for that will note to stop when it runs into eof. And I just thought it'd be a good idea to put that there. It's not necessary here, because it's a markdown file that we're creating. But I'll just go ahead and talk you through what we're doing here as well. We're using said to first look for an eof, which stands for end of a file. And if said finds an end of a file, it will delete it. Then we use echo immediately after that to place an end of a file. And the reason I did it in that order is because while testing the script, I was dumping multiple up system upgrades into the same file over and over again. And what would happen is there be tons of eof's at the end of each instance of the script running. So what I wanted to do is remove all the old ones and actually place one at the very bottom of the script. That's why this bit of, I don't know if you call it code, but whatever is in there. All right, let's go ahead and jump down here into the note.sh, which is another script that I wrote up. Remember I had the dump files, where I created that dump script or whatever to create the dump files, or no, everyone in the script, I think I just redirected the output into dump files. So I created a script that would just basically give you the exact same thing, but with markdown. So let me just go ahead and talk you through that really quickly here. This is a quick fast and a hurry one. We're not gonna be long here. So again, as you've seen in the previous script, we got our variables right at the top. You know about the Shabang with bash. So we got our variables. It's very familiar. I use the same file names. The only thing here is you'll see in the file name variable, I have that dollar sign two. That dollar sign two from my understanding is a shell built in variable, and it will accept user input. And I had to put the two there for a reason. Now in the script name, I have an example of how the script is to be ran. So I run the script by first typing the note dot SH, which will let it know that this is the note script that I'm running followed by a pair of double quotes. Now inside of that first pair of double quotes, I put the command that I'm actually running. And then I put a space after that and a second pair of double quotes where I then put the file name for this markdown file that's about to be created. Now you don't put an extension on it. You know the dot MD extension for markdown. No extension because that's handled by the script. Just the file name what you wanted to be called. So let's go ahead and run through this real quickly and I'll talk you through how that's all being done. So we already talked about the number two there and the file name that shell variable. That number two corresponds to the second bit of input that you're gonna be putting where I just talked about the note dot SH. So the first part and the first pair of double quotes would be the command later on you're gonna see a dollar sign one. So the part where the file name goes, that's where the dollar sign two is. That's where dollar sign two corresponds to. So now you'll see the variable, which has the dollar sign two plus the date dot MD. That's gonna be your file name, whatever you type in that second set of quotations. Now we got our directory which is well, you know, about that. Let's scroll down here, dry mouth, and get a drink real quick. So if you look down here you'll see the echo command with the number one in there. That's gonna correspond to that first set of quotation marks the command that you're running. So it's basically gonna give you a label with the word command using the markdown format and the markdown header number one, using that hashtag right there. Then it's gonna give you a nice little breakdown label and then it's gonna go into that next echo, which has the number one, the dollar sign one there. This is where the command you typed in gets broken down and formatted into, I guess sections that you can edit and provide details for. I use this to create show notes whenever I'm exploring commands for like say the apps, plonking episodes that I've been doing as I'm going through app and exploring packages and finding things that are cool and unique. I can just go ahead and make notes of them and have it automatically, you know, spelled out in markdown making my show notes a little bit easier to create with uniformed formatting. Yeah, I ran through a number of rabbit holes trying to get this done. And when Clad 2 said what he said is like yeah, I'm gonna definitely make sure I get something going to definitely talk to him about that. But yeah, you see the echo with the dollar sign one that you're command. Then we use TR to look for any spaces in the command so let's just use an LS command just for a second. Use LS, then you put a space and then the options you wish to run with LS. So LS, space, and then you do a hyphen L for long and put a H there for human readable, right? So what that TR is gonna do is find that space in between the LS and make it a new line. Then you see it gets piped into arc where we do a bit more formatting and we set up a nice little mark down section format that takes those different sections, spell them out with the word info after them. And that you just, when you wanna go back and edit up your show notes or whatever for these different packages that you're exploring, you can just delete the word info and type in what you wanna put there. I just put something there so that I can have a marker to know where I'm gonna be putting my information. Now step two of this command is super simple. We're basically just creating more mark down formatting the code blocks or whatever. And then we're taking the output of that command and storing it inside of those mark down code blocks. So you'll see down in there again, you see that echo with the dollar sign number one being piped into bash, meaning the command you just ran with note script will get piped into bash, ran in between that formatting so that it auto formats has marked down and it gives you a date in there as well. So that way when you go back through there to see, you know, when you last created this note, it will have a date there for you. Just a bit of a convenience and section three as you remember end of five. Oh man, there's nothing else after this point. Now I created that because when I'm doing notes, especially say for instance, I'm doing HPR episode eight. I've got several notes created for that show. I can store every single one of those notes in the same file called HPR episode eight and that end of file will get moved to the bottom of the file with every single note that gets added. So I do an LS command first followed by whatever else command like a X args command or something like that. Now you need to be careful here as well. Some commands, I think I give an example here with ARC. When you're using ARC, there's a lot of spaces in ARC and remember that TR commands are gonna take those spaces and turn them into new lines. So you'll see and some of the notes to the right of the command in step one. You'll see there a given example of ARC with the print command for ARC. Yeah, there's got a lot of spaces in it and each one of those spaces will turn those sections into a new element that requires information in the markdown. Now of course, you can go and just manually clean it up. So only commands like that will become something interesting. Outside of that, most commands will work out just fine with this and it's a very simple script, easy to read. I learned a lot in making it, working with variables and I wanted to do more with F statements and things of that nature, but they weren't necessary for such a simple script. I'll practice F statements a little bit later. When I'm done with this and I got my handle on bash because this is the end of the scripts here. When I'm done learning more about bash and getting very fluent with this learning, how to use F statements in case and the other things like that. I'm going to move into Python because I'm going to figure out how they made these books using Python and with ARL, I need to do. Because it sounded very good, but I just don't know how to use it. Plus Python will be good for other things as well. So I'm hoping to eventually learn that language and become proficient with it. I hope these show notes work out and I hope that you guys leave some comments down below telling Class 2 who's wrong, Markdown's awesome, especially for us new guys, we need it, love it. And even if you're not new, Markdown's still great. It's simple, it's not meant to replace anything. Just a wonderful format for doing very simple tasks and when used like this with scripts, you can just make your everyday upgrade just a little bit more pretty. You know what I'm saying? Get a nice Markdown layout of your output and it's just nice to have. Now, if you guys got some tips on different resources out there for learning more about bash scripting or whatever, definitely leave a comment or do a show responding. That'd be great for not only myself, but other new users getting into it. Definitely recommend this show and the notes as a nice resource. What I'm going to also do is try to link a Markdown file from my Google Drive. I figure since all my notes and everything is going to be CCBYSA, you know, Creative Commons, it'll be fine to put in a Google Drive and then just throw a link in there where people can download the actual Markdown files and just do whatever they want with it. I'll try to get that in there, hopefully it works because I don't know if you can get Markdown from hacker public radio, right? Like it'll just be the page, not the actual Markdown itself. I don't know, but I mean, either way, I'll try to put it in there and I'll put a link somewhere in here where hopefully you can get that if I've done it right. That's it for this show, ladies and gentlemen. Oh, yeah, one last thing. Mr. Fallon, I still owe you that other show and I am working on that as well. I bumped into Mr. Fallon over on, was it mastered on? I'm some guy on the internet, so I'm usually walking around on the internet just bumping into people. And I bumped into Mr. Fallon at Mastodon and we had some words about a thing. He may not remember because my name on Mastodon is not the same as hacker public radio, but we had a little conversation and if there are any Naruto fans out here, Mr. Fallon has the Fallon gone from, you remember Naruto had the sharring gone, the OGF family. Yeah, he's got the Fallon gone. When I bumped into him, he hit me with the Fallon gone and now I owe him a show. So when he hypnotized me, I gotta make a show for him now. And he sent me into like the HPR Sukuyome and Dave Morris was there too. And Dave Morris was there in the background saying, make sure you release your show every other week. And Ken was there constantly going, do a show. All right, guys, I'm some guy on the internet. I'll catch you guys in the next episode. Take it easy. You've been listening to Hacker Public Radio at HackerPublicRadio.org. We are a community podcast network that releases shows every weekday, Monday through Friday. Today's show, like all our shows, was contributed by an HPR listener like yourself. If you ever thought of recording a podcast and click on our contributing to find out how easy it really is. Hacker Public Radio was founded by the digital dog pound and the infonomicum computer club and is part of the binary revolution at binrev.com. If you have comments on today's show, please email the host directly. Leave a comment on the website or record a follow-up episode yourself. Unless otherwise stated, today's show is released under Creative Commons, Attribution, share a live 3.0 license.