Speaker 1: Welcome to the deep dive. You know that feeling, right? Speaker 2: Mhm. Staring at just mountains of information. Could be logs, code, maybe reports, research, and you just need that one specific thing, that key piece of insight, like right now. Speaker 1: Yeah. It's like being a digital detective sometimes, sifting through clues, and you feel like the clock's ticking. Speaker 2: Exactly. The problem isn't a lack of data. It's getting the right knowledge out of it quickly and, you know, effectively. So, today, we're basically equipping you with a digital toolkit. Think magnifying glass, tweezers, the whole setup. Speaker 2: We're diving deep into some really powerful Linux command line tools. These are the unsung heroes that can turn that raw data into real intelligence. We've pulled together some practical uh real world stuff from Adriana Holden Goa's great YouTube tutorials and web resources really focused on hands-on use and that's really the mission here for you. This isn't just, you know, a list of commands. It's about understanding how these tools actually empower you, how you can use them to like extract those important nuggets of knowledge to gain insight quickly but also thoroughly. It's about turning that information chaos into well clarity giving you control. Speaker 2: Okay, let's jump straight in then. The heart of digital sleuththing, you could say GP. Speaker 1: Ah, Grep. Yeah, for anyone who spends time on the command line, GP is just fundamental, isn't it? It's way more than just a simple text search. It's more like a precision instrument for finding patterns, anomalies, even in huge data sets. Speaker 2: That's a good way to put it. Its core job is basically to scan through information and tell you only about the bits that match exactly what you're looking for. And what's really fascinating here, I think, and maybe a bit underestimated sometimes, is why Grep is so much more powerful than just hitting strail plus F in a text editor. Speaker 1: Good point. Speaker 2: I mean, imagine you're debugging something complex. A system update's gone wrong, causing weird failures. You can't possibly scan thousands of lines of logs manually, right? Speaker 1: No way. Speaker 2: Grip lets you just slice through all that noise. pinpoint specific error codes, maybe user IDs, timestamps across loads of files, super fast. It directly tackles that need forgetting knowledge quickly by making your search incredibly targeted, finding the signal and the static, you know. Speaker 1: Absolutely. Let's take a basic example, but uh maybe raise the stakes a bit for just a datebook. Let's say we're looking at potentially compromised system logs. Speaker 1: Okay, Speaker 2: we suspect an intruder hit specific files. We've got a file access logs.txt to find all entries from a known bad P like say 192.168.11100. We just run grep 192.168.11100 tro access logs.txt. Speaker 1: Simple clear. Speaker 2: Yeah. Now a lot of you probably know you could also do cat accesslogs.txt. GP 192.168.1100000 Speaker 1: right the pipe method. Speaker 2: And for simple cases yeah same result. But it's worth remembering GP is actually optimized to read files directly. So for really really big files skipping the cat might be a bit more efficient. That's a subtle but important point for performance. Yeah. Speaker 1: Yeah. Speaker 2: And thinking bigger, maybe system configuration, you could GP allow users etc as config to quickly see who's allowed SSH access or check if uh like a guest account got left active and etc password. So it's not just finding stuff, it's verifying things, making sure the system state is what you expect. Speaker 1: Okay. So GP gets really interesting when you add options and especially uh regular expressions. Speaker 2: Mhm. Rex, for instance, maybe you're hunting for unusual activity. You might want to see what doesn't match a normal pattern. The FV option does that. Inverts the match. Speaker 1: Right. Speaker 2: So, GP-V normaluse activity system.log would show you everything but the routine stuff. Could highlight weird anomalies. Speaker 1: Useful. Speaker 2: Or if you're looking for an application name that might be capitalized inconsistently in logs. I for ignore case is your friend. Grep-b error.log saves you running multiple searches. Speaker 1: Yeah, case sensitivity trips people up all the time. Speaker 2: Okay, but here's where Grep goes. from just search to like true pattern identification, regular expressions, reax. Speaker 1: This is where the real power unlocks. Speaker 2: Many of you know the basics, but let's see how they give deeper insights in our access logs.txt. Maybe we want lines where an IP address starts with 192 followed by, you know, the rest of the numbers and dots, right? Speaker 1: You could use something like gp1 192.091909191 the access logs.txt that anchors it to the start of the line and the shapes the dot. So it means a literal dot, not any character. Speaker 2: And that's where that critical thinking Adriana Holden Ga talks about really comes in. Reax isn't just finding text. It's about defining the structure of what you want. Speaker 1: Exactly. Speaker 2: It lets you build super specific patterns like say you suspect a brute force attack, you could look for lines in off log saying failed password, then any username, then an IP. Something like Py1 in a failed password for Z9 plus from 09113 1013. Speaker 1: Okay. And the E enables the extended region. Yeah, makes some patterns easier to write. But the point is that reveals not just an error, but a specific type of attack pattern. A simple word search would totally miss that context. That's a real aha moment in security analysis. Speaker 2: And what takes GP to well another level entirely is using it with piping. Chaining commands together. Speaker 1: Feed the output of one into the next. Speaker 2: So sticking with the security example, we found all those potential brute force lines with our fancy reject script. Now maybe we only care about a attempts that happen around say 2:00 p.m. So 14 something. Speaker 1: Okay, narrowing it down. Speaker 2: We'd run that first grap, then pipe its output straight into another grap like grap-e failed password off. Grap 14.0-9.0 found 0-99. Speaker 1: See, no file name on the second grap. Speaker 2: Exactly. That's the crucial bit Adriana always flags. The pipe sends the results of the first command to the second one. If you put off.log again on the second GP, it would just search the whole original file again. and ignoring your first filter. You got to avoid that. Speaker 1: Yeah, that mistake happens. But getting it right, this sequential filtering, it's incredibly powerful, like surgical data extraction. Speaker 1: So this whole idea of chaining commands, how does that really make you a more well efficient analyst? Well, it's about building a sophisticated little information workflow right there on the command line on the fly. You're taking raw, often messy data and refining it step by step into targeted insights, building an evidence trail. It's fundamental to being proficient, letting you process complex info without needing to, you know, write a whole script for every single question you have. Speaker 2: Okay, so Grep helps us find the needle in the hay stack. We've filtered, we've found those critical insights, but what then? Once you've found that smoking gun, how do you actually capture it, save it, organize it for later? Speaker 1: Yeah, good question. Because just seeing it scroll past isn't always enough. This is where redirect and append come in. Vital concepts, right? They let you take that command output, the stuff that normally just appears on your screen and send it into a file instead. Permanently store it. It's about taking control, making a temporary finding into a lasting record. Speaker 2: Exactly. And the difference between the two symbols is really, really important to get right. The single greater than sign, that's redirect. If you do say ls-l directory listing.txt, it puts that detailed file list into the text file. But, and this is the key thing, if directory listing.txt already exists, poof, gone. Yeah, it's old content is completely wiped out, replaced by the new output. It's like starting fresh every time, Speaker 1: which is sometimes what you want, but not always. Speaker 2: Right now, for append, you use two greater than signs. Speaker 2: So, if you run maybe tail- 20 error.log daily errors.txt, it takes the last 20 lines from error.log and adds them to the very end of dailys.txt, Speaker 1: preserving whatever was already in there. Speaker 2: Exactly. Super useful for building up logs over time or creating reports piece by piece. Speaker 1: And the practical uses are well everywhere once you think about it. Automating system checks, maybe a cron job runs dsh dis spaceh history.log every day, builds up a history. Speaker 2: Nice example. Speaker 1: Or after you run a really complex grip that finds critical events, you immediately do critical events. Right after it on the command line, saves those findings instantly. It's how you go from just seeing something to actually documenting it. Speaker 2: Okay, so we've covered finding and capturing. Let's quickly touch on commands for actually viewing and uh comparing file content. There are different tools for different perspectives like Changing lenses on that digital magnifying glass. Speaker 1: Indeed, for just displaying files, several options. CAT, as most people know, just dumps the entire file content to the screen. Speaker 2: Good for small config files, maybe. Speaker 1: Yeah, quick checks. Then you have head. Shows you the top lines of a file. Default is 10. Great for checking headers or the start of logs. Speaker 2: And tail does the opposite. Speaker 1: Exactly. Shows the end lines. Again, default 10. But the killer feature for tail is tail-f. Speaker 2: Ah, the follow option, Speaker 1: right? Tail F var log SL log or whatever log file. It keeps the file open and just shows you new lines as they get added in real time like a live feed of what's happening. Essential for monitoring active systems. Speaker 2: Definitely use that one a lot. Speaker 1: Then there's odd octal dump. Now this might seem a bit niche. Speaker 2: A bit geeky Speaker 1: maybe a little but it's a real detective tool sometimes. Odd displays file content in uh hexadimal or other formats. If you suspect a file transfer got corrupted or a script is messing up because of invisible characters Speaker 2: like weird white space or something. Speaker 1: Exactly. Running odd see that problem file can actually show you non-printable characters, maybe encoding issues, bite order marks, stuff that a normal text editor just hides. It's for when you need to see the absolute raw data to figure out why something's broken. Speaker 2: That makes sense. Okay. Useful for those tricky cases. Speaker 1: Now, speaking of debugging, what if you need to see the difference between two versions of a file? Maybe a config file before and after a change. Speaker 2: Ah, yeah, that's diff. Many people know it from version control like git, right? Speaker 1: Git def uses it heavily. Speaker 2: Using diff directly. Well, its default output can look a bit intense. Speaker 1: That's fair. Adriiana mentioned it's often designed to be read by computer, not human. It can look pretty cryptic at first, Speaker 2: like a set of instructions for changing one file into the other. Speaker 1: Exactly. It's a compact lineby-line comparison showing additions, deletions, changes, but its real power is that programmatic detail. Imagine you just patched a server config. Before you push it everywhere, you run def originalconfig.com needconfig.com. Speaker 2: See exactly what changed Speaker 1: instantly. Every single modification for developers, code reviews rely on it. For CIS admins, it's amazing for diagnostics. Did these two config files somehow drift apart? Has someone tampered with the script? Diffu, the unified format makes it a bit friendlier by showing context lines around the changes. Speaker 2: Okay, that helps. Speaker 1: But fundamentally, its strength is highlighting even tiny alter. operations, spotting potential issues or just understanding what actually happened. Speaker 2: Right? So, we found data with GREP, saved it with redirects, viewed it with cat, head tail, debugged with odd, compared with BIFF. What about packaging things up like a whole project or lots of logs, maybe a directory structure for backups or moving stuff? Speaker 1: Okay. Yeah. Time to talk about tar, the tape archive tool. Let's uh bundle this up. Sorry, couldn't resist. Speaker 2: Huh. Well, yeah, the tape part is definitely historical now, but the idea is totally current. Tar is designed to bundle lots of files, whole directories into one single archive file. We often call it a tarball. Speaker 1: Tarbally Speaker 2: makes it super easy to move entire projects, complex data sets, backups. Makes them easily portable and manageable. It's really essential for making sure you grab everything you need when you're moving things or archiving. Speaker 1: So the basic command to create one is tar-cf, then the name for your tarball, and then what you want to put in it. Speaker 2: tar-c Tarball nametar source file sore directory. Yep. Let's break down those options. CVF. Adriana really flags these as important. Speaker 1: They are. C is for create. Pretty straightforward. You're making a new archive. Speaker 2: V is for verbose. Speaker 1: Yeah, highly recommended. It lists every file as it gets added. Good visual feedback that it's actually doing what you expect. Speaker 2: And F means file. Tells tar the next thing you type is the file name for the archive. Speaker 1: And here's that gotcha you mentioned. Speaker 2: Ah, yeah. The crucial detail. The F option has to be the last one in that block of options. Speaker 1: Absolutely. tar-c works. tar-c.nope. Speaker 2: You'll get some weird error message like files in the archive not dumped exiting with failures. It's cryptic if you don't know why. Speaker 2: Yeah, it's bitten probably everyone at some point. Yeah, Speaker 1: just remember F comes last when creating CVF or CF are common. Speaker 2: Good tip. And the arguments after the options, Speaker 1: right, first is the destination. That's the path and name for your new tarball. Could be my backup.tar in the current directory or mnt backups project data. date plus percent percent free dotar for something fancier using tar is the standard convention Speaker 2: makes sense and the source Speaker 1: that's what you actually want to archive super flexible you can use dot for everything in this current folder or a specific directory name like my project or wild cards likelog to grab all log files or maybe config files.com lots of options Speaker 2: and often you want to compress these tar balls right make them smaller Speaker 1: definitely especially for backups or transfers the most common way is using gzip you just add the z option to the command Speaker 2: so tar - CVZF. Speaker 1: Exactly. tar-cf myarch archive.tar.gz my stuff. And notice the file name change. Speaker 2: Ah, right. Tar.gz or sometimes people use tggz. Speaker 1: Yeah.gz is just a shortand fortarz. Good practice to use the right extension so you know it's compressed. Super useful for big log directories or code bases. Speaker 2: Makes sense. Speaker 1: And just briefly while we're focused on creating C tar obviously extracts archives too usually with the NAS option part actvf archive.tar. But the core idea is management. Tar helps you organize, package, and move collections of digital stuff reliably. Helps ensure you know we don't ever forget anything when dealing with groups of files. Speaker 2: Wow. Okay, that was quite a tour. We really covered some ground. Started with GP, taking it from basic search to a precision tool for patterns using rejects, Speaker 2: then redirect and append to control where that output goes. So important, Speaker 1: vital. Speaker 2: Look to viewing files with cat, head, tail, especially tail-flight obscure but useful for debugging weird characters. Speaker 1: Right. And different comparator. Speaker 2: Yeah. For spotting those critical changes. And finally, tar for bundling everything up with compression using Z. Speaker 1: Exactly. And the key takeaway, I think, is that these aren't just separate commands you learn in isolation. They're building blocks. Powerful interoperable building blocks. Speaker 2: Work together. Speaker 1: Yeah. Like we always say, knowledge is most valuable when you understand it and can apply it. Mastering these tools really lets you cut through all that information noise. Turn raw data into actual insights. They give you agility and precision in how you handle information. It's uh it's pretty gamechanging, honestly. Speaker 2: So, here's something to maybe chew on as we wrap up. Think about a tricky situation in your own work. Elusive bug, maybe a security incident review, a complex deployment. How could you chain these tools together? Speaker 2: Use GP with some clever rejects. Pipe that output, maybe redirect it to build a report on the fly, diff that against what you expected, and then tar up all the evidence. What hidden truths could you uncover in your data if you start building these kinds of command line workflows? Maybe the real deep dive starts right there. Crafting your own bespoke toolkit.
Excerpts from the transcript of the video "FAQ Grep: re-GREP-able, ohhh sooo re-GREP-able..." uploaded on the YouTube channel "Adrianna Holden-Gouveia"
Speaker 2: hi everyone today i would like to talk to you a little bit about grab grep is a search tool that we'll use in linux that has the ability to search through a set of information and then only tell us the information that it was able to find that matches what we're looking for so if we for example know that we are looking for a specific file or type of file somewhere but we're not exactly sure what the entire name is or if we're looking for a particular user on a system we would be able to use grep to search through and find that information and then have it get back to us so what you can see on the command line right now is i have the file called datebook and if we look at the file called datebook we can see that i have names phone numbers addresses states zip codes birthdays and then a number at the end now these are obviously all fake information there is no sir lancelot or zippy pinhead outside of the comic that i am aware of um so this is just some information for us to be able to practice on so if i wanted to look at the entirety of the file i can of course just cap the file however i probably don't want to do that especially when the files get bigger the file that i'm using is a sample file that is attached to the lab if you're one of my students and if not this is from the unix shells by example booklet which was published probably almost 20 years ago now for the last edition so you can usually find it online which i don't recommend you do because i think it's outside of copyright so anyway whatever you do don't google unix shells by example to look for this so with grep we actually have a couple ways that we can do this so i could for example grep for um something in my file so let's say i want to look at my file and i want to look for boston so i can grab for boston in my file and you can see i have my one result let me clear that so that you can see it a little better now the second way that i could do this is i could actually cat the file pipe it to grep and then search for boston and this will give me the same result this is not exactly the same thing it's the difference between standard in and searching the file but for your purposes right now for beginner stuff they're the same and you can basically have them function the same so these are two ways that we can use rep now for this example for this video i'm going to be doing the first way so i'm going to have grep and i'm going to pass it what i'm looking for and then the file that i'm looking for it in so i can do things like grep for boston in my file i can also do things where i can use options now if you look at the grep manual or the linux pocket guide or other resources you can see that grep does have other options so i could do something like look for graph for people that aren't in boston oops sorry about that and don't forget your file so you can see all of the results of people that aren't in boston now other things that i can do is grab can actually take regex as well as options so for grep i can look for you know for example not boston i can also look for ignoring case so if i have a lowercase boston something like that but i can also use regular expressions or regex so i could for example look for all files that have all lines that start with z in my datebook file and i can see that zippy pinhead is going to show up because zippy is the only one that has the start of line starting with a capital z now i could also look for things at the end of the line instead of the beginning of the line so i could for example graph for five seven zero zero zero at the end of the line of date book and you can see i was able to find that now grep does have a lot more possibilities than just this these are just a couple of examples there's going to be a lot more options there's a lot more regular expressions that you can do um you can even do things with like egrep and f grab which if you're using one of the newer new linuxes is probably rolled into grep but you can actually look into that if you're not sure um and so that you can start trying to work on how grep is able to search for things for you now one of the last things that i want to remind you of is it seems like a lot of people are forgetting about piping one of the things that makes the command line really nice is we can actually search for things and then pipe it to other commands so let's say for example i'm looking at my book and i actually wanted um but you can see that you can actually take your results and pipe them over to another command so that you have the ability to search through your results even further if you need to be able to do that so don't forget that you can combine commands pipe information to other commands and find your way around using graph so hopefully some of these basics will help you as you're learning how grep works and how to put it together if you feel like you're really struggling with the regular expression portion of grep don't forget that you can always refresh on your regex or regular expressions there are several websites out there that are really good for that and you can start putting together some more complex statements to be able to find what you're wanting i hope this was helpful for you learning grap if you have any questions please let me know
Excerpts from the transcript of the video "FAQ Tarball" uploaded on the YouTube channel "Adrianna Holden-Gouveia"
Speaker 2: hi everyone today i would like to talk to you about tarballs and how to create them in linux so when we talk about tarballs what we are talking about is a way of putting a bunch of information files and folders together so that we can work with it easily now this can give us the ability to do things like create backups that we can work with another thing that tar can actually be used for is you can create compressed files as well so if you wanted to save a bunch of let's say your homework and you wanted to compress it tar would actually give you the ability to do that now the way that tar is going to end up working is we use the command tar and then we have to give it some options so the options that i'm going to use today are cvf now the way that these options work is c means create v so let's create the tarball v means verbose so everything that's going into the tarball i want to make sure i can see it on the screen so that i know everything that's going in there and then f is file now an important note here is whatever set of options or switches that you would like to give tar you have to make sure f goes last i'll show you what happens if it doesn't in a moment now the next thing that we have with tar is we have to have two arguments we have a source and we have a destination now this is going to potentially look a little different to you depending on what you're used to but the order is actually destination and then source so in this case i'm going to have my destination be my current folder so what we remember from paths is current folder is dot slash and then i'm going to name it test tarball.tar now you can name it anything you want you don't technically have to say tar but i think it's a good idea because that way we can tell at a glance what this is going to look like and what kind of file this is the same way that i recommend doing things like dot sh or dot sed um things like that give us the ability to see at a glance what type of file this is is it a script file like python.py is it a shellscript file so dot sh is it a tarball dot tar and putting this in there gives us the ability to see that quickly now the path that we are using is dot slash so that's in the current folder but we could just as easily change it to do something like put it in the home folder and you can work with the paths there and then name it whatever you would like now after that we're going to have our source now in this case my source is just going to be everything in my current folder however i could make this a little bit more specific i could do things like i want everything in my current folder that's labeled a text file or i want everything in my current folder that's a shell script file or i want everything in my current folder that contains the word linux things like that um so you can work with wild cards and things to decide what you want to include in here for the purposes of this i'm actually just going to do everything so that you can see what'll happen now when i hit enter you can see everything that was in my folder so you can see my files and folders ended up getting in there now one of the things that you might see if you aren't careful about the switches or options if you for example switch this around and did f v instead of v f so remember c is create v is verbose that's how this information is showing up and then f is file so if i switch the order however you can see that this isn't necessarily going to work the same way now if i switch the order and let's say called it test2.tarball you can see i'm going to get an error it's going to say files in the archive not dumped exiting with failures due to previous errors so if i am creating a new tarball like this i have to make sure that i'm saying f as the last option so this f has to go at the end here and when you're creating make sure you have the c the v is the optional part i like to include it so i can see what's going in there but you don't have to so you can see what a big difference that can end up making now if we end up looking at our files we can see that we were able to create test tarball.tar and we can now work with it so you can also use tar to untar our files as in break them apart again and that's just a different option and if you want to look up the manual on how to use tar you can do that but this is the basic format of how you would end up creating a tarball i hope this was helpful for you today and i hope you're having a lovely day
Excerpts from the transcript of the video "TLDR CIS 117 Week 10" uploaded on the YouTube channel "Adrianna Holden-Gouveia"
Speaker 2: hi everyone and welcome to tldr week 10. the objectives this week is getting to know grep starting to work with redirect and append and explain the difference and also be able to use head tail and diff grep is one of the ways that we can filter text we would use grep so that we can look for specific pieces of text you can use it alone or part of a chain of commands some examples of grep so let's say for example we wanted to look at the etsy password file that contains all of the users on the server we could cat the etsy password file and that would give us all of the users however perhaps we would only like to use look for one of the users in this example jane so what we would do is cat etsy password and then pipe over to grep so that grep can search for jane pipe gives us the ability to send the results of one command to a second command so the pipe is actually above the enter key on an american style keyboard it looks like a little line and it very much functions like a pipe the way that you think about it so we can actually use that to pipe it over to grep so that grep can then do what it wants to now you might also see some examples where you would pass grep multiple arguments that's fine as well where you could say something like grep jane etsy password type of thing and then you would be able to just have grep use multiple arguments instead both formats are completely fine grep has a number of options or switches that you can use if you would like to go and try some of those out from either your pocket guide or the manual pages and you can see some of the other things that you can use grep for such as matches doesn't match case insensitive things like that redirects and appends are used so that instead of the results showing up on the screen what we can actually do is have the results get sent to a file now there is a difference between redirect and append a redirect is going to be the single greater than sign and that means all of the information that would normally be sent to the screen is instead going to be sent to a text file but it's going to erase the file and put the new information in there append which is two of the greater than signs one right after the other that's actually going to take the information and instead of showing it on the screen it's going to put it at the bottom of the file so if i did for example ls then that's going to show me all of the files and folders in my current folder if i did ls redirect to test file that would instead of showing those to me take all of those results and put that list in my text test file if i did ls append test file that would take all of that list of results and put it in the test file but it wouldn't erase the test file you should try both and see what happens what that looks like and make sure that you get familiar with that some things we can do with files is we can actually display them in different ways we can do things like use cat to display the entirety of a file we could also use head or tail if we only want to look at the top or bottom pieces of a file you can try this on the etsy password file on the server or any long files that you have on your computer if you're interested you can also try od which will show the file showing the hex code that can be kind of interesting to try out as well you also want to start playing a little bit with file comparisons so try using diff to compare a couple of files now this is not exactly designed to be human readable diff is usually integrated into other things for example git if you're going to be doing any coding it's important to know the difference so if you have made changes to your code you want to use diff so that you can see what those changes are you might also end up using diff if you're going to look through things like differences of settings or the difference in config files if you want to figure out what has changed the last one is tar tar is actually an archive format and an archive means that we're going to take a number of files and folders and put them all together so that we can move them around sort of as one lump we end up doing this so that we can move them around work with them and copy them very easily make sure that we don't ever forget anything the way that we're going to go about using tar is you actually run the command tar and then the options or switches cv and f now technically you do not have to use these switches c means create v means verbose and f means file you have to use c and f but you don't have to use v i choose to use v most of the time so that i know exactly what's going in my archive but you don't have to the first argument that will get passed to tar is going to be the name of the tarball and where you want it located in my example that you see on the screen here i am going to be calling my tarball name you can see i've named it name.tar and i want this to go live in my home directory so i can actually pass it that path as part of the argument so when i say in my home create a file called name that is going to be my tarball that's my first argument my second argument is actually the thing that i want to tar so in this case home user directory is the directory that i am looking to tar and that's going to be my second argument that i will pass it you can also play with gzip if you would like to do a compression instead of an archive add in a z and change the extension and that will give you the ability to do a compression instead the lab for the week is going to be you starting to play with grep i am having you play with grep on the etsy password file on the server because that has enough entries that it makes sense you're also going to play a little bit with head and tail and redirection and append if you have any questions please let me know
Please note that sources consist of text content (e.g., table of contents, learning outcomes, command examples) and therefore do not have spoken dialogue.