User Tools

Site Tools


journal:fall2016:nbrimme1:start

nbrimme1's fall2016 Journal

UNIX/Linux Fundamentals

CSCS 1730-A01/A71


Introduction

In this space you can provide a description of yourself, your pursuits, your interests. To fill out this section, click the edit button off to the right, delete this placement text, and type your original content.

I forgot to edit this part, and I can't think of a good description presently… so here's a cute bunny:

 ()()
 (°°)
(")(")

unix week14


MONTH Day, YEAR

Filler text- your entry goes here; remove this line to receive full credit.

unix week13


MONTH Day, YEAR

Filler text- your entry goes here; remove this line to receive full credit.

unix break2


November 21-25

During break, I worked on various parts of the EoCE. I've finished the puzzle box and I also started the two Palindrome bash scripts.

unix week12


Monday, November 14

Today was a general work day to work on anything needing completion. Project: UNIX DATA RECOVERY (udr2)'s due date was graciously changed again to Friday. There are only 5 weeks of class left with Project: UNIX DATA RECOVERY (udr2) being the last project, besides the EoCE: End of Course Experience.

Wednesday, November 16

Today was another work day. I spent the class and lab time pulling out different packet types for Project: UNIX DATA RECOVERY (udr2) using binhaxx and dd. I also started and completed part of the EoCE: End of Course Experience.

Friday, November 18

Today was another work day. I worked on and finished Project: UNIX DATA RECOVERY (udr2).

unix week11


Monday, November 7

Today we went over the binhaxx suite of tools that was provided by Matt. Since the data files for Project: UNIX DATA RECOVERY (udr2) are just files stuffed with packets, with no fixed offsets, binhaxx will help with some of the information required to complete Project: UNIX DATA RECOVERY (udr2). All the data blocks inside of the data files has a variable size, so searching for any pertinant data can be difficult. Any important data; like the start of a packet, the total time of each packet, and the datatype associated with each packet, is made much easier with binhaxx. Some math is required to convert the timestamps into hours:minutes:seconds. The time for each packet is stored in a 3-byte value; with 1 bit containing the lower 8 bits of the packets (Epoch) time, with the second and third byte values containing the 16-bit sub-second (LSB first) of the packets (Epoch) time. The first byte can then be used to search for and extract any timestamps from the raw data.

I spent the lab time working on Project: UNIX DATA RECOVERY (udr2), pulling out even more specific data from the packets I had already extracted.

Wednesday, November 9

Today we went over some basic usage of hexedit, a command-line hex editor. hexedit displays a file in both ASCII and Hexadecimal, and also allows editing. In hexedit; the first field contains the offsets or address of the current line. The second field contains 16 columns of 2 symbols, or 16 Hexadecimal bytes (00 through FF). This means that each line of a file has a byte offset of 16, so each line will have an offset that is 16 bytes higher than the previous line. The lines are also displayed at the bottom of the window in a special format: [offset:byte]. The third and final field displays an ASCII representation of the Hexadecimal displayed on each line. Any data that cannot be read is simply displayed as a dot (.). Another useful utility, called strings can also be used to output any ASCII data in a file. The due date for Project: UNIX DATA RECOVERY (udr2) was also extended to next Wednesday.

I spent the lab time trying to finish Project: UNIX DATA RECOVERY (udr2). I extracted more information from the data files using binhaxx and used grep and sed to convert the packet offsets into base10 addresses to make extracting any additional information easier.

Friday, November 11

Today, Matt spent the class time showing some useful examples of binhaxx and dd. He also showed the class how the date command could be used to convert the Epoch timestamps of the packets into a Human-readable date format. Matt also helped me form a glob that accomplishes the same task using sed.

unix week10


Monday, October 31

Today we learned the magic of sed. sed is a tool that is similar to grep, but can also transform parsed data. Sed also makes use of regular expressions when translating parsed data. Another tool that is similar to both sed and grep is awk, a programming language for parsing, extracting, and processing data. Project: UNIX DATA RECOVERY (udr1)'s due date was also extended one week.

Wednesday, November 2

Today we started a shell script, timelord.sh, that will parse /var/log/wtmp and calculate the total time spent on lab46.

Friday, November 4

Today we updated our timelord.sh script to fix some logic errors and alter it's current output into a more useable format.

unix week9


Monday, October 24

We discussed the next project; Project: UNIX DATA RECOVERY (udr1). We also discussed Bit Numbering, Endianness (Byte Order), and some differences in various processor Architectures.

Wednesday, October 26

Today we started learning one of the most useful tools on UNIX, Regular Expressions! A Regular Expression (or RegEx) is a set of metacharacters that describe a character pattern located inside a file. UNIX tools, UNIX shells, and some programming languages all have support RegExs, one of the most commonly used tool for RegExs is called grep (globally search a regular expression and print). Some RegEx examples:

  • deadbeef : matches the 8 letters deadbeef, anywhere on a line.
  • ^deadbeef : matches the 8 letters deadbeef, at the beginning of a line.
  • deadbeef$ : matches the 8 letters deadbeef, at the end of a line.
  • ^deadbeef$ : matches a line containing exactly the 8 letters deadbeef, and nothing else.
  • [Dd]eadbeef : matches either the 8 letters Deadbeef, or the 8 letters deadbeef, anywhere on a line.
  • dead.beef : matches the 4 letters dead, any character, and the 4 letters beef, anywhere on a line.
  • dead.*beef : matches the 4 letters dead, any sequence of 0 or more characters, and the 4 letters beef, anywhere on a line.

In addition to single pattern matches, Regex's can be used to search for multiple occurrences of a pattern:

  • \{n\} : matches exactly n occurrences of the preceeding regular expression. Matching exactly 4 occurrences of a: a\{4\}
  • \{n,\} : matches at least n occurrences of the preceeding regular expression. Matching at least 4 occurrences of a: a\{4,\}
  • \{n,m\} : matches between n and m occurrences of the preceeding regular expression. Matching between 4 and 8 occurrences of a: a\{4,8\}

Dictionary files can be useful when experimenting with RegExs, located in the /usr/share/dict directory.

Friday, October 28

We continued our journey into pattern matching with Regular Expressions. Regex's were further expanded upon with Extended Regular Expressions. Extended Regex's allow grouping and logic symbols to be included within Regexs. There are also 3 different types of greps, and the difference between them was explained:

  • grep: basic regular expression matching.
  • egrep: 'extended grep', extended regular expression matching.
  • fgrep: 'fast grep', searches for literal strings (Regex's are disabled).

unix week8


Monday, October 17

Today we discussed the next project Project: UNIX DATA RECOVERY (udr0). We also went over the dd command that copies blocks of data from an input file to an output file.

Wednesday, October 19

Today we started using loops in bash. There are various types of loops:

  • Numeric for loop: useful when dealing with numbers or mathematics.
  • for each loop: Used when there is a specific start, end, and translation (also known as iteration).
  • while loop: also known as a list-based for loop; is useful when something needs to be repeated but the end condition is not known.

We also discussed some of the differences between high-level programming languages and low-level programming languages. Machine Language is the lowest level, and bash is one of the higher level languages. The higher the Abstraction of a programming language, the less control a user has over data and performance degrades.

Friday, October 21

TODAY WE KILLED THINGS! MANY MANY THINGS. TREMENDOUS FUN WAS HAD. The ps command will list processes you can kill! There are exactly 64 different ways to kill a process; but alas, fire isn't listed as one of them (despite the fact that it is the BEST one). Process ID's will range from 0 - 65535, with the Operating System reserving the lower values for important system processes. We also toyed with child processes, zombie processes, and placing a process into the foreground and background. jobs can be used to view any background processes, which can then be killed!

There's also emerging proof that lab46 has achieved Sentience:

lab46:~$ love
-bash: love: not found
lab46:~$ happiness
-bash: happiness: not found
lab46:~$ peace
-bash: peace: not found
lab46:~$ kill
-bash: you need to specify whom to kill
lab46:~$

unix break1


October 10-15

During break, I worked on the Project: WEB PAGE ADVENTURE (wpa0), bash scripting, python programming, and some regular expressions.

I set up some Python virtual environments to play around in. Using a Python virtualenv allows a user to install Python packages separately from the Python environment the system uses. This helps separate projects and their required packages into separate directories.

I started a new bash script, arc.sh, that takes a single argument (the filename) and then uncompresses it based on it's file type. Some current improvements are:

  • * Fix file/directory check
  • * Support multiple archives being passed
  • * More archive formats can be added; currently only supports *.7z, *.tar.bz2, *.bz2, *.tar.gz, *.gz, *.tar, *.tbz2, *.tar.xz, *.tgz, *.rar, *.zip, and *.Z.

I also experimented with terminal colors, searching and printing ASCII and Unicode characters, a simple bash prime number sieve script, and some regular expressions.

unix week7


Monday, October 3

Today we had our first UNIX Knowledge Assessment, UKA0. The Knowledge Assessment was a single .tar.xz file that included 2 c programs, dec2asc.c and hex2asc.c, that needed to be compiled with gcc, or by using the provided Makefile. A data.file and README file were also included. Some of the commands that helped during UKA0:

  • xz
  • tar
  • file
  • chmod
  • gcc or make
  • I/O Redirection
  • cat
  • tac
  • rev
  • sort
  • grep.

The due date for Project: WEB PAGE ADVENTURE (wpa0) was also changed and is now due after break.

During lab, I worked on Project: WEB PAGE ADVENTURE (wpa0). I got the room transitions working, I had forgotten a character in the absolute path name. I also tried some SSL trickery to see if I could diagnose and/or fix my Lab46 Journal Entry problem.

Wednesday, October 5

Today we started bash scripting while exploring variables, logical expressions, and mathematical operations inside bash scripts. The if, if else, and else if statements are examples of logical expressions. A common use for an if/if else/else if statement is to check if a particular value exists. If the value exists, other operations can be performed. If the value does not exist, other operations can be performed or the loop can simply exit. Mathematical operations can also be used in bash scripts using bc. bc is a mathematical scripting language that can be used to perform mathematical operations from the command line or inside a shell script. Its manpage contains the language information for using statements with expressions, loops, and even functions. It can also do mathematical operations using different number bases like binary, octal, and hexadecimal by changing the ibase (input number base) and obase (output number base). If the number base is changed, bc will stay in that number base until ibase and obase are changed.

During Lab, I spent my time configuring i3. I've learned most of the keybindings for i3 but I still haven't configured it yet, beyond removing xclock. I also solved my lab46 Journal problem, it was an HTTPS Mixed Content issue. This occurs when only part of an HTML page is sent over an encrypted HTTPS connection, because other resources (in this case, an image in one of my Journal entries) are sent over an unencrypted HTTP connection. Since the HTTPS connection is only partially encrypted, any unencrypted resources can be viewed and/or altered. Web servers and browsers really don't like this, hence my problem.

The rest of lab was spent doing some more work on Project: WEB PAGE ADVENTURE (wpa0).

Friday, October 7

Today we spent some more time writing bash scripts using comparison operators, logical connectives, and loops. Comparison operators can be used in bash scripts to perform integer comparisons. Logical connectives are used to perform bitwise operations. Other types of loops, the for, while, and case loops were also introduced. The for and while loops are used when the end result is known. A case statement can be used to execute a command based on an entered value. The for loop also requires 3 fields:

  • Where to Start
  • Where to End, non-inclusive
  • How much to iterate

Lab time was spent working out any kinks in Project: WEB PAGE ADVENTURE (wpa0) before the break.

unix week6


Monday, September 26

Today we discussed the the omnipotent and ubiquitous series of tubes invented by Al Gore known as The Internet and the strange dialect it uses; HTML. HTML will be used in the next project; Project: WEB PAGE ADVENTURE (wpa0), a text-based interactive experience disguised as a webpage. There are some helpful links in the Lab46 documentation for using HTML, CSS, and PHP in the wpa0 project.

Those of us who are old enough to have experienced the beauty and magnificence of BBS systems with captivating ANSI art had a wonderful time reminiscing about the times of old. We briefly discussed bash aliases. One useful alias that I set was alias ptree='tree -p' to print file permissions along with a directory tree.

Wednesday, September 28

We spent class time exploring vi's various options and configuration files. I had already configured my .vimrc file but still learned some useful options and key bindings.

I spent the lab time playing with bash scripts, more specifically I was trying to mimic a Ternary operation. A Ternary operator is a conditional expression that evaluates another expression based on a result. I have found a few globs that will mimic a Ternary operator in bash:

lab46:~$ true && { echo true; } || { echo false; }
true
lab46:~$ false && { echo true; } || { echo false; }
false
lab46:~$ true && echo true || echo false
true
lab46:~$ false && echo true || echo false
false

I also enabled the i3 tiling window manager. I still need to configure it and learn the key bindings though.

Friday, September 30

This entry was lost due to a 'Connection Reset'. I had some HTTP links in my journal that broke HTTPS.

We have finally started our journey into Shell Scripting. A shell script is a series of instructions that a shell will blindly execute, one after another. Any bash commands, loops, or logic that can be executed on the current system can be used in a bash script. To execute a shell script, there are 2 steps that must be completed first: The execution bit should be enabled; the read and write bits can also be enabled but are not necessary. Then, #!/{absolute path to the shell interpreter} needs to be on the first line of the script file. Since we will be using bash, this line will be #!/bin/bash.

I spent my lab time getting familiar with some of the keybindings for i3. I also installed the NetSurf web browser from source code. I have more embedded experience than I do with full operating system installs, so this was my first time installing anything from source code. The process was suprisingly easier than expected, once Matt fixed all the dependency issues. Matt also helped with a glob to convert mm/dd/yyyy dates to the International date format yyyy-mm-dd:

lab46:~$ echo "09/30/2016" | sed 's:\([0-9][0-9]\)/\([0-9][0-9]\)/\([0-9][0-9][0-9][0-9]\):\3-\2-\1:g'
2016-30-09

unix week5


Monday, September 19

Matt gave us a new project today, Project: UNIX SKILLS REVIEW (usr0) along with a custom shell script named urev. The urev script will help to review, reinforce, and practice prior material covering file permissions. There is also a time limit that prevents more than one section from being completed every 2 hours. I don't like time limits so I decided to find urev and take a look inside the script for any useful information. This might help in getting around any trivial time limitations. Before editing anything; I also made back-ups of the original files in case something were to go wrong (or in the unlikely event that rules were broken). After modifying a few things in some files scattered here and there, I was able to run urev as many times as I needed to complete the assignment.

After discussing this weeks project, we started experimenting with command aliases. An alias is a user-specified command which enables a replacement of a word by another string. It can be used for abbreviating long system commands, combinations of system commands, or adding user-defined 'default arguments' that will be passed to a built-in UNIX command.

  • To list currently enabled aliases: alias
  • To set an alias: alias <alias_name>='syntax and commands'
  • To use an alias: <alias_name>
  • To unset an alias: unalias <alias_name>
  • To make an alias permanent (restored after a reboot): add any aliases to the .bashrc login file.

Aliases that are set using the command line are not permanent and will be removed after a reboot. After viewing the currently enabled aliases on lab46, and adding some of our own, bash's configuration file was revealed. The Bash shell uses a configuration file (.bashrc) in a users home directory to enable and disable options within bash. This configuration file is executed every time you log into lab46. The .bashrc login file is also used to store permanent aliases; aliases stored in a .bashrc login file are restored after a reboot. An alias in .bashrc can be temporarily be disabled by issueing unalias <alias_name>, but the alias will return after a reboot because it is in this configuration file.

The .bashrc file is an example of a dot-login file. Files starting with a dot (.) are considered hidden files on a UNIX system. These files will not be displayed when using a command like ls, but will be displayed if the -a flag is passed.

During Lab, Matt installed Python's virtualenv. Lets see if it worked:

lab46:~$ python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')

YAY! virtualenv is listed in the installed packages. Another ACHIEVEMENT UNLOCKED: Lab46 User Experience Improved!

Wednesday, September 21

Today we started our magnificently marvelous journey to enhance our productive laziness with vi. vi has a unique feature called moded editing, with two file editing modes:

  • command mode: vi's default mode that it starts in, all keys are interpreted as commands. Commands entered can be preceeded by a number, and the command will be executed the same number of times.
  • insert mode: All keys (except <escape>) are interpreted as characters. Pressing <i> while in command mode is used to enter insert mode, and the <escape> key is used to exit insert mode. While insert mode is enabled, – INSERT – will appear at the bottom of vi.

Navigation, Insertion, deletion, and substitution can be done individually or by groups of multiple characters, words, or lines. vi can also edit text to the beginning or end of a file. There is a slight difference between inserting and appending text; inserting will place entered characters before the current cursor position while appending will place entered characters after the current cursor position. A similar instance exists when deleting text; the text is removed but is actually cut and can be pasted to another part of a file. Quitting vi and saving any file modifications can only accomplished while in command mode.

A useful tool when learning vi is VIM Adventures. It is an online RPG based on Vim's keyboard shortcuts to teach text editing using vim.

Friday, September 23

We continued our journey of productive laziness enhancement with vi. Some of vi's various settings were demonstrated, it has a number of syntax options that can be set and unset. To toggle options from within vi, :set option will enable the option and :set nooption will disable the option. To make the settings permanent (a configuration file will be read every time vi is executed), edit (or create) a file in your HOME directory called .vimrc and type any options without the colon (:). I also left comments in my .vimrc file for later reference. The Vi online reference was also highly useful. vi also makes use of a unique and very useful feature, buffers. When text is cut; it is “yanked” into a buffer. Any text inside a buffer can then be pasted anywhere in the file. Seperate buffers can also be named and used. Text inside a file can also be substituted or changed. change replaces lines of text with some different text while substitute searches for a text pattern, and then replaces the first instance with a text string.

During lab, I experimented with my .vimrc configuration file. The first item on my vi agenda was to find a color scheme that was a bit more pleasant on the eyes. I entered command mode and enabled syntax highlighting (:syntax on) then used :colorscheme [tab] to cycle through the various color schemes.

unix week4


September 12, 2016

We started class with a discussion about the next project that will be similar to the previous one; PUZZLEBOX NEXT (pbx1). We then continued the previous discussion on declaring shell variables, viewing the current environment variables, parameter expansion, and command substitution. The shell we will be using for the class is called the bash shell. Shell variables can be used with shell commands or shell scripts, and they are declared and accessed a number of ways in bash:

lab46:~$ Variable1="variable1"
lab46:~$ variable2="variable2"
lab46:~$ echo $Variable1
variable1
lab46:~$ echo $variable2
variable2
lab46:~$ echo $Variable1
variable1
lab46:~$ echo "Variable1"
Variable1
lab46:~$ echo '$Variable1'
$Variable1

Declared variables can also be accessed using Parameter Expansion. Parameter Expansion gets a value from one variable and substitutes it into another command or variable. The basic syntax for using parameter expansion is ${parameter}, where parameter is what is substituted:

lab46:~$ echo ${Variable1}
variable1
lab46:~$ echo ${variable2}
variable2

These variables are temporary and will be removed after a logout or a reboot. The bash shell also has reserved variables that can be set, like the PATH variable. The bash PATH variable is the search path that bash uses to search for installed commands. This colon-separated list of directories is searched from left to right, with the first match being used. The bash shell also has list of built-in commands in the bash man page under the SHELL BUILTIN COMMANDS section. Default paths are system-dependent, but some common directories are /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, and /usr/local/sbin. Other programs can be also installed in one of these directories and can be executed simply by typing its name. Command Substitution allows the output of one command to replace another command name. One command is executed and the command substitution is replaced with the standard output of the second command. There are two ways to use command substitution: $(command) or `command`. Command substitutions can also be nested, with one substitution located inside another, seperate command substitution. One useful example from class:

lab46:~$ numfiles=`/bin/ls | wc -w`
lab46:~$ echo $numfiles

which displays the number of files in the current directory. A UNIX file descriptor can also be used as an argument to a shell command to redirect a data stream:

  • 0> : represents STDIN
  • 1> : represents STDOUT
  • 2> : represents STDERR
lab46:~$ echo "redirect STDIN" 0>/dev/null
redirect STDIN
lab46:~$ echo "redirect STDOUT" 1>/dev/null
lab46:~$ echo "redirect STDERR" 2>/dev/null
redirect STDERR

/dev/null is a special file that discards any data that it gets passed. We also learned a new command, tr, that 'translates' (substitutes) one character with another. The tr command will only translate 1 sequence of characters at a time and can also delete characters passed from STDIN. During lab, I started the next project, PUZZLEBOX NEXT (pbx1). Later, I have finished and submitted the project.

September 14, 2016

Today, we continued our study of shell variables, command expansion, and the magic and mysticism of UNIX wildcard characters. These special characters can be used with shell commands to accomplish various tasks based on a particular pattern, similar to command flags. Wildcards can also be used to construct patterns that a UNIX shell interprets. This can be exceptionally useful in many situations, and it's result is affectionately referred to as a glob. The four most common wildcards are:

  • ? : Matches any single character.
  • [] : Character Class. Matches one of any characters enclosed in the brackets.
  • [^] : Inverted Character Class. Matches characters not enclosed in the brackets.
  • * : Matches zero or more of any characters.

A Character Class is a sequences of characters or numbers that can be used to return multiple matches with a single set of characters. Some example character classes are:

  • [:alpha:]:[A-Za-z]
  • [:digit:]:[0-9]
  • [:alnum:]:[0-9A-Za-z]
  • [:punct:]:[! “ # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~]

A complete list of available Character Classes and Wildcardscan be found in the bash man page.

I've already completed PUZZLEBOX NEXT (pbx1) so lab was spent trying to (unsuccessfully) install busybox from source.

September 16, 2016

Today we learned some of the rich history of UNIX and some different shells. A UNIX shell is how a user interacts with the operating system. The default shell on lab 46 is known as bash. There are many variants of the original UNIX shell, each with its own purpose and syntax. To view the available shells on lab46:

lab46:~$ cat /etc/shells
# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/screen
/bin/zsh
/usr/bin/zsh
/usr/bin/tmux

Here is a comparison of the various types of shells and their features.

UNIX time was also discussed along with some commands that rely on time, such as the date, cal, and ncal commands. In UNIX, time is represented as the number of seconds that have passed since 00:00:00 UTC on 1 January 1970; known as epoch. Since UNIX stores this value as a signed 32-bit ingeger, this value will overflow on 03:14:07 UTC on Tuesday, 19 January 2038 (or earlier for programs that work with future dates). This issue is similar to the Y2K bug and is known as the UNIX Millennium Bug. The date command is used to set or print the system date and time, while both the cal and ncal programs are used to display various calanders, dates, and other information.

unix week3


September 05, 2016

I wanted to start the Project: THE PUZZLEBOX (pbx0) earlier, but the class did not have read permissions. The lab introduces the file and other diagnostic utilities. file attempts to determine the actual type of a file by checking a series of properties. There are three sets of tests, performed in order:

  • * Filesystem tests: Checks to see if the file is non-ordinary (such as a socket, symbolic link, or other special file).
  • * Magic number tests: A check of files conforming to existing fixed formats, typically by examining the file at the binary level. If using a hex editor, you will find that .gz files should always start with the same sequence of hexadecimal values. File Signatures are also known as 'Magic Numbers'
  • * Language tests: If the file is determined to be a simple ASCII file, file will attempt to analyze whether or not it conforms to some language (like C source code, HTML, etc.).

The first test that succeeds determines the file type that is printed.

First, I copied file.txt and riddle.html from /var/public/fall2016/unix/pbx0 to my home directory. Using file, the 'riddle.html' file appears to be a gzip compressed data file, using max compression, that was previously “shiny.tar”. cat'ing the file doesn't reveal any of the contents but opening the file in nano revealed a '^_^ ^H^Hap T^B^Cshiny.tar^@' string. Using gzip with default compression, the 'riddle.html.gz' file appears to be gzip compressed data file, using max compression, that was previously “riddle.html”. Using file after uncompressing, recompressing, and using arguments for fastest compression; the 'riddle.html.gz' file appears to be a gzip compressed data file, using max speed, that was previously “riddle.html”.

The final secret message seems to have been revealed, even after some output errors. I initially had trouble redirecting my input and output, so I simply modified the file manually using nano. Different path, same result.

September 07, 2016

We started class learning about the status command on lab46. This command only exists on lab46 and outputs your class submissions and grade details. After, we discussed the difference between echo and cat. The echo command will mimic any data that is passed while cat while display the contents of a file. We also started experimented with some bash programming examples. The bash shell uses 3 types of quotes, each with a separate function:

  • “ or half-quote allows parameter expansion. Parameter Expansion gets a value from a variable then prints it.
  • ' or full-quote is a literal quote that allows NO parameter expansion.
  • ` or back-tick allows command expansion or command substitution. Command substitution captures the output of one or multiple commands and assigns them into another command or variable. Any command that has output can be captured.

Command and Variable expansion can be useful when passed to other bash commands or assigned to a bash variable. These variables are only temporary and are removed after a reboot. The printenv command can be used to view temporary and system-wide environment variables.

During lab, I encountered an alternate method to achieve the correct result: Instead of creating a new file with the base64 encoded data and removing the first line (I found out that the first line contains the filename), I piped the results of cat to uudecode directly. There were no error messages and the file was created and viewed successfully.

September 09, 2016

We started class by discussing UNIX pipes (|) then finished our discussion of terminal commands, file system navigation and permissions, and IO redirection. A UNIX pipe reads the output (STDOUT) from one command and passes it directly as input (STDIN) to the next command. When using a pipe, the output generated from one command must be in a compatible data format to be passed as input to another command. Another requirement for using a pipe is that each command has both input and output. tee is a special type of pipe that reads input (STDIN) from one command and writes the data to both standard output (STDIN) and one or more files, duplicating its input.

After logging into lab46 and joining a wemux session, the tty, mesg, and write commands were further explored. The tty command displays the current terminal number. This number identifies the current terminal session. Information about the current terminal session can then be viewed using ls. The mesg and write commands are both used to display messages on another lab46 user's terminal. mesg controls access to the current terminal by other lab46 users. It will allow or deny other lab46 users the ability to write to the current terminal. write allows lab46 users to communicate with each other, by copying lines from one terminal to another. The recipient user will get a notification message on their terminal. Then, any further text entered will be copied to the recipient user's terminal. If the recipient user wants to reply, they must also run write in a seperate terminal.

Then, we were introduced to the wc, head, tail, and chmod commands. The wc command is used to count occurences of newlines, words, and characters in a data file. The counts returned by wc are always printed in the following order: newline, word, character, byte, maximum line length. Both the head and tail commands are examples of used to partially view a file. head will display data from the start of a file, and tail will display data from the end of a file. We continued experimenting with File permissions. File permissions can be used to limit the access to data within a file. For example, the ls command requires both read and execute permissions to function correctly, so 500 is the minimum file permission needed for ls. File permissions are set and changed using the chmod command.

The sed and awk commands were briefly explored, along with piping their outputs to grep. sed and awk are both very useful commands by themselves; but are particularly exceptional when used in conjunction with grep. We've finally started out decent into the rabbit hole known as grep. grep is a tool that searches for data matching a Regular Expression. grep, sed, and awk are examples of UNIX filter commands. These commands accept data as input, transform the data, and output the resulting data. We ended the class with a brief introduction to shell programming, specifically the bash shell.

unix week2


August 29, 2016

Everything was updating so we were introduced to links2. links2 is a terminal web browser similar but different to lynx (Links2 good. Lynx bad!). We were also introduced to wemux; which is a multi-user tmux session. We continued our discussion of file system directories, file permissions, and user commands. We discussed who, tty, cat, and ldd. The chmod command is used to set and change file permissions, that are stored in a files inode. There is a Regular Expression at the bottom of the chmod man page that looks a little confusing:

"Each MODE is of the form '[ugoa]*([-+=]([rwxXst]*|[ugo]))+'."

I should remember to pick it apart later when the class starts Regular Expressions and figure it out. We also started to explore more of the base file system and the /bin, /boot, /lib, /etc, and /dev directories. There are some peculiar but useful directories in /dev:

  • /dev0: A special file that serves an endless stream of '0's.
  • /dev/null: A special file that discards any and all data sent to it.
  • /dev/random: A special file that serves an endless stream of random data.
  • /dev/urandom: Similar to /dev/random but has more Entropy.

I've almost finished Lab0x2 but Since tree is not installed on lab46; I have an idea for an alternative. I've used du (disk usage) and grep before on a headless (no monitor) system. The output will not be as pretty as it is with tree, but it will do. The only problem is formatting it to display it as a pretty directory tree (which tree does wonderfully. HINT HINT Matt.) before adding it to my opus. But fret no more… MATT FINALLY INSTALLED TREE! ACHIEVEMENT UNLOCKED: Lab46 (UX) Improved! Now everyone can enjoy the awesomeness and beauty that is tree. I also started Project: ARCHIVE HANDLING (arc0), which is about file archives and file compression. The binaries for these commands are stored in /bin evidently. Since Matt finally installed the essential tree command during lab, I can finally place a beautiful directory structure in my Opus entry.

lab46:~$ man tree
lab46:~$ tree /home/nbrimme1 -L 3 -o tree.txt
lab46:~$ cat tree.txt
/home/nbrimme1/
|-- Desktop
|-- Documents
|-- Downloads
|-- Maildir -> /var/mail/nbrimme1
|-- Music
|-- Pictures
|-- Public
|-- Scratch
|   `-- template.cc
|-- Templates
|-- Typescripts
|-- Videos
|-- musings
|-- neato.jpg
|-- public_html
|-- src
|   |-- Makefile
|   `-- unix
|       |-- arc0
|       |-- lab0x0.txt
|       |-- lab0x1.txt
|       `-- lab0x2.txt
|-- src.bak
|-- tree.txt
`-- usr-bin.txt

22 directories, 87 files

August 31, 2016

Things are still broken/updating (mainly the web browsers) so links2 to the rescue again! During class, we continued exploring the base file system directories. We viewed the /bin, /lib32, /lib64, /lost+found, /media, /mnt, /opt, /proc, /run, /sbin, /sys, /tmp, and /usr directories and their contents. An interesting feature of the /tmp directory is that it's contents are erased after a reboot. We then discussed some of the differences between the /lib32 and /lib64 directories, and the reasoning behind their separation. /lib32 is for 32-bit libraries and /lib64 is for 64-bit libraries…clearly. A UNIX system will have at least one of these directories, but both are not required. Both directories contain .so (shared object) files that are used by many things in the operating system. There are some useful files in the /proc directory to display information about the system, like $ cat cpuinfo, $ cat meminfo, $ cat ioports, $ cat interrupts, and $ cat filesystems. I also finished and submitted Project: ARCHIVE HANDLING (arc0).

September 2, 2016

We continued exploring more of the base file system directories; /usr/sbin, /usr/share, /usr/local, /var (and the /var/lib, /var/log, /var/public, and /var/spool subdirectories), and of course /usr/games. One of the most important directories in all of UNIX (other than /var/log) can be found in /usr/share/man/. The /usr/share/ directory is used to store documentation and the systems man pages. The number after a man page is it's Section. As an example; to view section 8 of the cron man page: $ man cron 8 ($ man man has the entire list of section numbers). Manual pages also have some common section names between them like NAME, SYNOPSIS, CONFIGURATION, DESCRIPTION, OPTIONS, FILES, NOTES, BUGS, EXAMPLE, AUTHORS, and SEE ALSO. The apropos command can be used to search man pages if the needed man page is unknown. We also learned various ways to view and I/O Redirection in a terminal using cat, head, and tail. Using > (write/overwrite) and » (append), STDIN, STDOUT, and STDERR can be redirected:

  • <: Redirect STDIN
  • >: Redirect STDOUT
  • 1>: Redirect STDOUT
  • 2>: Redirect STDERR

I discovered that there was an error with my arc0 lab, so I recompressed and resubmitted a corrected file. While fixing it, I learned $ tar -t -f archive.tar.gz can be used to view the contents of an archive without decompressing it.

unix week 01

August 22, 2016

The first day of class was a basic introduction to UNIX. We started class by going over the online syllabus. After, everyone got acclimated with lab46, the personal wiki pages, and logging in with ssh. There are also some projects to complete, including one for using git with the lab46 Mercurial server. I also read some of UNIX for the Beginning Mage and it seems similar to another great book, Land of Lisp. I also noticed a long-forgotten public_html directory, I just don't know what to add to it yet. I looked at some of the other students web pages for some ideas, and some were quite interesting. Since I hate typing and remembering passwords, I might make an entry about using ssh keys with lab46. I also looked at the lab46 documentation and enabled logging for my public_html page. man pages were briefly mentioned, which made me recall a very useful website, Online Man Pages. In addition to man pages, there's also some useful beginner and advanced links. Some other websites that might be useful are List of UNIX Commands and List of UNIX Daemons.

Lab0x0 is complete. The project was just a brief introduction to lab46. I've used both UNIX and lab46 before so it was just a refresher. Some of the topics covered were traversing the file system, seeing who is logged on, sending messages and email, and joining the UNIX class mailing list.

Lab0x1 is complete. The project was about one of the most useful commands in UNIX, file. Everything on UNIX is a file… EVERYTHING! User and group file permissions and copying/moving/removing files were some of the other subjects covered. One of the more interesting subjects (that I have little experience with) was Symbolic Links f7ijz.jpg

I still don't fully understand them… So; like the great UNIX mages before me, I must consult the Tomes of Knowledge then venture deep into the catacombs beneath the monastery for deep meditation & contemplation.

Lab0x2 is complete. The project explained the contents of the Unix Filesystem tree in more detail. A great utility that isn't on lab46 (MATT. FIX. QUICK-LIKE-BUNNY. PLS-N-THX.) is the tree command. In short, “Tree is a recursive directory listing program that produces a depth indented listing of files.” The output of tree is so much prettier than that plain old ugly ls command. The provided diagram helps when visualizing the file system. For review:

  • /bin
    • Essential basic tools for normal system usage.
  • /etc
    • System-wide configuration files and system databases.
  • /home
    • Location of the system's user directories.
  • /lib, /lib64, /lib32
    • Contains important system libraries.
  • /mnt
    • An empty directory that is commonly used as a temporary mount point for additional filesystems.
  • /root
    • The superuser's home directory.
  • /sbin
    • Essential system administration utilities, usually needed to start/maintain/recover the system.
  • /tmp
    • Temporary directory. Files in this directory are removed after reboot.
  • /usr
    • Additional (secondary) system functionality & userspace tools.
  • /var
    • Misc. items (mail files & databases).

Another topic was absolute and relative path names. An absolute path name is a specific reference to a location on the filesystem, with the final destination immediately known. A relative path name is a directory reachable from the present working directory, with its final destination only making sense from the present working directory. This is similar, in concept, to Time Zones where the current time is relative to the current position on the Earth (or the current Time Zone). Different types of files and their user, group, and other File Access Permissions is another important topic that was introduced. Since they are represented in Octal, Unix File Permissions Calculator might be of some help to some, while saving some time for others.

August 24, 2016

Today started with an introduction to irssi and the class chat. I also managed to remove all the unnecessary files from my lab46 repository and commit-push'ed the changes. To recap:

  • hg status
  • hg add filename *Only if any new files were created.
  • hg commit -m “Descriptive comment.”
  • hg push

I've already completed the projects so all that's left is to submit them. Matt also help clarify the difference between hard- and soft-links: A hard-link creates another, separate copy of the file and a soft-link simply creates a pointer to the original file.

August 26, 2016

Today in class we learned some more about file system traversal and file access. There are 9 access modes: The first 3 places describe the file owner's permissions, the next 3 describe the file group's permissions, and the remaining 3 places describe what permissions anyone else on the system has (called global). Anyone else on the system refers to any other usernames that are not included with a system or in a system group. On some systems, the last group may be set to - - - for security purposes. We also discussed the setuid, setgid, and sticky bit file overlay permissions. The set user id bit is something that requires a certain user to execute, the set group id bit makes the user a member of a group for the run duration, and the sticky bit prevents file & directory deletion. These file permissions can be expressed in both octal or using letters: r for read access, w for write access, x for execute and search access, a - for no access, and l which are symlinks. Another important note to remember with the execute and search flag: Directories are searched, files are executed.


Portfolio

Data Structures

UNIX/Linux Fundamentals

Other Projects

Networking

Operating Systems

Cheat sheets/brain dumps for the installation, configuration, and maintenance of various OSs.


journal/fall2016/nbrimme1/start.txt · Last modified: 2016/08/22 05:32 (external edit)