1 # Introduction to the shell
3 * Name-drop [terminal][] as distinct from [shell][].
4 * Get folks to a terminal.
5 * Mention [Bash][] as a common default shell
6 * If you're in a different shell, run `bash` to get Bash.
7 * Mention commonality between shells, but existence of differences.
8 * Maybe mention [POSIX][].
9 * [Name-drop the Advanced Bash-Scripting Guide][ABSG].
10 * Have them clone the boot camp repository:
12 $ git clone -b YYYY-MM-PLACE --single-branch git://github.com/swcarpentry/boot-camps.git
14 * `echo` (e.g. `echo Hello, World`)
15 * Remind them to press enter.
16 * Useful for printing from a shell script, for displaying
17 variables, and for generating known values to pass to other
22 * `pwd` (print working directory)
23 * `ls` (list directory contents)
24 * Directory/folder equivalence
25 * Directories are just lists of files or other directories.
26 * New-terminals/shells start in the *home* directory.
27 * Every user has their own home directory.
28 * `whoami` (what is my user name)
32 * `ls` output is colored (run it in the home directory).
33 * Directories are usually blue (customize with `~/.dir_colors`).
34 * `touch` (create an empty file)
38 * `ls` again, see `testfile`. Note that it's white (a general file).
39 * `ls -F`, because some terminals don't support colors.
40 * Directories marked with a trailing slash (/)
41 * Executables marked with a trailing star (*)
42 * `ls -l` to get permissions, ownership, size, modification times, etc.
43 * If the entry is a directory, then the first letter will be a `d`.
44 * The fifth column shows you the size of the entries in bytes.
45 * Notice that `testfile` has a size of zero.
46 * `rm` to remove files and (with `-r`) directories.
50 * Run `ls` again, and see that `testfile` is gone.
52 # Changing Directories
54 * `cd` (change directory) into the directory they'd previously cloned
58 * `ls` to see what's in the newly current directory
59 * `cd` into the exercises section, then `ls` again, and `ls -F`.
60 * You may want to `cd "$LOC1"; cd "$LOC2"` instead of jumping
61 directly via `cd "$LOC1/$LOC2"`.
62 * `cd` without arguments takes you back to your home directory
63 * Change back into the examples directory
67 * `-F` and `-l` are arguments to `ls`
68 * `ls` takes lots of arguments
69 * `man` pages (when they exist) usually describe these options
73 * `man` displays the page using your `PAGER`. This is probably
76 * Page down with space or the "page down" key
77 * Page up with `b` or the "page up" key
78 * Search by regexp with `/`
79 * Get help with `h` with other `less` key bindings
82 * `man find` for a complicated page
83 * Don't worry about learning all of this, just refer back to it as
86 ## Exercise: `ls` options
88 1. Use the manual page for `ls` to guess what you would expect from
89 using the arguments `-l`, '-t', '-r' at the same time.
90 2. Try the following and see if you can figure out what they do,
91 either by examining the results or consulting the manual page.
92 * `ls -lS` (equivalent to `ls -l -S`)
93 * `ls -lt` (equivalent to `ls -l -t`)
94 * `ls -1` (that's the number one, not a letter 'ell')
96 # Examining the contents of other directories
98 * Instead of `cd "$LOC"; ls`, you can use `ls "$LOC"`
99 * Look at a few directories in this repository
100 * Use `cd "$LOC"` to change into a few directories as well
101 * Folks should be comfortable with `cd "$LOC1/$LOC2"` by now
103 # Full vs. relative paths
105 * Directories are arranged in a single hierarchy, based on the
106 *root directory* `/`.
107 * Full (aka absolute) paths include everything from the root down.
108 * Relative paths don't start with the root directory, and are
109 (usually) interpreted relative to the current working directory
111 * Examples with full paths:
113 $ cd /home/swc/boot-camps/modular
115 * Examples with relative paths:
118 $ cd boot-camps/modular
120 # Exercise: Listing `/bin`
122 List the contents of the `/bin` directory (a full path). Do you see
123 anything familiar in there?
125 # Saving time with shortcuts, wild cards, and tab completion
129 * `~` for your home directory
130 * `..` for the parent of the current working directory
131 * Chainable. For example: `ls ../../` or `ls ~/../`.
132 * `.` for the current directory
133 * Equivalent commands:
137 * These shortcuts are not necessary, they are provided for your
140 # Example: Cochlear implants
142 Introduce the [cochlear implant example][cochlear] to practice with
147 * Docs for wildcards are in in glob(7).
148 * For cochlear implants, cd into `data/THOMAS` for this.
149 * `ls`, there are a bunch of files with four-digit names
151 * `ls *1', all files ending with 1
152 * `ls /usr/bin/*.sh`, an absolute path example
153 * `ls *4*1`, compound wildcards
154 * These are expanded *by the shell*, before being passed to the
155 command, so the following are identical;
158 $ ls 0241 0341 0431 0481
160 ### Exercise: Globbing with `ls`
162 Do each of the following using a single `ls` command without
163 navigating to a different directory.
165 1. List all of the files in `/bin` that contain the letter `a`
166 2. List all of the files in `/bin` that contain the letter `a` or the letter `b`
167 3. List all of the files in `/bin` that contain the letter `a` AND the letter `b`
171 * Change to the home directory.
172 * Work back to the coclear implant example data using tab completion
176 $ cd boot-camps/<tab><tab>
179 * Tab completion for executables in the `PATH`.
187 * Work through history with up and down arrows.
188 * `^C` to cancel the command you are writing and give you a fresh
190 * `^R` to reverse-search through your command history.
191 * `history` to print your history
192 * `history | grep …` to grab some useful command
193 * `!123` to reuse the 123rd command in your history
195 ### Excercise: History
197 1. Find the line number in your history for the last exercise (listing
198 files in `/bin`) and reissue that command.
202 * `ls`, `rm`, `find`, … are stand-alone programs.
203 * `command`, `shift`, `cd`, `echo`, `pwd`, … are likely shell builtins
204 * `which COMMAND` to look up the command in your `PATH`
205 * `command -V COMMAND` to get useful info:
208 cd is a shell builtin
212 ls is aliased to `ls --color=auto'
213 * Explain `PATH` lookup (`echo $PATH`)
214 * In the example directory, there's a `hello` script
215 * Try to run it with `hello`. Oops, not in the `PATH`!
216 * Give a path with `./hello`. Success!
217 * Something that `.` is useful for!
218 * `/home/…/boot-camps/…/hello`
219 * `../exercises/hello/hello`
220 * Only get `PATH` lookups when there aren't any `/` in your command.
224 We now know how to switch directories, run programs, and look at the
225 contents of directories, but how do we look at the contents of files?
227 * `cat` (concatenate). In the cochlear implant example,
230 $ cat ex_data.txt ex_data.txt
234 1. Print out the contents of the `dictionary/dictionary.txt` file.
235 What does this file contain?
237 2. Without changing directories, use one short command to print the
238 contents of all of the files in the
239 `/home/…/boot-camps/…/exercises/hearing/data/THOMAS` directory.
243 * How to view large files?
246 $ less ~/boot-camps/…/exercises/dictionary/dictionary.txt
248 * `less` opens the file and lets you navigate through it.
249 * As mentioned with the `man` page stuff earlier, `h` for help, `q` to
251 * Searches with `/` don't wrap, search from the beginning of the file.
253 ### Example: paging through man pages
255 Use the commands we've learned so far to figure out how to search
256 in reverse while using `less`.
260 * Back to the hearing example!
261 * Enter the `Bert` subdirectory.
262 * Print all the experimental results
266 * Redirect that into a file:
268 $ cat au* > ../all_data
270 * Examine `all_data` with `less`.
271 * If `all_data` had already existed, we would overwritten it.
272 * `>` tells the shell to take the output from what ever is (stdout) on
273 the left and dump it into the file on the right.
274 * `>>` is the same, but it appends instead of overwriting.
276 ### Exercise: Redirection
278 Use `>>`, to append the contents of all of the files whose names
279 contain the number 4 in the `gerdal` directory to the existing
280 `all_data` file. Thus, when you are done `all_data` should contain all
281 of the experiment data from `Bert` and any experimental data file from
282 `gerdal` with filenames that contain the number 4.
284 ## Creating, moving, copying, and removing
286 We've created a file called `all_data` using the redirection operator
287 `>`. This file is critical—it's our analysis results—so we want to
288 make copies so that the data is backed up.
292 $ cp all_data all_data_backup
296 $ mv all_data_backup /tmp/
298 * Explain what `/tmp` is for, and that it doesn't (usually)
302 $ mv all_data all_data_IMPORTANT
306 $ rm /tmp/all_data_backup
308 * `mkdir` (make directory)
309 * By default, `rm`, will NOT delete directories. Use `-r` (carefully!)
310 * Also, `rmdir` (remove directory)
312 ### Excercise: Rename, create, and copy
314 1. Rename the `all_data_IMPORTANT` file to `all_data`.
315 2. Create a directory in the `data` directory called `foo`
316 3. Then, copy the `all_data` file into `foo`
320 * `wc` (word count). From `hearing/data`:
322 $ wc Bert/* gerdal/*4*
324 * Three columns for each file:
327 * Number of characters
328 * The final line contains this information summed over all of the
334 * Compare byte size with character count
338 ### Exercise: Longest line length
340 Figure out how to get `wc` to print the length of the longest line in
343 ## The awesome power of the Pipe
345 Suppose I wanted to only see the total number of character, words, and
346 lines across the files `Bert/*` and `gerdal/*4*`. I don't want to
347 see the individual counts, just the total. Of course, I could just do:
351 Since this file is a concatenation of the smaller files. Sure, this
352 works, but I had to create the `all_data` file to do this. Thus, I
353 have wasted a precious 10445 bytes of hard disk space. We can do this
354 *without* creating a temporary file, but first I have to show you two
355 more commands: `head` and `tail`. These commands print the first few,
356 or last few, lines of a file, respectively. Try them out on
362 The `-n` option to either of these commands can be used to print the
363 first or last `n` lines of a file. To print the first/last line of the
369 Let's turn back to the problem of printing only the total number of
370 lines in a set of files without creating any temporary files. To do
371 this, we want to tell the shell to take the output of the `wc Bert/*
372 gerdal/*4*` and send it into the `tail -n 1` command. The `|`
373 character (called pipe) is used for this purpose. Enter the following
376 wc Bert/* gerdal/Data0559 | tail -n 1
378 This will print only the total number of lines, characters, and words
379 across all of these files. What is happening here? Well, `tail`, like
380 many command line programs will read from the *standard input* when it
381 is not given any files to operate on. In this case, it will just sit
382 there waiting for input. That input can come from the user's keyboard
383 *or from another program*. Try this:
387 Notice that your cursor just sits there blinking. Tail is waiting for
388 data to come in. Now type:
395 then CONTROL+d. You should is the lines:
400 printed back at you. The CONTROL+d keyboard shortcut inserts an
401 *end-of-file* character. It is sort of the standard way of telling the
402 program "I'm done entering data". The `|` character is replaces the
403 data from the keyboard with data from another command. You can string
404 all sorts of commands together using the pipe.
406 The philosophy behind these command line programs is that none of them
407 really do anything all that impressive. BUT when you start chaining
408 them together, you can do some really powerful things really
409 efficiently. If you want to be proficient at using the shell, you must
410 learn to become proficient with the pipe and redirection operators:
414 ### A sorting example
416 Let's create a file with some words to sort for the next example. We
417 want to create a file which contains the following names:
424 To do this, we need a program which allows us to create text
425 files. There are many such programs, the easiest one which is
426 installed on almost all systems is called `nano`. Navigate to `/tmp`
427 and enter the following command:
431 Now enter the four names as shown above. When you are done, press
432 CONTROL+O to write out the file. Press enter to use the file name
433 `toBeSorted`. Then press CONTROL+x to exit `nano`.
435 When you are back to the command line, enter the command:
439 Notice that the names are now printed in alphabetical order.
444 Use the `echo` command and the append operator, `>>`, to append your
445 name to the file, then sort it and make a new file called Sorted.
449 Let's navigate back to `~/boot-camps/shell/data`. Enter the following command:
451 wc Bert/* | sort -k 3 -n
453 We are already familiar with what the first of these two commands
454 does: it creates a list containing the number of characters, words,
455 and lines in each file in the `Bert` directory. This list is then
456 piped into the `sort` command, so that it can be sorted. Notice there
457 are two options given to sort:
459 1. `-k 3`: Sort based on the third column
460 2. `-n`: Sort in numerical order as opposed to alphabetical order
462 Notice that the files are sorted by the number of characters.
467 1. Use the `man` command to find out how to sort the output from `wc` in
470 2. Combine the `wc`, `sort`, `head` and `tail` commands so that only the
471 `wc` information for the largest file is listed
473 Hint: To print the smallest file, use:
475 wc Bert/* | sort -k 3 -n | head -n 1
479 Printing the smallest file seems pretty useful. We don't want to type
480 out that long command often. Let's create a simple script, a simple
481 program, to run this command. The program will look at all of the
482 files in the current directory and print the information about the
483 smallest one. Let's call the script `smallest`. We'll use `nano` to
484 create this file. Navigate to the `data` directory, then:
488 Then enter the following text:
491 wc * | sort -k 3 -n | head -n 1
493 Now, `cd` into the `Bert` directory and enter the command
494 `../smallest`. Notice that it says permission denied. This happens
495 because we haven't told the shell that this is an executable
496 file. If you do `ls -l ../smallest`, it will show you the permissions on
497 the left of the listing.
499 Enter the following commands:
501 chmod a+x ../smallest
504 The `chmod` command is used to modify the permissions of a file. This
505 particular command modifies the file `../smallest` by giving all users
506 (notice the `a`) permission to execute (notice the `x`) the file. If
511 You will see that the file name is green and the permissions have changed.
512 Congratulations, you just created your first shell script!
516 You can search the contents of a file using the command `grep`. The
517 `grep` program is very powerful and useful especially when combined
518 with other commands by using the pipe. Navigate to the `Bert`
519 directory. Every data file in this directory has a line which says
520 "Range". The range represents the smallest frequency range that can be
521 discriminated. Lets list all of the ranges from the tests that Bert
529 Create an executable script called `smallestrange` in the `data`
530 directory, that is similar to the `smallest` script, but prints the
531 file containing the file with the smallest Range. Use the commands
532 `grep`, `sort`, and `tail` to do this.
539 The `find` program can be used to find files based on arbitrary
540 criteria. Navigate to the `data` directory and enter the following
545 This prints the name of every file or directory, recursively, starting
546 from the current directory. Let's exclude all of the directories:
548 find . -type f -print
550 This tells `find` to locate only files. Now try these commands:
552 find . -type f -name "*1*"
553 find . -type f -name "*1*" -or -name "*2*" -print
554 find . -type f -name "*1*" -and -name "*2*" -print
556 The `find` command can acquire a list of files and perform some
557 operation on each file. Try this command out:
559 find . -type f -exec grep Volume {} \;
561 This command finds every file starting from `.`. Then it searches each
562 file for a line which contains the word "Volume". The `{}` refers to
563 the name of each file. The trailing `\;` is used to terminate the
564 command. This command is slow, because it is calling a new instance
565 of `grep` for each item the `find` returns.
567 A faster way to do this is to use the `xargs` command:
569 find . -type f -print | xargs grep Volume
571 `find` generates a list of all the files we are interested in,
572 then we pipe them to `xargs`. `xargs` takes the items given to it
573 and passes them as arguments to `grep`. `xargs` generally only creates
574 a single instance of `grep` (or whatever program it is running).
579 Navigate to the `data` directory. Use one `find` command to perform each
580 of the operations listed below (except number 2, which does not
581 require a `find` command):
583 1. Find any file whose name is "NOTES" within `data` and delete it
585 2. Create a new directory called `cleaneddata`
587 3. Move all of the files within `data` to the `cleaneddata` directory
589 4. Rename all of the files to ensure that they end in `.txt` (note:
590 it is ok for the file name to end in `.txt.txt`
592 Hint: If you make a mistake and need to start over just do the
595 1. Navigate to the `shell` directory
597 2. Delete the `data` directory
599 3. Enter the command: `git checkout -- data` You should see that the
600 data directory has reappeared in its original state
604 Redo exercise 4, except rename only the files which do not already end
605 in `.txt`. You will have to use the `man` command to figure out how to
606 search for files which do not match a certain name.
614 * Command substitution (`$(command)` and `\`command\``).
617 * Variables (`PATH`, `PS1`, `PWD`)
621 * `ssh` and `scp` (secure shell and secure copy)
622 * Regular expressions (with `grep`, `sed`, …)
623 * Permissions (`ls -l`, `chmod`, `chown`)
624 * Chaining commands (pipes, file descriptors, fifos, …)
627 [terminal]: https://en.wikipedia.org/wiki/Terminal_emulator
628 [shell]: https://en.wikipedia.org/wiki/Unix_shell
629 [Bash]: http://www.gnu.org/software/bash/
630 [POSIX]: http://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html
631 [ABSG]: http://tldp.org/LDP/abs/html/
632 [cochlear]: exercises/hearing/