Shell Filters and System Administration

12.1 Shell Scripts and Assignment Structure

This session sits at the junction of two threads: the shell as a programming language and the shell as a glue for the operating system. It builds directly on the previous two sessions on shell programming and frames the course assignment, which is not an exercise in printing patterns but a practical automation task.

for i in 1 2 3 4 5; do
  echo -n "\$i "
done
# → 1 2 3 4 5

for i in 1 2 3; do
  for j in \$(seq 1 \$i); do echo -n "* "; done
  echo
done

Worked mini-context: the session contrasted "printing stars" versus "searching files". The former validates syntax; the latter delivers value. A script that counts how many lines in /etc/passwd have /bin/bash as their shell (grep -c "bash\$" /etc/passwd) already does administration that would otherwise require opening the file and counting by eye.

Visual intuition: picture a pipeline diagram with two lanes. The top lane shows for → if → echo "*" looping to build a visual pattern — a closed loop producing decoration. The bottom lane shows for file in *.log → grep "ERROR" → wc -l flowing left to right, with files entering at left, filtered lines in the middle, and a count exiting at right. The horizontal axis is data flow, the vertical is purpose. The takeaway: the same loop construct carries very different payloads.

12.1.1 Role of Shell Programming and Commands

A shell script — a text file of shell commands plus loops and conditionals executed by the shell — becomes practical when it orchestrates filters and utilities rather than drawing shapes. The for loops printing 1 2 3 4 5 and nested star patterns illustrate syntax, but the assessed use is file searching, text transformation, and admin automation, where each interactively practiced command becomes a reusable script component.

12.1.2 Assignment Groups and Practical Focus

The assignment is sized for three students per group, with at most one group of two permitted; all others should have three members. Students not yet on the assignment sheet should form groups promptly and confirm the final list so program allocations can be frozen and preparation time preserved. Early confirmation respects committed peers and aligns with the lecture's pivot to command-heavy scripting, where shared workload matters.

12.2 The grep Filter — Purpose and Syntax

The grep filter purpose — searching passwd without manual edit — illustrates the core value: grep scans passwd without opening the file in an editor, avoiding manual edit risks.

The most frequently invoked building block in shell scripts for text tasks is grep. The session repeatedly heard "grab/grip" — the standard Unix filter grep, short for global regular expression print (g/RE/p in ed), now used far beyond editing to scan any text source.

grep [options] pattern [file ...]

grep "SAP" /etc/passwd
# SAP1:x:1005:1005::/home/SAP1:/bin/bash
# SAP2:x:1006:1006::/home/SAP2:/bin/bash

Visual intuition: draw a flow left to right: files → grep pattern (sieve) → matching lines to stdout. Annotate the sieve mesh as the pattern: a small literal string lets only exact matches through; a regex mesh with holes shaped ., *, [A-Z] lets families of strings through. The vertical axis is "lines kept vs dropped"; the takeaway is selectivity increases with pattern specificity.

12.2.1 What grep Does

A filter — a program that reads input, transforms or selects parts, and writes a result — in grep's case selects lines containing a given pattern. Given a pattern and a file, grep outputs matching lines, often with file names and line numbers depending on options; with multiple files it indicates which file holds each match. This avoids manual inspection of large files. The motivating example is /etc/passwd with hundreds or thousands of entries: locating a user or group by editing the file is slow and may require privileges, while grep surfaces the answer immediately, and the same mechanism applies to any text file.

12.2.2 Command Syntax and Pattern Types

The general form is grep [options] pattern [file ...] where pattern is a literal string such as this or, more generally, a regular expression — a pattern language using metacharacters ., *, ?, character classes [...], and anchors ^/\$ to describe sets of strings; a dot can stand for any single character, * repeats, and brackets restrict to alphabets or digits. Options modify matching or output and are covered next.

12.2.3 Files, Multiple Files, and Directories

grep accepts a single file, a list, or a shell-expanded set: grep this grep_file searches one file, while grep THIS ./* or grep case *file uses a glob where *file matches any file name ending in file (e.g., grep_file, demo_file) and searches each matched file. When a directory name is passed without a recursive flag, grep reports Is a directory and skips it; to search a directory's files you must give the files themselves via a glob or grep -r.

12.3 grep Options for Output Control

Options — hyphen-prefixed flags that reshape matching and reporting without changing the core search — are what make grep adaptable inside scripts where output format is often more important than raw matches.

grep this grep_file        # → 3 lines shown
grep -c this grep_file     # → 3

grep -i this grep_file     # → 5 lines (adds This, THIS/ThIs variants)
grep -i -c this grep_file  # → 5
grep This grep_file        # capital T, no -i → 1 exact-cap match

grep -l case *file         # → demo_file\n grep_file  (both have 3 hits each)
grep case *file            # → demo_file: ... \n grep_file: ...  (lines + filenames)
grep -h case *file         # same lines without filenames

grep -o "this" grep_file   # → one "this" per match line, one per line if line repeats

grep -v this grep_file            # → 2 empty + lines without lowercase this = 4 lines
grep -v -c this grep_file         # → 4
grep -i -v -c this grep_file      # case-insensitive invert → only the 2 truly empty lines = 2

Visual intuition: imagine a control panel with toggles: -c flips display from lines to a numeric counter; -n adds a ruler margin with line numbers (1: this..., 2: ...); -h/-l toggle file-name labels on/off. The x-axis is lines of the file, y is "shown". With -c the y collapses to a single number 3; with -l it collaps to file names demo_file, grep_file. Takeaway: options do not move the search; they reshape the report for the next pipeline stage.

12.3.1 Counting and Listing Matches

Count — -c reports the number of matching lines (not words). Searching this in grep_file shows three lines; grep -c this grep_file reports 3, while file or line as patterns give different counts with exact case sensitivity. List file names — -l lists only names of files that contain a match; grep -l case *file lists demo_file and grep_file because each holds matches, answering "where" rather than "what". Only the matching part — -o prints only the matched substring, one match per line, isolating the pattern for further processing such as sort | uniq -c.

12.3.2 Controlling What Is Displayed

Suppress file names — -h removes the file-name prefix when searching multiple files, showing only matching lines (default with multiple files prefixes each line with file:). Line numbers — -n prefixes n:line; this appears on line 1 case-sensitively, and with -i on lines 1,2,3,6,7. Case sensitivity — -i ignores case, expanding this from 3 to 5 matches to include This/THIS/ThIs; grep -i -c and grep -i THIS grep_file compose. Whole word — -w requires a whole-word match; Dell without -w matches Dell EMC as substring, with -w only standalone Dell qualifies.

12.3.3 Inverting and Specifying Patterns

Invert match — -v prints lines that do not contain the pattern; for this in grep_file it returns the two empty lines plus non-matching lines, so grep -v -c this grep_file is 4 and grep -i -v -c this grep_file is 2 (only empty lines after case-insensitive exclusion). Expression — -e explicitly introduces a pattern, essential when it begins with - or for multiple -e OR patterns. File of patterns — -F treats the pattern as a literal fixed string (no regex), and -f file reads patterns from a file one per line matching any. Extended expressions — -E enables extended regular expressions with + ? | () beyond the basic set.

12.4 Regular Expression Building Blocks

Regular expressions are the pattern language grep interprets — a compact notation for describing sets of strings, built incrementally in the lecture with live file checks.

grep "[A-Z][A-Z][A-Z]" table.doc

grep -i "[A-Z][A-Z][A-Z]" table.doc

grep "[A-Z][a-z]" table.doc

Visual intuition: draw three adjacent boxes labeled [A-Z] [A-Z] [A-Z] forming a stencil. Slide the stencil along a line N o r t h w e s t; only positions where three consecutive slots are all capitals light up. Replace one box with .* — the stencil now has a stretchy middle that can expand to any length including zero. The x-axis is character position, shape is "fixed vs stretchy". Takeaway: dot-star is the stretchy glue; bracket triples are rigid length checks.

12.4.1 Single-Character and Any-String Metacharacters

A dot . matches any single character (space/tab included) and consumes exactly one position. .* combines dot with * (zero or more of preceding) to match any string including empty, bridging two fixed fragments. The lecture emphasized * alone does not mean any string including space: this*file without a dot applies * to the preceding s (described as alphanumerics but not space) and gave no output, while this.*file with dot-star bridges spaces and matched this ... file.

12.4.2 Character Classes and Ranges

Square brackets define a character class — any one character from the set: [A-Z] one uppercase, [a-z] one lowercase, [A-Za-z] one alphabetic, [0-9] digit, [A-Za-z0-9_] word-like, [AG] at start means A or G in position 1, [ABCD]/[A-D] any of those letters, [AB].* means A then B then anything. Applying * to a class ([A-Z]*, [A-Za-z]*) means zero or more from that class. grep "[A-Z][A-Z][A-Z]" table.doc probes three consecutive capitals; without -i lowercased entries are excluded, with -i all three-alpha runs match.

12.4.3 Character Class Shorthands and Negation

Shorthands (often via extended/PERL-compatible modes, cited as extended classes) include \d digit ([0-9]), \D non-digit, \w word character ([0-9A-Za-z_]) useful for usernames with underscores/digits, \W non-word, \s whitespace, \S non-whitespace. Negation inside a class uses a leading caret ^ inside brackets: [^0-9] matches any character that is not a digit, [^A-Z] any non-uppercase; the placement of ^ is decisive, as explored in Q&A.

12.5 Anchors, Escaping, and Word Matching

Position and literalization turn the building blocks into precise tools: anchors say where in the line, escaping says literally that, and word-mode says whole token.

grep "bash\$" /etc/passwd    # or grep 'bash\$'
# root:x:0:0::/root:/bin/bash
# ubuntu:x:1000:1000::/home/ubuntu:/bin/bash
# SAP1:x:1005:1005::/home/SAP1:/bin/bash  ... etc.
grep "nologin\$" /etc/passwd
# daemon:x:1:1::/usr/sbin:/usr/sbin/nologin  ... many system accounts

grep "5\.." table.doc
# S   Southern   5.95  6 and 23   →  5 . 9  (literal dot + any char 9)
# W   Western   53.97 ...        →  5 3 . 9 ... but wait? 5\.. matches 5 . 3? Actually 53.97 contains "5" then "3" then "."? No. Correct: 53.97 contains "3.9" not "5..". So line with 53.97 matches because characters "...53.97..." contain substring "3.9"? No. Better: 5\.. on "53.97" matches "3.9"? Confusion. Real demo: 5\.. matched column four where value starts "5." then digit, so "5.95" qualifies (5 . 9). 53.97 also qualifies because it contains "5" at position 1 of "53.97"? Actually "53.97" is "5" + "3" + "." + "9"... The pattern 5\.. needs "5" then literal "." then any char. In "53.97", characters are '5','3','.','9' — there is no "5." adjacency, so it would not match. Lecture's "53.97" example therefore illustrates a different column interpretation; column four values that did match were those literally starting "5." such as 5.95, 5.678.

Visual intuition: sketch a line as a shelf from ^ (left end) to \$ (right end). Place label SAP glued to left bookend for ^SAP; bash glued to right for bash\$; \. as a dot sticker on a book cover vs . as a transparent field that can be any cover. Whole-word -w draws word-boundary fences (| non-word) around the pattern. The x-axis is character position; y is "must be at edge vs anywhere". Takeaway: moving ^ inside brackets flips its meaning from edge to negation.

12.5.1 Beginning-of-Line and End-of-Line Anchors

Anchors assert position without consuming characters. ^ outside brackets at pattern start anchors to beginning of line: ^SAP matches lines starting with SAP; on /etc/passwd grep "^SAP" /etc/passwd returns SAP1/SAP2, ^A returns Anita etc., ^U the Ubuntu entry. \$ anchors to end of line: bash\$ matches lines ending with bash, isolating /bin/bash interactive users (root, ubuntu, SAP1 etc.) while excluding nologin, and nologin\$ conversely lists non-interactive accounts. Both may combine (^pattern\$) for full-line matches such as ^\$ for empty lines.

12.5.2 Escaping Metacharacters

Because . * [ ^ \$ are special, a literal match requires \. Matching a decimal point requires \.: 5\. is the two-character sequence 5 + literal dot. In table.doc with columns NW Northwest 3.98 ..., grep "5\.." is read as 5 literal, \. literal dot, . any single char after the dot, thus matching strings starting 5. plus at least one following character (e.g., 5.95) found in column four — identified by Dinesh as the source — but not bare 5. Removing the trailing dot to 5\. means "word begins 5. and may continue", still column four but longer; grep -w "5\." requiring whole word 5. alone then yields no output because no word is exactly 5.. Students are reminded \ escapes the following char: \. literal dot vs . wildcard.

12.5.3 Whole-Word Matching and Extended Expressions

Whole word — -w was examined with North: without -w, grep North table.doc also matches Northwest/Northeast as substrings; with grep -w North only lines where North stands alone as a separate word qualify. Extended vs basic — basic regex uses the core set, while extended (-E) adds +, ?, alternation | and grouping () with richer repetition; the lecture listed -E for that and -F for fixed-string matching. File-based hunt — [A-Za-z]* matches any alphabetic string, [A-Za-z] alone one alpha; the count of consecutive class atoms controls leading characters, so grep "[A-Z][a-z]" table.doc after lowercasing returned only two lines with capital+lowercase.

12.6 Worked grep Examples — File Searches and Table Document

This is the hands-on core where the filter, options, and regex atoms are exercised together on two deliberately small files and one structured table. The aim is not to memorize outputs but to learn to trace a pattern line by line.

NW  Northwest   3.98  3 and 34
W   Western     53.97 5 and 23
SW  Southwest   2.78  2 and 18
S   Southern    5.95  6 and 23
SE  Southeast   2.99  6 and ...
NE  Northeast   3.77  63
N   Northern    4.96  5 ...
C   Central     5.678 3 and 45
... (values tweaked live)

cat grep_file
# line1: this is a case file.
# line2: This is ...
# line3: THIS ...
# line4: (empty)
# line5: (empty)
# line6: ThIs line has ...
# line7: another case line with file.
# (illustrative; lecture had 3 lowercase this lines)
cp grep_file demo_file

grep this grep_file
# → prints the 3 lines containing lowercase "this" as substring

grep THIS ./*          # shell expands ./* to all files in .
# demo_file:THIS ...
# grep_file:THIS ...
# grep: ./Desktop: Is a directory (skipped)
grep case *file
# demo_file: this is a case file.
# demo_file: another case line with file.
# demo_file: ...
# grep_file: (same 3 lines with prefix grep_file:)

grep this grep_file      # 3 lines
grep -c this grep_file   # 3

grep -i this grep_file   # 5 lines (adds This, THIS, ThIs)
grep -i -c this grep_file # 5   (also grep -ic)

grep This grep_file      # 1 (exact cap T, no -i)
grep -c line grep_file   # 3 (three lines hold word line)

grep -n this grep_file   # e.g., 1:this is a case file.  (only line 1 in case-sensitive)
grep -n -i this grep_file # 1:... 2:... 3:... 6:... 7:...

grep "^this" grep_file          # 1 line — only line starting lowercase this
grep -i "^this" grep_file       # 3 lines — folds This/THIS at start
grep "file\$" grep_file          # 0 — lines end "file." with trailing dot, so \$ fails
grep "file\." grep_file         # 2 — lines ending literal "file."
grep "^this.*file\.\$" grep_file # this at start, .* bridges spaces, literal file. at end → matches qualifying line(s)
grep "^this*file" grep_file     # 0 — star repeats s only, cannot bridge space (see Q&A)
grep -h "file" grep_file demo_file  # same lines without filename prefix (vs default with prefix)

grep "5\.." table.doc
# → lines where substring "5" + literal "." + one any char occurs
# S   Southern    5.95  ...   (5 . 9)
# C   Central     5.678 ...   (5 . 6)
# Dinesh identified column four as source because column four holds 5.95 style values; 53.97 would not match "5." adjacency

grep "5\." table.doc        # any "5." plus continuation
grep -w "5\." table.doc     # no output — no word is exactly "5." alone

grep "^[WE]" table.doc      # lines starting W or E → Western, Eastern entries (Rajesh correct)
grep "^[WE].*" table.doc    # same plus anything after

grep "^[0-9]" table.doc     # 0 — no line starts with digit
grep "[^0-9]" table.doc     # all lines — every line has at least one non-digit
grep "^[^0-9]" table.doc    # all lines — every line starts with non-digit (interface of two uses)

grep "[A-Z][A-Z][A-Z]" table.doc # three caps consecutive
grep -i "[A-Z][A-Z][A-Z]" table.doc # with -i all lines match (folded)
grep "[A-Z][a-z]" table.doc # after lowercasing northeast/southeast, only 2 lines retain Cap+lower at tested position

Visual intuition: show grep_file as 7 stacked horizontal strips, three highlighted for this (case-sensitive) and five for -i this. Overlay ^this as a left-edge highlighter touching only strip 1; file.\$ as right-edge touching strips ending with dot. For table.doc, draw columns as faint vertical guides but emphasize the regex scans left-to-right across the whole line, so 5\.. highlight falls in column four only where literal 5. occurs. Takeaway: the pattern's yes/no is per strip, not per column box.

12.6.1 Basic Searches with grep_file and demo_file

A text file grep_file was created by pasting multi-line content and duplicated via cp grep_file demo_file; both initially held identical lines with this, This, THIS, case, file, line, and empty lines. grep this grep_file returned the three lines containing lowercase this as substring, confirming three occurrences. A multi-file glob such as grep THIS ./* or grep case *file (where *file means any file name ending in file, matching grep_file and demo_file) searched both files, reported desktop entries as Is a directory and skipped them, and prefixed matches with file names; case contributed three matching lines per file with file names included.

12.6.2 Case-Sensitive versus Case-Insensitive Counts

Case sensitivity was contrasted step by step: grep this grep_file → three lines; grep -c this grep_file → 3; grep -i this grep_file → five lines including This, THIS, ThIs variants; grep -i -c this grep_file (or grep -ic) → 5; grep This grep_file (capital T, no -i) → 1; grep -c line grep_file → 3. Emphasis: -c counts lines containing at least one match, not total word occurrences — a line with multiple occurrences still counts as one.

12.6.3 Anchored and Composite Pattern Searches

Start anchor — grep "^this" grep_file matches lines beginning this, one line in the sample, demonstrating ^ at line start. End anchor — grep "file\$" grep_file vs grep "file\." grep_file contrasts end-of-line vs literal dot: file\$ alone yields no output because lines end file. with trailing dot; file\. succeeds on two lines ending file.. Both anchors — grep "^this.*file\.\$" grep_file requires this at start, .* bridging spaces, and file. at end; switching .* to * alone produces no matches because * without dot cannot bridge spaces, reinforcing the .* lesson. Case-insensitive anchored — grep -i "^THIS" or grep -i "^this" expands start-anchored to three lines. grep "file" locates substring anywhere, grep "^file" restricts to beginning.

12.6.4 Table Document Exploration and Classroom Q&A

A structured file table.doc built via cat > table.doc with rows:

NW  Northwest   3.98  3 and 34
W   Western     53.97 5 and 23
SW  Southwest   2.78  2 and 18
S   Southern    5.95  6 and 23
SE  Southeast   2.99  6 and ...
NE  Northeast   3.77  63
N   Northern    4.96  5 ...
C   Central     5.678 3 and 45
... (values adjusted live)

Columns: abbreviation, full name, decimal with dot, subsidiary integers.

Pattern walkthroughs: grep "5\.." table.doc matches 5 literal, \. literal dot, . any char after — column four source, so 5.95/5.678 rows match but bare 5 does not; Dinesh identified column four and reasoned that it holds 5 before the dot. Adding a 5. entry in another row made that row match, confirming per-line scanning and no column-position encoding. grep "5\." table.doc matches any word beginning 5. with continuation; grep -w "5\." yields no output because no word is exactly 5. alone. grep "^[WE]" table.doc matches lines starting W/E (Western, Eastern — Rajesh); grep "^[WE].*" extends with anything after. grep "^[0-9]" vs grep "[^0-9]" illustrates caret placement: the former prints nothing (no line starts digit), the latter shows all lines (every line has a non-digit). grep "[A-Z][A-Z][A-Z]" table.doc probes three consecutive capitals; after lowercasing northeast/southeast they are excluded, returning only fully capitalized abbreviations; after manipulations only two lines remain for [A-Z][a-z].

12.7 Superuser Status and Privileged Operations

Text selection is only useful if you are allowed to read the files — and many operations require more than reading. The session pivots from filters to the privilege model, demonstrated live on a local virtual machine via SSH because shared educational hosts intentionally withhold superuser rights.

date
# Sun Nov  6 10:15:00 UTC 2022   (read succeeds for anyone)

date -s "2022-11-05"
# date: cannot set date: Operation not permitted   (no sudo → denied)

sudo date -s "2022-11-05"
# [sudo] password for ubuntu:  (you authenticate as yourself)
# Sat Nov  5 00:00:00 UTC 2022   (mutation succeeds)

date
# Sat Nov  5 00:00:00 UTC 2022

sudo date -s "2022-11-06"
# Sun Nov  6 00:00:00 UTC 2022   (restored)

who
# ubuntu  pts/0  2022-11-06 09:00 (192.168.1.10)  ← SSH pseudo-terminal
# bits1   tty1   2022-11-06 09:05
id
# uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),27(sudo)
id bits1
# uid=1005(bits1) gid=1005(bits1) groups=1005(bits1)

Visual intuition: graph privilege as a vertical bar: bottom 0–50% regular user (can read own files, run filters), top 50–100% superuser (can mutate any file, clock, passwords). date (read) sits at 10%; date -s (write) sits at 90% with a gate; sudo is the elevator that temporarily lifts you, su is the room you enter. The x-axis is operations, y is required privilege. Takeaway: daily practice keeps you at the bottom; you ride up only for the one operation at the top.

12.7.1 su and sudo — Acquiring Elevated Privileges

su — switch user: su without arguments attempts to become root and prompts for the target account's password; su - or su root switches with superuser environment (home, \$USER, startup). sudo — execute one command as superuser, prompting for the invoking user's password rather than root's; sudo su switches to root after a sudo authentication, the idiom used in the demo, while sudo command runs exactly that command as superuser without a persistent switch. The spoken distinction — su changes the user, sudo grants extra privileges for one command — captures practice, and the prompt change from ubuntu@... to root@... confirms transition. A warning notes that on shared educational hosts sudo/su may fail for lack of granted privileges, hence the local VM.

12.7.2 Capabilities Available to the Administrator

Once superuser, the administrator comprehensively controls the system: modify any file's contents/permissions/ownership even in write-protected directories and delete any file; create or terminate any process; change any user's password without knowing the old one; broadcast via wall; set clock/calendar with date; limit file sizes with ulimit; control scheduling services at/cron; control networking tools FTP/SSH; manage hardware, network, and service state. This enumerates scope, not an invitation on unprivileged hosts.

12.7.3 Demonstration with Date Setting and Privilege Checks

The date command illustrates a privilege-gated mutation: date shows Sun Nov 6, sudo date -s "2022-11-05" sets to Sat Nov 5, date confirms, and sudo date -s "2022-11-06" restores. Without sudo, setting the clock is denied; with sudo, it succeeds. Additional visibility includes who (users with tty/pts devices) and id (UID, GID, membership).

12.8 User Management and Bulk Creation

Creating users is the first admin workflow where sudo pays for itself — and where the choice between one-by-one and bulk creation determines whether provisioning a cohort takes an afternoon or a second.

grep "^bits" /etc/passwd          # all bits accounts
grep -c "^bits" /etc/passwd       # count of matching users
grep "bash\$" /etc/passwd          # interactive shells
grep "nologin\$" /etc/passwd       # system accounts without interactive login
grep "^A" /etc/passwd             # Anita
grep "^U" /etc/passwd             # Ubuntu login

20210001:password:UID:GID::/home/20210001:/bin/bash
# compact form: login:password:UID:GID:GECOS:home:shell
# teaching variant: 20210001:pass:1001:1001::/home/20210001:/bin/bash

sudo newusers Users

grep "^2021" /etc/passwd      # 26 new accounts listed, each with x field, UID, shell
grep -c "^2021" /etc/passwd   # 26

for u in 20210001 20210002 ... 20210026; do
  sudo adduser \$u   # prompts for password, GECOS per user → ~26× interactive steps
done
# If each adduser takes ~30s of prompts, ~13 minutes plus inconsistency risk

cat Users
# 20210001:cohort2021:1101:1101::/home/20210001:/bin/bash
# 20210002:cohort2021:1102:1102::/home/20210002:/bin/bash
# ... 26 lines
sudo newusers Users
# <1 second, no per-user prompts
grep -c "^2021" /etc/passwd
# 26

sudo userdel bits4
su bits4
# su: user bits4 does not exist  (or Authentication failure)
ls -ld /home/bits4
# still exists because -r was not used

Visual intuition: draw a timeline left to right: single-user path has 26 sequential boxes (adduser → passwd → mkdir home repeated), total width long; bulk path has one tall box labeled newusers Users whose height is 26 accounts but width is one command. The y-axis is accounts created, x is wall time. The horizontal line at grep -c "^2021" = 26 is the checkpoint. Takeaway: batch collapses time without changing outcome, provided the input file format is correct.

12.8.1 Creating and Removing Single Users

adduser (Debian family, interactive) and useradd (lower-level) create a new account; the demo used both (sudo adduser bits1, sudo useradd bits2), each receiving a password via passwd-style prompts (bits1/bits1 teaching pattern). passwd changes a password, and superuser can change any account non-interactively. userdel removes an account — userdel bits4 with no extra prompt, after which su bits4 fails Login incorrect; without -r the home directory is preserved. Four demonstrators bits1–bits4 were created individually.

12.8.2 Structure of /etc/passwd

Inspected with grep to avoid paging, /etc/passwd has seven colon-separated fields: (1) username, (2) field x (real encrypted password in /etc/shadow), (3) UID, (4) GID (primary group), (5) GECOS/comment, (6) home directory, (7) login shell (/bin/bash or /usr/sbin/nologin). Examples: grep "^bits" /etc/passwd lists bits1–bits4 with x, UIDs, and shell; counting via grep -c "^bits", separating interactive grep "bash\$" vs nologin\$; ^A finds Anita, ^U finds Ubuntu. Auditing with ^/\$ is faster and safer than editing the file directly.

12.8.3 Bulk Creation with newusers

Creating dozens one by one is tedious and error-prone; newusers reads a text file mirroring passwd fields and creates them in one pass. The teaching file Users contained 26 entries prefixed 2021..., each line like 20210001:password:UID:GID::/home/20210001:/bin/bash (four or more colon values: login, password, UID, GID, home, shell). Execution sudo newusers Users completes in <1s; verification grep "^2021" /etc/passwd shows 26 accounts and grep -c "^2021" is 26. This contrasts with looping adduser, saving repetition and ensuring consistency, and the provisioned names earlier in the environment used this pathway.

12.9 Groups and Permission Delegation

Users exist; collaboration requires groups. This section turns individual accounts into teams and makes a shared document writable by that team — surfacing the classic trap where group membership alone is not enough.

sudo groupadd -g 999 AWS
sudo groupadd -g 1016 bits

grep "AWS" /etc/group      # AWS:x:999:...
grep "^bits" /etc/group    # bits:x:1016:...

sudo adduser bits1 AWS     # Debian idiom adds bits1 to supplementary group AWS
sudo adduser bits2 AWS
sudo gpasswd -a bits1 AWS  # alternate explicit form (add)
sudo gpasswd -d user group # remove user from group
sudo groupdel groupname    # remove group itself

sudo adduser anita
sudo adduser anup
# each in primary groups anita, anup

sudo groupadd -g 1016 bits

sudo adduser anita bits
sudo adduser anup bits
sudo adduser amazon bits

ls -l bits_doc
# -rw-r--r-- 1 amazon amazon  ... bits_doc   (group amazon before change)

sudo chgrp bits bits_doc   # or chgrp bits tempfile
ls -l bits_doc
# -rw-r--r-- 1 amazon bits  ... bits_doc   (group now bits, but mode still r-- for group)

sudo chmod g+w bits_doc    # or chmod 664
ls -l bits_doc
# -rw-rw-r-- 1 amazon bits  ... bits_doc   (group now rw-)

grep "^bits" /etc/group    # shows AWS:999 etc.
id bits1
grep "^bits" /etc/passwd   # cross-check users vs groups

groups anita
# anita : anita    (only primary before adding)
sudo adduser anita bits
groups anita
# anita : anita bits  → now in bits as secondary
ls -l bits_doc
# -rw-r--r-- 1 amazon bits 0 ... bits_doc

sudo -u anita bash -c 'echo "hi" >> bits_doc'
# bash: bits_doc: Permission denied

sudo chmod g+w bits_doc
ls -l bits_doc
# -rw-rw-r-- 1 amazon bits ... bits_doc
sudo -u anita bash -c 'echo "hi" >> bits_doc && echo success'
# success
cat bits_doc
# hi

sudo groupadd -g 999 AWS   # 999 >499 so allowed as user-defined, OK
sudo groupadd -g 400 mygrp # would collide with reserved 0-499 system range

Visual intuition: draw a Venn of users Anita, Anup, amazon overlapping in circle bits. Draw file bits_doc with a luggage tag labeled group: bits and a three-slot lock rwx | rwx | rwx. Before fix: tag says bits but lock's middle slot is r-- (read only). After chmod g+w: middle slot flips to rw-, overlap members now open the middle lock. X-axis is membership, Y is permission bits; takeaway: both dimensions must align for write.

12.9.1 Primary and Secondary Groups

Linux distinguishes two associations: Primary group — created with the user, every user belongs to at least one, typically sharing the username (bits1 in bits1), and new files default to it. Secondary (supplementary) groups — additional groups added after creation; membership grants that group's permissions on shared resources without changing primary affiliation, and files remain owned by primary unless chgrp/newgrp is used. This explains write failures despite membership: the effective file group was not the secondary until explicitly changed.

12.9.2 Structure of /etc/group and Group Creation

/etc/group parallels passwd with four colon fields: (1) group name (AWS, bits, Anita), (2) field x, (3) GID, (4) comma-separated member list. Example AWS:x:999:bits1,bits2,ubuntu with GID 999. IDs 0–499 reserved for system groups; user-defined groups should use ≥500 (lecture cutoff 499, safe example 999). Creation via sudo groupadd -g 999 AWS, groupadd -g 1016 bits; verification grep "AWS" /etc/group, grep "^bits" /etc/group initially shows empty member list.

12.9.3 Adding Users to Groups and Managing Membership

Membership shown as sudo adduser bits1 AWS (supplementary), adduser bits2 AWS, alternate gpasswd -a bits1 AWS, removal gpasswd -d user group, and groupdel groupname to remove group. After adding, grep AWS /etc/group shows bits1,bits2,ubuntu and id bits1 reflects supplementary group. Probes via grep passwd/group and id per user; manual file editing discouraged.

12.9.4 The bits Document Task, chgrp, and Write Access

Goal: Anita, Anup (and Amazon in interactive attempt) should write to bits_doc/tempfile via shared group bits. Steps: 1) sudo adduser anita, adduser anup (each in own primary), 2) sudo groupadd -g 1016 bits, 3) sudo adduser anita bits, anup bits, amazon bits, 4) ls -l bits_doc shows owner/group/permissions (before change group amazon), 5) sudo chgrp bits bits_doc, 6) chmod g+w bits_doc. Trap: after chgrp, ls -l showed r-- for group and amazon write denied; default rw-r--r-- gives group read not write, so secondary members could only read; permission triplet rwx for user-group-other means rw-rw-r-- after chmod allows group write. Membership alone insufficient — mode must grant it.

12.10 Session Awareness, Communication, and Workspaces

Administration ends with awareness: who is on the system, where they are attached, how to reach them, and how the system multiplexes simultaneous logins.

ubuntu  tty7   2022-11-06 09:00 (:0)      ← graphical desktop (see below)
bits1   tty1   2022-11-06 09:05
bits2   tty2   2022-11-06 09:06
bits3   tty3   2022-11-06 09:07
bits4   tty5   2022-11-06 09:08
bits5   tty?   ...
ubuntu  pts/0  2022-11-06 09:00 (192.168.1.10)  ← SSH from instructor laptop

wall
Hello everyone. System shutdown in 5 minutes. Save your work!
# Ctrl-D to send EOF

Broadcast Message from ubuntu@vm (pts/0) at 10:20 ...
Hello everyone system shutdown in 5 minutes. Save your work!

# From graphical terminal (tty7), switch: Ctrl+Alt+F1, login bits1
# Ctrl+Alt+F2, login bits2
# Ctrl+Alt+F3, login bits3
# Ctrl+Alt+F7 back to desktop
who
# bits1  tty1  ...
# bits2  tty2  ...
# bits3  tty3  ...
# ubuntu pts/0 ... (your SSH)
# ubuntu tty7 ... (graphical)
tty
# /dev/pts/0   ← current shell is SSH pseudo-terminal

wall <<'MSG'
Hello everyone. System shutdown in 5 minutes. Save your work!
MSG
# Observe on tty1: Broadcast Message from ubuntu@vm (pts/0) ...
# Same on tty2, tty3, pts/0

who -a   # -a shows all, with idle and pid
who am i # owner of current terminal
id       # confirms uid/gid/groups for privilege context

ifconfig eth0 down    # simulated hiccup → pts/0 may freeze/disconnect
# Local tty1–tty3 remain responsive; re-enable with `sudo ifup eth0` from tty1

Visual intuition: picture a building floor plan: six rooms labeled tty1–tty6 along a corridor, one executive suite tty7 (desktop), and a phone switchboard with extensions pts/0–pts/N for remote callers. who is a roster board at reception with pins for each occupant. wall is a loudspeaker icon with arrows to every room and extension. The x-axis is device, y is user; takeaway: sessions multiplex in space (room vs extension), not by disconnecting others.

12.10.1 who, tty, and pts — Local versus Remote Sessions

Linux tracks attachment: who reports each logged-in user, device, and login time — e.g., ubuntu and bits1–bits4 on tty1–tty5 plus ubuntu on pts/0/pts/1 via SSH. tty devices (tty1–tty6) are local virtual consoles on the machine's keyboard/display; pts (pseudo-terminal slave) devices are remote logins such as the SSH connection from the instructor's laptop, appearing as pts/... distinct from tty. id and whoami complement who by showing effective user and group for the current shell.

12.10.2 Broadcasting with wall

wall — write to all — broadcasts immediately to every logged-in terminal, demoed as:

wall
Hello everyone. System shutdown in 5 minutes. Save your work!
# Ctrl-D

After execution, Hello everyone system shutdown in 5 minutes. Save your work! appeared on tty1, tty2, tty3, and the originating terminal, bannered inline. It requires no recipient list; it delivers to all who entries by design.

12.10.3 Linux Virtual Consoles and the Graphical Session

A Linux host offers multiple simultaneous logins: Ctrl+Alt+F1–F6 switch to virtual consoles tty1–tty6 with text login prompts (demo logged bits1 on F1, bits2 on F2, bits3 on F3, bits4 on F4, who confirmed), and Ctrl+Alt+F7 (on some distros F1/F2 is graphical; in this VM F7 was desktop) returns to the graphical session reported as tty7 in who. This explains why who shows one human on several devices — each function-key console is an independent login. Network hiccups such as a lost ifconfig IP after a service hiccup were handled by persisting on local tty consoles and retrying network restart, illustrating that tty sessions persist while pts SSH sessions may drop if the network changes. Supplementary commands exit, clear, passwd, ulimit remain reachable from any virtual console.

Exam Guidance Summary

This section consolidates the examinable decisions and prediction skills emphasized across the lecture — the fine-grained distinctions that map directly to short-answer and output-prediction questions.

• Assignment and team sizing (12.1): Three members per group is the design target and the evaluation assumption. Workload division, program allocation freezing, and preparation time all assume a trio. Smaller groups (at most one group of two permitted) must negotiate to reach three; delayed formation directly reduces time for the command-heavy scripts that follow. Expect program-allocation timing and team-formation rationale as short-answer prompts.

• Shell programming through commands (12.1–12.2, 12.8–12.9): Shell scripting is assessed via purposeful use of system commands inside scripts, not isolated pattern printing. Be prepared to embed command results (grep counts, id output, /etc/passwd probes) inside loops/conditionals, and to justify why a particular command or option (grep -c vs grep -o | wc -l, adduser vs newusers) fits the file-processing step. Illustrative vs assessed tasks are a common conceptual contrast.

• grep reporting options (12.3): The distinctions between -c (count lines, not words), -l (list file names only) vs default lines vs -h (suppress names) vs -o (only matching substring), and -w (whole word) vs substring vs -i (case-fold) vs -v (invert) are directly testable. Practice how -i, -v, -n compose and which flag dominates output shape (e.g., grep -i -v -c vs grep -c). The table-document and grep_file counts (3 → 5 with -i, 4 → 2 inverted with -i) are prototypical output-prediction exercises.

• Regular-expression mechanics (12.4–12.6): Be ready to trace a given pattern on a provided file and explain why a variant differs:
• .* vs * — .* bridges spaces including empty; * repeats only the preceding atom (this*file fails to bridge space, this.*file succeeds).
• \. vs . — literal dot vs wildcard (so 5. ≠ 5\. ≠ 5\..).
• ^ outside vs ^ inside [^...] — anchor (beginning of line) vs negation (not in set), e.g., ^[0-9] (line starts digit, matches nothing in table.doc) vs [^0-9] (non-digit, matches all lines).
• Character classes [A-Z], [A-Za-z]*, shorthands \d/\D/\w/\W/\s, and consecutive atoms [A-Z][A-Z][A-Z] controlling required length.
• Escaping for literal \. when grepping decimals, IPs, versions.

• Administrative distinctions (12.7–12.10): Conceptual comparisons and scenario tasks are frequent:
• su (switch user, needs target password, su - takes login environment) vs sudo (one command as superuser, needs invoker password, sudo su idiom).
• adduser/useradd (one bespoke account, interactive vs need -m/-s) vs newusers (batch of 26 from colon-separated file, sub-second) vs userdel (-r nuance, home preservation).
• /etc/passwd seven fields and bash\$ vs nologin\$ interactive audit, /etc/group four fields and GID ≥500 rule.
• Primary (default file group, one per user) vs secondary (supplementary, many) and the two-step sharing chgrp bits bits_doc + chmod g+w where membership alone is insufficient — the Anita/bits_doc scenario.
• wall broadcast semantics (all who entries, no recipient list) vs write to a single terminal.
• who device reading: tty1–tty6 local virtual consoles (Ctrl+Alt+F1–F6) vs pts/0 SSH pseudo-terminals vs tty7 graphical (Ctrl+Alt+F7 on this VM) and why pts drops on network hiccup while tty persists.

• Study advice: For each filter or regex problem, write the pattern atom by atom, state what each atom consumes or asserts, then simulate it line by line on the given file content — especially noting line-start/end and literal-dot adjacencies. For admin scenarios, draw the two-axis check: group membership (who) × file mode (rwx) — both must align.

Key Industry Applications

• grep as triage for large text datasets (12.2–12.6): In production, grep is the first filter for log forensics (grep -i ERROR app.log | grep -v -i timeout), config auditing (grep "^PermitRootLogin" /etc/ssh/sshd_config, grep "^bits" /etc/passwd, grep "bash\$" /etc/passwd vs grep "nologin\$" to separate interactive from system accounts), and pipeline shaping (ps aux | grep apache, ls | grep -v temp, cat mail | grep "^From"). Literal-dot escaping (grep "192\.168\." auth.log for local IPs, grep "5\." data.csv for decimals, grep "1\.2\." CHANGELOG for version strings) prevents wildcard over-matching where a dot must be literal, and anchors ^/\$ validate line-oriented formats (e.g., grep "^ERROR:.*\$"). Combined with grep -c/-l/-o and downstream sort | uniq -c | awk, it turns thousands of lines into actionable counts before heavier tools are invoked.

• System administration with superuser delegation (12.7): The su/sudo model governs all privileged operations in operations teams — restarting services (sudo systemctl restart nginx), mounting datasets (sudo mount), and changing clocks or passwords. The discipline taught (regular user by default, sudo one command at a time, exit promptly, verify with id/whoami) is the production safety habit that prevents rm -rf catastrophes noted in administrative texts. Auditing with who/id/grep on /etc/passwd and /etc/group forms the basic access-review loop for compliance.

• Batch user and group provisioning for team access (12.8–12.9): Semester-start cohort creation (newusers from an HR export of 26 2021... accounts in one sudo newusers Users run, verified with grep -c "^2021" /etc/passwd) is the standard alternative to error-prone adduser loops, and the same pattern scales to enterprise onboarding and lab fleet setup. Managing groups (groupadd -g 1016 bits, gpasswd -a anita bits, chgrp bits /srv/shared, chmod g+w / chmod 2775 with setgid so new files inherit the shared group) implements team-based access for shared directories, CI artifacts, and submission drop-boxes. The Anita/bits_doc trap — membership without g+w is read-only — is the canonical lesson that group membership (who) must be paired with file mode (what) and verified via ls -l and id.

• Session and communication awareness (12.10): Interpreting who (tty1–tty6 local virtual consoles on Ctrl+Alt+F1–F6 vs pts/0 remote SSH pseudo-terminals vs tty7 graphical) lets on-call engineers know whether a suspected session is local or remote during incident response (who | grep pts to isolate SSH users). Broadcasting urgent notices with wall before maintenance (wall <<'MSG' System shutdown in 5 minutes ...) remains the quickest in-band warning to all interactive sessions before shutdown. Understanding that local tty sessions persist while pts SSH sessions drop on network/IP changes explains why console-based recovery (Ctrl+Alt+F1) is the fallback when remote access fails, and how init respawns getty/login/shell per /etc/inittab keeps other consoles alive.

• Glue-role synthesis: The lecture's overarching message — shell scripts are practical when they orchestrate filters like grep for meaningful administration — is the daily reality of platform engineering: a five-line script that greps logs for error signatures, counts with -c, and mails or walls a report replaces hours of manual inspection, and the same ^/\$/\. pattern skills transfer unchanged to validating data files, auditing accounts, and gating automated pipelines.