Text: sed
Sed is a stream editor. Input flows into the sed program via stdin, is manipulated according to the commands provided, and output flows to stdout. By default, the files used as input to sed are not altered.
Everything you can do via sed could be accomplished using an editor. The power of sed comes from being able to package your edits as a script and perform those edits across a large number of files, or perhaps even a single file if this is a common or complex edit. You will find yourself building your own library of Sed scripts as they are just so damn useful when you need them. Sed can also work effectively on large files where interactive editors can struggle.
As with much of this site, this is not the place to learn sed. It’s designed as a quick refresher. If you are really looking for a shortcut, the sed cookbook might be a better place to look. For deeper learning check out the resources section below.
Resources
sed & awk, 2nd Edition, by Dougherty and Robbins, 1977. An old, but still relevant and useful book covering both sed and awk.
GNU sed manual, the GNU Software Provides a great manual for sed.
$ man sed
, the good old man pages are a fantastic resource for sed. We often
forget just how much effort the author’s put into the man pages.
Sed versions
There are many versions of sed. The good news is that they all operate in essentially the same way. But, there might be subtle differences in features, flags, and implementation of things like regular expressions. Make sure to read the man pages of the version you have installed.
Running sed
sed [-n] -e 'commands' inputfiles
sed [-n] -f scriptfile inputfiles
Use the -e
flag to include sed commands on the command line itself, and the
-f
flag to load commands from a script. You can combine multiple command and
script inputs and they will be combined in order. I avoid that, if things are
getting that complex it’s probably time for a different tool (awk or a scripting language).
The -n
flag is used to restrict output to only lines specified with the p
command or p flag.
You can supply multiple input files and use standard command line file globing. This is very common as you are often making edits over a set of files.
Sed scripts
Sed scripts are often invoked from a shell wrapper.
Sed command structure and basic processing
[address[,address]][!]command [arguments]
A script consists of a number of sed commands. Each command may be a block of commands contained within braces. Commands from multiple sources are combined in order.
Processing goes like this:
- Each input line is copied to what is known as the pattern space.
- If no address is supplied then the command is executed.
- If the address matches the pattern space then the command is executed.
- Subsequent commands in the block are applied against the changed line the in pattern space.
- All commands are applied in order to each line of input before moving on to the next line.
- The pattern space is printed (the
-n
flag overrides this default behaviour). - The original file is unchanged, a copy of the file is created.
Pattern space
The pattern space is a buffer where the input line is held whilst the commands are applied.
Hold space
Addressing
Addressing is used to specify which lines of input the commands will be executed against. An address can be one or two line numbers or patterns. Patterns are described using regular expressions enclosed in slashes.
[n[,n]][!]command [arguments]
[/pattern/[,/pattern/]][!]command [arguments]
Addressing rules:
- If no address is present the command is applied to all lines.
- If one number is present then the command is only applied to that line.
- If two numbers are present, the command is applied to all lines starting at the first line number and ending at the last line number.
- If one pattern is present the command will be applied to all lines matching that pattern.
- If two patterns are present the command is applied starting at the first line that matches the first pattern and ending at the first subsequent line that matches the second pattern (note that some commands only work with one address).
- The shebang character,
!
, following the address indicates to apply the command to lines that do not match the address.
Grouping
Commands can be grouped using braces ({}
). This enables you to apply multiple
commands at the same address. You can also nest addresses such the address only
applies within the original address scope.
[/pattern/[,/pattern/]]{
/pattern/command
command
command
}
Regular Expression Basics
Below is a quick summary of the common regular expression use cases in sed to get you moving.
. matches any single character (except newline)
* matches any number of the character that comes before it
[..] matches any of the class of characters in brackets
^ matches the beginning of the line
$ matches the end of the line
\ escapes special characters.
There is, of course, far more you can do with regular expressions. Head over here for a fuller discussion.
Commands
There are something like 25 sed commands. We are only covering the basics here, enough to get you going after a sed hiatus. You can do some pretty amazing stuff with sed, but to get deeper you will need to leverage the resources mentioned above.
Single line addressing
Some commands only operate on a single line rather than a range of lines. Where
that is the case [line-address]
is used in describing the command syntax.
Comments
If the first line starts with a #
it is a comment line. The original sed
versions only allowed a comment on the first line. It “seems” that this
restriction has been lifted, but the documentation about the various sed
versions can be confusing. To be safe, I only put comments on the first line.
Substitution : s
[address]s/pattern/replacement/flags
Flags:
n replace the nth occurence of the pattern
g replace all occurances of the pattern
p print the pattern space
Substitute the text matching the pattern with the replacement text.
Any character can be used as the delimiter. Thus, if you have slashes in your
pattern or replacement you could use another character as the delimiter. If your
delimiter does appear in your regular expression use the backslash (\
) to
escape it.
Delete : d
[address]d
Delete lines that match the address.
Example: Delete all blank lines
/^$/d
Example: Delete lines 2 through 4 inclusive
2,4d
Once the delete command is applied no more commands are executed (there is no pattern space). The next line of input is read in.
Append : a
[line-address]a\
text to add
The append command places test after the current line. The text must begin on the next line.
Example: Insert heading line
/^heading/a\
text to add\
can be multiple lines
Insert : i
[line-address]i\
text to add\
can be multiple lines
The insert command places text before the current line. The text must begin on the next line.
Example: Insert heading line
/^heading/i\
# The Heading
Change : c
[address]c\
text to add\
can be multiple lines
The change command replaces the current line. The text must begin on the next line.
Example: Add real address
/<name>/s\ 123 Whatever Street,\ Somewhere,
USA, 11111
Note that you can specify multi-line text by using the backslash
(\
) character at the end of the line.
Print : p
[address]p
The print command outputs the contents of the pattern space. Use the -n
flag
to avoid duplicate output.
Note that this is different from the substitution commands print option as the substitution command only prints if there us a successful substitution.
Print line number : =
[line-address]=
Prints the line number of the matched line. Only works for single line
addresses. Use the -n
flag to prevent also printing the line itself.
Next : n
[address]n
The next command will put the contents of the pattern space and then get the next line of input without starting at the top of the script (the script continues execution from where it is, but executing against a new line of input).
Quit : q
[line-address]q
The quit command will cause sed to stop reading input lines.