Text

Text: sed

Sed is a stream editor. Input flows into the sed program via stdin, is manipulated according to the commands provided, and output flows to stdout. By default, the files used as input to sed are not altered.

Everything you can do via sed could be accomplished using an editor. The power of sed comes from being able to package your edits as a script and perform those edits across a large number of files, or perhaps even a single file if this is a common or complex edit. You will find yourself building your own library of Sed scripts as they are just so damn useful when you need them. Sed can also work effectively on large files where interactive editors can struggle.

As with much of this site, this is not the place to learn sed. It’s designed as a quick refresher. If you are really looking for a shortcut, the sed cookbook might be a better place to look. For deeper learning check out the resources section below.

Resources

sed & awk, 2nd Edition, by Dougherty and Robbins, 1977. An old, but still relevant and useful book covering both sed and awk.

GNU sed manual, the GNU Software Provides a great manual for sed.

$ man sed, the good old man pages are a fantastic resource for sed. We often forget just how much effort the author’s put into the man pages.

Sed versions

There are many versions of sed. The good news is that they all operate in essentially the same way. But, there might be subtle differences in features, flags, and implementation of things like regular expressions. Make sure to read the man pages of the version you have installed.

Running sed

sed [-n] -e 'commands' inputfiles
sed [-n] -f scriptfile inputfiles

Use the -e flag to include sed commands on the command line itself, and the -f flag to load commands from a script. You can combine multiple command and script inputs and they will be combined in order. I avoid that, if things are getting that complex it’s probably time for a different tool (awk or a scripting language).

The -n flag is used to restrict output to only lines specified with the p command or p flag.

You can supply multiple input files and use standard command line file globing. This is very common as you are often making edits over a set of files.

Sed scripts

Sed scripts are often invoked from a shell wrapper.

Sed command structure and basic processing

[address[,address]][!]command [arguments]

A script consists of a number of sed commands. Each command may be a block of commands contained within braces. Commands from multiple sources are combined in order.

Processing goes like this:

Pattern space

The pattern space is a buffer where the input line is held whilst the commands are applied.

Hold space

Addressing

Addressing is used to specify which lines of input the commands will be executed against. An address can be one or two line numbers or patterns. Patterns are described using regular expressions enclosed in slashes.

[n[,n]][!]command [arguments]

[/pattern/[,/pattern/]][!]command [arguments]

Addressing rules:

Grouping

Commands can be grouped using braces ({}). This enables you to apply multiple commands at the same address. You can also nest addresses such the address only applies within the original address scope.

[/pattern/[,/pattern/]]{
    /pattern/command
    command
    command
}

Regular Expression Basics

Below is a quick summary of the common regular expression use cases in sed to get you moving.

.    matches any single character (except newline)
*    matches any number of the character that comes before it
[..] matches any of the class of characters in brackets
^    matches the beginning of the line
$    matches the end of the line
\    escapes special characters. 

There is, of course, far more you can do with regular expressions. Head over here for a fuller discussion.

Commands

There are something like 25 sed commands. We are only covering the basics here, enough to get you going after a sed hiatus. You can do some pretty amazing stuff with sed, but to get deeper you will need to leverage the resources mentioned above.

Single line addressing

Some commands only operate on a single line rather than a range of lines. Where that is the case [line-address] is used in describing the command syntax.

Comments

If the first line starts with a # it is a comment line. The original sed versions only allowed a comment on the first line. It “seems” that this restriction has been lifted, but the documentation about the various sed versions can be confusing. To be safe, I only put comments on the first line.

Substitution : s

[address]s/pattern/replacement/flags

Flags:
 n  replace the nth occurence of the pattern
 g  replace all occurances of the pattern
 p  print the pattern space

Substitute the text matching the pattern with the replacement text.

Any character can be used as the delimiter. Thus, if you have slashes in your pattern or replacement you could use another character as the delimiter. If your delimiter does appear in your regular expression use the backslash (\) to escape it.

Delete : d

[address]d

Delete lines that match the address.

Example: Delete all blank lines

/^$/d

Example: Delete lines 2 through 4 inclusive

2,4d

Once the delete command is applied no more commands are executed (there is no pattern space). The next line of input is read in.

Append : a

[line-address]a\
text to add

The append command places test after the current line. The text must begin on the next line.

Example: Insert heading line

/^heading/a\
text to add\
can be multiple lines

Insert : i

[line-address]i\
text to add\
can be multiple lines

The insert command places text before the current line. The text must begin on the next line.

Example: Insert heading line

/^heading/i\
# The Heading

Change : c

[address]c\
text to add\
can be multiple lines

The change command replaces the current line. The text must begin on the next line.

Example: Add real address

/<name>/s\ 123 Whatever Street,\ Somewhere,
USA, 11111

Note that you can specify multi-line text by using the backslash (\) character at the end of the line.

[address]p

The print command outputs the contents of the pattern space. Use the -n flag to avoid duplicate output.

Note that this is different from the substitution commands print option as the substitution command only prints if there us a successful substitution.

[line-address]=

Prints the line number of the matched line. Only works for single line addresses. Use the -n flag to prevent also printing the line itself.

Next : n

[address]n

The next command will put the contents of the pattern space and then get the next line of input without starting at the top of the script (the script continues execution from where it is, but executing against a new line of input).

Quit : q

[line-address]q

The quit command will cause sed to stop reading input lines.

Tools

Web

Languages

Data