Vim Tip of the Day: Aggregate Operations

Having returned from that one-dimensional land of Ex, we now know that it is another way of looking at our document, as a collection of lines, which is different from Vim’s usual (more fine-grained, multifaceted) way of looking at it. We have also seen hints that this linear perspective could make it easier for us to make large-scale, sweeping changes to our document, changes that might take us a lot longer to do in Normal mode, if it is even feasible at all. In this tip, we will look at a few such categories where Ex mode is simply the right tool for the job.

As we know that typing : in Vim momentarily takes us to Ex to execute a command, we will leave out the : prefix in this tip, and it should be understood that these commands would need a : prefix if executed while in normal mode. Additionally, all of these commands can be prefixed with any line range (such as, commonly, % for “all lines”) to apply the command only to those lines. This line range can even be indicated visually using Visual Line selection prior to hitting :, which is a very common and useful way to do it. We will therefore also omit line ranges from the commands below, so that for instance, if you see s/old/new/g, you might execute an instance of it as :% s/old/new/g (with the space here being optional).

Batch Processing Like It’s ’89

Whenever you want to do something to “all of these” lines or “for all” instances that match some criteria, you are looking to do a batch operation. This is accomplished by either (1) prefixing an Ex command (like s) with a line range (like %, or implicitly via a visual selection), or (2) by using g (or v) to identify the lines using pattern-based selection instead, or both. Here are some common batch operations:

• Search and replace: s/old/new/g
  • You can also specify additional parameters here, such as case (i)nsensitivity, and prompting to (c)onfirm each change, etc.
  • Recall that the perhaps unfortunately-named parameter g used here is unrelated to the g Ex command featuring in the remaining examples, except that it also suggests “global,” that all matches (on the line) are to be replaced.
• Delete all lines matching or not matching a pattern: g/re/d or v/re/d
  • e.g. g/^$/d deletes all empty lines
  • Remember that Ex mode’s v is the in(v)erse of g and has nothing to do with visual selection in Normal mode
• On all matching lines, substitute old with new: g/re/s/old/new/g
  • g can be composed with any Ex command, and in this instance it is composed with s

Since Vim defaults to using the last search term in all regex-based commands if you don’t specify one, an efficient way to start your search-based batch operation is to hit * or # on a word of interest (which we learned about in Search) and then run the batch command as something like:

g//d or s//new/g

If you like grep, you’ll love grevrep! Since g and v are commands that identify a selection of lines on which we may apply any command, we may use g or v themselves as the command to apply, which has the effect of narrowing the selection further. We can think of grevrep as an Ex “phrase” which means, “for all lines matching this but not matching that, do this.”

g/re/v/re/

And of course, you can use any command on the final selection and not only p.

“Multiple Cursors”

Multiple cursors have become all the rage ever since Sublime Text introduced (or at least popularized) them a few years ago. While the paradigm is certainly fascinating and perhaps still underutilized, there’s an alternative to it that comes out of the box with Vim. It is the norm(al) Ex command, which allows you to enter normal mode commands in Ex mode. Just think about that — normal mode is an Ex command. This means that you can compose normal mode commands with Ex mode addressing, an extremely simple notion that is, partly for that very reason, incredibly useful. One may even call it … sublime. 🥁

If, say, you wanted to delete trailing whitespace on a line, you might use the “delete trailing whitespace” phrase g_lD that we learned earlier. Now to do it on a selection of lines (such as all lines), it’s simply:

norm g_lD

With multiple cursors, you may also like to place cursors on every line matching a pattern instead of indicating a contiguous selection visually. In Vim you would use norm with g to accomplish the same thing. For instance, if you want to duplicate every line in the file that matches a pattern, try:

g/re/ norm yyp

which simply uses the “duplicate line” phrase that we already know. For more complicated operations, you could define a macro to do it on a single line, and then apply that macro globally using norm, the same as you would any other normal command.

Another example, if you want to collapse all contiguous empty lines in a file to single lines:

g/^$/norm dipO (or use ^\s*$ to also match blank lines in addition to empty lines. And that’s an “O” as in “Oscar”)

By virtue of the expressiveness of regexes and the power of Vim’s normal mode, there are many things you can do with Ex’s “multiple cursors” (i.e. norm in combination with visual selection and g) that you cannot do with multiple cursors in other editors.

But in a sense, if Vim can do some things better than other editors, that’s not very interesting. We already know that. The interesting question is, is multiple cursors truly a different paradigm, one whose wholehearted inclusion in Vim could make Vim even better, even more expressive? And I don’t mean norm which operates on batches sequentially. I mean cursors, operating independently with isolated and namespaced views of the world, each completely unaware of the others and each in full possession of all of the machinery of Vim — change histories, registers, marks, everything — cursors as first-class citizens. There is an undeniably intuitive quality to multiple cursors as a natural generalization of the cursor-based editing experience one is already familiar with. But beyond this intuitive quality, it has the other benefit of arbitrary placement. We can place such cursors anywhere, based on match criteria or not, for any reason or no reason — it’s easy. I am not aware of a way to do this with norm. There are plugins that provide multiple cursors in Vim, such as this one, but I admit I haven’t tried them. If you have or if you decide to, share your experience in the comments!

The Sieve of Ex-osthenes

The move command (m) is handy for sifting through a document and collecting its lines into distinct piles based on whatever criteria we like. For instance, if you’d like to divide your file into two distinct parts based on some pattern, use:

g/re/m$

This says, “for every line matching this pattern, move it to the bottom of the file.” Since pattern-based batch operations conducted using g (or v) operate sequentially, we cannot use 0 instead of $ here, because each successive matching line would be placed at the beginning of the file in sequence and this would have the effect of reversing the order of the lines (while still accomplishing the partitioning into two sets). This feature is handy, though, when reversing the order of some lines is just what we want to do, for instance:

g/^/m0

… reverses the entire file.

In using m as a “sieve,” we can divide the file into as many sets as we like, not just two. We simply repeat the same sieve operation on each created partition, using the last line of that partition instead of $, to further subdivide it into two, and so on.

The Ultimate Verb

Here’s the thing about Vim. Since most schemes compose with one another (e.g. Normal nouns and verbs, visual selection and normal commands, Ex addresses and Normal commands, Ex addresses and Ex commands, Ex global and local commands …) and aren’t just one-offs, each new thing you learn pays compounding dividends. Learning about regexes sounds boring, until you realize it can help you be more efficient at pretty much everything you do when you use Vim. Ex commands, there be dragons! But wait, you can use all the normal commands you already know in Ex mode too, and it’s magic! Where does the composition end? It turns out that it doesn’t, since it extends beyond the frontiers of Vim itself, into the broader shell environment.

Vim integrates with the shell via the Ex commands w (write), r (read), and ! (transform, or “bang”).

We already know that Vim allows you to write to a file on disk using w. But w can also write input to shell commands, which we can indicate via the prefix !. For instance, to find out how many characters, words, and lines are in the document,

w !wc (note the space! w! without the space means overwrite file, which you wouldn’t want to do accidentally)

And this accepts a line range or visual selection like any Ex command (e.g. “Translate this selection into Spanish,” if you happen to have a language translation shell utility). Likewise, the r command allows you to read the contents of any file into the current document. For instance,

r otherfile.txt

… inserts the contents of otherfile.txt beginning at the current line.

And just like w, r can also read in the output of a shell command.

r !date

… inserts the system date at the current line.

Finally, ! in normal mode allows you to use any shell command as a verb! For instance, !ip prompts you to enter a shell command to use to transform a paragraph. Although we can use it this way, the truth is, ! is actually implemented as an Ex command. This means that you can’t use it on individual words or the contents of delimiters the way you can other normal verbs. ! operates only on full lines and sets of lines, even if you provide it something smaller. As with any normal mode verb, repeating it twice, !!, operates on the current line.

As a verb, ! can be used with any shell command that accepts text input and produces text output. As text is the primary I/O paradigm in Unix, this includes a lot of built-in shell utilities, and it also includes any such executable program written by you or anybody else. Here are some examples using handy built-in Unix utilities (equivalents likely exist for non-Unix platforms, too):

!sort — sort lines alphabetically
!fold -s -w WIDTH — wrap lines to WIDTH columns
!cut -d ' ' -f1-2 — retain the first two columns on lines containing columns of text separated by spaces
!column -ts, — format (in this case, comma-separated) text as a table with aligned columns
!nl — number the lines

Note that these “bang” commands must be prefixed with a line range (or executed in the context of a visual selection) in order to function as verbs or transformations of text. Otherwise, they simply execute the shell command without passing it any input, and the output is simply printed rather than inserted into the buffer. When used as a verb, ! is roughly a composition of w followed by r, that is, writing the text as input to a shell command and then reading back the output to replace the original.

When you know a way to do something with a shell utility and don’t immediately know how to do it in Vim natively, don’t hesitate to use !.