• Next chapter
• Previous chapter
• Table of contents

Chapter 5: Substitutions

This chapter will appeal above all to people who are excited by the fact that

  print ${array[(r)${(l.${#${(O@)array//?/X}[1]}..?.)}]}

For those who don't do a lot of function writing and spend most of the time at the shell prompt, the most useful section of this chapter is probably that on filename generation (i.e. globbing) at the end of the chapter. This will teach you how to avoid wasting your time with find and the like when you want to select files for a command.

5.1: Quoting

I've been using quotes of some sort throughout this guide, but I've never gone into the detail. It's about time I did, since using quotes is an important part of controlling the effects of the shell's various substitutions. Here are the basic quoting types.

5.1.1: Backslashes

The main point to make about backslashes is that they are really trivial. You can quote any character whatsoever from the shell with a backslash, even if it didn't mean anything unquoted; so if the worst comes to the worst, you can take any old string at all, whatever it has in it --- random collections of quotes, backslashes, unprintable characters --- quote every single character with a backslash, and the shell will treat it as a plain string:

  print \T\h\i\s\ \i\s\ \*\p\o\i\n\t\l\e\s\s\*\ \ 
      \-\ \b\u\t\ \v\a\l\i\d\!

However, zsh has an easier way of making sure everything is quoted with a backslash when that's needed. It's a special form of parameter substitution, just one of many tricks you can do by supplying flags in parentheses:

  % read string
  This is a *string* with various `special' characters
  % print -r -- ${(q)string}
  This\ is\ a\ \*string\*\ with\ various\ \`special\'\ characters

The flag is not very useful there, because zsh usually (remember the GLOB_SUBST option?) doesn't do anything special to characters from substitutions anyway. Where it is extremely useful is if you are going to re-evaluate the text in the substitution but still want it treated as a plain string. So after the above,

  % eval print -r -- ${(q)string}
  This is a *string* with various `special' characters

You can strip off quotes in parameters, too; the flag (Q) does this. It doesn't care whether backslashes or single or double quotes are used, it treats them all the way the shell's parser would. You only need this when the parameter has somehow acquired quotes in its value. One way this can happen is if you try reading a file containing shell commands, and for this there's another trick: the (z) flag splits a line into an array in the same way as if the line had been read in and was, say, being assigned to an array. Here's an example:

  % cat file
  print 'a quoted string' and\ another\ argument
  % read -r line <file
  % for word in ${(z)line}; do
  for> print -r "quoted:    $word"
  for> print -r "unquoted:  ${(Q)word}"
  for> done
  quoted:    print
  unquoted:  print
  quoted:    'a quoted string'
  unquoted:  a quoted string
  quoted:    and\ another\ argument
  unquoted:  and another argument

5.1.2: Single quotes

The only thing you can't quote with single quotes is another single quote. However, there's an option RC_QUOTES, where two single quotes inside a single-quoted string are turned into one. Apparently `RC' refers to the shell rc which appeared in plan9; it seems to be one of those programmes that some people get fanatically worked up about while the rest of us can't quite work out why. Zsh users may sympathise. (This was corrected by Oliver Kiddle and Bart Schaefer after I guessed incorrectly that RC stood for recursive, although you're welcome to think of it that way anyway. It doesn't really work for RC_EXPAND_PARAM, however, which is definitely from the rc shell, and if you look at the source code you will find a variable called `plan9' which is tested to see if that option is in effect.)

You might remember something like this from BASIC, although in that case with double quotes --- in zsh, it works only with single quotes, for some reason. So,

  print -r 'A ''quoted'' string'

The standard way of quoting single quotes is to end the quote, insert a backslashed single quote, and restart quotes again:

  print -r 'A '\''quoted'\'' string'

You can make parameter flags produce strings quoted with single quotes instead of backslashes by doubling the `q': `${(qq)param}' instead of `${(q)param}'. The main use for this is that the result is shorter if you know there are a lot of special characters in the string, and it's also a bit more easy to read for humans rather than machines, but usually it gains nothing over the other form. It can tell whether you have RC_QUOTES set and uses that to make the string even shorter, so be careful if you might use the resulting string somewhere where the option isn't set.

5.1.3: POSIX quotes

There's a relative of single quotes which uses the syntax $' to introduce a quoted string and ' to end it; I refer to them as `POSIX quotes' because they appear in the POSIX standard and I don't know what else to call them; `string quotes' is one possibility, but sounds a bit vague (what else would you quote?) The difference from single quotes is that they understand the same backslash sequences as the print builtin. Hence you can have the convenience of using `\n' for newline, `\e' for escape, `\xFF' for an arbitrary character in hexadecimal, and so on, for any command:

  % cat <<<$'Line\tone\nLine\ttwo'
  Line    one
  Line    two

  % print 'Line\tone\n\Line\ttwo'
  Line    one
  Line    two

This is as good a point as any to mention that the shell is completely `eight-bit clean', which means you can have any of the 256 possible characters anywhere in your string. For example, $'foo\000bar' has an embedded ASCII NUL in it (that's not a misprint --- officially, ASCII non-printing characters have two- or three-letter abbreviations). Usually this terminates a string, but the shell works around this when you are using it internally; when you try and pass it as an argument to an external programme, however, all bets are off. Almost certainly the first NUL in that case will cause the programme to think the string is finished, because no information about the length of arguments is passed down and there's nothing the shell can do about it. Hence, for example:

  % echo $'foo\000bar'
  foobar
  % /bin/echo $'foo\000bar'
  foo

There are actually uses for embedded NULs: some versions of find and xargs, for example, will put or accept NULs instead of newlines between their bits of input and output (as distinct from command line arguments), which is much safer if there's a chance the input or output can contain a live newline. Using $'\000' allows the shell to fit in very comfortably with these. If you want to try this, the corresponding options are -print0 for find (print with a NUL terminator instead of newline) and -0 for xargs (read input assuming a NUL terminator).

In older versions of the shell, characters with the top bit set, such as those from non-English character sets found in ISO 8859 fonts, could cause problems, since the shell also uses such characters internally to represent its own special characters, but recent versions of the shell (from about 3.0) side-step this problem in the same way as for NULs. Any remaining problems --- it's quite tricky to handle this completely consistently --- are bugs and should be reported.

You can force parameters to be quoted with POSIX quotes by the somewhat absurd expedient of making the q in the quote flag appear a total of four times. I can't think why you would ever want to do that, except that it will turn newlines into `\n' and hence the result will fit on a single (maybe rather long) line. Plus you get the replacement of funny characters with escape sequences.

5.1.4: Double quotes

Double quotes allow some, but not all, forms of substitution inside. More specifically, they allow parameter expansion, command substitution and arithmetic substitution, but not any of the others: process substitution doesn't happen, braces and initial tildes and equals signs are not expanded and patterns are not special. Here's a table; each expression on the left is some command line argument, and the results show what is substituted if it appears outside quotes, or in double quotes.

  Expression      Outside quotes  In double quotes
  ------------------------------------------------
  =(echo hi mum)  /tmp/zshTiqpL     =(echo hi mum)
  $ZSH_VERSION    4.0.1             4.0.1
  $(echo hi mum)  hi mum            hi mum
  $((6**2 + 6))   42                42
  {a,b}cd         acd bcd           {a,b}cd
  ~/foo           /home/pws/foo     ~/foo
  .zl*            .zlogin .zlogout  .zl*

You might already have guessed that `${(qqq)string}' forces $string to use double quotes to quote its special characters. As with the other forms, this is all properly handled --- the shell knows just which characters need quoting inside double quotes, and which don't.

Word-splitting in double quotes

Where the substitutions are allowed, the (almost) invariable side effect of double quotes is that word-splitting is suppressed. You can see this using `print -l', which prints one argument per line:

  % array=(one two)
  % print -l $(echo foo bar) $array
  foo
  bar
  one
  two
  % print -l "$(echo foo bar) $array"
  foo bar one two

This is extended for other parameters in the following way:

  % array=(one two three)
  % print -l "${array[@]}"
  one
  two
  three

  % print -l "${(@)array}"
  one
  two
  three

Digression on subscripts

The version with flags is perhaps less clear than the other, but it can appear in lots of different places. For example, here is how you pick a slice of an array in zsh:

  % print -l ${array[2,-1]}
  two
  three

  % print -l "${(@)array[2,-1]}"
  two
  three

Although it's probably not obvious, you can use the other notation in this case:

  % print -l "${array[@][2,-1]}"
  two
  three

  % print -l "${array[@][2,-1][1]}"
  two

  % print -l "${array[@][2,1][1][2,-1]}"
  wo

We've now strayed severely off topic: the subscripts will of course work quite independently from whether the word is being split or appears in double quotes. Despite the joining of words that occurs in double quotes, subscripts of arrays still select array elements. This is a consequence of the order in which the rules of parameter expansion apply. There is a long, involved section on this in the zshexpn manual entry (look for the heading `Rules' there or in the `Parameter Expansion' node of the corresponding Info or HTML file).

Word-splitting of quoted command substitutions

Zsh has the useful feature that you can force the shell to apply the rules of parameter expansion to the result of a command substitution. To see where that might be useful, consider the case of the special `command substitution' (although it's handled entirely in the shell, not by running an external command) which puts the contents of a file on the command line:

  % args() { print $#; }    # report number of arguments
  % cat file
  Words on line one
  Words on line two
  % args $(<file)
  8
  % args "$(<file)"
  1

It's very common, however, that you want one line per argument, not splitting on spaces within the line. This is where parameter expansion can come in. There is a flag (f) which says `split the result of the expansion, one word per line'. Here's how to use it in this case:

  % args "${(f)$(<file)}"
  2

5.1.5: Backquotes

The main thing to say about backquotes is that you should use the other form of command substitution instead. There are two good reasons.

First, the other form can be nested:

  % print $(print $(print a word))
  a word

This is clearly not true with `...`, even though the basic effect is the same. Any unquoted ` which happens to appear in a chunk of code within the backquotes will be treated as the end of the quotes.

The second reason, which is closely related, is that it can be quite difficult to decide how many levels of quotes are required inside a backquoted expression. Consider:

  % print "`echo \"hello\"`"
  hello
  % print "$(echo \"hello\")"
  "hello"

As a consequence, in

  % print "$(echo "hello")"
  hello

  % unset foo
  % print "${foo:-"a string"}"
  a string

A third, less good, reason for using the form with parentheses is that your more sophisticated friends will laugh at you otherwise. Peer pressure is so important in this complex world.

That's all I have to say about command substitution, since I already said a lot about it when I discussed the basic syntax in chapter 3.

5.2: Modifiers and what they modify

Modifiers were introduced in chapter 2 when I talked about `bang history', since that's where they came from. In zsh, however, they can be used in a couple of other places. They have the same form in each case: a colon, followed by a letter which is the code for what the modifier does, possibly (in the case of substitutions) followed by some other string. So, to jog your memory, unless you have NO_BANG_HIST set:

  % print ~/file
  /home/pws/file
  % print !-1:t
  file

The second use is in parameters. This follows on very naturally. Note that neither this nor any of the later uses of modifiers rely on the NO_BANG_HIST option; that's purely for history.

  % param=~/file
  % print ${param:t}
  file

  % print foo
  foo
  % ^foo^bar
  bar
  % param='this sentence contains a foo.'
  % print ${param:&}
  this sentence contains a bar.

Modifiers are also smart enough to handle arrays in a useful fashion. Note this is not true of sets of arguments in history expansions; `:t' will only extract one tail in that case, which may not be quite what you're expecting:

  % print a sentence with a /real/live/bogus/path in it.
  % print !!:t
  path in it.

  % array=(~/.zshenv ~/.zshrc ~/.zlogout)
  % print ${array:t}
  .zshenv .zshrc .zlogout

The same logic is applied with substitutions. This means that the first match in every element of the array is replaced:

  % array=('a bar of chocolate' 'a bar of barflies' 
  array> 'a barrier of barns')
  % print ${array:s/bar/car/}
  a car of chocolate a car of barflies a carrier of barns

  % print ${array:gs/bar/car/}
  a car of chocolate a car of carflies a carrier of carns

The final use of modifiers is in filename generation, i.e. globbing. Since this usually works by having special characters on the command line, and modifiers just consist of ordinary characters, the syntax is a little different:

  % print *.c
  parser.c lexer.c input.c output.c
  % print *.c(:r)
  parser lexer input output

  % print -l ~/stuff/*
  /home/pws/stuff/onefile.c
  /home/pws/stuff/twofile.c
  /home/pws/stuff/subdir
  % print ~/stuff/*(.:r:t)
  onefile twofile

Two final points to note about modifiers with filenames. First, it is the only form of globbing where the result is no longer a filename; it is always performed right at the end, after all normal filename generation. Presumably, in the examples above, the word which was inserted into the command line doesn't actually correspond to a real file any more.

Second, although it does work if the word on the command line isn't a pattern but an ordinary word with a modifier tacked on, it doesn't work if that pattern, before modification, doesn't correspond to a real file. So `foo.c(:r)' will only strip off the suffix if foo.c is there in the current directory. This is perfectly logical given that the attempt to match a file kicks the globbing system, including modifiers, into action. If this is a problem for you, there are ways round; for example, insert the right value by hand in a simple case like this, or more realistically store the value in a parameter and apply the modifier to that.

5.3: Process Substitution

I don't have much new to say on process substitution, but I do have an example of where I find it useful. If you use the pager `less' you may know it has the facility to preprocess the files you look at, for example uncompressing files temporarily via the environment variable $LESSOPEN (and maybe $LESSCLOSE). Zsh can very easily and, to my thoroughly unbiased way of looking, more conveniently do the same thing. Here's a subset of my zsh function front-end to less --- or indeed any pager, which is given here by the standard environment variable $PAGER with the default less. You can hard-wire any file-displaying command at that point if you prefer.

  integer i=1
  local args arg
  args=($*)

  for arg in $*; do
    case $arg in
      (*.bz2) args[$i]="=(bunzip2 -c ${(q)arg})"
              ;;
      # this assumes your zcat is the one installed with gzip:
      (*.(gz|Z)) args[$i]="=(zcat ${(q)arg})"
                 ;;
      (*) args=${(q)arg}
          ;;
    esac
    (( i++ ))
  done

  eval command ${PAGER:-less} $args

The reason the eval is there is so that the process substitutions are evaluated on the command line when the pager is run, and not before. They are assigned back to elements of $args in quotes, so don't get evaluated at that point. The effect will be to turn:

  less file.gz file.txt

  less =(zcat file.gz) file.txt

In case you haven't come across it, bzip2 is a programme very similar to gzip, and it is used almost identically, but it provides better compression.

There's an infelicity in output process substitutions, just as there is with multios.

  echo hello > >(sed s/hello/goodbye)

5.4: Parameter substitution

You can probably see from the above that parameter substitutions are at the heart of much of the power available to transform zsh command lines. What's more, we haven't covered even a significant fraction of what's on offer.

5.4.1: Using arrays

The array syntax in zsh is quite powerful (surprised?); just don't expect it to be as efficient as, say, perl. Like other features of zsh, it exists to make users' lives easier, not to make your computer run blindingly fast.

I've covered, somewhat sporadically, how to set arrays, and how to extract bits of them --- the following illustrates this:

  % array=(one two three four)
  % print ${array}
  one two three four
  % print ${array[3]}
  three
  % print ${array[2,-1]}
  two three four

You can use the array index and array slice notations for assigning to arrays, in other words on the left-hand side of an `=':

  % array=(what kind of fool am i)
  % array[2]=species
  % print $array
  what species of fool am i
  % array[2]=(a piece)
  % print $array
  what a piece of fool am i
  % array[-3,-1]=(work is a man)
  % print $array
  what a piece of work is a man

You can remove bits in the middle, too, but note you should use an empty array:

  % array=(one two three four)
  % print $#array
  4
  % array[2]=
  % print $#array
  4
  % array[2]=()
  % print $#array
  3

Just as parameter substitutions have flags for special purposes, so do subscripts. You can force them to search through arrays, matching on the values. You can return the value matched ((r)everse subscripting):

  % array=(se vuol ballare signor contino)
  % print ${array[(r)s*]}
  se
  % print ${array[(R)s*]}
  signor

  % array=(some words)
  % args() { print $#; }
  % args ${array[(r)s*]}
  1
  % args ${array[(r)X*]}
  0
  % args "${array[(r)X*]}"
  1

You can also return the index matched; (i) to start matching from the beginning, and (I) to start from the end.

  % array=(se vuol venire nella mia scuola)
  % print ${array[(i)v*]}
  2
  % print ${array[(I)v*]}
  3  

  array=(three egregious words)
  for pat in '*e*e*' '*a*a*'; do
    if [[ ${array[(i)$pat]} -le ${#array} ]]; then
      print "Pattern $pat matched in array: ${array[(r)$pat]}."
    else
      print "Pattern $pat failed to match in array"
    fi
  done

  Pattern *e*e* matched in array: three.
  Pattern *a*a* failed to match in array

There is currently no way of extracting a complete set of matches from an ordinary array with subscript flags. We'll see other ways of doing that below, however.

5.4.2: Using associative arrays

Look back at chapter 3 if you've forgotten about associative arrays. These take subscripts, like ordinary arrays do, but here the subscripts are arbitrary strings (or keys) associated with the value stored in the element of the array. Remember, you need to use `typeset -A' to create one, or one of typeset's relatives with the same option. This means that if you created it inside a function it will be limited to the local scope, so if you want to create a global associative array you will need to give the -g flag as well. This is particularly common with associative arrays, which are often used to store global information such as configuration details.

Retrieving information from associative arrays can get you into some of the problems already hinted at in the use of subscript flags with arrays. However, since normal subscripting doesn't make patterns active, there is a way round here: make the subscript into another parameter:

  % typeset -A assoc
  % assoc=(key value Shlüssel Wert clavis valor)
  % subscript='key'
  % print ${assoc[$subscript]}
  value  

  % assoc=(']' right\ square\ bracket '*' asterisk '@' at\ sign)
  % subscript=']'
  % print ${assoc[$subscript]}
  right square bracket

I introduced two parameter flags, (k) and (v) in chapter 3:

  % print ${(k)assoc}
  * ] @

  % print ${(kv)assoc}
  * asterisk ] right square bracket @ at sign

  % print ${(v)assoc}
  asterisk right square bracket at sign
  % print ${assoc}
  asterisk right square bracket at sign

  % print -r ${(qkv)assoc}
  \* asterisk \] right\ square\ bracket @ at\ sign

  % print -l ${(kv)assoc}
  *
  asterisk
  ]
  right square bracket
  @
  at sign

  % print -l "${(@kv)assoc}"
  *
  asterisk
  ]
  right square bracket
  @
  at sign

Apart from the subscripts, the second major difference between associative and ordinary arrays is that the former don't have any order defined. This will be entirely familiar if you have used Perl; the principle here is identical. However, zsh has no notion at all, even as a convenience, of slices of associative arrays. You can assign individual elements or whole associative arrays --- remembering that in the second case the right hand side must consist of key/value pairs --- but you can't assign subgroups. Any attempt to use the slice notation with commas will be met by a stern error message.

What zsh does have, however, is extra subscript flags for you to match and retrieve one or more elements. If instead of an ordinary subscript you use a subscript preceded by the flag (i), the shell will search for a matching key (not value) with the pattern given and return that. This is deliberately the same as searching an ordinary array to get its key (which in that case is just a number, the index), but note this time it doesn't match on the value, it really does match, as well as return, the key:

  % typeset -A assoc
  % assoc=(fred third\ man finnbar slip roger gully trevor long\ off)
  % print ${assoc[(i)f*]}
  fred

  % print ${(kv)assoc[(i)f*]}
  fred third man

Because of the essentially random ordering of associative arrays, you couldn't tell here whether fred or finnbar would be chosen. However, you can use the capital form (I) to tell the shell to retrieve all matches. This time, let's see the values of the elements for which the keys were matched:

  % print -l ${(v)assoc[(I)f*]}
  third man
  slip

You probably aren't surprised to hear that the subscript flags (r) and (R) try to match the values of the associative array rather than its keys. These, too, print out the actual part matched, here the value, unless you use the parameter flags.

  % print ${assoc[(r)*i*]}
  third man
  % print ${(k)assoc[(R)*i*]}
  fred finnbar

There's one more pair of subscript flags of particular relevance to associative arrays, (k) and (K). These work a bit like a case statement: the subscripts are treated as strings, and the keys of the associative arrays as patterns, instead of the other way around. With (k), the value of the first key which matches the subscript is substituted; with (K), the values of all matching keys are substituted

  % typeset -A assoc
  % assoc=('[0-9]' digit '[a-zA-Z]' letter '[^0-9a-zA-Z]' neither)
  % print ${assoc[(k)0]}
  digit
  % print ${assoc[(k)_]}
  neither

  case 0 in
    ([0-9]) print digit
            ;;
    ([a-zA-Z]) print letter
               ;;
    ([^0-9a-zA-Z]) print neither
                   ;;
  esac

I said storing information about configuration was a common use of associative arrays, but the shell has a more powerful way of doing that: styles, which will figure prominently in the discussion of programmable completion in the next chapter. The major advantage of styles over associative arrays is that they can be made context-sensitive; you can easily make the same style return the same value globally, or make it have a default but with a different value in one particular context, or give it a whole load of different values in different places. Each shell application can decide what is meant by a `context'; you are not tied to the same scheme as the completion system uses, or anything like it. Use of hierarchical contexts in the manner of the completion system does mean that it is easy to create sets of styles for different modules which don't clash.

Here, finally, is a comparison of some of the uses of associative arrays in perl and zsh.

      perl                          zsh
  -----------------------------------------------------------------
  %hash = qw(key value);         typeset -A hash; hash=(key value)
  $hash{key}                     ${hash[key]}
  keys %hash                     ${(k)hash}
  values %hash                   ${(v)hash}
  %hash2 = %hash;                typeset -A hash2; hash2=("${(@kv)hash}")
  unset %hash;                   unset hash
  if (exists $hash{key}) {       if (( ${+hash[key]} )); then
    ...                            ...
  }                              fi

One final reminder: if you are creating associative arrays inside a function which need to last beyond the end of the function, you should create them with `typeset -gA' which puts them into the surrounding scope. The `-g' flag is of course useful with all types of parameter, but the associative array is the only type that doesn't automatically spring into existence when you assign to it in the right context; hence the flag is particularly worthy of note here.

5.4.3: Substituted substitutions, top- and tailing, etc.

There are many transformations which you can do on the result of a parameter substitution. The most powerful involve the use of patterns. For this, the more you know about patterns, the better, so I will reserve explanation of some of the whackiest until after I have gone into more detail on patterns. In particular, it's useful if you know how to tell the shell to mark subexpressions which it has matched for future extraction. However, you can do some very useful things with just the basic patterns common to all shells.

Standard forms: lengths

I'll separate out zsh-specific forms, and start off with some which appear in all shells derived from the Bourne shell. A more compact (read: terse) list is given in the manual, as always.

A few simple forms don't use patterns. First, the substitution ${#param} outputs the length of $param. In zsh, you don't need the braces here, though in most other shells with this feature you do. Note that ${#} on its own is the number of parameters in the command line argument array, which is why explicit use of braces is clearer.

$# works differently on scalar values and array values; in the former case, it gives the length in characters, and in the latter case the length in elements. Note that I said `values', not `parameters' --- you have to work out whether the substitution is giving you a scalar or an array:

  % print ${#path}
  8
  % print ${#path[1]}
  13

Earlier versions of zsh always returned a character count if the expression was in double quotes, or anywhere the shell evalauted the expression as a single word, but that doesn't happen any more; it depends only on the type of the value. However, you can force the shell to count characters by using the (c) flag, and to count words (even in scalars, which it will split if necessary) by using (w):

  % print ${#PATH}
  84
  % print ${(c)#path}
  84
  % foo="three scalar words"
  % print ${(w)#foo}
  3

  % foo="three  well-spaced  word"
  % print ${(w)#foo}
  3
  % print ${(W)#foo}
  5

Note that if you use lengths in an arithmetic context (inside ((...)) or $((...))), you must include the leading `$', which you don't need for substituting the parameters themselves. That's because `#foo' means something different here --- the number in the ASCII character set (or whatever extension of it you are using if it is an extended character set) of the first character in $foo.

Standard forms: conditional substitutions

The next group of substitutions is a whole series where the parameter is followed by an option colon and then `-', `=', `+' or `?'. The colon has the same effect in each case: without a colon, the shell tests whether the parameter is set before performing the operation, while with the colon it tests whether the parameter has non-zero length.

The simplest is `${param:-value}'. If $param has non-zero length (without the colon, if it is set at all), use its value, else use the value supplied. Suppose $foo wasn't set at the start of the following (however unlikely that may seem):

  % print ${foo-bar}
  bar
  % foo=''
  % print ${foo-bar}
  
  % print ${foo:-bar}
  bar
  % foo='please no anything but bar'
  % print ${foo:-bar}
  please no anything but bar

`${param:=value}' is similar to the previous type. but in this case the shell will not only substitute value into the line, it will assign it to param if (and only if) it does so. This leads to the following common idiom in scripts and functions:

  : ${MYPARAM:=default}  ${OTHERPARAM:=otherdefault}

`${param:+value}' is the opposite of `:-', logically enough: the value is substituted if the parameter doesn't have zero length. In this case, value will often be another parameter substitution:

  print ${value:+"the value of value is $value"}

  print ${1+"$@"}

The final type isn't that often used (meaning I never have): ${param?message} tests if param is set (no colon), and if it isn't, prints the message and exits the shell. An interactive shell won't exit, but it will return you immediately to the prompt, skipping anything else stored up for execution. It's a rudimentary safety feature, a little bit like `assert' in C programmes; most shell programmers seem to cover the case of missing parameter settings by more verbose tests. It's quite neat in short shell functions for interactive use:

  mless() { mtype ${@:?missing filename} | $PAGER }

Standard forms: pattern removal

Most of the more sophisticated Bourne-like shells define two pairs of pattern operators, which I shall call `top and tail' operators. One pair (using `#' and `##') removes a given pattern from the head of the string, returning the rest, while the other pair (using `%' and `%%') removes a pattern from the tail of the string. In each case, the form with one symbol removes the shortest matching pattern, while the one with two symbols removes the longest matching pattern. Two typical uses are:

  % print $HOME
  /home/pws
  % print ${HOME##*/}
  pws
  % print ${HOME%/*}
  /home

  % print ${HOME#*/}
  home/pws
  % print ${HOME%%/*}

How do you remember which operator does what? The fact that the longer form does the longer match is probably easy. Remembering that `#' removes at the head and `%' at the tail is harder. Try to think of `hash' and `head' (if you call it a `pound sign', when it's nothing of the sort since a pound since looks like `£', you will get no sympathy from me), and `percent' and `posterior'. It never worked for me, but maybe I just don't have the mental discipline. Oliver Kiddle points out that `#' is further to the left (head) on a standard US keyboard. On my UK keyboard, `#' is right next to the return key, unfortunately, although here the confusion with `pound sign' will jog your memory.

The most important thing to remember is: this notation is not our fault. Sorry, anyway. By the way, notice there's no funny business with colons in the case of the pattern operators. (Well --- except for the zsh variant noted below.)

Zsh-specific parameter substitutions

Now for some enhancements that zsh has for using the forms of parameter substitution I've just given as well as some similar but different ones.

One simple enhancement is that in addition to `${param=value}' and `${param:=value}', zsh has `${param::=value}' which performs an unconditional assignment as well as sticking the value on the command line. It's not really any different from using a normal assignment, then a normal parameter substitution, except that zsh users like densely packed code.

All the assignment types are affected by the parameter flags `A' and `AA' which tell the shell to perform array and associative array assignment (in the second case, you need pairs of key/value elements as usual). You need to be a little bit careful with array elements and word splitting, however:

  % print -l ${(A)foo::=one two three four}
  one two three four
  % print ${#foo}
  1

  % print -l ${(A)=foo::=one two three four}
  one
  two
  three
  four
  % print ${#foo}
  4

You should be careful, however, as more sophisticated attempts at putting arrays inside parameter values can easily lead you astray. It's usually much easier to use the `array=(...)' or `set -A ...' notations.

One extremely useful zsh enhancement is the notation `${+foo}' which returns 1 if $foo is set and 0 if it isn't. You can use this in arithmetic expressions. This is a much more flexible way of dealing with possibly unset parameters than the more standard `${foo?goodbye}' notation, and consequently is better used by zsh programmers. The notation `plus foo' for `foo is set' should be fairly memorable, too. A more standard way of doing this (noted by David Korn) is `0${foo+1}', giving 0 if $foo is not set and 01 if it is.

Parameter flags and pattern substitutions

Zsh increases the usefulness of the `top and tail' operators with some of its parameter flags. Usually these show you what's left after the removal of some matched portion. However, with the flag (M) the shell will instead show you the matched portion itself. The flag (R) is the opposite and shows the rest: that's not all that useful in the normal case, since you get that by default. It only starts being useful when you combine it with other flags.

Next, zsh allows you to match on substrings, not just on the head or tail. You can do this by giving the flag (S) with either of the `#' or `%' pattern-matching forms. The difference here is whether the shell starts searching for a matching substring at the start or end of the full string. Let's take

  foo='where I was huge lizards walked here and there'

  % print -l ${(S)foo#h*e} ${(S)foo##h*e} ${(S)foo%h*e} ${(S)foo%%h*e}
  wre I was huge lizards walked here and there
  w
  where I was huge lizards walked here and tre
  where I was huge lizards walked here and t

You can extend this by using the (I) flag to specify a numeric index. The index needs to be delimited, conventionally, although not necessarily, by colons. The shell will then scan forward or backward, depending on the form used, until it has found the (I)'th match. Note that it only ever counts a single match from each position, either the longest or the shortest, so the (I)'th match starts from the (I)'th position which has any match. Here's what happens when we remove all the matches for `#' using the example above.

  % for (( i = 1; i <= 5; i++ )); do
  for> print ${(SI:$i:)foo#h*e}
  for> done
  wre I was huge lizards walked here and there
  where I was  lizards walked here and there
  where I was huge lizards walked re and there
  where I was huge lizards walked here and tre
  where I was huge lizards walked here and there

It's obvious what happens with `##': it will find matches at all the same points, but they will all extend to the `e' at the end of the string. It's probably less obvious what happens with `%%' and `%', but if you try it you will find they produce just the same set of matches as `##' and `#', respectively, but with the indices in the reverse order (4 for 1, 3 for 2, etc.).

You can use the `M' flag to leave the matched portion rather than the rest of the string, if you like. There are three other flags which let you get the indices associated with the match instead of the string: (B) for the beginning, using the usual zsh convention where the first character is 1, (E) for the character after the end, and (N) for the length, simply B-E. You can even have more than one of these; the value substituted is a string with the given values with spaces between, always in the order beginning, end, length.

There is a sort of opposite to the `(S)' flag, which instead of matching substrings will only match the whole string; to do this, put a colon before the `#'. Hence:

  % print ${foo:#w*g}
  where I was huge lizards walked here and there
  % print ${foo:#w*e}

  % 

Pattern replacement

The most powerful of the parameter pattern-matching forms has been borrowed from bash and ksh93; it doesn't occur in traditional Bourne shells. Here, you use a pair of `/'s to indicate a pattern to be replaced, and its replacement. Lets use the lizards again:

  % print ${foo/h*e/urgh}
  wurgh

  % print ${(S)foo/h*e/urgh}
  wurghre I was huge lizards walked here and there

  % print ${(S)foo//h*e/urgh}
  wurghre I was urgh lizards walked urghre and turghre

The index flag (I) is useful here, too. In the case of `/', it tells the shell which single match to substitute, and in the case of `//' it tells the shell at which match to start: all matches starting from that are replaced.

Overlapping matches are never replaced by `//'; once it has put the new text in for a match, that section is not considered further and the text just to its right is examined for matches. This is probably familiar from other substitution schemes.

You may well be thinking `wouldn't it be good to be able to use the matched text, or some part of it, in the replacment text?' This is what you can do in sed with `\1' or `\&' and in perl with `$1' and `$&'. It turns out this is possible with zsh, due to part of the more sophisticated pattern matching features. I'll talk about this when we come on to patterns, since it's not really part of parameter substitution, although it's designed to work well with that.

5.4.4: Flags for options: splitting and joining

There are three types of flag that don't look like flags, for historical reasons; you've already seen them in chapter 3. The first is the one that turns on the SH_WORD_SPLIT option, ${=foo}. Note that you can mix this with flags that do look like flags, in parentheses, in which case the `=' must come after the closing parenthesis. You can force the option to be turned off for a single substitution by doubling the symbol: `${==foo}'. However, you wouldn't do that unless the option was already set, in which case you are probably trying to be compatible with some other shell, and wouldn't want to use that form.

More control over splitting and joining is possible with three of the more standard type of flags, (s), (j) and (z). These do splitting on a given string, joining with a given string, and splitting just the way the shell does it, respectively. In the first two cases, you need to specify the string in the same way as you specified the index for the (I) flag. So, for example, here's how to turn $PATH into an ordinary array without using $path:

  % print -l ${(s.:.)PATH}
  /home/pws/bin
  /usr/local/bin
  /usr/sbin
  /sbin
  /bin
  /usr/bin
  /usr/X11R6/bin
  /usr/games

Although the split or join string isn't a pattern, it doesn't have to be a single character:

  % foo=(array of words)
  % print ${(j.**.)foo}
  array**of**words

The (z) flag doesn't take an argument. As it handles splitting on the full shell definition of a word, it goes naturally with quoted expressions, and I discussed above its use with the (Q) flag for extracting words from a line with the quotes removed.

It's possible for the same parameter expression to have both splitting and joining applied to it. This always occurs in the same order, regardless of how you specify the flags: joining first, then splitting. This is described in the (rather hairy) complete set of rules in the manual entry for parameter substitution. There are one or two occasions where this can be a bit surprising. One is when you have SH_WORD_SPLIT set and try to join a string:

  % setopt shwordsplit    
  % foo=('another array' of 'words with spaces')
  % print -l ${(j.:.)foo}
  another
  array:of:words
  with
  spaces

  % print -l "${(j.:.)foo}"
  another array:of:words with spaces

Specifying a split for a particular parameter substitution not only sets the string to split on, but also ensures the split will take place even if the expression is quoted:

  % array=('element one' 'element two' 'element three')
  % print -l "${=array}"
  element
  one
  element
  two
  element
  three

I will talk shortly about nested parameter substitution; you should also note that splitting and joining will if necessary take place at all levels of a nested substitution, not just the outermost one:

  % foo="three blind words"
  % print ${#${(z)foo}}
  3

5.4.5: Flags for options: GLOB_SUBST and RC_EXPAND_PARAM

The other two flags that don't use parentheses affect options for single substitutions, too. The second is the `~' flag that turns on GLOB_SUBST, making the result of a parameter substitution eligible for pattern matching. As the notation is supposed to indicate, it also makes filename expansion possible, so

  % foo='~'
  % print ${~foo}
  /home/pws

There's one other of these option flags: `^' forces on RC_EXPAND_PARAM for the current substitution, and `^^' forces it off. In chapter 3, I showed how parameters expanded with this option on fitted in with brace expansions.

5.4.6: Yet more parameter flags

Here are a few other parameter flags; I'm repeating some of these. A very useful one is `t' to tell you the type of a parameter. This came up in chapter 3 as well. It's most common use is to test the basic type of the parameter before trying to use it:

  if [[ ${(t)myparam} != *assoc* ]]; then
    # $myparam is not an associative array.  Do something about it.
  fi

Another very useful type is for left or right padding of a string, to a specified length, and optionally with a specified fill string to use instead of space; you can even specify a one-off string to go right next to the string in question.

  foo='abcdefghij'
  for (( i = 1; i <= 10; i++ )); do
   goo=${foo[1,$i]}
   print ${(l:10::X::Y:)goo} ${(r:10::X::Y:)goo}
  done

  XXXXXXXXYa aYXXXXXXXX
  XXXXXXXYab abYXXXXXXX
  XXXXXXYabc abcYXXXXXX
  XXXXXYabcd abcdYXXXXX
  XXXXYabcde abcdeYXXXX
  XXXYabcdef abcdefYXXX
  XXYabcdefg abcdefgYXX
  XYabcdefgh abcdefghYX
  Yabcdefghi abcdefghiY
  abcdefghij abcdefghij

Two parameters tell the shell that you want something special done with the value of the parameter substitution. The (P) flag forces the value to be treated as a parameter name, so that you get the effect of a double substitution:

  % final=string
  % intermediate=final
  % print ${(P)intermediate}
  string

A more powerful flag is (e), which forces the value to be rescanned for all forms of single-word substitution. For example,

  % foo='$(print $ZSH_VERSION)'
  % print ${(e)foo}
  4.0.2

The remaining flags are a few simple special formatting tricks: order array elements in normal lexical (character) order with (o), order in reverse order with (O), do the same case-independently with (oi) or (Oi) respectively, expand prompt `%'-escapes with (%) (easy to remember), expand backslash escapes as print does with p, force all characters to uppercase with (U) or lowercase with (L), capitalise the first character of the string or each array element with (C), show up special characters as escape sequences with (V). That should be enough to be getting on with.

5.4.7: A couple of parameter substitution tricks

I can't resist describing a couple of extras.

Zsh can do so much on parameter expressions that sometimes it's useful even without a parameter! For example, here's how to get the length of a fixed string without needing to put it into a parameter:

  % print ${#:-abcdefghijklm}
  13

Sometimes you don't need anything more than the flags. The most useful case is making the `fill' flags generate repeated words, with the effect of perl's `x' operator (for those not familiar with perl, the expression `"string" x 3' produces the string `stringstringstring'. Here, you need to remember that the fill width you specify is the total width, not the number of repetitions, so you need to multiply it by the length of the string:

  % print ${(l.18..string.)}
  stringstringstring

5.4.8: Nested parameter substitutions

Zsh has a system for multiple nested parameter substitutions. Whereas in most shells or other scripting languages you would do something like:

  % p=/directory/file.ext
  % p2=${p##*/}            # remove longest match of */ from head
  % print $p2
  file.ext
  % print ${p%.*}          # remove shortest match of .* from tail
  file

  % p=/directory/file.ext
  % print ${${p##*/}%.*}
  file

  % p=(/dir1/file1.ext1 /dir2/file2.ext2)
  % print ${${p##*/}%.*}
  file1 file2

   # Perl code.
   @p = qw(/dir1/file1.ext1 /dir2/file2.ext2);
   @q = map { m%^(?:.*/)(.*?)(?:\.[^.]*|)$%; } @p;
   print "@q\n";'

  print ${p:t:r}

Note one feature of the nested substitution: you might have expected the `${...}' inside the other one to do a full parameter substitution, so that the outer one would act on the value of that --- that's what you'd get if the substitution was on its own, after all. However, that's not what happens: the `${...}' inside is simply a syntactic trick to say `here come more operations on the parameter'. This means that

  bar='this doesn'\''t get substituted'
  foo='bar'
  print ${${foo}}

5.5: That substitution again

Finally, here is a brief explanation of how to read the expression at the top of the chapter. This is for advanced students only (nutcases, if you ask me). You can find all the bits in the manual, if you try hard enough, even the ones I didn't get around to explaining above. As an example, let's suppose the array contains

  array=(long longer longest short brief)

  print ${array[(r)${(l.${#${(O@)array//?/X}[1]}..?.)}]}

1. Always start from the inside. The innermost expression here is

       ${(O@)array//?/X}
   

   Not much clearer? Start from the inside again: there's the parameter we're operating on, whose name is array. Before that there are two flags in parenthesis: (O) says sort the result in descending alphabetic order, (@) treat the result as an array, which is necessary because this inner substitution occurs where a scalar value (actually, an arithmetic expression) would usually occur, and we need to take an array element. After the array name, `//?/X' is a global substitution: take the pattern `?' (any character) wherever it occurs, and replace it with the string `X'. The result of this is an array like $array, but with all the elements turned into strings consisting of `X's in place of the original characters, and with the longest first, because that's how reverse alphabetic order works for strings with the same character. So

       long longer longest short brief
   

   would have become

       XXXXXXX XXXXXX XXXXX XXXXX XXXX
   

2. Next, we have `${#result[1]}' wrapped around that. That means that we take the first element of the array we arrived at above (the `[1]': that's why we had to make sure it was treated as an array), and then take the length of that (the `#'). We will end up in this case with 7, the length of the first (and longest element). We're finally getting somewhere.

3. The next step is the `${(l.result..?.)}'. Our previous result appears as an argument to the `(l)' flag of the substitution. That's a rather special case of nested substitution: at this point, the shell expects an arithmetical expression, giving the minimum length of a string to be filled on the left. The previous substitution was evaluated because arithmetic expressions undergo parameter substitution. So it is the result of that, 7, which appears here, giving the more manageable

       ${(l.7..?.)}
   

   The expression for the `(l)' flag in full says `fill the result of this parameter substitution to a minimum width of 7 using the fill character `?'. What is the substitution we are filling? It's empty: zsh is smart enough to assume you know what you're doing when you don't give a parameter name, and just puts in an empty string instead. So the empty string is filled out to length 7 with question marks, giving `???????'.

4. Now we have `${array[(r)???????]}'. It may not be obvious (congratulations if the rest is), but the question marks are active as a pattern. Subscripts are treated specially in this respect. The subscript flag `(r)' means `reverse match', not reverse as in backwards, but as in the opposite way round: search the array itself for a matching value, rather than taking this as an index. The only thing that will match this is a string of length 7. Bingo! that must be the element `longest' in this case. If there were other elements of the same length, you would only get the first of that length; I haven't thought of a way of getting all the elements of that length substituted by a single expression without turning $array into an associative array, so if you have, you should feel smug.

After I wrote this, Sven Wischnowsky (who is responsible for a large fraction of the similar hieroglyphics in the completion functions) pointed out that a similar way of achieving this is:

  print ${(M)array:#${~${(O@)array//?/?}[1]}}

If you are wondering about how to do that in perl in a single expression, probably sorting on length is the easiest:

  # Perl code
  @array = qw(long longer longest short brief);
  @array = sort { length $b <=> length $a } @array;

  # Perl code
  $elt = '';
  $l = 0;
  foreach (@array) {
    $newl = length $_;
    $elt = $_, $l = $newl  if $l > $newl;
  }
  print $elt, "\n";

  local val elt
  integer l newl
  for val in $array; do
    newl=${#val}
    if (( newl > l )); then
      elt=$val
      (( l = newl ))
    fi
  done
  print $elt

If you enjoyed that expression, there are many more like it in the completion function suite for you to goggle at.

5.6: Arithmetic Expansion

Performing mathematics within the shell was first described in chapter 3 where I showed how to create numeric parameters with variants of `typeset', and said a little about arithmetic substitution.

In addition to the math library, loadable with `zmodload zsh/mathfunc', zsh has essentially all the operators you expect from C and other languages derived from it. In other words, things like

  (( foo = bar ? 3 : 1, ++brr ))

One extra operator has been borrowed from FORTRAN, or maybe Perl, the exponentiation operator, `**'. This can take either integers or floating point numbers, though a negative exponent will cause a floating point number to be returned, so `$(( 2 ** -1 ))' gives you 0.5, not rounded down to zero. This is why the standard library function pow is missing from zsh/mathfunc --- it's already there in that other form. Pure integer exponentiation, however, is done by repeated multiplication --- up to arbitrary sizes, so instead of `2 ** 100', you should use `1 << 100', and for powers of any other integer where you don't need an exact result, you should use floating point numbers. For this purpose, the zsh/mathfunc library makes `casts' available; `float(num)' forces the expression num to interpreted as a floating point number, whatever it would otherwise have given, although the trick of adding `0.0' to a number works as well. Note that, although this works like a cast in C, the syntax is that of an ordinary function call. Likewise, `int(num)' causes the number to be interpreted as an integer --- rounding towards zero; you can use floor and ceil to round down or up, and rint to round to the nearest integer, although these three actually produce floating point numbers. They are standard C library functions.

For completeness, the assignment form of exponentiation `**=' also works. I can't remember ever using it.

The range of integers depends on how zsh was configured on your machine. The primary goal is to make sure integers are large enough to represent indexes into files; on some systems where the hardware usually deals with 32-bit integers, file sizes may be given by 64-bit integers, and zsh will try to use 64-bit integers as well. However, zsh will test for large integers even if no large file support is available; usually it just requires that your compiler has some easy to recognise way of defining 64-bit integers, such as `long long' which may be handled by gcc even if it isn't by the native compiler. You can easily test; if your zsh supports 64-bit integers, the largest available integer is:

  % print $(( 0x7FFFFFFFFFFFFFFF ))
  9223372036854775807

The range of floating point numbers is always that of a C `double', which is usually also 64 bits, and internally the number is highly likely to be in the IEEE standard form, which also affects the precision and range you can get, though that's system specific, too. On most systems, the math library functions handle doubles rather than single precision floats, so this is the natural choice. The cast function is called `float' because, unlike C, the representation of a floating point number is chosen for you, so the generic name is used.

5.6.1: Entering and outputting bases

I'll say a word or two about bases. I already said you could enter a number with any small base in a form like `2#101010' or `16#ffff', and that the latter could also be `0xffff' as in C. You can't, however, enter octal numbers just by using a leading `0', which you might expect from C. Here's an example of why not. Let's set:

  % foo=${(%):-%D}
  % print $foo
  01-08-06

  mlens=(31 28 31 30 31 30 31 31 30 31 30 31)
  date=(${(s.-.)foo})    #  splits to array (01 08 23)
  typeset -Z 2 incr
  if (( ${date[3]} < ${mlens[${date[2]}]} )); then
    # just increment day
    (( incr = ${date[3]} + 1 ))
    date[3]=$incr
  else
    # go to first of next month
    date[3]=01
    if (( ${date[2]} < 12 )); then
      (( incr = ${date[2]} + 1 ))
      date[2]=$incr
    else
      # happy new year
      date[2]=01
      (( incr = ${date[3]} + 1 ))
      date[3]=$incr
    fi
  fi
  print ${date[1]}-${date[2]}-${date[3]}

 
  typeset -Z 2 incr
  (( incr = ${date[2]} + 1 ))
  date[2]=$incr

  date[2]=${(l.2..0.)$(( ${date[2]} + 1 ))}

Now, finally, the point. In that `$(( ${date[2]} + 1 ))', the `${date[2]}' is simply the scalar `08' --- the result of splitting an arbitrary string into an array. Suppose we used leading zeroes to signify octal numbers. We would get something like:

  % print $(( ${date[2]} + 1 ))
  zsh: bad math expression: operator expected at `8 + 1 '

However, there is an option you can set, OCTAL_ZEROES; this is required for compatibility with the POSIX standard. That's how I got the error message in the previous paragraph, in fact.

Floating point numbers are never octal, always decimal:

  % setopt octalzeroes
  % print $(( 077 ))
  63
  % print $(( 077.43 ))
  77.430000000000007

The other option to do with bases is C_BASES, which makes hexadecimal (and, if you have OCTAL_ZEROES set, octal) numbers appear in the form that you would use as input to a C (or, once again, Perl) program.

How do you persuade the shell to print out numbers in a particular base anyway? There are two ways. The first is to associate a base with a parameter, which you do with an argument after the `-i' option to typeset:

  % typeset -i 16 hexnum=32
  % print $hexnum
  16#20

  % integer anynum
  % (( anynum = 16#20 ))
  % print $anynum
  16#20

Often, however, you just want to print out an expression in, say, hexadecimal. Zsh has a shorthand for this, which is only in recent versions (and not in other shells). Preceding an expression by `[#base]' causes the default output base to be set to base with the the usual prefix showing the base, and `[##base]' will do the same but without the prefix, i.e. `$(( [##16]255 ))' is simply `FF'. This has no effect on assignments to a parameter, not even on the parameter's default output base, but it will affect the result of a direct substitution using $((...)).

5.6.2: Parameter typing

Just as creating a parameter with an ordinary assignment makes it a scalar, so creating it in an arithmetic substitution makes it either an integer or a floating point parameter, according to the value assigned. This is likely to be a floating point number if there was a floating point number in the expression on the right hand side, and an integer otherwise. However, there are reasons why a floating point number on the right may not have this effect --- use of int, for example, since it produces an integer.

However, relying on implicit typing in this fashion is bad. One of the reasons is explained in the manual entry, and I can't do better than use that example (since I wrote it):

  for (( f = 0; f < 1; f += 0.1 )); do
    print $f
  done

You could turn the first 0 into 0.0, but a better way is to declare `float f' before the loop. In a function, this also ensures $f is local to the function.

If you use a scalar to store an integer or floating point, everything will work. You don't have the problem just described, since although $f contains what looks like an integer to start with, it has no numeric type associated with it, and when you store 0.1 into $f, it will happily overwrite the string `0'. It's a bit more inefficient to use scalars, but actually not that much. You can't specify an output base or precision, and in versions of zsh up to 4.0.x, there is a problem when the parameter already has a string in it which doesn't make sense as a numeric expression:

  % foo='/file/name'
  % (( foo = 3 ))
  zsh: bad math expression: operand expected at `/file/name'

You need to be careful with scalars that might contain an empty string. If you declare `integer i', it will immediately contain the value 0, but if you declare `typeset s', the scalar $s will just contain the empty string. You get away with this if you use the parameter without a `$' in front:

  % typeset s
  % print $(( 3 * s ))
  0

  % print $(( 3 * $s ))
  zsh: bad math expression: operand expected at `'

  % fn() { print "Twice $1 is $(( 2 * $1 ))"; }
  % fn 3
  Twice 3 is 6
  % fn
  fn: bad math expression: operand expected at `'

  % fn() { print "Twice ${1:=0} is $(( 2 * $1 ))"; }
  % fn
  Twice 0 is 0

Note that you need to do this even if it doesn't look like the number will be needed:

  % fn() { print $(( ${1:-0} ? $1 : 3 )); }
  % fn
  fn: bad math expression: operand expected at `: 3 '

You should note that just as you can put numbers into scalar parameters without needing any special handling, you can also do all the usual string-related tricks on numeric parameters, since there is automatic conversion in the other direction, too:

  % float foo
  % zmodload -i zsh/mathfunc
  % (( foo = 4 * atan(1.0) ))
  % print $foo
  3.141592654e+00
  % print ${foo%%.*}${foo##*.[0-9]##}
  3e+00

Starting from 4.1.1, a calculator function called zcalc is bundled with the shell. You type a standard arithmetic expression and the shell evaluates the formula and prints it out. Lines already entered are prefixed by a number, and you can use the positional parameter corresponding to that number to retrieve that result for use in a new formula. The function uses vared to read the formulae, so the full shell editing mechanism is available. It will also read in zsh/mathfunc if that is present.

5.7: Brace Expansion and Arrays

Brace expansion, which you met in chapter 3, appears in all csh derivatives, in some versions of ksh, and in bash, so is fairly standard. However, there are some features and aspects of it which are only found in zsh, which I'll describe here.

A complication occurs when arrays are involved. Normally, unquoted arrays are put into a command line as if there is a break between arguments when there is a new element, so

  % array=(three separate words)
  % print -l before${array}after
  beforethree
  separate
  wordsafter

  % print -l before${^array}after
  beforethreeafter
  beforeseparateafter
  beforewordsafter

These two ways of expanding arrays interact differently with braces; the more useful version here is when the RC_EXPAND_PARAM option is on. Here the array acts as sort of additional nesting:

  % array=(two three)
  % print X{one,${^array}}Y
  XoneY XtwoY XoneY XthreeY

If RC_EXPAND_PARAM is not set, you get something at first sight slightly odd:

  % array=(two three)
  % print X{one,$array}Y
  X{one,two three}Y

There's no obvious simple way of forcing the $array to be embedded in the braces at the same level, instead of like an additional set of braces. There are more complicated ways, of course.

  % array=(two three)
  % print X${^=:-one $array}Y
  XoneY XtwoY XthreeY

Braces allow numeric expressions; this works a little like in Perl:

  % print {1..10}a
  1a 2a 3a 4a 5a 6a 7a 8a 9a 10a

  % print {01..10}b
  01b 02b 03b 04b 05b 06b 07b 08b 09b 10b

  % print {10..1}c
  10c 9c 8c 7c 6c 5c 4c 3c 2c 1c

There's also an option BRACE_CCL which, if the braces aren't in either of the above forms, expands single letters and ranges of letters:

  % setopt braceccl
  % print 1{abw-z}2
  1a2 1b2 1w2 1x2 1y2 1z2

An important point to be made about braces is that they are not part of filename generation; they have nothing to do with pattern matching at all. The shell blindly generates all the arguments you specify. If you want to generate only some arguments, depending on what files are matched, you should use the alternative-match syntax. Compare:

  % ls
  file1
  % print file(1|2)
  file1
  % print file{1,2}
  file1 file2

This point is particularly worthy of note if you have come from a C-shell world, or use the CSH_NULL_GLOB option:

  csh% echo file{1,2}
  file1 file2
  csh% echo f*{1,2}
  file1

5.8: Filename Expansion

Filename expansions consists of just `~/...', `~user/...', `~namedir/...' and `=prog', where the `~' and `=' must be the first character of a word, and the option EQUALS must be set (it is by default) for the `=' to be special. I told you about all this in chapter 3.

There's really only one thing to add, and that's the behaviour of the MAGIC_EQUAL_SUBST option. Assignments after typeset and similar statements are handled as follows

  % typeset foo=~pws
  % print $foo
  /home/pws
  % typeset PATH=$PATH:~pws/bin
  % print ${path[-1]}
  /home/pws/bin

It's often useful to have this feature with commands of your own. There is an option, MAGIC_EQUAL_SUBST, which spots the forms `...=~...' and `...=...:~...' for any command at all and expands ~-expressions. Commands where this is particularly useful include make and the GNU configure command used for setting up the compilation of a software package from scratch.

A related new option appeared in version 4.0.2 when it became clear there was an annoying difference between zsh and other shells such as ksh and bash. Consider:

  export FOO=`echo hello there`

  export FOO=hello there

Normally, KSH_TYPESET applies only after parameter declaration builtins, and then only in the values of an assignment. However, in combination with MAGIC_EQUAL_SUBST, you will get the same behaviour with any command argument that looks like an assignment --- actually, anything following an `=' which wasn't at the start of the word, so `"hello mother, => I'm home "$(echo right now)' qualifies.

It seems that bash behaves as if both KSH_TYPESET and MAGIC_EQUAL_SUBST are always in effect.

5.9: Filename Generation and Pattern Matching

The final topic is perhaps the biggest, even richer than parameter expansion. I'm finally going to explain the wonderful world of zsh pattern matching. In addition to patterns as such, you will learn such things as how to find all files in all subdirectories, searching recursively, which have a given name, case insensitive, are at least 50 KB large, no more than a week old and owned by the root user, and allowing up to a single error in the spelling of the name. In fact, the required expression looks like this:

  **/(#ia1)name(LK+50mw-1u0)

To repeat: filename generation is just the same as globbing, only longer. I use the terms interchangeably.

5.9.1: Comparing patterns and regular expressions

It can be confusing that there are two rather different sorts of pattern around, those used for matching files on a command line as in zsh and other shells, and those used for matching text inside files as in grep, sed, emacs, perl and many other utilities, each of which, typically, has a slightly different form for patterns (called in this case `regular expressions', because UNIX was designed by computer scientists). There are even some utilities like TCL which provide both forms.

Zsh deals exclusively with the shell form, which I've been calling by its colloquial name, `globbing', and consequently I won't talk about regular expressions in any detail. Here are the two classic differences to note. First, in a shell, `*' on its own matches any set of characters, while in a regular expression it always refers to the previous pattern, and says that that can be repeated any number of times. Second, in a shell `.' is an ordinary (and much used) character, while in a regular expression it means `any character', which is specified by `?' in the shell. Put this together, and what a shell calls `*' is given by `.*' in a regular expression. `*' in the latter case is called a `Kleene closure': it's those computer scientists again. In zsh, art rather than science tends to be in evidence.

In fact, zsh does have many of the features available in regular expressions, as well as some which aren't. Remember that anywhere in zsh where you need a pattern, it's of the same form, whether it's matching files on the command line or a string in a case statement. There are a few features which only fit well into one or another use of patterns; for example the feature that selects files by examining their type, owner, age, etc. (the final parenthesis in the expression I showed above) are no use in matching against a string.

5.9.2: Standard features

There is one thing to note about the simple pattern matching features `*' and `?', which is that when matching file names (not in other places patterns are used, however) they never match a leading `.'. This is a convention in UNIX-like systems to hide certain files which are not interesting to most users. You may have got the impression that files begining with `.' are somehow special, but that's not so; only the files `.' (the current directory) and `..' (the parent directory, or the current directory in /) are special to the system. Other files beginning with `.' only appear special because of a conspiracy between the shell (the rule I've just given) and the command ls, which, when it lists a directory, doesn't show files beginning `.' unless you give the `-a' option. Otherwise `.'-files are perfectly normal files.

You can suppress the special rule for an initial `.' by setting the option GLOB_DOTS, in which case `*' will match every single file and directory except for `.' and `..'.

In addition to `*' and `?', which are so basic that even DOS had them (though I never quite worked out exactly what it was doing with them a lot of the time), the pattern consisting of a set of characters in square brackets appears in all shells. This feature happens to be pretty much the same as in regular expressions. `[abc]' matches any one of those three characters; `[a-z]' matches any character between a and z, inclusive; `[^a-z]' matches any single character except those 26 --- but notice it still matches a single character.

A recent common enhancement to character ranges features in zsh, which is to specify types of characters instead of listing them; I'm just repeating the manual entry here, which you should consult for more detail. The special syntax is like `[:spec:]', where the square brackets there are in addition to the ones specifying the range. If you are familiar with the `ctype' macros use in C programmes, you will probably recognise the things that spec can be: alnum, alpha, blank, cntrl, digit, graph, lower, print, punct, space, upper, xdigit. The similarity to C macros isn't just for show: the shell really does call the macro (or function) `isalpha' to test for [:alpha:]ness, and so on. On most modern systems which support internationalization this means the shell can tell you whether a character is, say, an alphabetic letter in the character set in use on your machine. By the way, zsh doesn't use international character set support for sorting matches --- this turned out to produce too many unexpected effects.

So `[^[:digit:]]' matches any single character other than a decimal digit. Standards say you should use `!' instead of `^' to signify negation, but most people I know don't; also, this can clash with history substitution. However, it is accepted by zsh anywhere where history substitution doesn't get its hands on the `!' first (which includes all scripts and autoloaded functions).

5.9.3: Extensions usually available

Now we reach the bits specific to zsh. I've divided these into two parts, since some require the option `EXTENDED_GLOB' to be set --- those which are most likely to clash with other uses of the characters in question.

Numeric ranges

One possibility that is always available is the syntax for numeric ranges in the form `<num1-num2>'. You can omit either num1, which defaults to zero, or num2, which defaults to infinity, or both, in which case any set of digits will be matched. Note that this really does mean infinity, despite the finite range of integers; missing out num2 is treated as a special case and the shell will simply advance over any number of digits. (In very old versions of zsh you had to use `<>' to get that effect, but that has been removed and `<>' is now a redirection operator, as in other shells; `<->' is what you need for any set of digits.)

I repeat another warning from the manual: this test

  [[ 342 = <1-30>* ]]

  [[ 342 = <1-30>(|[^[:digit:]]*) ]]

The numbers in the range are always positive integers; you need extra pattern trickery to match floating point. Here's one attempt, which uses EXTENDED_GLOB operators, so come back and look when you've read the rest of this section if it doesn't make sense now:

  isfloat() {
    setopt localoptions extendedglob
    if [[ $1 = ([-+]|)([0-9]##.[0-9]#|[0-9]#.[0-9]##)\ 
([eE]([-+]|)[0-9]##|) ]]; then
      print -r -- "$1 is a floating point number"
    else
      print -r -- "$1 is not a floating point number"
    fi
  }

The range operator is the only special zsh operator that you can't turn off with an option. This is usually not a problem, but in principle a string like `<3-10>' is ambiguous, since in another shell it would be read as `<3-10 >', meaning `take input from file 3-10, and send output to the file formed by whatever comes after the expression'. It's very unlikely you will run across this in practice, however, since shell code writers nearly alwys put a space after the end of a file name for redirection if something else follows on the command line, and that's enough to differentiate it from a range operator.

Parentheses

Parentheses are quite natural in zsh if you've used extended regular expressions. They are usually available, and only turned off if you set the `SH_GLOB' option to ensure compatibility with shells that don't have it. The key part of the expression is the vertical bar, which specifies an alternative. It can occur as many times as necessary; `(a|b|c|d|e|f|g|h|i|j|k|l|m)' is a rather idiosyncratic way of writing `[a-m]'. If you don't include the vertical bar (we'll see reasons for not doing so later), and you are generating filenames, you should be careful that the expression doesn't occur at the end of the pattern, else it would be taken as a `glob qualifier', as described below. The rather unsightly hack of putting `(|)' (match the empty string or the empty string --- guess what this matches?) right at the end will get around that problem.

The vertical bar usually needs to be inside parentheses so that the shell doesn't take it as a pipe, but in some contexts where this won't happen, such as a case statement label, you can omit any parentheses that would completely surround the pattern. So in

  case $foo in
    (bar|rod|pipe) print "foo represents a piece of metal"
              ;;
    (*) print "Are you trying to be different?"
        ;;
  esac

Indeed, ksh has slightly different ways of specifying patterns: to make the use of parentheses less ambiguous, it requires a character before the left parenthesis. The corresponding form for a simple alternative is `@(this|that)'. The `@' can also be a `?', for zero or one occurrences of what's in the parentheses; `*' for any number of repetitions, for example `thisthisthatthis'; or `!' for anything except what's in the parentheses. Zsh allows this syntax if you set the option KSH_GLOB. Note that this is independent of the option SH_GLOB; if you set KSH_GLOB but not SH_GLOB, you can actually use both forms for pattern matching, with the ksh form taking precedence in the case of ambiguities. This is probably to be avoided. In ksh emulation, both options are set; this is the only sensible reason I know of for using these options at all. I'll show some comparisons in the next section.

An important thing to note is that when you are matching files, you can't put directory separators inside parentheses:

  # Doesn't work!
  print (foo/bar|bar/foo)/file.c

5.9.4: Extensions requiring EXTENDED_GLOB

Setting EXTENDED_GLOB makes three new types of operator available: those which excluded a particular pattern from matching; those which specify that a pattern may occur a repeated number of times; and a set of `globbing flags', a little bit like parameter flags which I'll describe in a later section since they are really the icing on the cake.

Negative matches or exclusions

The simpler of the two exclusions uses `^' to introduce a pattern which must not be matched. So a trivial example (I will assume for much of the rest of the chapter that the option EXTENDED_GLOB is set) is:

  [[ foo = ^foo ]]
  [[ bar = ^foo ]]

  [[ foo = *^foo ]]

Note that the ^ applies to the whole pattern to its right, either to the end of the string, or to the end of the nearest enclosing parenthesis. Here's a couple more examples:

  [[ foo = ^foo* ]]

  [[ foo = (^foo)* ]]

In filename generation, the ^ has a lower precedence than a slash:

  % print /*/tmp
  /data/tmp /home/tmp /usr/tmp /var/tmp
  % print /^usr/tmp
  /data/tmp /home/tmp /var/tmp

Note one point mentioned in the FAQ --- probably indicating the reason that `^' is only available with EXTENDED_GLOB switched on. Some commands use an initial `^' to indicate a control character; in fact, zsh's bindkey builtin does this:

  bindkey '^z' backward-delete-word

There's another reason this isn't available by default: in some versions of the Bourne shell, `^' was used for pipes since `|' was missing on some keyboards.

The other exclusion operator is closely related. `pat1~pat2' means `anything that matches pat1 as long as it doesn't also match pat2'. If pat1 is *, you have the same effect as `^' --- in fact, that's pretty much how `^' is currently implemented.

There's one significant difference between `*~pat' and `^pat': the ~ has a lower precedence than `/' when matching against filenames. What's more, the pattern on the right of the ~ is not treated as a filename at all; it's simply matched against any filename found on the left, to see if it should be rejected. This sounds like black magic, but it's actually quite useful, particularly in combination with the recursive globbing syntax:

   print **/*~*/CVS(/)

Note that in that example, any subdirectory of a directory called CVS would have matched successfully; you can see from the pattern that the expression after the `~' wouldn't weed it out. Slightly less obviously, the `**/*' matches files in the current directory, while the `*/CVS' never matches a `CVS' in the current directory, so that could appear. If you want to, you can fix that up like this:

   print **/*~(*/|)CVS(/*|)(/)

   print ./**/*~*/CVS(/)

You can restrict the range of the tilde operator by putting it in parentheses, so `/(*~usr)/tmp' is equivalent to `/^usr/tmp'.

A `~' at the beginning is never treated as excluding what follows; as you already know, it has other uses. Also, a `~' at the end of a pattern isn't special either; this is lucky, because Emacs produces backup files by adding a `~' to the end of the file name. You may have problems if you use Emacs's facility for numbered backup files, however, since then there is a `~' in the middle of the file name, which will need to be quoted when used in the shell.

Closures or repeated matches

The extended globbing symbols `#' and `##', when they occur in a pattern, are equivalent to `*' and `+' in extended regular expr