path: root/public/bash-arrays.md

Bash arrays
===========
---
date: 2013-10-13
---

Way too many people don't understand Bash arrays.  Many of them argue
that if you need arrays, you shouldn't be using Bash.  If we reject
the notion that one should never use Bash for scripting, then thinking
you don't need Bash arrays is what I like to call "wrong".

The simple explanation of why everybody who programs in Bash needs to
understand arrays is this: command line arguments are exposed as an
array.  Does your script take any arguments on the command line?
Great, you need to work with an array!

Normal array syntax
-------------------

<table>
  <caption>
    <h1>Setting an array</h1>
    <p><var>tokens...</var> is expanded and split into array elements
       the same way command line arguments are.</p>
  </caption>
  <tbody>
    <tr>
      <td><code>array=(<var>tokens...</var>)</code></td>
      <td>Set the contents of the entire array.</td>
    </tr><tr>
      <td><code>array+=(<var>tokens...</var>)</code></td>
      <td>Appends <var>tokens...</var> to the end of the array.</td>
    </tr><tr>
      <td><code>array[<var>n</var>]=<var>value</var></code></td>
      <td>Sets an individual entry in the array, the first entry is at
          <var>n</var>=0.</td>
    </tr>
  </tbody>
</table>

Now, for accessing the array.  The most important things to
understanding arrays is to quote them, and understanding the
difference between `@` and `*`.

<table>
  <caption>
    <h1>Getting an entire array</h1>
    <p>There is almost <em>no</em> valid reason to not wrap these in
       double quotes.</p>
  </caption>
  <tbody>
    <tr>
      <td><code>"${array[@]}"</code></td>
      <td>Returns every element of the array as a separate token.</td>
    </tr><tr>
      <td><code>"${array[*]}"</code></td>
      <td>Returns every element of the array in a single
          whitespace-separated string.</td>
    </tr>
  </tbody>
</table>

It's really that simple—that covers most usages of arrays, and most of
the mistakes made with them.

To help you understand the difference between `@` and `*`, here is a
sample.

<pre><code>#!/bin/bash
array=(foo bar baz)
for item in "${array[@]}"; do
        echo " - &lt;${item}&gt;"
done<hr> - &lt;foo&gt;
 - &lt;bar&gt;
 - &lt;baz&gt;</code></pre>

<pre><code>#!/bin/bash
array=(foo bar baz)
for item in "${array[*]}"; do
        echo " - &lt;${item}&gt;"
done<hr> - &lt;foo bar baz&gt;</code></pre>

To get individual entries, the syntax is
<code>${array[<var>n</var>]}</code>, where <var>n</var> starts at 0.

<table>
  <caption>
    <h1>Getting a single entry from an array</h1>
  </caption>
  <tbody>
    <tr>
      <td><code>"${array[<var>n</var>]}"</code></td>
      <td>Returns the <var>n</var>th entry of the array, where the
          first entry is at <var>n</var>=0.</td>
    </tr>
  </tbody>
</table>

To get a subset of the array, there are a few options (like normal,
switch between `@` and `*` to switch between
getting it as separate items, and as a whitespace-separated string):

<table>
  <caption>
    <h1>Getting subsets of an array</h1>
    <p>Substitute <code>*</code> for <code>@</code> to get the subset
       as a whitespace-separated string instead of separate tokens, as
       described above.</p>
    <p>Again, there is almost no valid reason to not wrap each of
       these in double quotes.</p>
  </caption>
  <tbody>
    <tr>
      <td><code>"${array[@]:<var>start</var>}"</code></td>
      <td>Returns from <var>n</var>=<var>start</var> to the end of the array.</td>
    </tr><tr>
      <td><code>"${array[@]:<var>start</var>:<var>count</var>}"</code></td>
      <td>Returns <var>count</var> entries, starting at <var>n</var>=<var>start</var>.</td>
    </tr><tr>
      <td><code>"${array[@]::<var>count</var>}"</code></td>
      <td>Returns <var>count</var> entries from the beginning of the array.</td>
    </tr>
  </tbody>
</table>

Notice that `"${array[@]}"` is equivalent to `"${array[@]:0}"`.

<table>
  <caption>
    <h1>Getting the length of an array</h1>
    <p>The is the only situation where there is no difference
       between <code>@</code> and <code>*</code>.</p>
  </caption>
  <tbody>
    <tr>
      <td>
        <code>${#array[@]}</code>
        <br>or<br>
        <code>${#array[*]}</code>
      </td>
      <td>
        Returns the length of the array
      </td>
    </tr>
  </tbody>
</table>

Argument array syntax
---------------------

Accessing the arguments is mostly that simple, but that array doesn't
actually have a variable name.  It's special.  Instead, it is exposed
through a series of special variables (normal variables can only start
with letters and underscore), that *mostly* match up with the normal
array syntax.

Setting the arguments array, on the other hand, is pretty different.
That's fine, because setting the arguments array is less useful
anyway.

<table>
  <caption>
    <h1>Accessing the arguments array</h1>
    <aside>Note that for values of <var>n</var> with more than 1
           digit, you need to wrap it in <code>{}</code>.
           Otherwise, <code>"$10"</code> would be parsed
           as <code>"${1}0"</code>.</aside>
  </caption>
  <tbody>
    <tr><th colspan=2>Individual entries</th></tr>
    <tr><td><code>${array[0]}</code></td><td><code>$0</code></td></tr>
    <tr><td><code>${array[1]}</code></td><td><code>$1</code></td></tr>
    <tr><td colspan=2 style="text-align:center">...</td></tr>
    <tr><td><code>${array[9]}</code></td><td><code>$9</code></td></tr>
    <tr><td><code>${array[10]}</code></td><td><code>${10}</code></td></tr>
    <tr><td colspan=2 style="text-align:center">...</td></tr>
    <tr><td><code>${array[<var>n</var>]}</code></td><td><code>${<var>n</var>}</code></td></tr>
    <tr><th colspan=2>Subset arrays (array)</th></tr>
    <tr><td><code>"${array[@]}"</code></td><td><code>"${@:0}"</code></td></tr>
    <tr><td><code>"${array[@]:1}"</code></td><td><code>"$@"</code></td></tr>
    <tr><td><code>"${array[@]:<var>pos</var>}"</code></td><td><code>"${@:<var>pos</var>}"</code></td></tr>
    <tr><td><code>"${array[@]:<var>pos</var>:<var>len</var>}"</code></td><td><code>"${@:<var>pos</var>:<var>len</var>}"</code></td></tr>
    <tr><td><code>"${array[@]::<var>len</var>}"</code></td><td><code>"${@::<var>len</var>}"</code></td></tr>
    <tr><th colspan=2>Subset arrays (string)</th></tr>
    <tr><td><code>"${array[*]}"</code></td><td><code>"${*:0}"</code></td></tr>
    <tr><td><code>"${array[*]:1}"</code></td><td><code>"$*"</code></td></tr>
    <tr><td><code>"${array[*]:<var>pos</var>}"</code></td><td><code>"${*:<var>pos</var>}"</code></td></tr>
    <tr><td><code>"${array[*]:<var>pos</var>:<var>len</var>}"</code></td><td><code>"${*:<var>pos</var>:<var>len</var>}"</code></td></tr>
    <tr><td><code>"${array[*]::<var>len</var>}"</code></td><td><code>"${*::<var>len</var>}"</code></td></tr>
    <tr><th colspan=2>Array length</th></tr>
    <tr><td><code>${#array[@]}</code></td><td><code>$#</code> + 1</td></tr>
    <tr><th colspan=2>Setting the array</th></tr>
    <tr><td><code>array=("${array[0]}" <var>tokens...</var>)</code></td><td><code>set -- <var>tokens...</var></code></td></tr>
    <tr><td><code>array=("${array[0]}" "${array[@]:2}")</code></td><td><code>shift</code></td></tr>
    <tr><td><code>array=("${array[0]}" "${array[@]:<var>n+1</var>}")</code></td><td><code>shift <var>n</var></code></td></tr>
  </tbody>
</table>

Did notice what was inconsistent? The variables `$*`, `$@`, and `$#`
behave like the <var>n</var>=0 entry doesn't exist.

<table>
  <caption>
    <h1>Inconsistencies</h1>
  </caption>
  <tbody>
    <tr>
      <th colspan=3><code>@</code> or <code>*</code></th>
    </tr><tr>
      <td><code>"${array[@]}"</code></td>
      <td>→</td>
      <td><code>"${array[@]:0}"</code></td>
    </tr><tr>
      <td><code>"${@}"</code></td>
      <td>→</td>
      <td><code>"${@:1}"</code></td>
    </tr><tr>
      <th colspan=3><code>#</code></th>
    </tr><tr>
      <td><code>"${#array[@]}"</code></td>
      <td>→</td>
      <td>length</td>
    </tr><tr>
      <td><code>"${#}"</code></td>
      <td>→</td>
      <td>length-1</td>
    </tr> 
  </tbody>
</table>

These make sense because argument 0 is the name of the script—we
almost never want that when parsing arguments. You'd spend more code
getting the values that it currently gives you.

Now, for an explanation of setting the arguments array.  You cannot
set argument <var>n</var>=0.  The `set` command is used to manipulate
the arguments passed to Bash after the fact—similarly, you could use
`set -x` to make Bash behave like you ran it as `bash -x`; like most
GNU programs, the `--` tells it to not parse any of the options as
flags. The `shift` command shifts each entry <var>n</var> spots to the
left, using <var>n</var>=1 if no value is specified; and leaving
argument 0 alone.

----

2013-12-06 update: When it's okay to not quote arrays
-----------------------------------------------------

I mentioned that there is "almost no" valid reason to not wrap
`${array[@]}` in double-quotes.  Seriously, you probably want to put
it in quotes.  In an earlier version, I wrote "no", and in an even
earlier draft I wrote "no reason that I can think of".  Well, I just
changed it to "almost no", because I thought of a reason:

It is okay to not quote it when you are doing field-separator
manipulations.  For example:

	# Usage: path_la PATH1 PATH2...
	# Description:
	#   Takes any number of PATH-style values; that is,
	#   colon-separated lists of directories, and prints a
	#   newline-separated list of executables found in them.
	# Bugs:
	#   Does not correctly handle programs with a newline in the name,
	#   as the output is newline-separated.
	path_ls() {
		local dirs
		IFS=: dirs=($@) # The odd-ball time that it needs to be unquoted
		find -L "${dirs[@]}" -maxdepth 1 -type f -executable -printf '%f\n' 2>/dev/null | sort -u
	}

The explanation is that no field-separator evaluation is done when you
quote the array.  This is almost always what you want—the array is
already field-separated; you don't want that to be re-evaluated, and
possibly changed.  However, if you do have character-separated fields
that you want to get at, you do want field-separation to be
re-evaluated.

But seriously, quote your arrays by default.  If they need to be
unquoted, you should think long and hard before doing so.