INSTALL: Make the use of [, ], { and } consistent and explain it

The diverse notations used in INSTALL are not as self explanatory as
we might imagine, so let's attempt a consistent notation for mandatory
and optional pieces of a command line, and to explain the meaning of
each notation.

This does away with the bash notation used in one spot, as it isn't
universally understood and will only confuse the unknowing more.

Reviewed-by: Rich Salz <rsalz@openssl.org>