upvote
Not quoting the tilde doesn't help:

    : tmp; echo $PATH:~
    /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:~
Edit: mjmas points out that I am wrong, because different Calvinball rules apply to variable assignments:

    : tmp; x=$PATH:~
    : tmp; echo "$x"
    /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user
Also in general your advice is very bad advice. In command-line arguments, which is the vast majority of the code of any shell script, you should always quote strings that contain variable expansion unless you want them to be implicitly split on spaces after variable expansion, because the thing you're storing in the variable is not an atomic string but rather a space-separated list.

This is almost never what you actually want, and in the rare cases that you do want to store a list, the shell's implicit space splitting usually breaks on filenames containing spaces. Bash has actual array variables which make it possible, but horrible, to handle this in a first-class way:

    : tmp; x=("foo bar" baz)
    : tmp; echo "${x[@]}"
    foo bar baz
    : tmp; touch "${x[@]}"
    : tmp; ls -l "${x[@]}"
    -rw-r--r-- 1 user user 0 Oct 11 17:52  baz
    -rw-r--r-- 1 user user 0 Oct 11 17:52 'foo bar'
This ksh feature is absent in the Bourne shell and in dash, but MirBSD ksh, Bash, and zsh all have it.

In general, whenever you see a $variable $expansion in a shell script outside of double quotes, you should suspect that the shell script will probably fail if your filenames or directory names contain spaces. In very many cases, this results in path injection security vulnerabilities. There are contexts where unquoted $variable $expansion is safe, but they are relatively rare.

However, relevant detail here! One of those safe contexts is actually variable assignment, where as mjmas pointed out in their helpful comment below, unquoted variable expansion is actually perfectly safe:

    : tmp; a='x  y'
    : tmp; b=α:$a:ω
    : tmp; echo "$b"
    α:x  y:ω
reply
What's the `: tmp;` incantation for?
reply
Sorry, that's my shell prompt. It's designed to be harmless if you accidentally copy and paste it — the opposite extreme from tcsh's default prompt of >.
reply
Bash does it only inside variable assignments:

  a=123:~
  echo $a
  echo 123:~
results in:

  123:/home/me
  123:~
reply
Oh, thank you for the correction. I was wrong about that. Calvinball.
reply
I feel like this is common knowledge and should not be worth mentioning. But then it apparently is not common knowledge, as the article proves.

Have I run into this at some point?

I certainly have.

Have I learned to quote better and only where appropriate from it?

I certainly have.

Bourne compatible shells take a while to learn and require some experience. This won't change, but alternatives exist, with their own caveats.

reply
Almost nobody that I've ever worked with knows how the shell works. I send them [the docs][1] but what kind of world would this be if people read the docs? Also, not exactly a page turner.

[1]: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V...

reply
I learned this stuff from friends, O'Reilly books and endless nights of failure and trial and error. I'm certainly not a super wizard, but I feel like I'm not missing any shell skills to be able to do what I need to do.

[Insert mild rant on kids these days having no attention span any more]

reply
also poorly organized and formatted.
reply
This feature, or at least the syntactic unit in question has a name here "bare words". You also have these in some contexts in Perl and Ruby, YAML, and probably some other languages, idk.

I've also seen this even with people who seem like generally competent shell users. Idrgi

reply
Unless you're running a shell command from python. That was the first time I saw a command string broken down into "string" arguments for every thing like that.
reply
In that case, you're not running a "shell command" from python, you're passing arguments to exec. A shell command would be a string interpreted by the shell, and you'd use that for shell syntax things like having the shell do variable interpolation or redirections as part of executing the command.
reply
[dead]
reply
The trick is to quote explicitly and correctly. “ and ‘ are different.
reply
why would you use smart quotes in a terminal like that?
reply
They probably fell foul of some browser text box auto-correct.
reply
' "

I don't use WYSIWYG editors anymore, so I have to remind myself to use those when writing raw HTML text. Although, most browsers correct raw ' and " symbols in text now, I still try to use them to have compliant HTML

reply
why would anyone use smart quotes ever
reply
They are typographically the right thing to have been using all along.

Using ' and " to pretend to be ‘/’ or “/” is on par with the typewriter days where people would use the l key to stand in for 1 also. A justifiable approximation when technology limitations prevented using the real deal, but an approximation all the same.

reply
In fact, if we had been using left and right quotes from the beginning in shells, most “quoting problems” go away, as they’re all inherently rooted in not being able to know what level of nesting a quote character is at.
reply
m4 did in fact use left and right quotes from the beginning, but it still had other quoting problems. Really terrible ones, to the point of making the language unusably bug-prone for anything beyond very basic tasks, which is why it's mostly forgotten.
reply
Can't that be handled by the typesetting/rendering software and displayed appropriately regardless of which specific character was typed?
reply
No, because it's missing semantic information carried by the distinction. The ’ at the beginning of

    ’Tis brillig, and the slithy toves
is not a left quote ‘ but an apostrophe ’, like the apostrophe in “can’t”; but ‘tis certainly possible that someone might want to put single quotes around a phrase beginning with the word “tis”, such as the Latin ‘tis misereri’. By contrast to “can’t”, the ‘ in “Hawai’i” is the ’okina used to represent the glottal-stop sound in the Hawai’ian language, not an apostrophe.
reply
Except l has a specific meaning that isn't 1, while the entire purpose of ' and " is quoting (and apostrophe).

The equivalent of l for 1 is doing font-specific pseudo smart quotes with ` and '

reply
" and ' are themselves conjoined with other uses.

" can be inches or arcseconds from cartography.

' can be feet (of measure) or arcminutes from cartography.

These actually all have slightly different symbols, and the symbols for the left/right quotation marks are actually farthest from this approximation, even if they are the most frequent usage.

It’s always been wrong in some respect to use a single available symbol to emulate three or more different typographical tasks, but quotation marks may actually be the one where the emulation is the most wrong.

reply
Overstriking ' was commonly used (on typewriters and ASCII printers) for acute accents, and overstriking " was commonly used for a diaeresis. The glyphs used in typewriter fonts and traditional (pre-Unicode) Unix terminal fonts are compromises between these different uses.
reply
Aside: in Unicode we now have ″ and ′ for that. Also the triple variant: ‴
reply
That equivalency only holds if there is a visual distinction in the output between the l and the 1. There often wasn't when this practice was popular.
reply
As others have commented, for the same reason that we use smart parentheses instead of the more utilitarian |.
reply
People quote both too often and too little.

GENERAL RULE

1. Double-quote dollar sign expressions, and nothing else.

  foo

  "$bar"/foo

  baz:"$(cat example.txt)"

  exec cmd "$@"
2. Single-quote words with a literal special character, and nothing else.

  'Die Hard'

  'ke$ha'
---

I should point out that the author's example is NOT fixed by different quoting though.

  # original
  export PATH="$PATH:~/.local/bin/"

  # without unnecessary quotes
  export PATH="$PATH":~/.local/bin/
Because tilde expansion only happens at the beginning of the word.
reply
> I should point out that the author's example is NOT fixed by different quoting though.

Sure, but I would say you almost always want to add to the start of the PATH, not to the end.

reply
I used to put it first for convenience, but got increasingly paranoid about something same-named getting slipped into my ~/.local/bin; now it lives at the end.
reply
The Z shell's, C shell's, and others's syntaxes for setting the PATH environment variable via a shell array variable alias also does the tilde expansion.

    path=( $path ~/bin )
It's worth noting, also, that the path and manpath settings in login.conf(5) expand leading tildes in individual search path items.

* https://man.freebsd.org/cgi/man.cgi?query=login.conf&sektion...

So putting the addition of things like ~/bin to PATH in /etc/login_conf and ~/.login_conf instead of shell scripts is another way to address it.

    :path=~/bin /usr/local/bin /usr/pkg/bin /usr/bin /bin:
It's particularly handy when there are multiple login shells in use.

* http://jdebp.uk./FGA/BSDs-for-Linux-users/login-conf.html

reply
Or just stop using bash. It’s a terrible language to write and has tons of footguns.
reply
For the things shell is good at (running other commands, pipes, and shuffling files), I have yet to find anything even close to as good.
reply
I agree that shells are uniquely good for that but there are plenty of better shells than Bash.

Such as Fish, nushell, Elvish, or the project I help maintain, “murex”

reply
None that I can rely on being available wherever I see a terminal. It's bash or sh as far as I'm concerned.
reply
Fortunately, 99.9% of the time I’m on my computer, where I can install whatever I want.
reply
Then you ssh into an instance and have to remember how bash/sh work
reply
I can’t think of many occasions where I’ve needed SSH access to a box (ie they’re not just a fleet of ephemeral systems) but I’ve also not had the permission to install an alternative shell on it. I’m not saying it never happens. But it’s not as common a problem as it was 20 years ago.

Unfortunately though, it’s not generally that hard to switch between different shells. I hear the argument you made a lot from other people too and yet we all acknowledge you should be able to code in Bash and a “proper” programming language (Python, Go, JavaScript, C, Rust, whatever). In fact I’ve learned probably 2 dozen languages in my career and it’s rare when I need to pause and remind myself how to do x in y.

Or to put it another way: you don’t say “I’m not going to use $IDE on my dev machine because I’m stuck with vi on the remote server”.

But if people don’t feel comfortable working in two different shells and that is the dealbreaker for using murex, or any other alternative shell, then I’m not going to hold that against them. Everyone has their own preference when it comes to workflows and productivity tools.

reply
At what point does an alternate shell get baked into the image?
reply
deleted
reply
A different shell??? Nushell, fish, powershell, etc.
reply
Oh, you mean bash specifically. Then yeah, that's reasonable enough. Although I agree with the other commenter that sh/bash have the advantage of ubiquity.
reply
None of those have the #1 best thing bash is good at: bash is already installed, nushell and fish are not.

Powershell is, as I understand it, not available on Linux but is omnipresent on Windows, so hopefully the windows folks can use it like they would bash.

reply
No, powershell runs on Linux, it just sucks because it wants objects and everything on *nix speaks streams of text.
reply
PowerShell works on Windows because of WMI, to a first approximation.

Because Windows is tied together through APIs and databases and not text files (unlike Linux), everything is text is not as useful as something that can talk objects. And a lot of Windows is object oriented, from the window message dispatching system through COM and down to NT's object manager.

reply
deleted
reply
I did.

Ruby replaced all my shell needs. Almost 25 years ago. I even have a shell written in ruby (it handles both bash-like behaviour as well as ruby code as-is); admittedly it is not quite perfect for everything, but I improve on it steadily. And it works on Windows too, which was one reason I wrote it in the first place (need to have it work via cmd.exe as-is).

Never looked back to shell. It is too awful to use.

reply
Unfortunately half of its badness isn't isn't the shell itself, but the convention of how parameters are passed to processes on Unix systems.
reply
That’s not correct because POSIX passes what is ostensibly an array of strings.

Windows, on the other hand, only passes one string. So it’s up to the application to choose how to handle whitespace, quotation marks, and other nuances with parsing parameters.

Variable expansion in Bash is lazy. But there’s no reason why variables cannot be tokenised so that strings with spaces aren’t treated as multiple parameters. And in fact that’s exactly how some other shells work, such as the one I maintain.

reply
While unix handles splitting into an array of strings, it still leaves parsing those strings up to the application. One of the bigger problems is the ambiguity caused by filenames starting with `-`. Some applications support the `--` marker to end such parsing, but it's inconsistently implemented and the caller has to actively remember to use it.

A typed array (e.g. by having a required single byte marker at the start of each string), or even nested structures similar to s-expressions would have avoided this.

reply
Funny enough, I’ve actually spent a long time trying to figure out how to solve this problem in murex but it always comes back to the same problem: anything smart I implement into the shell is immediately lost the moment I call fork().

I even considered writing a wrapper around some commands to send -- regardless of whether the user includes it or not. But that’s error prone too because

1. it’s a GNUism so you can’t guarantee compatibility with any tools that don’t call GNUs flag parsing library (which is particularly problematic outside of Linux)

2. If there a file called “-f” (for example), how do you know if the user intended that to be called as a file reference or a command flag? You then need to bake in a bunch of additional syntax sugar to make that explicit. Which results in a pretty awful user experience

3. The maintenance overhead for this would be astronomical

If UNIX were designed from day one to have that parameter type passed then this would be a very simple problem to solve. But the PDP machine UNIX was originally designed to run on wouldn’t have been powerful enough for that anyway. So once again we are limited by an architecture designed to run on mid-range systems of the 1970s.

reply
Array of arguments is vastly superior and more secure than every program/runtime inventing a slightly different way of splitting a command string into an array of arguments. No debate. A real problem is the related birth defect in ssh2.
reply
Well, the only other way I can think of that it could be done is the Windows way, whereby you pass the unparsed command line, spaces and all, as a string to the new process. And while this is arguably the cleaner interface, in practice it has meant even worse quote handling, since how -- or even whether -- double quotes are parsed now depends on the probably undocumented process startup code chosen by the program's compiler vendor.

Want to quote a command line that may already contain double quotes, in order to pass it as an argument to some other program? No, you don't. It isn't right to want that.

reply
The other other way would be more structured. Arguments are not just an array, they also contain `--switch` and key/value pairs (e.g. `--key value` or `--key=value` depending on who you ask). One problem with the flat array approach is there's no way to distinguish a literal value starting with `-` from a switch, which means there needs to be some way to workaround that (and users have to remember the workaround; how often do people remember to use `rm -- "$FILENAME"`).

In practice almost every application does still need to do its own parameter parsing; a flat array is not enough.

reply
yeah but... that convention is so much of a security/bug headache

sometimes, its footguns seem worse than javascript...

hope some typescript-like "typed shell" becomes mainstream someday

reply
> hope some typescript-like "typed shell" becomes mainstream someday

The trouble with trying to invent a new, more robust shell language is you basically just end up re-inventing any number of scripting languages (eg Perl, Python, awk, ...), so you might as well use one of those.

reply
Most regular programming languages aren’t well suited for shells because they have a verbose syntax due to their readability goals. But with a shell, the vast majority of times you’re typing in stuff that you have no intention of reading back ever again.

I’ve done a fair amount of research here and I actually think we do need a new programming language for the shell (and then I created one).

I wrote a blog about this problem: https://murex.rocks/blog/split_personalities.html#conclusion

reply
Excellent article, thanks for the link. I do agree with the premise. But every time I write some Bash code and think there should be a better way to do this, I ask myself why I don't just learn Perl. I dunno. Feels like inventing another solution would just hit the old "there are now 14 standards" problem where it would solve some problems but introduce others (see: PowerShell).
reply
I don’t disagree with you per se. But if the new shell solves enough annoying problems then I think it has merit in existing.
reply
> hope some typescript-like "typed shell" becomes mainstream someday

Might want to check out nushell (https://www.nushell.sh/)

reply
PowerShell
reply
The widwit answer: "Dont do X" (and nothing more)

The enlightened answer "You should use A, B or C for these reasons"

Even though bash is installed on many systems, I try to encourage people to use better designed shells that have less of these footguns :

Fish (https://fishshell.com/): No implicit word splitting : spaces in variables won't unexpectedly become separate arguments.

Zsh (I use this: https://ohmyz.sh/): Arrays start at 1 by default, but crucially, unquoted variables don't implicitly split into multiple arguments.

Nushell (https://www.nushell.sh/): Passes structured tables and records between commands : avoids fragile parsing of text with awk/grep.

reply
You need to read the manual.
reply
I think the appropriate method by POSIX rules would be:

  export PATH="$PATH:"~/.local/bin
I may be wrong. If I'm not, that works in any POSIX-compliant shell.

Edit: this appears to work properly with mksh on my phone:

  :/ $ export PATH="$PATH:"~/.local/bin
  :/ $ print $PATH
  /product/bin:/apex/com.android.runtime/bin:/apex/com.android.art/bin:/system_ext/bin:/system/bin:/system/xbin:/odm/bin:/vendor/bin:/vendor/xbin:~/.local/bin
reply
That's a literal ~ in your path, the exact problem the blog post talks about.

Your use doesn't count as a "word", per man bash. Tilde is only expanded to home at the start of a typically whitespace-separated word, and your tilde is in the middle of one.

> If a word begins with an unquoted tilde character (‘~’), all of the characters up to the first unquoted slash (…) are considered a tilde-prefix. (…)

> word A sequence of characters considered as a single unit by the shell. Also known as a token.

reply
My initial syntax does work in dash (and bash), but all these shells appear to rely on expanding ~ inside a word, that your source asserts is not POSIX (which I do not contest).

Perhaps a succinct and compliant expression could be:

  export PATH="$PATH:$(printf %s ~/.local/bin)"
That comes at the cost of forking a subsell.

Edit: 2.6.1 Tilde Expansion in the POSIX standard says that ~ may be expanded "following any unquoted <colon>".

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

That being so, the most succinct and compliant version is my first variant, with the colon moved outside the quotes:

  export PATH="$PATH":~/.local/bin
Thank you for prompting me to look this up.
reply