Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.)
Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation.
As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]:
$ podman image ls --all
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/prom/prometheus latest 937690d77350 2 months ago 367 MB
quay.io/keycloak/keycloak latest da9433c9fac3 2 months ago 466 MB
registry.fedoraproject.org/fedora-toolbox 43 a32da54355ca 4 months ago 2.19 GB
docker.io/powerdns/pdns-auth-49 latest 8c1385c9deed 4 months ago 208 MB
docker.io/testcontainers/ryuk 0.13.0 b75bc7ce94c3 6 months ago 7.21 MB
Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline:
podman image ls --all |
Replace-SpacesWithTabs |
ConvertFrom-Csv -Delimiter "`t" |
Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending
podman image ls --all | sed 's/\s\s\+/\t/g' | tee >(head -n 1) >(tail -n +2 | sort -hrk 5) >/dev/null
docker image ls -a | stdbuf -oL sed -r 's/\s{2,}/\t/g' | { head -n1; tail -n+2 | sort -hrk5 -t$'\t'; } | column -ts$'\t'
sort's -t is set to tab for field separation.
stdbuf sets sed's output to only buffer a line at a time and flush, so the head in the {...} command group doesn't completely consume stdin's contents before it's passed to tail.
The column command recreates the space-aligned table based on tab-delimited input.
I like using column to format the table. Appending it to alloyed's command fixes their header problem.
The stdbuf to multi-command block (term.?) is a neat trick. Although, one time when I ran this, I only got a couple lines of output. No idea why and I can't replicate it, but there could be some flakiness that results from the buffering somehow?
Question: how do the ? markers on the sort and column invocations work/what do they do?
Unfortunately, it's not 100% correct, due to misaligned headers:
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.fedoraproject.org/fedora-toolbox 44 5a36f433c691 2 months ago 2.14 GB
quay.io/keycloak/keycloak latest 1361d6e49205 9 days ago 478 MB
...
> I'd probably just end up dropping the header and living with worse output in reality
This pretty much sums up plain text and unix shell imo. It's very much the pragmatic solution here, and it's what ~100% of shell scripters would choose to do. And it should make anyone question the orthodoxy around the "power" of plain text in shells.
$ podman image ls --all --sort=size
Baking --sort flags into shell tools is a sign that the tools do not compose well.
Basically Unix has a long tradition of "everything is a file" and a big ecosystem of coreutils that are based around text and windows.. didn't. You can't look at /dev or /etc and learn anything about the machine. They had a few generations of APIs and wanted to give admins and power users any shell at all instead of a GUI. So the shell is centered around making those APIs accessible, rather than piping grep and sed or whatever.
Linux tools that are starting to output raw JSON by default are indeed a nuisance, but how else can you achieve structured output if no standard shell supports it? It's a chicken and egg problem.
here is an elvish shell command that converts a freetube playlist from json into a list of urls grouped by author:
for i (cat 'freetube-playlist-favorites.db' | from-json)["videos"] {
mkdir -p $i['author']
print http://youtu.be/$i['videoId'] >> $i['author']/get }
curl -s -H "Authorization: token <redacted>" "https://api.zerotier.com/api/v1/network/<redacted>/member" |
all (from-json) |
order &key={|d| put $d[name]} |
each { |device|
var t = (printf "%.0f" (/ $device[lastSeen] 1000))
if (> 20000000 (- (date '+%s') $t)) {
print (date -R --date='@'$t) $device[config][ipAssignments] $device[name] "\n" } }
In the AWS case the tools talk to an API server, so sure, you can call the API server directly or use a wrapper that does, but what about all the other CLI apps that don't? The CLI program is the API.
I built a CLI program that wraps luks + btrfs, and they only offer a `--json` output option for a few commands. I have to write an ad hoc parser for each command since raw text includes arbitrary formatting and presentation lipstick that the creator came up with. And I have to do extra work to avoid breaking changes at the parser level instead of the higher data level.
If I had to pick between the two, json would at least solve the data representation part so that I can build on top of it. And it's trivial to go data (json) -> pretty print rather than pretty print -> data.
I can see it being annoying if all you care about is using CLI programs by hand, but it seems like a mild upside compared to the downsides the second you want to consume it programmatically, even if it's with a chain of awk, cut, and tr.
It's got a lot of the unpleasant clunkiness that something like the Bourne shell owes to decades of compatibility, but Microsoft doesn't have that excuse. Despite this, it's gratuitously incompatible with itself, I had code which worked fine, we upgraded Powershell oh, now that won't work, just fix all the scripts. Crazy. It clearly wants an NPM-style experience where you seamlessly incorporate other people's work, but then it doesn't really deliver this well and so you often end up manually copy-pasting.
If Powershell was a one man project and this was their Beta I'd say it is promising. But it's a project from the Microsoft corporation for 20 years. Do better.
But the actual UX is just convoluted and horrible. For example, the person who decided that commands without a ; at the end of the line should just vomit their outputs to the enclosing function's clearly hasn't heard of the principle of least surprise.
PS just does so many plain weird and unintuitive things, which is worsened by the fact that it looks like a boring old programming language, while clearly not being one.