undefined

points

[-]

I remember when I was coming up on the command line and I'd browse the forums at unix.com. Someone would ask how to do a thing and CFAJohnson would come in with a far less readable solution that was more performative (probably replacing calls to external tools with Bash internals, but I didn't know enough then to speak intelligently about it now).

People would say, "Why use this when it's harder to read and only saves N ms?" He'd reply that you'd care about those ms when you had to read a database from 500 remote servers (I'm paraphrasing. He probably had a much better example.)

Turns out, he wrote a book that I later purchased. It appears to have been taken over by a different author, but the first release was all him and I bought it immediately when I recognized the name / unix.com handle. Though it was over my head when I first bought it, I later learned enough to love it. I hope he's on HN and knows that someone loved his posts / book.

https://www.amazon.com/Pro-Bash-Programming-Scripting-Expert...

by noisy_boy3 hours ago|

parent|

[-]

Wow that takes me back. I used to lurk on unix.com when I was starting with bash and perl and would see CFAJohnson's terse one-liners all the time. I enjoyed trying my own approaches to compare performance, conciseness and readability - mainly for learning. Some of the awk stuff was quite illuminating in my understanding of how powerful awk could be. I remember trying different approaches to process large files at first with awk and then with Perl. Then we discovered Oracle's external tables which turned out to be clear winner. We have a lot more options now with fantastic performance.

by mememememememo10 hours ago|

prev|

[-]

Also as someone who looks at latency charts too much, what happens is a request does a lot in series and any little ms you can knock off adds up. You save 10ms by saving 10 x 1ms. And if you are a proxyish service then you are a 10ms in a chain that might be taking 200 or 300ms. It is like saving money, you have to like cut lots of small expenses to make an impact. (unless you move etc. but once you done that it is small numerous things thay add up)

Also performance improvements on heavy used systems unlocks:

Cost savings

Stability

Higher reliability

Higher throughput

Fewer incidents

Lower scaling out requirements.

by lock110 hours ago|

parent|

[-]

Wait what? I don't get why performance improvement implies reliability and incident improvement.

For example, doing dangerous thing might be faster (no bound checks, weaker consistency guarantee, etc), but it clearly tend to be a reliability regression.

by spiffyk9 hours ago|

parent|

[-]

First, if a performance optimization is a reliability regression, it was done wrong. A bounds check is removed because something somewhere else is supposed to already guaratee it won't be violated, not just in a vacuum. If the guarantee stands, removing the extra check makes your program faster and there is no reliability regression whatsoever.

And how does performance improve reliability? Well, a more performant service is harder to overwhelm with a flood of requests.

by johnisgood7 hours ago|

parent|

[-]

"Removing an extra check", so there is a check, so the check is not removed?

by spiffyk6 hours ago|

parent|

[-]

It does not need to be an explicit check (i.e. a condition checking that your index is not out of bounds). You may structure your code in such a way that it becomes a mathematical impossibility to exceed the bounds. For a dumb trivial example, you have an array of 500 bytes and are accessing it with an 8-bit unsigned index - there's no explicit bounds check, but you can never exceed its bounds, because the index may only be 0-255.

Of course this is a very artificial and almost nonsensical example, but that is how you optimize bounds checks away - you just make it impossible for the bounds to be exceeded through means other than explicitly checking.

by cwaffles9 hours ago|

parent|

prev|

[-]

Less OOMs, less timeouts, less noisy neighbors problems affecting other apps

by tclancy10 hours ago|

prev|

[-]

Which is fine, but the vast majority of the things that get presented aren’t bothering to benchmark against my use (for a whole lotta mes). They come from someone scratching an itch and solving it for a target audience of one and then extrapolating and bolting on some benchmarks. And at the sizes you’re talking about, how many tooling authors have the computing power on hand to test that?

by NoSalt8 hours ago|

prev|

[-]

> "somebody, somewhere, is doing stream processing of TB-sized JSON objects"

That's crazy to think about. My JSON files can be measured in bytes. :-D

by j1elo8 hours ago|

parent|

[-]

Well obviously that would happen mostly only on the biggest business scales or maybe academic research; one example from Nvidia, which showcases Apache Spark with GPU acceleration to process "tens of terabytes of JSON data":

https://developer.nvidia.com/blog/accelerating-json-processi...

by DoctorOW5 hours ago|

parent|

prev|

[-]

All files can be measured in bytes. :)

by NoSalt3 hours ago|

parent|

[-]

You, sir or ma'am, are a first class smarty pants.

by lbj1 hours ago|

prev|

[-]

That's a really great perspective. Thanks for sharing!

by Chris204810 hours ago|

prev|

[-]

But even in this example, the 2ms vs 0.2 is irrelevant - its whatever the timings are for TB-size objects.

So went not compare that case directly? We'd also want to see the performance of the assumed overheads i.e. how it scales.

by 7bit8 hours ago|

prev|

[-]

Who is the target audience? I truly wonder who will process TB-sized data using jq? Either it's in a database already, in which case you're using the database to process the data, or you're putting it in a database.

Either way, I have really big doubts that there will be ever a significant amount of people who'd choose jq for that.

by simonw8 hours ago|

parent|

[-]

There was a thread yesterday where a company rewrote a similar JSON processing library in Go because they were spending $100,000s on serving costs using it to filter vast amounts of data: https://news.ycombinator.com/item?id=47536712