upvote
From the article, it sounds like the passwords are indeed stored in cleartext:

> On Feb. 1, 2025, Muneeb Akhter asked Sohaib Akhter for the plaintext password of an individual who submitted a complaint to the Equal Employment Opportunity Commission’s Public Portal, which was maintained by the Akhters’ employer. Sohaib Akhter conducted a database query on the EEOC database and then provided the password to Muneeb Akhter. That password was subsequently used to access that individual’s email account without authorization.

reply
It still blows my mind. Shouldn't the government audit their contracting companies for egregious issues like this? Seems extremely reckless not to.
reply
I've been through a handful of SOC2 audits and they've never asked us to _prove_ that we aren't storing passwords in plaintext or with reversible encryption (we weren't).

This is why so much of vetting & compliance is toothless. You can have robust change management, physical security, network security, identity management, etc. policies but absolutely nobody wants to spend enough on audit & enforcement to make them meaningful.

The gov't will make you _claim_ that you do all of these things before awarding a contract, but they won't ever check.

Good actors will do the right thing regardless because they know the consequences of cutting corners.

reply
I'm pretty shocked as well. I thought every company stopped doing this like 20 years ago? Even for a legacy system that is a long time to continue storing credentials like that.
reply
20 years is rookie numbers in these systems. I guarantee it’s been at least 40 years since a single fuck was given.
reply
My wife works in IT at a mid sized city. They still store credentials in source control.
reply
Policy and practice might not be the same thing. The company and the entire management staff should be on somebody’s blacklist for future procurement.
reply
The tighter your security is, the more inconvenient it is for legitimate users, and the more you have to do audits because it's easy to justify going around security in the name of efficiency.

It's not just information security, either. I've seen vault doors propped open because the people working inside didn't want to do all the sign-in/sign-out paperwork to take a leak.

reply
The whole point of stuff like SOC2 and audit to verify that policy is actually implemented. Seems like nobody actually checked.
reply
SOC2 requires an audit. But one of the weaknesses of SOC2 is that the audit mostly checks to determine that you are following whatever your policy is. It doesn't verify that your policy is rigorous.
reply
I don’t think you understand what SOC2 is.

First of all, it is viral, and it is almost never adopted based on its own technical merit.

Second, it has lots of levels. The first level is “we wrote down a plan explaining how we’re going to secure stuff”.

The second level is when you start implementation or maybe tracking or something.

The key thing is that first level: When your SOC2 dept says you have to do something idiotic for SOC2 compliance, it is because someone at your company invented the idiocy, and should be fired. However, you still need to follow their dumb plan because that’s the process.

In this case, the “how do we fire people” process, and “how do we prevent one llm from dropping 96 prod DBs in a single session” very well could have had answers in the plan, the plan could have been implemented, and therefore the company is still soc2 compliant, and this is exactly what a working soc2 process is supposed to look like.

reply
Depends on what their offboarding policy is. If it's 72 hours or something they would not breach policy.
reply
I can only think of a scenario where this is still valid: spying.

The minimum one can do is have a different randomized password for every service on a possibly completely offline password manager.

Yes, you will depend on a password manager at all times, but at least the blast radius is minimized to the affected service.

reply
And how exactly do you want to store passwords if not in plain text (and then encrypted of course)? 5k is a lot, the authorization process is broken, but this is not related to how the passwords are stored.

The only solution is correct access segregation and a bastion

reply
You should never store passwords in plain-text, encrypted or not, you should always use a one-way cryptographic hash like bcrypt [0], scrypt [1], or PBKDF2 [2], combined with a single use salt [3] and optionally a pepper [4], and then store the output of the hash in the database.

To confirm a user supplied password matches you run input into the same hash function again with the salt+pepper and compare it to the value in the database.

That way if the database is stolen, the attacker cannot recover the contents of the passwords without brute forcing them. Encrypting passwords is not recommended because too often attackers are able to recover the encryption keys during the same attack where the password data is extracted.

[0] https://en.wikipedia.org/wiki/Bcrypt

[1] https://en.wikipedia.org/wiki/Scrypt

[2] https://en.wikipedia.org/wiki/PBKDF2

[3] https://en.wikipedia.org/wiki/Salt_(cryptography)

[4] https://en.wikipedia.org/wiki/Pepper_(cryptography)

reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
Hashed, you store them hashed (and salted). A breach should never reveal passwords.
reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
You speak very authoritatively on something you don’t know.

Hashing passwords has been a thing for at least 50 years now. V3 unix had /etc/passwd which hashed all user passwords. Notably, these hashed passwords in early unix have been cracked: https://arstechnica.com/information-technology/2019/10/forum...

I guess you got your answer.

reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
Typically you store a hash of user passwords instead, then when logging in you hash the user password client-side and compare the hashes. This acts like a one-way function that protects the password while letting the user authenticate themselves.
reply
Hashing passwords client-side is generally a bad idea, since it means that the hash effectively becomes the password. For example, if I have a database row that has the hash of the password and a bad-guy gets access to the database, they will get the hash. The benefit of a hash is that it is a one-way operation, I can't figure out the plaintext from the hash, so my account is safe. If the password is hashed on the client, and sent to the server the attacker doesn't need to reverse the hash, they can just send the hash in the request. Instead, you should send the password to the server (using TLS encryption) and do the hash and compare on the server.
reply
You actually want to one-way passwords both client-side, for transport, and again server-side, for storage/comparison.

Otherwise, there's a hole, between the end of the TLS connection and where the server-side encryption happens, where the password is in plain text. Think logs and load-balancers and proxies.

While the client-side hashing doesn't help protect your site a lot (as you say, the hashed value the client sends effectively becomes the password), it helps protect the users who use the same password across multiple sites.

Notice in this case, that's exactly what the brothers are accused of doing: using credentials harvested from their site to log into other, potentially more lucrative accounts.

I didn't see if that's the hole the brothers exploited but it very well could have been.

The client-side encryption may have been all that was missing in this case.

reply
If you're worried about MITM in the TLS web connection between client and server, you already lost and no prevention method client side will work, because if you own the connection you can just give client malicious JS to extract the password when they enter it
reply
Hashing client side is sufficient because the only service you can breach with the hash is the one you already had to breach in order to read the database.

Of course performing an additional server side hash on top of the client side one is good defense in depth because there's at least some chance that it might make things more difficult for a rogue insider and doing so costs approximately nothing. But it certainly isn't critical because by the time you're dealing with a rogue insider things are already looking quite bad.

reply
Also, you need to add salt. Otherwise every person using "Password123" has the exact same hash. Before they broke their search engine, it was common to google the MD5/MD4 hashes to "decrypt" or "unhash" them.
reply
And rainbow tables appeared too.
reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
People shouldn't be downvoting this...

Hashing client-side is a good idea. You must also hash server-side, for storage/comparison.

Otherwise, an insider may be able to harvest the original password, from logs, proxies, load balancers, etc. that requests pass through after the end of the TLS connection, on the way to the db.

They can then try the credentials on other, perhaps more lucrative sites. That's what the brothers are accused of doing here, so client-side hashing (or just simple encryption) may have been the missing piece of security that would have thwarted the credential stealing.

reply
I wonder how common are setups where an internal person has access to the TLS private key part of the certificate or access to a network equipment that all traffic passes through, yet they cannot access the inputs required for hashing/encryption client-side?

This seems to mostly prevent accidental logging and is thus a matter of defense in depth, stopping malicious actors from exploiting it later — but an actively malicious IT person would not be deterred.

reply
> This seems to mostly prevent accidental logging

Yes, and that's not uncommon, IME. There's generally a lot of logging that's at least potentially available, and it gets turned on, and the logs shared when there's a problem that needs to be fixed (especially when it needs to be fixed quickly, which is usual).

This is going to make more sense for "enterprise"-type deployments, where there's a significant distinction between the people who might have access to request logs at times, and the people who can push code to production.

reply
Yes limited protection against insiders is good defense in depth but not the primary purpose which is to protect end user accounts on other services in the event that you are breached.
reply
My question still stands: how do you disallow cleartext password extraction if you are breached, assuming all your IT infrastructure and code is now accessible to an attacker?

I am talking about not logging them ever, using internal TLS and strong hashing in general, and wondering what exact value is added on top with client side hashing.

reply
There are substantial differences between database access, snooping the logs, internal (no TLS) wiretap, and full MITM of the frontend.

Hashing client side minimizes the risk of any blast radius exceeding the bounds of your own service. There's obviously no way to prevent an adversary who achieves full MITM from gradually harvesting credentials over time. The only solution there is to use keys instead of passwords.

reply
We are not disagreeing, but I am not getting my answer: how is client side hashing really helping, what are the circumstances it helps with if you do have the basics right?

In your enumeration, what is breached for this to be meaningfully impactful for other services where customers might be reusing credentials?

reply
As opposed to what? Your question seems unclear to me.

I already answered you that if you assume full MITM of the frontend then it is physically impossible to prevent gradual credential harvesting. Did you have a different scenario in mind?

> how is client side hashing really helping

Compared to what? Server side hashing? It prevents the plaintext from ever hitting your infra which minimizes to the greatest extent possible an insider or intruder gaining access to the plaintext. Since preventing access to the plaintext is the primary purpose of hashing it's the sensible option if you're forced to pick only one.

Of course there's really no good reason to pick only one of the two. You might as well elect for defense in depth against rogue insiders with full db access even though it's difficult to imagine what use they would have for a password based login at that point.

reply
There is certainly good reason to do server-side hashing: you do not keep a persistent record of the customer's secret, yet keep the ability for them to authenticate with it.

If you are never logging a clear-text secret and storing a hashes version to validate against, and using TLS between client and server, client-side hashing does not bring much benefit other than protecting customers' reused passwords against people who have sufficient access to the infrastructure to MITM the client but no ability to modify the client side code where they could extract the password directly.

When IT has write access to code, client side hashing protects only against accidental log leakage and similar.

reply
> There is certainly good reason to do server-side hashing: you do not keep a persistent record of the customer's secret, yet keep the ability for them to authenticate with it.

That is not a property of server side hashing but rather hashing in general.

> If you are never logging a clear-text secret

Yeah good luck with that. /s

It's better not to need to worry about logging plaintext. The less of your infra that falls within any given security boundary the easier it will be to properly secure. The difference between server and client side hashing is how much of your infra touches the plaintext. Less is always better.

Sure, an insider could make direct use of hashes pulled from the logs. However if the attacker already has access to your systems then they are already inside the security boundary (or at least most of them) and likely have many other (probably better) options available to them.

It's important to keep in mind that the primary purpose behind hashing credentials is to minimize the degree to which your systems come into contact with the plaintext. The goal here is to contain the blast radius of any intrusion to only your own systems and only the immediate intrusion (ie to prevent future abuse after you've cleaned everything up) given the unfortunate reality that end users will frequently reuse credentials.

It's also important to keep in mind that hashing is cheap enough that you should probably be doing both. Hash it on the client so that you don't need to worry about logs (or snooping the LAN or whatever other way an employee might come up with to obtain plaintext passwords) and then hash it again before it enters the db so that you don't need to worry about the logs that contain hashes leaking.

reply
I am not disputing client side hashing brings some value, just claiming that it is the least useful of the techniques applied.
reply
I hope youre joking
reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
Assuming you're serious? Store passwords with salted one-way hashes.
reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
You don't store passwords.

You store safely crafted hashes.

reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
Thanks for the follow up! And yeah, I share your gripe with the next target :)
reply
I don't think those words mean what you think they mean.
reply
(I will be copy/paste this answer for the other comments)

My bad - I misread the post.

To clear things up: I am completely aware about how to store passwords in services that check against them. You are likely to have read some of my prose on that topic in OWASP or at a conference :)

My point, after misreading the article, was that in order to authenticate to a service (the one that holds the hashed version of that password) you need to have access to its cleartext version. This is VERY bad, should never be stored without special considerations etc.

I read the articlae as if they accessed the source of the passwords, the one used to access to services (a vault, with its encryption, access restrictions etc.). 5k was a lot but that could have been bearers or similar ones.

So my comment, and the comments to it, actually yelled at me (that's good!) the way I yell at actual implemententions sometimes :)

In all seriousness - thanks for the reaction, we need more of these. My next obsession are servies that require "only digits" or "strictly 8 to 11 chars" for credentials :)

reply
You should have stuck with the 'My bad...' and left out the eye-roll inducing humble brags and inscrutable non-clarifications. But what do I know...I've only been an infosec practitioner since the early 90s...I'm sure 'conference experience' trumps that.

(I will not 'be copy/paste' (?) this response everywhere you spammed someone who pointed out the glaringly obvious)

reply
Or I could have let it go.

See, I respect people who point out mistakes, and explain why I did it.

Why did I mention that I do security? Because I spent the last, 25 years trying to push proper practices and did not want to jump into discussions where over 10 comments we would end up flexing about details.

Since you are an infosec practitioner since the early 90s you either are a saint, or did not have to yell through best practices to just let it go.

Not sure what you don't like about conferences? Never got anything from them? I did and I am sure glad to have listened to great presentations.

reply
You should have stuck with "Or I could have let it go.".
reply
And yet I have not, sorry for having dissapointed you.
reply