Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day.
We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience.
Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post.
Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.
It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours.
`mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post.
I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
Sorta odd you don't support one of Europe's most popular distros.
How to report a bug or vulnerability
... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net
It should always be assumed that someone else (if not several someone elses) have already discovered the same flaw and are currently taking advantage of it while users remain totally unaware of their actual risk. By going public immediately, you give as many of those users as possible a chance to protect themselves.
Waiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up.
That's not what they said though. They said "please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away" (emphasis mine)
The flipside of course is ... does your disclosure increase the risk?
> aiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up
I don't think it's like this at all. The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred. To stretch your analogy, I'd say its more like you've found the gas leak and instead of turning off the gas supply are instead running around outside the building shouting about how there's a gas leak.
When you've got that much on the line you have to assume that the risk is already present for all users. It's true that there's always a chance that some users won't find your disclosure in time and additional would-be attackers who weren't aware of it already will start taking advantage of the flaw, but the alternative is that no users are safe.
> The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred.
It's true that warning people not to enter wouldn't make the gas more dangerous, but it limits the death count of the impending explosion. It keeps at least some people from entering the building and walking into a death trap.
There's no way to shut off the gas supply when you can't control what's already running on user's devices and more users are downloading and installing the buggy code all the time. It's really not a perfect analogy. The point is that immediate action will save some people, while waiting around means that nobody has a chance of being saved.
why are companies so entitled to get free security research/audits?
If so, I guess we just have different opinions on the ethics involved here.
As for our support team they are responsive and experienced. Several of them have worked with us for many years and do offensive security research in their free time.
Unlike many organisations we don't see customer support as a cost center, just like we don't see security as a cost center. Our support team represent our customers, and as a consequence contribute a lot to how we prioritise our roadmap.
I second this.
Clearly the person who wrote "Oof" has never emailed Mullvad support.
Whenever I have emailed Mullvad support I have received a prompt reply from a human being who clearly actually cares about taking ownership of the question and seeing it through to resolution.
I have also witnessed first-hand the support person taking the question to an internal team member where it requires additional input. So there are clear paths for escalation if circumstances require it.
Finally the support mail allows for PGP encryption of communications too.
(I am not a Mullvad shill. Not a Mullvad employee. Just a satisfied customer)
I'm not familiar with how you run your company -- without the context you gave most people would hesitate emailing support@ for security issues.
"Just email support@" feels like you don't care. That you do, and that your support team is awesome, doesn't change the fact that there are other companies out there who's aren't. Security people are human with human egos, and they want to feel special, so giving them a special way to reach you, even if it's the same thing behind the scene, makes a world of difference.
This sounds like how I'd design a VPN if I were an intelligence agency.
Make it look like an accidental misconfiguration and if an insider who isn't an NSA mole does somehow discover the logging, there's a fair chance they'll turn a blind eye anyway. After all, if you work at a VPN, publicly outing your employer for logging will tank the business, then you and your colleagues will all be out of a job.
I guess we’ll see how they respond.
Mullvad have been taken to court over this in relation to a copyright infringement case.
TL;DR The judge permitted people to take a fine-tooth comb to Mullvad's infrastructure and no logging was found[1].
[1] https://mullvad.net/en/blog/mullvad-vpn-was-subject-to-a-sea...
I recall a PRISM slide showing the diagram of Google and the public internet, with a big arrow on GFE saying, quote, “SSL added and removed here! :-)”
If NSA aren’t installed at Cloudflare, I wonder what they are even doing.
Hmm do we want them to decide what stuff is shady and what isn't?
We're already allowing payment processors to do that and it's not good.
That nonetheless doesn't help them unless they are doing active MITM. In order to do that they'd have to have at least some physical presence at Cloudflare or on the path to Cloudflare.
People didn’t care when they learned about PRISM, why would they care now when it’s a known fact? The sane stance would be to assume Cloudflare is in cahoots with NSA.
The NSA leaks dominated news cycles for the entirety of 2013.
This is as helpful as Whatsapp's so called E2E encryption comms (that just happens to not be applicable by default in certain situations).
it does give better peering. reduces latency a bit for me.
but you can also see from curl or traceroute, that the endpoint you talked to was a cloudflare ip and your ssl ended there. after that you can't see inside cloudflare.
I think more people than you would expect would be happy to accept that as the price for protection against malicious actors
That doesn't mean collusion
Either way, if they were directly colluding with Google, they would have had a much simpler time siphoning off that data.
The funny thing about that era is you knew they started using Cloudflare because they went from stable with constant uptime to going down and showing a Cloudflare banner randomly all the time for a good year or so. They ran worse with Cloudflare than they did while they were allegedly getting DDoSed. The whole company glows, as the late great HN commenter Terry Davis would've said.
Oh my god, this is how & when I realize that Terry Davis (Rest in peace) used to use Hackernews too: https://news.ycombinator.com/threads?id=TerryADavis
https://news.ycombinator.com/item?id=10061171 (From this comment written by terry):
"I wrote all the code from scratch, including a 20,000 line of code compiler that makes x86_64 machine code from HolyC or Asm and operates AOT and JIT.
My JIT mode is not interpreted. It optimizes and compiles to x86_64 machine code.
I was chosen by God because I am the best programmer on the planet and God boosted my IQ with divine intellect." -Terry A Davis.
Anyone with a few crypto currencies in their wallet that can click a button on any of the booter services with botnets for hire.
This is a massive issue in my view, it allows correlation across multiple VPNs exit nodes, but that’s it. It doesn’t allow to identify you automatically. It does significantly lower the bars for identifying you though, but the requirements are still high.
Hopefully they fix this soon.
I can’t believe this type of “let’s make it a hash or something sensitive” still happen, and at mullvad, of all places. Why not randomise it simply?
If you squint a bit, it looks a lot like a "Nobody But US" (NOBUS[1]) scheme. A few more identifying bits could tip the scale for party that has a whole host of other bits on a list of suspects, without being useful to most other people.
Their ads on San Francisco's public transit are good.
Security is always a balance. Always
AI is showing that everything has a weak spot (wondering where are the "I don't make mistakes with C" now people are - but that's for another discussion)
There's another commenter mentioning this makes sense because exactly it avoids them keeping information on which customer is matched to which server. You know, one of the things you don't want to log
Could it be done better? Probably.
Here's a better idea, logging off is 100% safe
Meanwhile 99% of the normies will go for NordVPN
Let me specify: The user must have entered his data on one site which the attacker has control of. That is a high bar still.
Sure, there are other intelligence agencies, but that's the one I'd be the most worried about. Since either they run it, or they would know of it and want to emulate the idea, or know of it and have access to it from the partner agency running it. Or they are not a threat to me.
There's also the issue of no publicly known cases where someone that used Mullvad being deanonymized through the VPN but instead being discovered through some other opsec failure. If an intelligence agency has this capability they have been sitting on it for almost 2 decades without making use of the data. Hard to believe.
Wow, I didn't realize Mullvad was this old! Then again, maybe they weren't popular enough back then for intelligence agencies to target them? For instance, Mullvad kinda rode WireGuard's popularity wave by being the first(?) VPN provider to implement the protocol. Big ads on billboards came even later. So maybe they only became a target in recent years?
So does your comment...
I think its safe to assume that intelligence agencies have other options available to them, such as country-wide timing attacks.
I don't know the answer, but there are two ways to take it:
1. Submarining to destroy confidence in an actually trustworthy, decent VPN company
2. They're an intelligence front.
For me, Mullvad have the appearance of the greatest likelihood of being legit since they're not aggressively pushing their product with lies and fear mongering. That gels with my vibe. If they're an intelligence front, well, most VPNs probably are as well, so I'm no worse off.
Luckily I'm not doing anything that would get me in the kind of trouble for which multi-jurisdictional cooperation is worthwhile.
I don't see how the author is arriving at this ">99% chance" purely from the numbers provided in the article. Assuming the first (banned IP) seed and the second seed are both in the range 0.4423 - 0.4358 (a stronger assumption than is justified by the example), all this tells us is that the first and second IP addresses both have seeds in a range that would contain 0.4423 - 0.4358 = 0.65% of all Mullvad users, which 0.0065 * 100,000 = 650 users. We've eliminated >99% of users as "suspects", but we haven't actually gotten >99% accuracy in identifying an individual across multiple exit IPs.
In more Bayesian thinking, the overlap in potential seeds is great evidence to think these IP addresses represent one and the same person (or Mullvad VPN account at least), but as far as I can tell, that's not what the author is saying.
What are the chances that someone uses this vpn, joins your forum the day after someone was banned, and has an ip in a similar range?
For most small websites this would be strong evidence.
Why can't it aim to solve what it can do? TOR is a great example: the TOR network itself can't perfectly anonymize you due to browser fingerprinting, but users of the TOR Browser get both the TOR network resisting deanonymization on a network level and a browser with plenty of anti-fingerprinting measures built in. A VPN could aim to prevent deanonymization on a network level so that users who want to stay anonymous can use the VPN in combination with fingerprinting-resistant software.
If I'm on a public VPN, I don't want anyone to know who is making the request, including the terminating IP.
Think about it. By your logic, VPNs shouldn't be used for torrents because VPNs shouldn't anonymize you to the terminating IP. Whereas they work gangbusters for that.
If you are talking about private VPNs.. Mullvad isn't one.
But today’s internet is essentially a giant ad network.
The fact that Tor does not intend to tackle the timing problem is plainly stated on the Tor website.
Edit: In hindsight I regret making this comment. It was unnecessary, but removing it now would look weird.
Putting aside the IP correlation across multiple servers, at first I wondered why even keep the user IP stable on one server. But I think it makes sense because as the author states other VPNs usually have only one IP per server so they are essentially simulating that. The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
The IP correlation across multiple servers they should fix though with something like rand.seed(user_pub_key + server_id)
On the flip side, if they’re getting banned by a service because of a noisy neighbor on the same IP, they’d have no way to work around that, no?
All things considered, there are just an incredibly small number of IPs shared among all users, no matter the allocation strategy.
Also if the threat model you're addressing w/ VPN usage is anything other than "I don't want my ISP to know what I'm doing" you need to use/do something else.
I'm a little confused on this... what is stopping third parties from doing key rotations like the main app clients if it is detailed in the repo how to do it?
Knowing to do so, primarily.
It does seem ridiculous once you spell it out like that, and then you have to realize that it’s plausible to de-anonymize even Tor users by controlling exit nodes.
Things you connect to or log in to are clearly going to be able to ID you at least with in the context of the login that you use regardless of what the VPN does.
I'm logged into HN through Mullvad as it happens. I usually leave it on regardless of what I'm doing because what I'm doing isn't my ISP's business even though I'm pretty happy with them.
Most likely these people just look to hide their torrenting, saying political shit on Twitter from employer and not share their choice of porn with local ISP. Also just adding one more layer between them and occasional scammer who can sometimes infer more broad geodata from their IP leaked from yet another database. Oh and now to avoid "Show your ID" page on the same porn sites.
It works well enough for this goal. Not everyone needs NSA-proof solution.
PS: Obviously more tech savvy people understand importance of hiding traffic on public WiFi, but I doubt average Joe the VPN user will buy VPN for this.
What percent on people on Hacker News who say they care about privacy live without Google, Apple, Microsoft and Facebook accounts?
How many people outside of HN do you think care about privacy for real? Like about adtech surveillance and not about their naked photos leaking?
I doubt either % is very high sadly. We tend to say we care, but very few people actually do anything or use self hosted solutions or not tied to Apple or Google ecosystems.
You really cant blame VPN providers for selling on "privacy" hype and not delivering because most people dont care either way.
Might be I wrong, but I feel in west for most normal people use VPNs for torrents, watching porn and hidding activity from school or employeer. Small subsets are also sport fans who bypass geo blocking and people scheming for cheaper regional prices on netflix / steam / consoles.
I blame mullvad for messing up, but I do not suspect them of working with some state sponsored surveillance programme at the moment.
Do you have any facts? I know they really on _additional_ stuff, but do you have sources showing that they never use cookies or source IPs?
On that topic, though, is the Mullvad Browser, who's entire intention is to defeat browser fingerprinting.
1. It's the preferred VPN of TeamPCP.
"23034 IPs to blocklist.txt"
blocked IPs they contain all VPN providers. Often VPN providers seed Geofeeds with wrong data, this is why i use traceroute and ping network to locate their real location.
If they're checking my locked doors, I don't want them coming in my unlocked doors.
Because I'm quite curious on where the IPs are from. Usually residential IPs is a fancy wording for malware infested devices from regular people.
Ohh, that makes sense haha.
@m00dy: please disclose when you’re talking about your own projects! It’s okay to plug your stuff sometimes, just be honest about it :-)
> I’m not here to promote anything just wanted to share a valid use case in the right context.
There’s a small difference: if one of your users did this it would be totally fair, but when a founder does this I think it’s a polite thing to disclose it. That’s what I’ve been doing when talking about my own project on HN [1], and I think in most cases other legit founders just say that upfront, too. I’m not sure if that breaks any rules, but it feels juuuuust a bit shady not to :-)
[1]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
> Since you've made seven posts to HN about it
(Seems to have some weird cache issues though, had to play around with the ?querystring part to get more results)
Yes I know it comes from pirating/torrenting/scrapping. Are you saying you acknowledge your IPs come from malware, and that is OK because OpenAI is shady too?
Search for “mobile proxy” – those are usually cheap-ish monthly subscriptions, with unlimited traffic, and often an API to rotate the IP programmatically if you need it. No KYC, but you usually do have to sign up with an email.
yes, it's a bit more expensive because it's for different use cases. You can't use VPNs or Mullvad for anything mission critical. Just try to log in to your bank in US, it will increase your risk score on their end because VPNs by nature are very easy to detect whereas "residential proxies" much harder.
Naturally! I’m just saying there’s residential proxy providers that are a LOT cheaper than that.
(IIRC, you can usually reply to fresh comments if you click on the “n minutes ago” – the reply link should be visible there even if it isn’t shown in the main comments tree)
I’ve been implementing an Instagram liker service back in... 2018 was it? So a stable pool of non-flagged residential proxies was important here, and it was my client who introduced me to the concept of “mobile proxies”. Basically, they use regular 3G/4G/5G modems with regular SIM cards, and expose that as a SOCKS proxy. You get a normal-looking IP from a pool of mobile operator’s IPs. Since mobile devices reconnect all the time (and are behind a CGNAT mostly nowadays), you can’t really flag an IP like that – and if it is flagged, you can get a fresh one in a moment.
I’m not using this mostly because I’m too lazy to research. Here’s a random one I found (so not an endorsement!) which is $1/GB, seems to only require email to sign up, and takes crypto (including XMR): https://floppydata.com/
That is a binary thought process with a lot of assumptions. You might introduce even more attack surface in pursuit of this "security" measure by installing additional software like fail2ban, for example. Close your ports, maybe assign a non-standard port to the popular ones (like SSH) to reduce log spam, and patch your server often. Anything more complicated than that is not worth it, IMO.
Like when I was travelling, sites would routinely use the language of my IP address location, not the language preference as I set it in my browser. So I would be served a site that I couldn't read. My only option was to use a VPN to spoof my location so that it would serve me a site in a language I understand.
Seems like a good deal to me. I don't care if they know I use mullvad, I care they don't know I'm me, and that's not something mullvad will easily disclose.
That's exactly what the article is about, a side channel information leak that de-anonymises users, did you read it?
I'll go ahead and answer that it can't. It knows I'm mullvad user X, thus deanonimization, "it knows I use mullvad", but it doesn't know my original IP, so "it doesn't know I'm me".
But when you connect to the site from via server A and later via server B they can tell that you're the same person.
And they can deanonymise you through data brokers. All Mullvad IPs are traceable back to the same number (acting as a pseudo account identifier) so if you ever entered your PII on any website when using Mullvad, it can be linked to the same Mullvad account.
And if you ever visited any of those sites without using a VPN, your home IP can be linked to your Mullvad ID through browser fingerprinting.
And if you ever entered any PII on any website from your home IP, you can once again be deanonymised.
Now the existence of browser fingerprinting isn't Mullvad's fault, but this flaw makes it a lot easier to accidentally deanonymize yourself.
What's the point of this? This seems more complicated to implement than mapping exit ips at the server level, so surely they must be doing this for a good reason?
If you get a new exit IP each time you connect, you need something like a NAT table to look up "key 0xabc exits ip 1.2.3.4", and that grows to be the size of the number of users you have active, and you need to save it forever so that when the NSA asks who used the IP for what duration you can tell them.
With a static mapping derived from the key, you don't need a table like that.
It's also better UX since it means reconnecting your VPN software (say you switch wifi hotspots) doesn't give you a different IP address, so things like SSH sessions can resume, which wouldn't be possible if it were a different public IP each time.
It's a practical measure, but definitely has a privacy cost though.
It seems more likely this is just about load-balancing use against their available nodes.
Given how much of the world is stuck behind CGNAT now, I would expect any major sites to handle it.
I'm also stuck in a 2 year ISP contract
This is an AI comment from an AI account.
>Should I use a VPN?
Yes, almost certainly. A VPN has many advantages, including:
1. Hiding your traffic from only your Internet Service Provider.
2. Hiding your downloads (such as torrents) from your ISP and anti-piracy organizations.
3. Hiding your IP from third-party websites and services, helping you blend in and preventing IP based tracking.
4. Allowing you to bypass geo-restrictions on certain content.
What power is in $2.99/month that it offers so much security?
Why is that at least 40% of sponsorship to YouTube Creators seem to be from VPN industry?
What is that they know and we don't know?
Many many examples out there. "We don't keep logs" is not good enough neither realistic because how else a VPN provider is supposed to protect itself if it doesn't keep a log of what's happening inside and through its own systems.
Well, my ISP sent me a nice letter saying they intend to monetize my metadata, and mullvad has demonstrated in court that they don't have user data to give up.
> and how do you expect them to protect your identity in face of determined state actors that are afer you?
That's moving the goalposts; your parent comment didn't say anything about determined state actors. And defending against commercial actors is useful even if it doesn't help against state actors. I tend to assume the NSA can compromise anything. I'd like to ensure only the NSA can compromise my stuff.
One at least has open source software clients, and publishes audits from other 3rd-party audit organizations.
The other open source... nothing. Their client apps have dozens of trackers inside. And it's a dream to see any of the ISPs in my county publish any 3rd-party audits. Their other products (going with the service) have trackers and personalized targeting ads inside.
Yeah, in my 1 million alternate universes should I trust my ISP more.
In theory, but as someone who uses Mullvad in the UK on a day-to-day basis on my personal laptops (not my phone) - I'm using it now, I'm afraid there's quite an additional downside I've found, in that because Mullvad's (at least UK, but also French and Dutch ones I've tried) exit IPs are known, many companies (Cloudflare, Akamai) at the very least know about them, and several sites block access when using Mullvad, returning 403s.
Santander bank for example, I can't always (sometimes I can) connect to when using Mullvad, and sometimes have to turn it off, as I get 403 responses from the bank otherwise (using Firefox).
Sometimes using IPv6 in the Mullvad settings gets around this, but more and more recently I've found it doesn't, so there sites where I'm having to stop using Mullvad to actually access sites.
(I'm still a happy customer, and 1 to 3 are still true and why I use it otherwise).
Rotating your VPN endpoint will resolve the issue. It might take two or three tries.
Local law enforcement can tap a local ISP for their records, but it would take a scale more effort to then tap a non-local service provider for their records. Each additional level of difficulty adds a cost, and at some point those costs aren't worth the potential results.
(assuming that the VPN provider doesn't just roll over due to an email inquiry, or isn't a front for very cooperative law enforcement).
From outside the US I should be using a VPN end-point within the US, so that my browsing traffic doesn't hit the NSA - only my encrypted VPN traffic does.
I mean, let's be real.
All known US VPN servers and Tor exit nodes--and probably all US Tor relays regardless of exit policy--are going to be considered a totally legitimate "communications facility" target for the warrantless wiretapping system due to exactly the scenario you just posited.
From that perspective you'd be better off using US residential proxies. Of course, while they'll never admit it in court, NSA just does whatever they want, laws be damned, and are almost certainly logging everything. So while such a scheme might theoretically hinder the introduction of evidence in a court case, it doesn't really matter; NSA is still gonna see your traffic and they're still gonna either drone strike you or "parallel construction" your ass, anyway.
When you share the evidence for this, it will be international news.
VPNs are useful for the reasons you mentioned.
This is highly subjective statement.
Almost all commercial VPN services farm and sell your data. Just by that, my ISP is definitely high trust point while any commercial VPN is a low trust.
Most VPNs are untrustworthy, but unlike ISPs, you can choose from any VPN provider in the world, not just the two or three that are local to you. And there are VPN providers in the world that have been proven not to retain data by audits + actual court cases where the court determined that the VPN provider did not have the data authorities were seeking. Do your research and choose a court-proven VPN, it's that simple.
And then sells it?
What gives you the confidence that Bigfoot does not exist?
What gives you the confidence we're not ruled by Reptile overlords?
What gives you the confidence we're not just in the Matrix and nothing matters?
What gives you the confidence you're not just a dream by a dog in Sicily?
What gives you the confidence I even exist and you're not talking to yourself?
Doesnt't surprise me that they're counting on gullible, useful idiots defending "Chat Control" and eIDAS.
Neither of those is possible with my ISP.
Phone doesn't even need data if you have access to wifi wherever you stash it.
VPN chaining easier though.
Like you need to physically be there, need ability to connect phone it to electricity and somehow maintain if it e.g reboots. And stay anonymous while doing so? I'd say that Hollywood kind of solution.
That said, you might still want to use a VPN on top of that, depending on what you’re doing.
Citation needed.
For everyone in the industry it is le secret de Polichinelle.
Should I trust my ISP than Mullvad? LMFAO.
(yes, I've been raided)
(I started using Mullvad after - because of - that)
(I don't do illegal shit, I just like some obfuscation of my trail because I enjoy fiddling with this stuff - which may have been why I ended up a raid target in the first place)
They have their own tools + tor, they do not need mullvad.
>Also. This is how they ruined any meaningful talks about privacy
There is so much noise
"Use braive. Don't use braive. Use vpn. Don't use vpn"
Then the debate spreads to all other aspects password managers, emails and etc
VPNs are a technical tool for technical people. You need to know exactly why you need it in order for it to be useful.
The most generous way of reading that would be the fact that every YouTube pushing for a VPN as an essential tool just to use the internet outside of your house without getting hacked is a big exaggeration or fear mongering but there's good reasons for using a VPN for a lot of reasons and it's not snake oil.
Yes, obviously.
> VPNs are snake oil
Huh?