upvote
I've done some work in this space and it's mostly that Big Tech has regulators and parent advocacy groups already very angry with them and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.

Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.

I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".

reply
I'm unsure if there's any particular size requirement for a company to not care about individual customers. Barring an exceedingly small amount of businesses who rely on a handful of customers for their entire bottom line, probably almost all B2C and most B2B companies have a certain point at which any particular customer is simply not worth the overhead to serve. For OAI I would assume anything that even smells like CSAM is probably more than enough for them to justify acting first and (maybe) asking questions later, especially with the regulators and IPO to consider.
reply
> I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".

On the contrary, that's an incredibly elegant solution.

reply
I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.
reply
I would posit the exact opposite: it's really important to fight child sex abuse, which is why it's so frustrating that people are co-opting that cause to promote surveillance and data collection.
reply
Oh you misunderstood me; it’s important for sure.

I don’t think this is something that can be solved by more surveillance though.

Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)

Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.

reply
You know, when you stop to think about it, what does forcing the major file/image hosters to overtly delete CSAM (and ban the poster accounts) exactly do? That's right, it forces the people who create/share/exchange CSAM to go for the secured/encrypted messengers where that content is pretty much impossible to intercept and detect — which obviously makes the task of finding and arresting those people much harder. Like, just make those hosters to tacitly contact the relevant authorities and not tip off the criminals that they've been detected. Nah, let's push those guys to use Signal and TOR.
reply
Typically they do report people to the police as well. But once the image is uploaded, they have as much info as they're getting probably. Hosting it for months until police get an indictment doesn't really do any good and just perpetrates these images' spread. I think criminals already know that most mainstream image hosts try to detect CSAM, whether they delete it immediately or not.
reply
Obviously there's all flavors of coordination with law enforcement to do exactly what you're talking about when it's appropriate.
reply
It also pushes big players (like FB or Apple) to add e2e encryption in their services so they can avoid being blamed for "knowing" that someone was planning an act of violence.

AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.

Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.

reply
> AI unfortunately currently can't be e2e encrypted at scale

It can actually, there are even inference providers that offer this. It isn't strictly E2EE but running a published image on attested hardware is the next best thing. The customer can inspect the image and the hardware manufacturer attests that this is the image that the inference provider is actually running.

That said the latest generation of attested hardware is quite vulnerable (see https://tee.fail/) so you're still taking the provider at their word that they haven't intentionally compromised their own machines.

reply
> Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.

Of interest for those who might not know, the USPS does this for _metadata_ today.

The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.

But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]

Modern problems, modern solutions, etc.

[0]: https://en.wikipedia.org/wiki/Mail_Isolation_Control_and_Tra...

[1]: https://www.usps.com/manage/informed-delivery.htm

reply
Saying it's important is not the opposite of saying that it's rare to actually care.
reply
How else do you prevent it tho?
reply
One key means of preventing the exploitation of children is placing children in situation where they aren't powerless, where their caregivers aren't in such a marginal position that they can't prevent predators from exploiting them.

Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.

But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".

reply
Another great way to prevent child exploitation is age-appropriate sex education for the children. But we don't see politicians promoting that.
reply
It is complex and controversial. What is appropriate to what age? Who should do it? How much should it be driven by children's questions and how much regardless? Should all children by taught the same things at the same age? Should it be purely biological or do you teach them values at the same time? If so whose?
reply
It’s not so hard.

> What is appropriate to what age?

Pretty much everything. You can talk to children about sex without issues.

> Who should do it?

A person the child can trust, can be a teacher or parent for example

> How much should it be driven by children's questions and how much regardless?

If they have questions answer them.

> Should all children by taught the same things at the same age?

Yea I don’t see why not

> Should it be purely biological or do you teach them values at the same time?

Both

reply
A lot of countries seem to manage just fine.
reply
No more of that, people are made Brave New World style in batches of clones with various oxygen levels.

In veil of ignorance Rawlsian style you teach them enough where if you were in their position you wouldn't feel slighted by the adults setting you up for the world at large.

reply
deleted
reply
It may be important, but remains that nobody really cares.
reply
I'm not sure your definitions of "posit" and "exact opposite" align with common usage.
reply
Reminds me of a newsroom (the AFP) who was constantly publishing on their social media account conspiracy theory debunking. I told them it was a problem, because they were actually exposing me to those theories: I would not have heard of them without reading the AFP feed, and now I would remember something about it months or years later, maybe not remembering clearly if it was true or not. I recommended than instead of posting their debunking in their main feed, they should post it in discussions on the debunked subject. They never answered and never stopped. And I understood why: they don't want to stop conspiracy theories. They want to be seen fighting them. It's all posturing as well.
reply
That’s dark. For sure it’s used as a pretense for surveillance, however that is not mutually exclusive with caring about child sex abuse.
reply
I'd agree but I'd add a lot of things are done to get the child sex crusaders off a given corporation's back. Moreover, the crusaders will use every victory to widen their definition of bad things (which isn't me saying anything "good" is here) and number of areas they can trawl for this bad stuff. Enlarged enough, the agenda essentially would include criminalizing pornography and any "deviant" sexuality.

But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.

reply
I agree. Because if they do they hold some sort of view on what is or isn’t in an image/video. I think it is simpler and easier to put your hands over your eyes and look the other way.
reply
To be fair, they're also trying to use it as PR to polish their rapidly tarnishing corporate image.
reply
[flagged]
reply
I think the good faith interpretation of GP's comment is that almost no one among big tech companies (and maybe governments) cares.
reply
I think the good faith interpretation of his comment is that most people and companies dont care, which Im responding to. If the commenter doesn’t agree with his own words he should clarify.

I think virtually every person and company does care. In fact I think its exceedingly obvious.

Even if OpenAI, Facebook, etc dont care be cause they are evil, they do care because pf liability.

None of this even excludes the possibility that CSAM controls are co-opted for power sometimes. Which is why it’s absurd to suggest these companies don’t care - you can concede that point and still argue its being co opted.

reply
So they don't actually care about CSAM existing, they only care about not losing money due to it being on their platforms. Did I get that right? In that case you are confirming what icantevenhold is expressing.

To actually care about something is not the same as doing arbitrary actions due to financial reasons. That's caring about money, and nothing else.

reply
Im not confirming what icantevenhold said because what he said is broader than the narrow point of liability=caring. Executing on operational requirements to remove CSAM looks a lot more like caring than actively fearmongering people about CSAM to justify new powers.

What do you think? Do you think Facebook, OpenAI, etc. are fine with child porn on their platforms save for liability? A liability which of course exists because virtually everyone agrees that it's a real problem and not just an excuse to grab power.

reply
Personally, I think they are fine with it. If there was no law against it, and if they benefited even 1 cent from it, they would be all for it. These tech giants are so profit-oriented, they don't care one bit, who has to suffer for it.
reply
I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.

The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.

reply
Though of course fastest growing is not any excuse to not be prepared. Maybe the growing should happen in other directions too, than only investor money and evaluation. There are other axes of growth than the financial one.
reply
uh actually it's good that they're bad and they're only bad because they're so good, cut them some slack will you it's not easy being rich
reply
I had the same experience with OAI. We were all-in with OpenAI as our primary provider. We used it exclusively for python and go programming requests. Received a ban for "inciting violence". Appeal denied, no refund, no human response or explanation, 3 years of data lost (privacy requests for data denied). We're on team Anthropic now. Still stings when I think about it.
reply
Anthropic acts the same way (from experience), so take the learning of backing up your data from this, if nothing else.
reply
So just go from bad to worse? I thought the lesson should be trust neither and pick an open weight provider from the dozens?
reply
Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.
reply
I run a moodboard website (sort of like pinterest, but no social features) and I have been worrying about this as well. The site is picking up pretty decent traffic and has hit nearly 1TB in uploaded images, but I haven't yet started scanning incoming images for CSAM content.

Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).

reply
If OpenAI actually flagged something as CSAM, they were supposed to report it to NCMEC. I wonder whether they will confirm having done so, even if they won't say which image it was.

I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.

reply
> OpenAI does not care about fighting child sexual abuse, this is completely performative.

That doesn’t really follow from your experience.

OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.

reply
But part of that was trying to identify which content was flagged so that the uploader could be identified, but since it sounds like they didn't identify the content now the OP cannot report it to the authorities.
reply
OpenAI was given a chance to help someone fight child sexual abuse. They declined to do so, potentially violating the law in the process. It's a pretty straight line, given the facts of the post.
reply
If they were serious about stopping CSAM they'd spend 10 seconds looking at the prompt we used, the emails we've sent and the fact we're trying to get an NCMEC case number and report the uploader, but we can't because we literally don't know what image they're referencing. If a human was involved in any part of this process or decision, it isn't apparent because all we've received is automated responses.

Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.

reply
Because CSAM is illegal to transmit in many jurisdictions, and sending a link to it is considered transmitting, it's quite possible that they don't think they can legally inform you. The presumption being, if NCMEC think it's CSAM, you will find out then, when they ask you to remove it.
reply
They informed us our account was being deactivated for CSAM. We were seeking literally any metadata to help us track it down (a batch number, a content hash, a description, a date and time it happened so we could narrow down the search by looking in our logs, etc), and they have refused to provide any information to us, at all, despite our eagerness to quarantine the asset(s) and report the person who uploaded it to the proper authorities.

At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".

reply
Transmitting to a third party is not at all the same as reporting a url (or hash or what have you) directly to the host. Consider the real world analogy of informing a property owner that you noticed something illegal in his backyard versus telling other unrelated parties where they can obtain the illegal thing.
reply
His point is that even if they do care, they are being performative in going completely silent, doing "nothing" as opposed to working together with others to remove content and report bad actors.
reply
Thanks for running ytmnd. When I was a child, it brought me a great deal of (juvenile) amusement.

Duck roll…

reply
Classifying unreviewed user-uploaded content with a service that does not accept CSAM is a footgun, even if they misclassified something this time.
reply
Imagine you have a small forum that can run cheaply on one server/k3s cluster. You want to classify new content to detect most spam and ease administrative burdens. You rate limit classification[1] so you won't be hit with surprise bills. Sounds fine? Nope, because now any idiot can submit CSAM and have your AI provider drop you.

So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud). Otherwise you're at the mercy of opaque and kafkaesque (by necessity) service termination without warning for CSAM (real or hallucinated), and subsequent refusal to communicate about anything related to it.

[1] either by queuing all posts for classification before showing them publically, or just best-effort after-the-fact detection of spam or abuse, and upon detection they're moved to a spam queue.

reply
> So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud).

Or just use one of the many third party inference providers that markets an image classification endpoint intended for user generated content.

reply
What is the price differential between those providers and the frontier labs, on the capability/cost curve? Last I checked, the price difference is on the order of 4-5x for equivalent Artificial Analysis scores.
reply
> We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using)

The fact that people have autonomous agents making these kinds of decisions is still mind-boggling to me. Maybe I'm stuck in the past. But could this agent have been racking up things of dollars in charges because of this decision GP didn't even know about?

reply
deleted
reply
If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?
reply
Performative and just try to shed liability.
reply
Hope it wasn't Batman-is-a-dick huwehweheheheheheh, ahh the memories!
reply
Performative would be giving the people (everyone) the means to expand the crime they seek to punish
reply
why even have openai account then?

stay away, use deepseek and stuff

reply
it's not performative if it's done to reduce or eliminate liability. Performative is about signally for the purpose of signalling. Businesses dont signal for it's own sake. They do it to either increase business or decrease liability.
reply
Oh my, ytmnd!

I used to view shit there a lot sometime in the mid-2000s.

reply
Seems like they want to play in the big boys club (Google, Facebook, ... ) where this kind of treatment is standard.
reply
[dead]
reply
[dead]
reply
Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.

How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?

reply
I don't need to imagine. The postal service, telephone company, and internet service providers have all existed, and contnue to exist.
reply
And you have insight into their level of tolerance for CSAM?
reply
This is understandable. I think the core problem is that most big tech/social media have no working appeals system which would allow a false-positive situation like this to be resolved in a sensible fashion.
reply
I mean, you're holding it wrong. You uploaded CSAM to OpenAI which is presumably against their TOS.
reply
So nobody can classify any user-submitted image content using cloud AI providers.

Is this really the world you want?

reply
deleted
reply
Allegedly. Or the discrepancy in Cloudflare's and OpenAI's detection is caused by a false positive on OpenAI's side. Who knows?
reply