We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.
OpenAI does not care about fighting child sexual abuse, this is completely performative.
EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.
I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
On the contrary, that's an incredibly elegant solution.
I don’t think this is something that can be solved by more surveillance though.
Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)
Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.
AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.
Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
It can actually, there are even inference providers that offer this. It isn't strictly E2EE but running a published image on attested hardware is the next best thing. The customer can inspect the image and the hardware manufacturer attests that this is the image that the inference provider is actually running.
That said the latest generation of attested hardware is quite vulnerable (see https://tee.fail/) so you're still taking the provider at their word that they haven't intentionally compromised their own machines.
Of interest for those who might not know, the USPS does this for _metadata_ today.
The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.
But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]
Modern problems, modern solutions, etc.
[0]: https://en.wikipedia.org/wiki/Mail_Isolation_Control_and_Tra...
Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.
But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".
In veil of ignorance Rawlsian style you teach them enough where if you were in their position you wouldn't feel slighted by the adults setting you up for the world at large.
But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
I think virtually every person and company does care. In fact I think its exceedingly obvious.
Even if OpenAI, Facebook, etc dont care be cause they are evil, they do care because pf liability.
None of this even excludes the possibility that CSAM controls are co-opted for power sometimes. Which is why it’s absurd to suggest these companies don’t care - you can concede that point and still argue its being co opted.
To actually care about something is not the same as doing arbitrary actions due to financial reasons. That's caring about money, and nothing else.
What do you think? Do you think Facebook, OpenAI, etc. are fine with child porn on their platforms save for liability? A liability which of course exists because virtually everyone agrees that it's a real problem and not just an excuse to grab power.
The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).
I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.
That doesn’t really follow from your experience.
OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.
Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".
Duck roll…
So basically, table stakes for running a user-submitted-content site will be to pay for your own GPU node/cluster and manage your own inference stack (running a worse model than you could use in the cloud). Otherwise you're at the mercy of opaque and kafkaesque (by necessity) service termination without warning for CSAM (real or hallucinated), and subsequent refusal to communicate about anything related to it.
[1] either by queuing all posts for classification before showing them publically, or just best-effort after-the-fact detection of spam or abuse, and upon detection they're moved to a spam queue.
Or just use one of the many third party inference providers that markets an image classification endpoint intended for user generated content.
The fact that people have autonomous agents making these kinds of decisions is still mind-boggling to me. Maybe I'm stuck in the past. But could this agent have been racking up things of dollars in charges because of this decision GP didn't even know about?
stay away, use deepseek and stuff
I used to view shit there a lot sometime in the mid-2000s.
How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?
Is this really the world you want?
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
At least with AWS they're clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it's actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.
The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes "actual knowledge" of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there's also a public optics element too I'm sure.
This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.
Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.
In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.
"ZDR, except for _______" is not ZDR.
https://www.nytimes.com/2022/08/21/technology/google-surveil...
I often take photos of my kids in the bath but they are neither babies nor is it their first so I guess that's CSAM then?!
They imply a 28% false-positive rate on their "high-confidence" flags. And that's an improvement? Yikes.
When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)
A model capable of shielding investigators from additional trauma is an excellent thing.
Just to counter any obvious moralizing, I am not saying using models like that is okay, but it's an entirely different kind of crime.
The problem is, this viewpoint is infinitely flexible, infinitely elastic and can do a lot of gymnastics. I'm not sure it's compatible with the very concrete and procedural orientation of Anglo-American jurisprudence, particularly.
One closely related example was the punishment introduced for (legal-age) deepfake porn for which legislation already exists - this could be handled as a special (more severe) case of that more appropriately.
Only if you think right/wrong sex is a function of consent and harm. Which is a very modern idea. And also cant condemn incest between consenting adults.
I see your perspective. It's a very coherent path to normalized ai generated child porn.
The law already recognizes nuance. Calling for the death of someone is still a serious crime, but an entirely different ballpark to actual murder.
Saying that you need to persecute these people as a full-on child predators or not at all is not consistent with how basically all of existing laws work.
Which parts of my comment do you feel I mischaracterized you? Are you saying you think there may still be something wrong with child porn even when the subject doesn’t exist?
The law is already clear on computer generated child porn. Been explicitly illegal since 2003 if we are talking about the US. And shouldn’t it be?
* in the sense that there was not many laws against it specifically; as I understand it the imagery may have been considered illegal under broader obscenity statutes
On the issue of Epstein, that probably would warrant more nuance. I would be willing to bet that many people still today draw a very significant distinction between the teenagers Epstein preferred and, for example, prepubescent children.
They weren't, people's brains have just rotted. They've started to call people who are attracted to 17 year olds pedophiles (rather than calling most people saying that they're not attracted to 17 year olds liars who you shouldn't be alone with), fighting for the right of 18 year olds to be prostitutes, and meanwhile you can get sued to sharing copyrighted videos of 18 year olds being paid $1200 to do triple-anal penetration by one of the people who was doing the penetration and also owns the copyright.
It's not a new moral framework, it's a bunch of slogans and a bunch of legal contradictions. And somehow, things become more legal and more moral when they're filmed or done in the name of commerce.
In the 80s, or to be more accurate pre-9/11, you were expected to be able to make a coherent argument about things that you believed. After 9/11, coherent arguments became a sign of privilege and possible foreign propaganda.
The kids born after 9/11 just didn't have a chance. We raised them to say the words, and told them if they interrogated the words that they would be expelled from school, fired, and socially shunned. We had eroded workers' rights (and worker's consciousnesses) so completely by then that firing anybody for anything was trivial. We had eroded free speech so much that any reaction to anything you had said or done was not only fair but actually good, and maybe obligatory.
Now, AI is helping (and being allowed to help) enforce this strategic, authoritarian confusion. I'm sure that will end well.
Facebook: it started off censoring nipples, and ended up censoring anti-wingsuit posts and wishes that people would die. Then it branched out into surveillance devices and nuclear reactors. Are you happy about how power develops in this new world, and how it starts with the centralized control of "truth"?
/rant
That is an entirely different line of discussion, and please, please, please do not go there, as it's unlikely you'll find support for these opinions
Isn't CSAM only material based on abuse of real humans? How could computer models multiply CSAM by that definition?
This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.
If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
https://link.springer.com/article/10.1007/s11896-023-09611-4
Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
It's still seen as extremely contraband. We're talking one of the most universal taboos of Western society. Heck, it's equally detested by the US Right and Left. How many things can we say that about?
I don't think anybody is going to suddenly think it's normal, cool, or acceptable to possess or view real or fake CSAM images or to abuse children, just because we aren't prosecuting people for using a diffusion model in a gross way.
The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_
So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.
Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.
Just don't put it in MY face.
With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.
Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?
I don't think the fake is legal if depicting the same act. One could be at least liable in a civil context.
> It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
Can you tell me which ethical framework you are using to make your assertion? Right now it seems like an appeal to emotion, which is how we get things like the Patriot Act, KOSA, and all these id verification laws.
Given the choice between someone doing actual child abuse and generating fake pictures on a computer, every time someone chooses the latter over the former, the world becomes a slightly better place.
You may argue that it would be even better to do neither, that's nice, it's called the Nirvana fallacy. That's already the law today, and people ignore it anyway.
"So you don't think it's morally wrong to generate _______ using AI?"
"So you don't think it's morally wrong to _______ using AI?"
The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.
As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.
To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.
What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I guess calling it a cost recognizes there is something wrong with it at least.
I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.
Im not asking about if it should be allowed, I'm asking if you find anything morally wrong about making child porn via AI, or consuming it.
Why can't you just say "yeah it's morally wrong?" Why is that not obvious?
A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.
We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.
I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).
I'm not entirely sure what to think. I don't have a great theory of mind for the kind of person who generates AI child porn. I'm sure many people think they do, but I can confidently say that I don't. Are they prone to real-life sex crimes against children? Is there a high degree of overlap with sundry groomers and predators? Is there a correlation between their interest in this and sexual abuse of their own family members? Their own children?
Or are they just curious techno-libertarians, maybe with a contrarian streak or a few unnerving kinks? What's a child in this context, anyway? Are we talking about toddlers or teenagers? Are they a child, or the kind of quasi-adult Judith Levine had in mind in Harmful to Minors (1994)? Is the perpetrator here more in the style of a 'predator', or just a 19 year-old horsing around? I don't really know what these folks are thinking or making, as I've never met anyone with such open proclivities. I'd guess, as with most things sexual, that many things are a lot more common than one might glean from the readily visible surface. I'd also guess that the truly depraved extreme stuff is at the margins. I think? I'm not a sex researcher.
Sex crime against children is a big dragnet, though: in my day, they put away Genarlow Wilson here in Georgia for 10 years:
https://en.wikipedia.org/wiki/Wilson_v._State
Most people seem to agree today that this was an instance of justice run amok and massively cruel and unusual punishment. The Georgia Supreme Court did, too, if only by 4-3. However, I was in high school and college in those days, and back in the day, advocating for this teen earned one few friends, and a "paedo" label, even though we all had girlfriends and boyfriends when we were in high school and ... I guess we were just supposed to pretend Genarlow Wilson was some apex sex predator because he was black, not sure what the message was. It's hard for me to imagine that a similar spectrum of possibilities, and shifting norms, doesn't exist in the world of child porn, even though it's an area that rarely sees the light of any kind of nuance. I refuse to believe it's just this opaque monolith, just as I refuse to believe the socialist bloc was one univocal monolith. There have got to be things that common sense can agree are somehow worse than other things, even if "better" is not a word one would reach for.
Anyway, my intent isn't to muddy the waters with "troubled" distinctions, but rather to say that without that kind of theory of who we're talking about, some details to enrich a mental picture and a moral penumbra, I can't really say if I think it's wrong. I don't know who's doing what in your example, nor the why. I'm reduced to mostly talking in the narrowly procedural mood of Anglo-American jurisprudence, which deals with rights and harms and boundaries and definitions, not, by and large, a central idea of the good. That works in court, which deals with whether one is allowed to do something, but doesn't settle the moral question much.
My first instincts are quite liberty-leaning on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.
Not true in terms of the law and philosophy. Congress passed a law already making child porn illegal for computer generated images of children.
In terms of social norms its also not novel. People have debated if pedophilia is morally wrong absent the harm to the child. Some people believe sex is right/wrong based on personal details like consent and harm and others factor in more natural order (such as roles - children, spouse etc). Maintaining integrity of roles makes it trivial to exclude incest and cases like this.
In a different time, place and era in technology, well before it was possible to create, at scale, lifelike and photorealistic images of almost anything, none of it incorporating real people. Back then, I think the idea was to criminalise well-done "Photoshop" jobs, or something like that.
The scale does matter. If computers, for example, become extensions of our imagination, so that we can wish a mental example into vivid, lifelike being with the touch of a button, and this becomes routine, then prohibiting a computer operation may tread, in many instances, close to prohibiting a category of expression, maybe even a category of thought (who knows how far the man-machine synthesis will go if Elon has his way...)
I don't have all the answers, I don't think Congress does, and I don't think you do.
This is a tendentious framing, but legally detailed:
https://enoughabuse.org/ai-csam-ruling-what-the-seventh-circ...
Specifically:
> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.
Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
/s
>A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
Privacy, unfortunately, is largely the result of it being impractical to violate it.
I see zero reason to believe that states wont read your thoughts if they can easily manage to.
Tech Company: At long last, we have created the Precrime Division from classic sci-fi novelette The Minority Report.
I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!
It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
If that is a worry you carry around, I really hope you work harder to make it never be the case lol. FYI it’s not a normal thing to worry about.
It's been like that for the entire 20 years I have run these kinds of websites. It sucks, but it's just a sad reality of the world we live in.
https://developers.openai.com/api/docs/models/omni-moderatio...
It requires all devices integrated into one single consciousness.
Remember when they wanted Apple to scan everybody’s devices locally?
I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.
If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
Similar to Roblox ignoring child grooming in their platform because they make money off of it. https://www.bloomberg.com/features/2024-roblox-pedophile-pro...
Similar to Facebook and other platforms feeding self-harm content to vulnerable children and teenagers, and even connecting them to dealers. These companies know their platforms are not safe for kids. https://www.bloomberg.com/features/cant-look-away/
Could be a great start for "rogue AI"
Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
>In SafetyKit’s evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases
>This is the first CSAM classifier with broad availability that has a specific signal on indicating if CSAM may be AI generated.
What children are being abused by drawings or AI generated images, stories, etc?
I understand the cases where it's problematic to make such content based on real children and share it, and I think it's probably a bad idea to allow photorealistic child porn to be publicly posted on the internet for the sake of not confusing law enforcement (in the same way posting misinformation is harmful), but if these images are not realistic, or if it's just text, or if they are realistic but they aren't being shared on the internet, who is being harmed?
Whatever you say about "non-offending" (and possessing child porn IS an offence in most places). I would definitely not have wanted you anywhere near my kids.
Worth remembering, though, that the most likely offenders are family & friends. We spend a lot of time worrying about stranger danger because we cannot come to terms with just how common this predilection really is.
I believe AI is one of the only ways to engage with these desires in an ethical way.
Of course, you can always recommend abstinence, but unless you're rather conservative in your views, you probably don't think that's a good way to deal with teenage pregnancy, homosexuality, etc. So why would this be the exception?
I understand the distinction between offending and non offending.
But pedophile is inclusive of both.