upvote
Depends on the definition. It's become more frequent for predators to download images of real children from their parents' social media, then edit them with diffusion models to be nudes. This should still be illegal, and it is in the US.
reply
So deep fakes? That does sound horrific and challenging to reliably detect and stop.
reply
Legally, if the computer generated content looks too real, its still criminal.
reply
Well it's against the ToS. Whether or not it's criminal (in the US, I don't know how europe handles it) depends on if it would qualify as "obscene" which means that looking realistic isn't actually relevant (it might seem bizarre but people have been sent to jail for cartoons).
reply
I assume Europe depends from country to country. In this one Simpsons porn is illegal. You heard that right. A bright yellow four fingered character is illegal to draw without clothes.
reply
Yup. This is such hysteria. Personally I’d argue that the existence of diffusion models “multiplying the supply of this material” means that the value of it goes way down, meaning it’s harder for the truly monstrous parties who have for decades harmed actual human beings to create that “content” now face collapsing value of what they make, which makes it far less lucrative to be in that business and should make it less attractive to get into it, and remove some incentive to make more and more of it.

This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.

If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.

reply
Not sure that multiplying the supply of content that pedophiles jerk it to is a great idea. Not only do you risk exposing people who (hopefully) never would've been exposed to it before, but research shows that increased access to CSAM can actually help normalize the abuse and support deviant behavior in online communities dedicated to it.

https://link.springer.com/article/10.1007/s11896-023-09611-4

reply
Some have argued the opposite, that "multiplying the supply of this material" further normalises sexual abuse of children, whereas before, it lurked in the shadows of the dark web and was treated like serious contraband, which imbued it with a certain gravity.

Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.

reply
(Recognizing that it's not your argument but responding to it in general)

It's still seen as extremely contraband. We're talking one of the most universal taboos of Western society. Heck, it's equally detested by the US Right and Left. How many things can we say that about?

I don't think anybody is going to suddenly think it's normal, cool, or acceptable to possess or view real or fake CSAM images or to abuse children, just because we aren't prosecuting people for using a diffusion model in a gross way.

reply
The reason why it's a specific problem is in my original comment: Signal to noise ratio.

The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_

So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.

reply
According to your reasoning I'm left wondering, if the synthetic can't be readily distinguished from the genuine then what are you supposing would be the remaining motivation for criminal parties to produce and distribute such things? By producing and distributing they commit a very serious crime and for what personal benefit exactly?
reply
> Isn't CSAM only material based on abuse of real humans?

If only.

The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.

reply
It is indeed a very dangerous ratchet. Who in their right mind would take an extremely unpopular position that is affront to the SafeTy of children?

And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.

Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.

reply
Huh. So you don’t think it’s morally wrong to generate child porn using AI?
reply
I do not. I would much rather that those interested do exactly that rather than trade in the real thing.

Just don't put it in MY face.

reply
Are you not confusing your preferences with moral here? It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.

With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.

reply
But wait, why is the bar different for this subject matter than for other depictions of immoral acts?

Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.

So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?

reply
> Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.

I don't think the fake is legal if depicting the same act. One could be at least liable in a civil context.

reply
You edited your comment. Below is the version I replied to in case of further edits

> It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.

Can you tell me which ethical framework you are using to make your assertion? Right now it seems like an appeal to emotion, which is how we get things like the Patriot Act, KOSA, and all these id verification laws.

reply
No need to signal your virtue, we are all just anonymous strangers on the internet.

Given the choice between someone doing actual child abuse and generating fake pictures on a computer, every time someone chooses the latter over the former, the world becomes a slightly better place.

You may argue that it would be even better to do neither, that's nice, it's called the Nirvana fallacy. That's already the law today, and people ignore it anyway.

reply
What a bizarre strawman
reply
I don't think OP thinks that. I hear him as pointing out that this question can't be considered in acontextual, ahistorical vacuum. It's a set of simultaneous equations, where the other equations and terms are of a more general form:

"So you don't think it's morally wrong to generate _______ using AI?"

"So you don't think it's morally wrong to _______ using AI?"

The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.

As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.

reply
I don't really understand what you are saying.

To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.

What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?

I guess calling it a cost recognizes there is something wrong with it at least.

reply
> What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?

I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.

reply
>I don't have a great answer on whether it should be "allowed"

Im not asking about if it should be allowed, I'm asking if you find anything morally wrong about making child porn via AI, or consuming it.

Why can't you just say "yeah it's morally wrong?" Why is that not obvious?

reply
Why are you focused on whether one particular commenter has the correct answer to this question and not whether their concern about unintended consequences has merit?
reply
I think that's just the nature of caustic moral litmus test issues, of which child sexual abuse is certainly the main one. Any qualms and you're "pro-child porn".
reply
Offhand, I'm not sure what's special about it There are all kinds of horrors one can generate with AI: beheadings, murder fantasies, extreme sadism, material promotional of darkly extremist ideology, text and literature intended to mislead/manipulate/deceive/brainwash, depict or glorify self-harm and suicide, etc, for example, and one could argue that all of these further actual socially harmful impulses and behaviours, feed and nurture social clubs built around them, etc.

A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.

We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.

I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).

I'm not entirely sure what to think. I don't have a great theory of mind for the kind of person who generates AI child porn. I'm sure many people think they do, but I can confidently say that I don't. Are they prone to real-life sex crimes against children? Is there a high degree of overlap with sundry groomers and predators? Is there a correlation between their interest in this and sexual abuse of their own family members? Their own children?

Or are they just curious techno-libertarians, maybe with a contrarian streak or a few unnerving kinks? What's a child in this context, anyway? Are we talking about toddlers or teenagers? Are they a child, or the kind of quasi-adult Judith Levine had in mind in Harmful to Minors (1994)? Is the perpetrator here more in the style of a 'predator', or just a 19 year-old horsing around? I don't really know what these folks are thinking or making, as I've never met anyone with such open proclivities. I'd guess, as with most things sexual, that many things are a lot more common than one might glean from the readily visible surface. I'd also guess that the truly depraved extreme stuff is at the margins. I think? I'm not a sex researcher.

Sex crime against children is a big dragnet, though: in my day, they put away Genarlow Wilson here in Georgia for 10 years:

https://en.wikipedia.org/wiki/Wilson_v._State

Most people seem to agree today that this was an instance of justice run amok and massively cruel and unusual punishment. The Georgia Supreme Court did, too, if only by 4-3. However, I was in high school and college in those days, and back in the day, advocating for this teen earned one few friends, and a "paedo" label, even though we all had girlfriends and boyfriends when we were in high school and ... I guess we were just supposed to pretend Genarlow Wilson was some apex sex predator because he was black, not sure what the message was. It's hard for me to imagine that a similar spectrum of possibilities, and shifting norms, doesn't exist in the world of child porn, even though it's an area that rarely sees the light of any kind of nuance. I refuse to believe it's just this opaque monolith, just as I refuse to believe the socialist bloc was one univocal monolith. There have got to be things that common sense can agree are somehow worse than other things, even if "better" is not a word one would reach for.

Anyway, my intent isn't to muddy the waters with "troubled" distinctions, but rather to say that without that kind of theory of who we're talking about, some details to enrich a mental picture and a moral penumbra, I can't really say if I think it's wrong. I don't know who's doing what in your example, nor the why. I'm reduced to mostly talking in the narrowly procedural mood of Anglo-American jurisprudence, which deals with rights and harms and boundaries and definitions, not, by and large, a central idea of the good. That works in court, which deals with whether one is allowed to do something, but doesn't settle the moral question much.

My first instincts are quite liberty-leaning on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.

reply
>We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content

Not true in terms of the law and philosophy. Congress passed a law already making child porn illegal for computer generated images of children.

In terms of social norms its also not novel. People have debated if pedophilia is morally wrong absent the harm to the child. Some people believe sex is right/wrong based on personal details like consent and harm and others factor in more natural order (such as roles - children, spouse etc). Maintaining integrity of roles makes it trivial to exclude incest and cases like this.

reply
> Congress passed a law already making child porn illegal for computer generated images of children.

In a different time, place and era in technology, well before it was possible to create, at scale, lifelike and photorealistic images of almost anything, none of it incorporating real people. Back then, I think the idea was to criminalise well-done "Photoshop" jobs, or something like that.

The scale does matter. If computers, for example, become extensions of our imagination, so that we can wish a mental example into vivid, lifelike being with the touch of a button, and this becomes routine, then prohibiting a computer operation may tread, in many instances, close to prohibiting a category of expression, maybe even a category of thought (who knows how far the man-machine synthesis will go if Elon has his way...)

I don't have all the answers, I don't think Congress does, and I don't think you do.

reply
I'd prefer it be illegal to generate porn that looks like children. I also understand that threshold can be difficult to define. What I'd heard most recently was courts said CSAM must involve real people or it's just free speech.
reply
It's been reported and discussed that way, but the ruling was actually quite a bit more narrow and contingent.

This is a tendentious framing, but legally detailed:

https://enoughabuse.org/ai-csam-ruling-what-the-seventh-circ...

Specifically:

> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.

reply
I cannot imagine being pro-child pornography.
reply
Few people can. However, when it comes to digital freedom and privacy, you can't surgically cherry-pick what you're for and against. You need to consider larger principles, and operate in generalities.
reply
You definitely can. It's technically and morally consistent to say "I support digital freedom, and I oppose the sexual abuse of children".
reply
You can't say "I oppose the sexual abuse of children, but nothing concrete follows from this whatsoever." What am I consenting to, in regard to my digital freedom, in voicing that sentiment?
reply
Good. Then you wouldn’t be able to imagine being pro-not having a state sponsored surveillance agent on every phone and computer you have.
reply
Yes, who will cry for the people making virtualized child pornography which is, of course, sourced from actual child pornography? WHO???
reply
This is a real moral dilemma for people who think consent and harm are all that matters. Vitriolic disgust at the idea of generating child porn using AI, but no good way to explain why its wrong.
reply