Welcome to Incremental Social! Learn more about this project here!
Check out lemmyverse to find more communities to join from here!

‘IRL Fakes:’ Where People Pay for AI-Generated Porn of Normal People

A Telegram user who advertises their services on Twitter will create an AI-generated pornographic image of anyone in the world for as little as $10 if users send them pictures of that person. Like many other Telegram communities and users producing nonconsensual AI-generated sexual images, this user creates fake nude images of celebrities, including images of minors in swimsuits, but is particularly notable because it plainly and openly shows one of the most severe harms of generative AI tools: easily creating nonconsensual pornography of ordinary people.

Cris_Color , (edited )
@Cris_Color@lemmy.world avatar

God, generative ai is such a fucking caustic technology. I honestly don't see anything positive and not disgusting enabled by this tech.

Edit: I see people don't agree, but like why can't ai stick to translating stuff and being useful rather than making horrifically unethical porn, taking the humanity out of art, and replacing peoples jobs with statistical content generation. I hate it here.

reddithalation ,

i liked ai when it was a bunch of researchers messing around, but commercialized ai is horrifying.

Mubelotix ,
@Mubelotix@jlai.lu avatar

You can call people disgusting over what they do with a tool, but the tool itself is just a tool, it can't be disgusting

Cris_Color ,
@Cris_Color@lemmy.world avatar

The distinction is that I can see worthwhile use cases for non-generative ai, and not for generative ai, and generative ai is built on theft of creative labor

I'm not angry at people who use generative ai, I'm angry at the people who built it by stealing from creatives to build a commercial tool that can seemingly only be used in awful ways.

Mubelotix ,
@Mubelotix@jlai.lu avatar

That I can understand

Kuinox ,

The root problem is government not enforcing the law on internet. Deepfakes existed for years.
The law enforcement should be more proactive on internet.

Ultragigagigantic ,
@Ultragigagigantic@lemmy.world avatar

It's gonna suck no matter what once the technology became available. Perhaps in a bunch of generations there will be a massive cultural shift to something less toxic.

May as well drink the poison if I'm gonna be immersed in it. Cheers.

VinnyDaCat ,

I was really hoping that with the onset of AI people would be more skeptical of content they see online.

This was one of the reasons. I don't think there's anything we can do to prevent people from acting like this, but what we can do as a society is adjust to it so that it's not as harmful. I'm still hoping that the eventual onset of it becoming easily accessible and useable will help people to look at all content much more closely.

afraid_of_zombies ,

It's stuff like this that makes me against copyright laws. To me it is clear and obvious that you own your own image, and it is far less obvious to me that a company can own an image whose creator drew multiple decades ago that everyone can identify. And yet one is protected and the other isn't.

What the hell do you own if not yourself? How come a corporation has more rights than we do?

LodeMike ,

This stuff should be defamation, full stop. There would need to be a law specifically saying you can't sign it away, though.

spez_ ,

Get out of the way of progress

ulterno ,
@ulterno@lemmy.kde.social avatar

I say stop antagonizing the AI.
The only difference between a skilled artist making it with Photoshop and someone using a Neural Net, is the amount of time and effort put into creating the instance.

If there are to be any laws against these, they need to apply to any and every fake that's convincing enough, no matter the technology used.

Don't blame the gunpowder for the dead person.
People were being killed by thrown stones before that.

Ultragigagigantic ,
@Ultragigagigantic@lemmy.world avatar

The laws that oppress us on a daily basis suck ass I'll give yall that for fucking sure.... but downvoting someone wishing for the law equally being applied to all?

Maybe I should go back to 4chan.

ulterno , (edited )
@ulterno@lemmy.kde.social avatar

OooOo!
That's some high number of dwnv0t3s!
I wouldn't have realised unless you had replied here.

Nice, but it's also good that everyone is at least free to downvote and see the number of downvotes, unlike YouTube.

All over history, there has been this trend of people misusing technology and then blaming the technology instead of those that misuse it.
This trend is detrimental to the technological progress of a civilisation and is one of the driving forces, causing the cycle that our civilisation is stuck in (of losing all tech and history every once a while and then having to start over from the dark ages).
Technology, gives someone the ability to do something, but it is their will that makes them want to do so. If the "something" is considered "bad" for society, then instead of taking away the ability, we need to insist on getting the person to understand, why and how, said "something" is a problem for the society.

Until this problem is fixed, we are going to be stuck at the barrier and the next levels of civilisation shall stay a part of Fiction.

anticurrent ,

We are acting as if through out history we managed to orient technology so as to to only keep the benefits and eliminate negative effects. while in reality most of the technology we use still comes with both aspects. and it is not gonna be different with AI.

boatsnhos931 ,

Oooo puter' bobs and vagenes. Scissor me timbers that gets me hot n bothered

RoseTintedGlasses ,
@RoseTintedGlasses@lemmy.blahaj.zone avatar

We need to shut this whole coomer thing down until we work out wtf is going on in their brains.

RoseTintedGlasses ,
@RoseTintedGlasses@lemmy.blahaj.zone avatar

Unironically though, anyone who does this should just be locked up

Mastengwe ,

As long as there are simps, there will always be this bullshit. And there will always be simps, because it isn’t illegal to be pathetic.

ItsAFake ,
@ItsAFake@lemmus.org avatar

Come on, prisons are over populated as it is, if that happens then you me and everyone here are fucked.

Mastengwe ,

I’m not a simp so it’s not a problem for me.

GrymEdm , (edited )
@GrymEdm@lemmy.world avatar

To people who aren't sure if this should be illegal or what the big deal is: according to Harvard clinical psychiatrist and instructor Dr. Alok Kanojia (aka Dr. K from HealthyGamerGG), once non-consensual pornography (which deepfakes are classified as) is made public over half of people involved will have the urge to kill themselves. There's also extreme risk of feeling depressed, angry, anxiety, etc. The analogy given is it's like watching video the next day of yourself undergoing sex without consent as if you'd been drugged.

I'll admit I used to look at celeb deepfakes, but once I saw that video I stopped immediately and avoid it as much as I possibly can. I believe porn can be done correctly with participant protection and respect. Regarding deepfakes/revenge porn though that statistic about suicidal ideation puts it outside of healthy or ethical. Obviously I can't make that decision for others or purge the internet, but the fact that there's such regular and extreme harm for the (what I now know are) victims of non-consensual porn makes it personally immoral. Not because of religion or society but because I want my entertainment to be at minimum consensual and hopefully fun and exciting, not killing people or ruining their happiness.

I get that people say this is the new normal, but it's already resulted in trauma and will almost certainly continue to do so. Maybe even get worse as the deepfakes get more realistic.

Regrettable_incident ,
@Regrettable_incident@lemmy.world avatar

I'm wondering if this may already be illegal in some countries. Revenge porn laws now exist in some countries, and I'm not sure if the legislation specifies how the material should be produced to qualify. And if the image is based on a minor, that's often going to be illegal too - some places I hear even pornographic cartoons are illegal if they feature minors. In my mind people who do this shit are doing something pretty similar to putting hidden cameras in bathrooms.

lud ,

once non-consensual pornography (which deepfakes are classified as) is made public over half of people involved will have the urge to kill themselves.

Not saying that they are justified or anything but wouldn't people stop caring about them when they reach a critical mass?
I mean if everyone could make fakes like these, I think people would care less since they can just dismiss them as fakes.

Drewelite ,

I think this is realistically the only way forward. To delegitimize any kind of nudes that might show up of a person. Which could be good. But I have no doubt that highschools will be flooded with bullies sending porn around of innocent victims. As much as we delegitimize it as a society, it'll still have an effect. Like social media, though it's normal for anyone to reach you at any time, It still makes cyber bullying more hurtful.

afraid_of_zombies ,

Well if you are sending nudes to someone in high school you are sending porn to a minor. Which I am pretty confident is illegal already. I just would rather not search for that law.

Drewelite ,

https://lemmynsfw.com/pictrs/image/64ccabf7-40de-4a6c-950a-9f0ef3d493ba.jpeg

eatthecake ,

The analogy given is it’s like watching video the next day of yourself undergoing sex without consent as if you’d been drugged.

You want a world where people just desensitise themselves to things that make them want to die through repeated exposure. I think you'll get a whole lot of complex PTSD instead.

stephen01king ,

People used to think their lives are over if they were caught alone with someone of the opposite sex they're not married to. That is no longer the case in western countries due to normalisation.

The thing that makes them want to die is societal pressure, not the act itself. In this case, if societal pressure from having fake nudes of yourself spread is removed, most of the harm done to people should be neutralised.

eatthecake ,

The thing that makes them want to die is societal pressure, not the act itself.

That's an assumption that you have no evidence for. You are deciding what feelings people should have by your own personal rules and completely ignoring the people who are saying this is a violation. What gives you the right to tell people how they are allowed to feel?

too_much_too_soon , (edited )

Agreed.

"I've been in HR since '95, so yeah, I'm old, lol. Noticed a shift in how we view old social media posts? Those wild nights you don't remember but got posted? If they're at least a decade old, they're not as big a deal now. But if it was super illegal, immoral, or harmful, you're still in trouble.

As for nudes, they can be both the problem and the solution.

To sum it up, like in the animate movie 'The Incredibles': 'If everyone's special, then no one is.' If no image can be trusted, no excuse can be doubted. 'It wasn't me' becomes the go-to, and nobody needs to feel ashamed or suicidal over something fake that happens to many.

Of course, this is oversimplifying things in the real world but society will adjust. People won't kill themselves over this. It might even be a good thing for those on the cusp of AI and improper real world behaviours - 'Its not me. Its clearly AI, I would never behave so outrageously'.

spez_ ,

The technology will become available everywhere and run on every device over time. Nothing will stop this

SendMePhotos ,

I'd like to share my initial opinion here. "non consential Ai generated nudes" is technically a freedom, no? Like, we can bastardize our president's, paste peoples photos on devils or other characters, why is Ai nudes where the line is drawn? The internet made photos of trump and putin kissing shirtless.

LadyAutumn , (edited )
@LadyAutumn@lemmy.blahaj.zone avatar

They're making pornography of women who are not consenting to it when that is an extremely invasive thing to do that has massive social consequences for women and girls. This could (and almost certainly will) be used on kids too right, this can literally be a tool for the production of child pornography.

Even with regards to adults, do you think this will be used exclusively on public figures? Do you think people aren't taking pictures of their classmates, of their co-workers, of women and girls they personally know and having this done to pictures of them? It's fucking disgusting, and horrifying. Have you ever heard of the correlation between revenge porn and suicide? People literally end their lives when pornographic material of them is made and spread without their knowledge and consent. It's terrifyingly invasive and exploitative. It absolutely can and must be illegal to do this.

cley_faye ,

It absolutely can and must be illegal to do this.

Given that it can be done in a private context and there is absolutely no way to enforce it without looking into random people's computer unless they post it online publicly, you're just asking for a new law to reassure people with no effect. That's useless.

WaxedWookie ,

Strange of you to respond to a comment about the fakes being shared in this way...

Do you have the same prescriptions in relation to someone with a stash of CSAM, and if not, why not?

cley_faye ,

No. Because in one case, someone ran a program on his computer and the output might hurt someone else feelings if they ever find out, and in the other case people/kid were exploited for sexual purpose to begin with and their live torn appart regardless of the diffusion of the stuff?

How is that a hard concept to understand?

LadyAutumn , (edited )
@LadyAutumn@lemmy.blahaj.zone avatar

How can you describe your friends, family, co-workers, peers, making and sharing pornography of you, and say that it comes down to hurt feelings??? It's taking someone's personhood, their likeness, their autonomy, their privacy, and reducing them down to a sexual act for which they provide no knowledge or consent. And you think this stays private?? Are you kidding me?? Men have literally been caught making snapchat groups dedicated to sharing their partner's nudes without their consent. You either have no idea what you're talking about or you are intentionally downplaying the seriousness of what this is. Like I said in my original comment, people contemplate and attempt suicide when pornographic content is made and shared of them without their knowledge and consent. This is an incredibly serious discussion.

It is people like you, yes you specifically, that provide the framework by which the sexual abuse of women is justified.

afraid_of_zombies ,

I agree. For any guy here who doesn't care imagine one of your "friends" made an AI porn of you where you have a micropenis and erection problems. I doubt you would be over the moon about it. Or if that doesn't work imagine it was someone you love. Maybe you don't want your grandma's face in a porn.

WaxedWookie ,

What's hard to understand is why you skipped the question I asked, and answered a different one instead.

The creation of the CSAM is unquestionably far more harmful, but I wasn't talking about the *creation *- I was talking about the possession. The harm of the creation is already done, and whether or not the material exists after that does nothing to undo that harm.

Again, is your prescription the same as it relates to the possession, not generation of CSAM?

LadyAutumn ,
@LadyAutumn@lemmy.blahaj.zone avatar

No, I'm saying make it so that you go to prison for taking pictures of someone and making pornography of them without their consent. Pretty straightforward. If you're found doing it, off to rot in prison with you.

GhostTheToast ,

Don't get me wrong it's unsettling, but I agree, I don't see the initial harm. I see it as creating a physical manifestation of someone's inner thoughts. I can definitely see how it could become or encourage dangerous situations, but that's like banning alcohol because it could lead to drunk driving or sexual assault.

Mastengwe ,

Innocently drinking alcohol is in NO WAY compared to creating deepfakes of people without consent.

One is an innocent act that has potentially harsh consequences, the other is a disgusting and invasively violating act that has the potential to ruin an innocent persons life.

abhibeckert , (edited )

The internet made photos of trump and putin kissing shirtless.

And is that OK? I mean I get it, free speech, but just because congress can't stop you from expressing something doesn't mean you actually should do it. It's basically bullying.

Imagine you meet someone you really like at a party, they like you too and look you up on a social network... and find galleries of hardcore porn with you as the star. Only you're not a porn star, those galleries were created by someone who specifically wanted to hurt you.

AI porn without consent is clearly illegal in almost every country in the world, and the ones where it's not illegal yet it will be illegal soon. The 1st amendment will be a stumbling block, but it's not an impenetrable wall - congress can pass laws that limit speech in certain edge cases, and this will be one of them.

WaxedWookie ,

The internet made photos of trump and putin kissing shirtless.

And is that OK?

I'm going to jump in on this one and say yes - it's mostly fine.

I look at these things through the lens of the harm they do and the benefits they deliver - consequentialism and act utilitarianism.

The benefits are artistic, comedic and political.

The "harm" is that Putin and or Trump might feel bad, maaaaaaybe enough that they'd kill themselves. All that gets put back up under benefits as far as I'm concerned - they're both extremely powerful monsters that have done and will continue to do incredible harm.

The real harm is that such works risk normalising this treatment of regular folk, which is genuinely harmful. I think that's unlikely, but it's impossible to rule out.

Similarly, the dissemination of the kinds of AI fakes under discussion is a negative because they do serious,measurable harm.

Mananasi ,

I think that is okay because there was no intent to create pornography there. It is a political statement. As far as I am concerned that falls under free speech. It is completely different from creating nudes of random people/celebrities with the sole purpose of wanking off to it.

SendMePhotos ,

Is that different than wanking to clothed photos of the same people?

RageAgainstTheRich ,

The difference is that the image is fake but you can't really see that its fake. Its so easily created using these tools and can be used to harm people.

The issue isn't that you're jerking off to it. The issue is it can create fake photos of situations of people that can be incredibly difficult to deny it really happened.

UsernameIsTooLon ,

Lemme put it this way. Freedom of speech isn't freedom of consequences. You talk shit, you're gonna get hit. Is it truly freedom if you're infringing on someone else's rights?

John_McMurray ,

Yeah you don't have the right to prevent people from drawing pictures of you, but you do have the right not to get hit by some guy you're drawing.

antlion ,

Seems to fall under any other form of legal public humiliation to me, UNLESS it is purported to be true or genuine. I think if there’s a clear AI watermark or artists signature that’s free speech. If not, it falls under Libel - false and defamatory statements or facts, published as truth. Any harmful deep fake released as truth should be prosecuted as Libel or Slander, whether it’s sexual or not.

Maggoty ,

It's a far cry from making weird memes to making actual porn. Especially when it's not easily seen as fake.

douglasg14b ,
@douglasg14b@lemmy.world avatar

I think their point is where is the line and why is the line where it is?

Maggoty ,

Psychological trauma. Normal people aren't used to dealing with that and even celebrities seek help for it. Throw in the transition period where this technology is not widely known and you have careers on the line too.

Ookami38 ,

I think the biggest thing with that is trump and Putin live public lives. They live lives scrutinized by media and the public. They bought into those lives, they chose them. Due to that, there are certain things that we push that they wouldn't necessarily be illegal if we did them to a normal, private citizen, but because your life is already public we turn a bit of a blind eye. And yes, this applies to celebrities, too.

I don't necessarily think the above is a good thing, I think everyone should be entitled to some privacy, having the same thing done to a normal person living a private life is a MUCH more clear violation of privacy.

afraid_of_zombies ,

Public figures vs private figures. Fair or not a public figure is usually open season. Go ahead and make a comic where Ben Stein rides a horse home to his love nest with Ben Stiller.

noxy ,
@noxy@yiffit.net avatar

I wonder how holodecks handle this...

shasta ,

They send you to therapy because "it's not healthy to live in a fantasy."

antlion ,

Don’t be like Lt Reg Barclay

9488fcea02a9 ,

Probably the same types of guardrails chatGPT has when you ask it to tell you how to cook meth or build a dirty bomb

And maybe Data was distributing jailbroken holodeck programs for pervs on the ship

curiousaur ,

You can get 300 tokens in pornx dot ai for $9.99. This guy is ripping people off.

flower3 ,

I doubt tbh that this is the most severe harm of generative AI tools lol

Luisp ,

Israeli racial recognition program for example

CeeBee ,

FR is not generative AI, and people need to stop crying about FR being the boogieman. The harm that FR can potentially cause has been covered and surpassed by other forms of monitoring, primarily smartphone and online tracking.

BleatingZombie ,

I wholeheartedly disagree on it being surpassed

If someone doesn't have a phone and doesn't go online then they can still be tracked by facial recognition. Someone who has never agreed to any Terms and Conditions can still be tracked by facial recognition

I don't think there's anything as dubious as facial recognition due to its ability to track almost anyone regardless of involvement with technology

neatchee ,

You don't need to be online or use a digital device to be tracked by your metadata. Your credit card purchases, phone calls, vehicle license plate, and more can all be correlated.

Additionally, saying "just don't use a phone" is no different than saying "just wear a mask outside your house". Both are impractical, if not functionally impossible, in modern society

I'm not arguing which is "worse", only speaking to the reality we live in

BleatingZombie ,

I am arguing which is worse. There are people in Palestine who don't have the internet, don't have a phone, and don't have a credit card. How are they being tracked without facial recognition?

I also didn't say don't use a phone. I don't know where you got that

neatchee , (edited )

I know what you're arguing and why you're arguing it and I'm not arguing against you.

I'm simply adding what I consider to be important context

And again, the things I listed specifically are far from the only ways to track people. Shit, we can identify people using only the interference their bodies create in a wifi signal, or their gait. There are a million ways to piece together enough details to fingerprint someone. Facial recognition doesn't have a monopoly on that bit of horror

FR is the buzzword boogieman of choice, and the one you are most aware of because people who make money from your clicks and views have shoved it in front of your face. But go ahead and tell me about what the "real threat" is 👍👍👍

BleatingZombie ,

I didn't say "real threat" either. I'm not sure where you're getting these things I'm not saying

I think facial recognition isn't as much of a "buzzword" as much as it is just the most prevalent issue that affects the most people. Yes there are other ways to track people, but none that allow you to easily track everybody regardless of their involvement with modern technology other than facial recognition

(Just to be clear I'm not downvoting you)

neatchee , (edited )

That's why I put "real threat" in quotes ; I was paraphrasing what I consider to be the excessive focus on FR

I'm a security professional. FR is not the easiest way to track everybody/anybody. It's just the most visible and easily grok'd by the general public because it's been in movies and TV forever

To whit, FR itself isn't what makes it "easy", but rather the massive corpus of freely available data for training combined with the willingness of various entities to share resources (e.g. Sharing surveillance video with law enforcement).

What's "easiest" entirely depends on the context, and usually it's not FR. If I'm trying to identify the source of a particular set of communications, FR is mostly useless (unless I get lucky and identify, like, the mailbox they're using or something silly like that). I'm much more interested in voice identification, fingerprinting, geolocation, etc in that scenario

Again, FR is just...known. And visible. And observable in its use for nefarious purposes by shitty governments and such.

It's the stuff you don't see on the news or in the movies that you should really be worried about

(and I'm not downvoting you either; that's for when things don't contribute, or deserve to be less visible because of disinformation; not for when you disagree with someone)

sbv ,

I'm pretty sure the AI enabled torture nexus is right around the corner.

Sanctus ,
@Sanctus@lemmy.world avatar

Pretty sure we will see fake political candidates that actually garner votes soon here.

SnotFlickerman ,
@SnotFlickerman@lemmy.blahaj.zone avatar

The Waldo Moment manifest.

captain_aggravated ,
@captain_aggravated@sh.itjust.works avatar

Re-elect Deez Nuts.

bigkahuna1986 ,

This business is going to get out of control. It's going to get out of control and we'll be lucky to live through it.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • technology@lemmy.world
  • incremental_games
  • meta
  • All magazines
    •