Workshop of a range of experts on non-consentual intimate imagry.

As always, the below is my interpretation of what people said, and should not be interpreted as a directly quote by them.

Keynote: The Image is Synthetic, The Abuse is Real

Speaker: Eleonora Esposito

Title: The Image is Synthetic, The Abuse is Real: AI-Generated Image-based Sexual Abuse as a form Technology-Facility Gender-Based Violence

The design of technology can have a large impact on how technology is used, but also the ways it can be abused, and the ways people can (or cannot) protect themselves.

Technology-facilitated violence does tend to impact women disproportionally. 1 in 3 women in the EU has experienced some form of gender-based violence.

There are many forms of violence that focus at impacting women such as NCII Abuse, Deepfakes, and Unwanted Sexualization. The negative impacts of the abuse do not stop at the online world and often spill over into dailly life.

There is an economic argument here as well beyond the costs to mental wellfare. The annual cost of digital violence in the EU is 49-89.3 Billion Euro. These costs come from loss of life quality, labour market impact, and health and legal cost.

Image abuse has evolved with the introduction and evolution of the Interernet and digitization of communication. Image manipulation has always been a thing. For example, swapping out some of the contents of an image, wasy to do on say a white paper to change the message in a sign. More complex to swap out complex image elements.

Memes are also a way for large communities to have a conversation. Each person edits the meme with different messages. We see this same approach happen with image based abuse with false attribution of images to the wrong person.

Image manipulation is getting much easier…

Before GenAI: There was a need for a real underlying image. As a side effect, the perpitrator had to capture or steal a real image somehow. Also the image editing tended to be low quality. Motivations were also things like status, gamification, control, humiliation, sexual gratification, and retribution. Impacts are a range and include things like social rupture which breaks up how the victim interacts with others, constancy since the abuse never really ends, isolation, and constrained liberty because victims have to self-edit to protect themselves. (McGlynn et al. (2020))

After GenAI: Now a base image containing sensitive content is not needed. Nudification sold as a acheap consumer service, peer-to-peer deepfakes amoung minors at schools scales up, genreation of content embedded in mainstream platforms. There are now more motivations: amusement, curiosity, peer pressure among teams, private consumption, and financial extortion at a professional scale. Imact is shifting. The fact that the image is fabrication does not necessarily reduce the harm.

Two problems:

1. Now fake images can be passed off as real
2. Now real images can be passed off as fake

Child Sexual Abuse Material (CSAM) is very different from Non-consentual Intimate Imagry (NCII). CSAM is contraband, it is considered illegal. The EU also considers GenAI of CSAM to also be illegal. Consent is not possible for CSAM, there is no issue of consent and if it exists it is illegal, there is no possibility of a minor consenting in the law. This is very different from NCII where consent is an issue. CSAM is well protected under the law and consequently there is a good bit of technology that backs up that protection. NCII is very different with consent opening the possibility for dissent about if the image should or should not be up. It also puts the burden and blame on the victim rather than assuming that they should be protected.

• StopNCII - allows perceptual hashes, but not many social media take down images based on this dataset

EU regulatory framework

• 2011 - The council of Europe Istanbul Convention - on Prevent and Combating Violence Against Women and Domestic Violence
• 2021 Grevio - Group of Experts on Action against Violence against Women and Domestic Violence - General Recommendation #1
• 2022 - Regulation (EU) 2022/2065 Digital Services Act
• 2024 - Directive (EU) 2024/1385 - On Violence Against Women And Domestic Violence
  • It is illegal to share images, or threaten to share intimate images, without their consent
• 2024 - Regulation (EU) 2024/1689 - Artificial Intelligence Act

Harm is an important concept in law. But IBSA harm is accumulated over time. The problem with intimate imagry is constancy, that the images do not go away even when they seem to for a time. Responsibility is also a challenge, if an image exists we may not know who made it or what made it. If an system made it, how do we do legal responsibility. Consent is also complex. Just because a person consented to the creation of an image, that doesn’t mean that they consented to later uses or distribution. Consent gets more complex with GenAI where there may not be any point where consent is done.

Future challenges

• Theory of harms - Can IBSA Theory account for AI-Generated Harms?
  • GenAI is mostly used for private consumption - so distribution is no longer a component and the victim may never know about it.
• Substitution, Escalation and Offender Composition - it is now much easier to generate intimate images. There is an open question if it is a new set of offenders or if it is the same set of offenders as before.
• Statutory Thresholds in GenAI - Lots of laws hindge on there being an identifiable person. How to attribute likeness to an image. Open question on what counts as “identifiable person” to the general public.
  • What is “effective safeguards”?
• Protective Infrastructure - hashes and watermarks are the solutions that are most put forward
  • These are victim-focused: the victim has to state what images are problematic
• Measurement of the Nudification Ecosystem - Most of the time victims have to bring forward the problem. Detection is outsourced to victims. Abuse that victims never discover is not being measured or studied because we do not know about it.
  • There should be some ability to stop these images automatically.
  • There is no good way to detect nudes at scale.
• Protecting and Safeguarding Victims - can we avoid re-victimisation
  • Right now victims have to provide the image they want to take down or have their naked body scanned.

Mini presentations

Forensics reasoning

Forensics thinks about how events happened in the past based on evidence.

A typical NCEI before GenAI is: Explicit imagery is recorded during the relationship between Malleroy & Alice. After the relationship ends, Malleroy uploads the explicit material to a hosting site.

• Traces:
  • On Malleroy’s device of the filming
  • Traces on the metadata
  • May not have Malleroy’s device
  • Traces in the depected imagry (backtround)
  • Traces fro mthe uploaded
• Traces with GenAI
  • Traces on Malleroy’s device (may not have access devices)
  • Metadata has vanished
  • Traces in the detected imagry has vanished.
  • Still traces from the uploaded

Profit incentives of companies

Companies are driven by profit and social media is built by users. The users put lots of effort, data, and time into createing the communities and content. The companies get data to refine algorithums, sell data, and use the data for other purposes.

Moderation of content is considered a cost by companies. So they have minimal incentive to do the moderation.