1It is necessary to define clearly the scope of the prohibition, including in particular the extent of providers’ and deployers’ obligations. 2This prohibition should not prevent providers from developing the technical capabilities of AI systems to generate or manipulate images, videos, audio or similar material. 3As concerns providers, the prohibition should be limited to the placing on the market or the putting into service of AI systems that generate or manipulate non-consensual intimate material or child sexual abuse material in two cases. 4First, it should cover systems intended to generate or manipulate such material. 5Second, it should cover systems where such generation or manipulation is a reasonably foreseeable and reproducible outcome and there are no reasonable and adequate technical safety measures and other safeguards in place, taking into account reasonably foreseeable misuse, to reliably prevent, and where necessary correct, that outcome and correct observed or reported misuse, including circumvention of such measures. 6Technical measures and other safeguards to prevent the generation of such material could include data cleaning, refusal training, safe prompt design and output controls, runtime prompt guardrails, content classification and filtering mechanisms, usage restrictions, abuse detection mechanisms, and notice and action mechanisms. 7Such preventive measures should be reasonable for the specific AI system, and are considered adequate if they align with the state-of-the-art measures and demonstrably prevent or sufficiently reduce in each specific case the likelihood of generating or manipulating such material, taking into account known and reasonably foreseeable misuse, including reasonably foreseeable circumvention of the preventive measures without significant technical modification. 8For providers retaining effective control over AI systems, for instance through a platform or a web interface, that could include following and reporting methods for misuse cases in full compliance with Union privacy and data protection law. 9In cases of observed or reported circumvention of the preventive measures or other safeguards, adequate corrective measures should be taken provided that such measures are reasonable, taking into account the specific AI system, including its release and distribution strategy (such as open-source releases). 10As concerns deployers, the use of an AI system should be prohibited only where the deployer uses an AI system for the purpose of generating or manipulating non-consensual intimate material or child sexual abuse material. 11This includes cases where a deployer uses or misuses AI systems placed on the market or put into service that lack reasonable and adequate preventive measures or where a deployer circumvents the preventive measures or uses for such purposes lawful AI systems not intended to generate or manipulate such material. 12The prohibition on use therefore does not cover the use of an AI system for lawful purposes, such as the generation or manipulation of material other than non-consensual intimate material or child sexual abuse material, even in cases where the AI system lacks reasonable and adequate safeguards that should have been put in place by the provider, nor does it cover accidental generation or manipulation of such content. 13Concerning the prohibition regarding non-consensual intimate material, where an AI system is intended for generation or manipulation of material falling under this prohibition, measures and other safeguards should include appropriate means for the distribution of the AI system aimed at enabling the reliable collection and demonstration of consent of the depicted person to such generation or manipulation, in compliance with Regulation (EU) 2016/679. 14The prohibition regarding non-consensual intimate material should be limited to realistic depictions of intimate parts, notably the genitals, pubic area, anus, exposed buttocks or exposed female breasts, nipples or areolae, or of sexually explicit activity. 15This ‘realism’ refers to the depiction of the person’s face, voice or their body in a credible real-life manner, regardless of the realism of the context of that depiction or whether it fully corresponds to the actual voice or appearance of the depicted person. 16Conversely, it excludes cartoonish or physically impossible depictions of a person’s body. 17The prohibition of non-consensual intimate material does not affect the generation or manipulation of other forms of nude material, such as material that does not depict identifiable natural persons, realistic partially nude depictions where intimate parts are not revealed and sexually explicit activities are not depicted, or non-realistic artistic nude works that do not realistically depict identifiable natural persons engaged in sexually explicit activity or depict their intimate parts. 18It also does not cover generative AI applications where intimate parts are not exposed or, if exposed, this is subject to the freely given, specific, informed, unambiguous and explicit consent of the depicted person (for example, try-on applications and medical applications, such as medical anatomical simulations and mammograms); this prohibition does not preclude the exceptional use of AI systems generating or manipulating nude depictions of the intimate parts of an identifiable person, in accordance with fundamental rights law, including data protection law, and applicable medical law, for the purpose of medical diagnosis and treatment by medical professionals where the person concerned is incapable of consent (for instance in an emergency situation). 19Finally, the prohibition on ‘manipulating’ non-consensual intimate material excludes cases where pre-existing intimate material is manipulated in a way that does not increase the exposure of any depicted intimate parts or alter the nature of any depicted sexually explicit activities, for instance the mere enhancement of an existing image depicting intimate parts or video depicting sexually explicit activities, such as changing the background, adding a text heading or enhancing the contrast or the brightness. 20Conversely, any manipulation of material, including material that already depicts intimate parts or sexually explicit activity, that increases the level of exposure of any depicted intimate parts or alters the nature of any depicted sexually explicit activities, falls under the scope of this prohibition.
* This title is an unofficial description.