The online platform X has provoked significant backlash after users utilized its AI chatbot Grok to manipulate pictures of women by stripping their attire.
Numerous instances have surfaced where the system has been used to digitally undress female subjects to render them in swimwear without consent, as well as to put them in inappropriate scenarios.
Developer Reaction and Legal Stance
XAI, the firm behind Grok, offered no comment to questions, other than an automatically-generated message stating "legacy media lies".
A Home Office official stated that legislation is being brought forward to prohibit nudification tools. Under a proposed criminal offence, anyone who supplied such tools would "receive a jail term and substantial penalties".
"The platform could prevent these forms of abuse if they desired to," said a law professor, adding they "seem to have impunity."
Regulator Stance and Tool Capabilities
The regulator Ofcom said platforms must "assess the potential" of people viewing unlawful material on their networks, but did not specify whether it was presently investigating X or Grok in regarding AI-generated pictures.
Grok is a no-cost artificial intelligence—with some additional functions—which answers X users' queries when they tag it in a tweet.
It is often used to offer commentary or further context to other users' remarks, but people on X are also can modify an posted photo through its artificial intelligence picture modification function.
Background of Criticism and Rules
The tool has been condemned for permitting users to produce pictures and videos with nudity and sexually suggestive material, and it was earlier accused of making graphic content of public figures.
"X has been allowing the generation and spread of these images for months without taking any action and we have yet to see any intervention by regulators," Professor McGlynn added.
XAI's own terms of service bans "portraying likenesses of persons in a graphic manner".
In a statement, Ofcom said it was illegal to "create or share deepfakes or illegal content" and stated this included sexual deepfakes made with AI.
It said platforms such as X were obligated to take "necessary steps" to "lessen the danger" of users encountering illegal material on their platforms and to take down it promptly when they are notified of it.