This is a theoretical question. I just watched a certain video in which the author apparently unmasks a chatbot AI that is likely trying to harvest data and spread influence in a cult-like manner on a given social network. The video is hosted on YouTube: Who or What is Stephanie Lawson Stevens?

What really caught my attention was that at some point one of the investigators, who has an IT background, starts asking some questions to the user (who up to that point was just a person, not a supposed AI) and then infers from the answers that that might be an actual chatbot. The video I linked above shows the process.

I'm no specialist in AI at all but it made me wonder: The AI reads information coming from a text input (the chatbox). Would it be theoretically possible for a user to craft a certain coded message through a chat that, once read by the AI, acts as if a reflected XSS attack was made on the AI (the same way that people can reflect code that is executed by a victim's browser)?

In this scenario, if the programming of the AI itself doesn't include the necessary precautions in sanitizing the text input, the AI might read a snippet of code from the chatbox and end up executing it.

Is this an actual possibility?


XSS? Highly unlikely. General injection attacks? Sure.

XSS is essentially a vulnerability that allows an attacker to execute Javascript in another users browser. But it's unlikely that an AI would simulate real-world users to a degree where it logs in with a browser, gains a session, etc.

It is more likely that the AI would run on the server itself (or some other server and communicate via API).

Without (user/AI) interaction in a web browser, there's by definition no XSS. But you can have various other injection vulnerabilities. OS command injection, SQL injection, etc.

A very simple example:

$userinput = [input]
// do complex stuff. Find an unknown word $word (defined as a string of chars in-between spaces) and look it up via OS command:
$definition = system("dict " + $word);

What is also possible is that user input can contain some HTML (eg for links), and that the HTML parser which feeds data to the AI contains vulnerabilities (SSRF comes to mind). But again, without executing Javascript in a browser and abusing the current authorization of the attacked user, any vulnerability here wouldn't be an XSS vulnerability.


actually on your scenario you can have 2 possible weakness.

weakness 1 :

Sended query (as string) will be probably be prompt on the chatbox. If it is not properly sanitize, you can inject malicious code which will runned by your browser and maybe by human operator which will browse your interaction with the chat bot.

weakness 2 :

As this kind of AI is base on linguistic analyze, string will be transform to token, like peace of intel to understand the behavior of the string. Depend on the technology used you can also have possible malicious code injection on this piece.

For your suggested case where is the AI himself which have point of failure, i don't think it is possible by this way, means inject special command which counter the original behavior of the AI. It will probably only return a regular failure from the AI which doesn't understand what it have to do.

the actual main issue with AI is the quality of the data for training, not for exploitation. If you can pollute by massive wrong data when AI is trained, then you can influence the final behaviour. Interesting case was Tay, A microsoft AI which has started to have unexpected answser on twitter after a massive attack.

  • Welcome on the SecSE! What is an "IA"?
    – peterh
    Dec 29 '19 at 20:19
  • my apology "IA" is the acronym for Artificial Intelligence in french, i have fixed this typo mistake. About the downvote, i will appreciate if you can also drop me a comment to explain to me why my answser was wrong, like this i can improve myself next time :)
    – Yanis-git
    Dec 30 '19 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.