The Superior Court of Justice (STJ) fined lawyers R$ 20,000 for hiding instructions in a legal filing to influence the artificial intelligence used by the court. The practice, known as prompt injection, was considered an unlawful attempt to interfere with the technological infrastructure of the Judiciary.
The 1st Panel also ordered that the documents be sent to the Paraná branch of the OAB and to the Federal Public Prosecutor’s Office. According to the source, this is the largest fine imposed by the STJ in a case involving an attempt of this kind.
The use of artificial intelligence is permitted in legal filings, but attempting to deceive the Court’s algorithms is unlawful. – Image: Digineer Station/Shutterstock
How the attack with hidden commands works
A prompt injection is a technique that seeks to steer the responses of an artificial intelligence system through instructions inserted into texts. Instead of merely analyzing the requested content, the system can be induced to follow malicious commands that attempt to alter the outcome of the task.
In the case heard by the STJ, the lawyers distributed hidden instructions at different points on a page of the special appeal petition. One of them was inserted into the decorative footer banner, in the same color as the graphic element, making it impossible to read with the naked eye.
The commands sought to favor the lawyers’ own argumentation.Among the instructions, there were requests for the AI to consider that the challenged ruling disregarded the limits of res judicata and conclude that the appeal should be accepted in full.
The risk lies in the possibility that the tool will interpret these instructions as a legitimate part of the task, rather than recognizing that someone is attempting to manipulate its analysis. The technique specifically exploits the difficulty of separating the content that should be examined from the commands inserted to influence the response.
What are the risks for decisions and documents
When AI is used to examine documents or produce recommendations, malicious commands may compromise the analysis if the system cannot identify them properly. In a legal context, this could affect the interpretation of arguments, the synthesis of information, and the recommendations produced to support human work.
Possible risks include:
- Biased legal analyses: hidden instructions may favor a specific interpretation of the arguments presented.
- Distorted summaries: relevant information may receive less attention if the tool is influenced by commands inserted into the document.
- Compromised recommendations: AI-generated responses may depart from the criteria established for the task.
When AI is used to analyze legal documents, hidden commands may bias the interpretation of arguments, distort summaries, or influence recommendations.This may undermine the reliability of the information produced by the system.
The rapporteur, Minister Paulo Sérgio Domingues, classified the conduct as exceptionally serious and considered the substantial fine an “appropriate response.”
Concealing commands in digital texts raises new questions about the security of automated adjudication systems. – Image: TANYARICO / Shutterstock
STJ distinguishes manipulation from legitimate use of AI
The decision does not prohibit lawyers from using artificial intelligence to prepare legal documents. The court distinguished the use of technology as a support tool from the covert insertion of commands intended to influence the system used by the adjudicating body.
The decision states that it does not intend to discourage the use of AI in preparing legal documents, but rather to prevent these tools from being unlawfully used to convey hidden commands to the court’s artificial intelligence, with the aim of affecting its technological infrastructure.
In addition to the R$20,000 fine, the panel ordered that the case files be sent to the Paraná branch of the OAB for assessment of possible administrative sanctions, and to the Federal Public Prosecutor’s Office for analysis of any potential criminal implications. The source also reports that the Federal Supreme Court had already fined a lawyer for a similar reason, in a decision by Minister Alexandre de Moraes.
Valdir Antonelli is a journalist specializing in digital marketing and consumer issues.
