What happened?
The Federal Office for Information Security (BSI) and the Police Crime Prevention Programme of the German States and Federal Government (ProPK) have published the Cybersecurity Monitor, a current survey that reveals a significant discrepancy between self-assessment and actual behaviour: Almost half of internet users in Germany believe they can recognise AI-generated content. In practice, however, only a minority actually checks whether a piece of content is genuine. Only 28 percent have ever searched for inconsistencies in an image, and only 19 percent have researched the source of a piece of content.
The Details
The Cybersecurity Monitor provides further figures that illustrate the extent of this knowledge gap. Around a third of respondents (32 percent) have not yet taken a single one of the common measures to detect AI-generated content. At least 40 percent have considered whether a depicted scene appears realistic – a low-threshold but important first step.
Knowledge of specific fraud scenarios is also limited. Only 38 percent of respondents consider it possible for cybercriminals to deliberately manipulate an AI program in order to make it disclose sensitive data. Even fewer – only 40 percent – consider it plausible that criminals could embed invisible instructions for AI systems into documents. According to the BSI, both attack scenarios are technically possible.
BSI President Claudia Plattner emphasises the importance of recognising AI-generated content for consumer protection: "AI-generated content has long become part of everyday life for consumers. To identify risks and misinformation, it is therefore essential to recognise which content, posts, and depictions in the online world are AI-generated." The BSI provides guidance on this and raises awareness of what is possible with AI and how such content can be identified.
Dr. Stefanie Hinz, State Police Commissioner and Chair of ProPK, provides a concrete practical example: so-called cyber trading fraud. In this scheme, criminals promise quick profits and high returns from online trading. They often use AI to create videos in which apparent celebrities advertise supposedly lucrative investment opportunities. Hinz advises that, when faced with such offers, one should first check whether the promise is even realistic – or whether it is simply too good to be true.
Assessment
The figures from the Cybersecurity Monitor are relevant to businesses and security officers for several reasons. First, the discrepancy between perceived and actual competence reveals a fundamental problem: those who feel confident without applying appropriate verification mechanisms are less likely to question content critically and are therefore more susceptible to deception attempts. This affects not only private individuals but also employees within companies, who in their daily work are confronted with emails, documents, videos, or calls whose authenticity is not readily apparent.
Furthermore, the low level of awareness regarding technical manipulation scenarios – such as the deliberate insertion of invisible instructions into documents or the manipulation of AI programs to leak data – makes clear that corporate security concepts should not only include classic IT protection measures but also training on how to handle AI-generated content. The cyber trading fraud scenario described above illustrates in particular how deliberately AI is used to gain trust and cause financial damage.
Practical Tips
- Actively look for inconsistencies in suspicious content, such as incorrect shadows or unnatural limbs in images and videos.
- Always research the source of content before considering the information credible.
- Critically assess whether promised returns on online investment offers are realistic – according to ProPK, an essential first step against cyber trading fraud.
- Use the checklist provided by the BSI listing indicators for identifying AI-generated images.
- Incorporate ProPK's information resources on online investment fraud into training and awareness programmes.
- Keep the joint BSI and ProPK checklist for emergencies involving online banking fraud readily accessible.
Outlook
The Cybersecurity Monitor makes clear that technical protective measures alone are not sufficient to counter the growing risk posed by AI-generated content. Consumer protection and awareness-raising, as offered by the BSI and ProPK through their checklists, are becoming increasingly important. For businesses, this means that awareness measures addressing AI fraud should become a fixed component of their security strategy, complementing traditional topics such as perimeter security, access control, or access management. Given the technical possibilities for manipulating AI systems or embedding invisible instructions in documents, this topic is likely to gain further relevance in the coming months – both for private users and for security officers within organisations.
