AI Act and Information Security: What You Need to Know About the AI Regulation

TL;DR

The AI Act (Regulation (EU) 2024/1689) has been in force since August 2024 and will become fully applicable in stages through August 2027. Prohibited AI practices have applied since February 2025.
The AI Act classifies AI systems into four risk classes: prohibited practices, high-risk systems, limited-risk systems (transparency obligations), and minimal-risk systems.
Comprehensive requirements apply to high-risk AI systems: risk management, data quality, documentation, transparency, human oversight, accuracy, robustness, and cybersecurity.
The cybersecurity requirements of the AI Act mandate that high-risk AI systems be protected against attacks that could manipulate their functioning (adversarial attacks, data poisoning, model extraction).
For the ISM, the AI Act means: add AI systems to the asset register, assess AI-specific risks, create an AI policy, and inventory AI usage across the organization.

Why the AI Act is relevant for information security

If you work as an ISM or information security officer, you might initially think about the AI Act: "That's a topic for the AI developers and the legal department, not for me." This assessment is understandable but wrong.

The AI Act contains explicit cybersecurity requirements for AI systems. It demands that high-risk AI systems be robust against attacks, and it addresses AI-specific threats such as adversarial attacks, data poisoning, and model extraction — threats that haven't appeared in traditional ISMS risk assessments until now. At the same time, more and more organizations are using AI tools in their daily work, from ChatGPT to AI-powered data analytics to automated decision systems, creating new risks that the ISMS must address.

The AI Act affects you in at least three roles: as the person responsible for the security of IT systems in the organization (including AI systems), as a risk manager who must assess AI-specific threats, and as an advisor to senior management making decisions about AI system deployment.

The AI Act at a glance

The AI Act (Regulation (EU) 2024/1689) was published in the Official Journal of the EU on July 12, 2024, and entered into force on August 1, 2024. Full applicability is being phased in gradually.

Timeline

February 2, 2025: Prohibitions on unacceptable AI practices apply
August 2, 2025: Requirements for general-purpose AI models (GPAI) apply, including obligations for providers like OpenAI, Google, and Anthropic
August 2, 2026: Most requirements are fully applicable, including the rules for high-risk AI systems
August 2, 2027: Requirements for high-risk AI systems embedded in other EU-regulated products

What is an AI system?

The AI Act defines an AI system as a machine-based system that operates with varying levels of autonomy and that pursues explicit or implicit objectives to generate outputs such as predictions, recommendations, decisions, or content that can influence physical or virtual environments.

This definition is deliberately broad and covers not only the obvious AI applications (large language models, image generators, autonomous systems) but also many systems you might not consider "AI": algorithms for credit scoring, automated resume screening tools, predictive maintenance systems, or fraud detection in payment processing.

The four risk classes

The AI Act follows a risk-based approach. The higher the risk an AI system poses to fundamental rights and the safety of individuals, the stricter the requirements.

Prohibited AI practices (Article 5)

Certain AI applications are entirely prohibited because they are considered incompatible with European values:

Manipulative techniques: AI systems that use subliminal techniques or manipulative methods to influence individuals' behavior in a way that causes them harm
Exploitation of vulnerabilities: AI systems that deliberately exploit vulnerabilities of specific groups (age, disability, social situation)
Social scoring: Scoring of individuals based on their social behavior that leads to unjustified disadvantage
Real-time remote biometric identification: In public spaces by law enforcement (with narrow exceptions)
Emotion recognition: In the workplace and educational institutions
Biometric categorization: Based on sensitive characteristics (race, political opinion, trade union membership, religious belief, sexual orientation)
Untargeted facial recognition databases: Building facial recognition databases through untargeted scraping of images from the internet or surveillance cameras

For the ISM, this is relevant: if your organization uses AI systems that could fall under one of these prohibitions, you must escalate. The prohibitions have applied since February 2025, and violations are subject to fines of up to EUR 35 million or 7 percent of global annual turnover.

High-risk AI systems (Article 6 and Annex III)

High-risk AI systems are the core of the AI Act. They are subject to the most comprehensive requirements. An AI system is classified as high-risk if it is used in one of the following areas:

Biometrics: Remote biometric identification, biometric categorization, and emotion recognition (to the extent not prohibited)
Critical infrastructure: Safety components in road traffic, water, gas, electricity and heat supply, and digital infrastructure
Education: Access to educational institutions, assessment of learning performance, examination monitoring
Employment: Recruitment, performance evaluation, promotion decisions, task assignment based on individual behavior
Access to essential services: Credit scoring, insurance risk assessment, emergency dispatch evaluation
Law enforcement: Risk assessment of offenders, lie detectors, evidence evaluation
Migration: Risk assessment at border controls, asylum proceedings
Justice: Support for judicial decisions

Additionally, AI systems are classified as high-risk if they are embedded as safety components in products falling under specific EU harmonization legislation (Machinery Regulation, Medical Devices Regulation, Toy Safety Regulation, and others).

Limited-risk systems (transparency obligations)

Certain AI systems are subject to transparency obligations even if they are not classified as high-risk:

Chatbots and virtual assistants: Users must be informed that they are interacting with an AI system
Deepfakes: AI-generated or manipulated image, audio, or video content must be labeled as such
AI-generated text: Text published for informational purposes on matters of public interest must be labeled as AI-generated

Minimal-risk systems

All remaining AI systems (the vast majority) are not subject to specific requirements. The AI Act encourages providers to voluntarily adopt codes of conduct.

Requirements for high-risk AI systems

For high-risk AI systems, the AI Act defines comprehensive requirements in Articles 9 to 15. Here are the most relevant for information security.

Risk management system (Article 9)

Providers of high-risk AI systems must establish a risk management system covering the entire lifecycle of the system. This system must: identify and analyze risks associated with the AI system, define and implement risk mitigation measures, assess and communicate residual risks, and conduct tests to verify the effectiveness of the measures.

For the ISM, there's a natural integration: AI risk management should be part of the ISMS risk management, not a separate system. The methodology can be the same (asset assessment, threat analysis, risk treatment), but the risk catalogs need to be expanded with AI-specific scenarios.

Data quality and data governance (Article 10)

Training, validation, and test data for high-risk AI systems must meet certain quality criteria: they must be relevant, representative, as error-free as possible, and complete. Data collection and processing must comply with data protection regulations.

From an information security perspective, training data represents a particularly sensitive asset. Classification of such data is the first step. If attackers gain access to training data, they can manipulate the model (data poisoning) or extract confidential information contained in the data. The ISMS must capture training data as assets to be protected and define appropriate safeguards.

Cybersecurity and robustness (Article 15)

Article 15 contains the explicit cybersecurity requirements of the AI Act. High-risk AI systems must exhibit an appropriate level of accuracy, robustness, and cybersecurity and function consistently throughout their lifecycle.

Robustness against errors and inconsistencies. AI systems must be robust against errors, disruptions, and inconsistencies in the environment in which they are deployed. Technical redundancy, fail-safe mechanisms, and fallback solutions are part of this requirement.

Resilience against AI-specific attacks. This is the most relevant point for information security. High-risk AI systems must be resilient against:

Adversarial attacks: Targeted inputs that trick the AI system into producing incorrect outputs (for example, a slightly modified image that deceives an image recognition system)
Data poisoning: Manipulation of training data to alter the behavior of the AI system
Model extraction: Attempts to reconstruct the AI model or its training data through targeted queries
Model inversion: Attempts to derive information about the training data from the AI system's outputs

These attack vectors are new territory for most ISMs. They require an understanding of the specific workings of AI systems and collaboration with AI developers and data scientists.

Human oversight (Article 14)

High-risk AI systems must be designed so they can be effectively overseen by natural persons. The overseeing individuals must understand the capabilities and limitations of the system, be able to detect anomalies, and be able to interrupt the use of the system or override its outputs.

Provider vs. deployer: Who is responsible for what?

The AI Act distinguishes between providers (who develop and place AI systems on the market) and deployers (who use AI systems in their organizations). Both have obligations, but to different extents.

Provider obligations

Providers of high-risk AI systems bear the main compliance burden: establish a risk management system, conduct conformity assessment, prepare technical documentation, affix CE marking, conduct post-market monitoring, and report serious incidents.

Deployer obligations

Deployers have less comprehensive but nonetheless substantial obligations:

Use according to instructions: Deployers must use high-risk AI systems in accordance with the provider's instructions for use
Human oversight: Deployers must ensure that competent individuals oversee the AI systems
Input data quality: Deployers are responsible for the quality of input data
Monitoring and reporting: Deployers must monitor operations and report incidents or malfunctions to the provider and, where applicable, the supervisory authority
Data protection impact assessment: For certain high-risk AI systems, deployers must conduct a data protection impact assessment
Fundamental rights impact assessment: Certain deployers (public bodies, private deployers of public services) must conduct a fundamental rights impact assessment

When does a deployer become a provider?

A critical distinction: if a deployer substantially modifies an AI system (for example, adapts a pre-trained model for a new purpose or performs fine-tuning), it can legally become a provider and thereby assume full provider obligations. This is relevant if your organization adapts AI models or develops its own AI solutions based on third-party platforms.

What the AI Act means for your ISMS

Capture AI systems as assets

The first step: create an inventory of all AI systems used or developed in your organization. This includes not only obvious AI applications but also embedded AI features in existing software (for example, AI-powered fraud detection in ERP systems, AI-based spam filters, predictive features in CRM systems).

For each AI system, document: the purpose of use, the provider, the risk class under the AI Act, the data processed, the users, and the responsibilities. In ISMS Lite, AI systems can be captured as a separate asset category and directly linked to the risk assessment.

Assess AI-specific risks

Expand your risk assessment with AI-specific scenarios:

Data leakage to AI services: Employees enter confidential company data into ChatGPT, Copilot, or similar services — a typical shadow IT problem
Adversarial attacks on business-critical AI systems: An attacker manipulates the inputs of an AI-powered credit assessment
Bias and discrimination: An AI-powered recruiting tool disadvantages certain applicant groups
Hallucinations and misinformation: An AI-generated report contains false information that feeds into business decisions
Dependency on AI third-party providers: The provider of a business-critical AI function changes its terms, raises prices, or discontinues the service

Create an AI policy

Create an AI policy that governs: Which AI tools may be used in the organization? What data may be entered into which AI services? Who approves the use of new AI systems? How are AI outputs verified? What transparency obligations apply?

The AI policy should be created as a standalone document or as an extension of the existing IT usage policy. It must be regularly updated because the AI landscape is changing rapidly.

Collaborate with data science and development

If your organization develops its own AI systems, you as the ISM must work closely with data scientists and developers. The cybersecurity requirements of the AI Act (Article 15) require technical know-how that typically doesn't reside with the ISM. Define clear interfaces: the ISM defines the security requirements, the AI developers implement them, and the ISMS monitors compliance.

Penalties

The fines under the AI Act are tiered:

Up to EUR 35 million or 7 percent of turnover for violations of the prohibitions
Up to EUR 15 million or 3 percent of turnover for violations of other requirements
Up to EUR 7.5 million or 1 percent of turnover for providing inaccurate information

For SMEs and startups, reduced fines apply (the lower of the two alternatives in each case).