Description
Automated risk-level flagging enables compliance officers to identify and classify AI systems according to the EU AI Act’s risk categories: Prohibited, High-risk, Limited-risk, and Minimal-risk. The system uses automated detection based on content analysis of personal information, biometric data, healthcare information, financial data, critical infrastructure contexts, and law enforcement applications. Each automated flag includes a confidence score of 0-100%, with low-confidence flags below 70% highlighted for mandatory manual review. Compliance Officers can review automated classifications, validate confidence scores, and override flags when necessary, with all changes logged in the audit trail. The system tracks use-case categorization per the EU AI Act Annex III and monitors high-risk categories, including Biometric Identification, Critical Infrastructure, Education, Employment, Essential Services, Law Enforcement, Migration/Border Control, and Justice/Democracy. This feature exists to support systematic compliance with the EU AI Act classification requirements and to ensure appropriate oversight of high-risk AI systems. It is primarily used by compliance officers, legal counsel, and risk management teams.Example
A compliance officer reviews the automated risk assessment dashboard and identifies three prompts flagged as high risk for processing biometric identification data, with confidence scores of 85%, 68%, and 92%. She validates the two high-confidence flags and examines the 68% confidence flag in detail, reviewing the prompt content and request logs. After determining the prompt actually processes general facial recognition for photo organization rather than biometric authentication, she overrides the classification to Limited-risk, documents the rationale in the audit trail, and notifies the prompt engineering team to add clearer context metadata for future automated assessments.Risk Level Categories
PromptMetrics classifies AI systems according to EU AI Act risk levels:Prohibited
AI systems banned under EU AI Act Article 5, including:- Social scoring systems
- Subliminal manipulation techniques
- Exploitation of vulnerabilities of specific groups
- Real-time remote biometric identification in public spaces (with limited exceptions)
High-Risk
AI systems listed in the EU AI Act Annex III require stringent compliance:- Biometric identification and categorization
- Management of critical infrastructure
- Educational/vocational training (student assessment, admission)
- Employment decisions (recruitment, promotion, termination)
- Essential private/public services (creditworthiness, emergency services)
- Law enforcement (crime prediction, polygraph alternatives, evidence evaluation)
- Migration, asylum, and border control management
- Administration of justice and democratic processes
Limited-Risk
AI systems with specific transparency obligations:- Chatbots and conversational AI
- AI-generated content (text, images, audio, video)
- Emotion recognition systems
- Biometric categorization systems
Minimal-Risk
AI systems with no specific obligations beyond general law:- AI-enabled video games
- Spam filters
- Recommendation systems without high-risk context
The vast majority of AI applications fall into the minimal-risk category but may escalate to higher risk based on specific use cases and data processed.
Automated Detection Criteria
PromptMetrics uses pattern matching and content analysis to automatically detect risk indicators:Personal Information
- Full names, addresses, contact information
- National identification numbers
- Passport or driver’s license numbers
- Date of birth in combination with other identifiers
Biometric Data
- Facial recognition patterns
- Fingerprint or iris scan data
- Voice biometric analysis
- Gait recognition or behavioral biometrics
Healthcare Information
- Medical diagnoses or treatment plans
- Prescription medication data
- Mental health information
- Genetic or genomic data
Financial Data
- Credit scores or creditworthiness assessments
- Bank account or payment card numbers
- Loan application or approval decisions
- Investment or insurance recommendations
Critical Infrastructure
- Energy grid or utility management
- Transportation system control
- Water supply or waste management
- Telecommunications infrastructure
Law Enforcement
- Crime prediction or risk assessment
- Evidence evaluation or case prioritization
- Suspect identification or tracking
- Polygraph alternatives or deception detection
Reviewing Confidence Scores
Each automated risk flag includes a confidence percentage indicating the system’s certainty:High Confidence (70-100%)
- System is highly certain of the risk classification
- Based on clear content indicators and pattern matches
- Typically requires validation but rarely needs override
- Action: Review for accuracy and validate classification
Medium Confidence (50-69%)
- System identifies potential risk indicators but with ambiguity
- May include borderline cases or mixed content signals
- Action: Manual review required - examine request details and context
Low Confidence (0-49%)
- System detects possible risk indicators but with significant uncertainty
- May include false positives or edge cases
- Action: Detailed manual review required - likely needs reclassification
Manual Override Process
Compliance Officers can override automated risk classifications when necessary:When to Override
Override automated classifications when:- Context indicates different risk level: Prompt purpose doesn’t match detected content patterns
- False positive detection: System misidentified content (e.g., test data vs. real personal information)
- Use case clarification: Actual application context changes risk assessment
- Regulatory interpretation: Legal counsel provides different risk classification guidance
How to Override
1
Access Risk Assessment Dashboard
Navigate to Compliance → Risk Assessment to view all flagged requests and prompts.
2
Select Flagged Item
Click on a specific request or prompt to open the detailed risk assessment view.
3
Review Context
Examine the full prompt text, request logs, response data, and metadata to understand actual use case.
4
Document Rationale
In the override form, provide detailed rationale for the classification change citing specific EU AI Act articles or internal policies.
5
Submit Override
Select the correct risk level and submit. The change is logged in the audit trail with your user ID, timestamp, and rationale.
6
Notify Stakeholders
If significant, notify prompt engineers or legal counsel about the reclassification for awareness and future improvements.
Override Audit Trail
All manual overrides are permanently logged:- Original classification and confidence score
- New classification and rationale
- User ID of Compliance Officer who made the change
- Timestamp of the override
- Supporting documentation links or references
Use Case Categorization
Classify each AI system by EU AI Act Annex III use case:Biometric Identification
- Real-time biometric identification
- Post (non-real-time) biometric identification
- Biometric categorization
Critical Infrastructure
- Water, gas, heating, or electricity management
- Road traffic management
- Aviation or maritime systems
Education and Vocational Training
- Student assessment or admission decisions
- Examination scoring or evaluation
- Educational institution access control
Employment
- Recruitment or candidate selection
- Employment termination or demotion decisions
- Task allocation or performance monitoring
Essential Services
- Creditworthiness assessment
- Emergency service prioritization
- Benefit eligibility determination
Law Enforcement
- Risk assessment for crime prediction
- Polygraph alternatives or lie detection
- Evidence reliability evaluation
- Crime analytics or pattern detection
Migration and Border Control
- Asylum application assessment
- Border control risk evaluation
- Visa application processing
Justice and Democracy
- Legal interpretation or case law search
- Electoral process influence detection
- Democratic decision-making support
High-Risk Categories Tracking
The Risk Assessment dashboard provides filtering and monitoring for each high-risk category:- Volume tracking: Number of requests per high-risk category
- Trend analysis: Changes in high-risk usage over time
- Confidence distribution: Percentage of high vs. low confidence flags per category
- Override frequency: How often manual overrides occur by category
- Compliance status: Percentage meeting transparency and oversight requirements
Related Documentation
Compliance Overview
Return to the main compliance officer introduction.
Request History
Search and filter requests by risk level for detailed investigation.
Compliance Monitoring
View real-time metrics for high-risk system performance.
Transparency Disclosure
Manage transparency requirements for high-risk AI systems.