Automated Automotive Cybersecurity Testing Process: Bridging the ISO/SAE 21434 Verification Gap

About the Authors and This Research

This paper was presented at IEEE VTC2021-Spring and has accumulated 12 citations since publication, indicating meaningful uptake in the automotive cybersecurity research community. The lead contributor Stelios Karagiannis carries a notable h-index of 15 with 775 cumulative citations, establishing him as a credible voice in vehicle cybersecurity and embedded systems security testing. Co-author Hayk Hamazaryan (h-index: 3, 43 cumulative citations) contributes a practical engineering perspective on automated testing process design.

The timing of this research is significant. ISO/SAE 21434 was finalized in 2021, and UNECE WP.29 Regulation 155 began requiring OEMs to establish Cybersecurity Management Systems (CSMS) with verifiable testing evidence. Against this backdrop, the paper identifies a critical gap: while standards define what needs to be secured, they do not prescribe how to systematically test security properties. This paper directly addresses that methodological void.

Core Finding: A Structured Framework for Automotive Cybersecurity Testing Verification

The research identifies a structural deficiency in current automotive cybersecurity practice: even when organizations complete rigorous Threat Analysis and Risk Assessment (TARA) as required by ISO/SAE 21434, the subsequent testing phase often lacks standardized execution methodology. This creates a situation where test results are difficult to compare, reproduce, or present as auditable evidence—a significant liability during TISAX assessments or OEM supplier audits.

Finding 1: The Structural Gap in Cybersecurity Testing Methodology

The authors demonstrate that while the attack surface of modern vehicles has expanded dramatically—with ADAS systems, over-the-air update mechanisms, and V2X communications creating new threat vectors—testing processes have not evolved proportionally. Manual, ad-hoc testing approaches cannot provide the consistency, speed, or comparability required by emerging regulatory frameworks. The paper quantifies this in qualitative terms: without structured testing processes, organizations face inherent inconsistency in how security properties are verified across different development teams, projects, and audit cycles.

Finding 2: A Tool-Neutral, V-Model-Aligned Testing Process Architecture

The proposed framework is structured around the V-model logic familiar to automotive engineers, mapping testing activities to corresponding development phases from system concept through implementation. Critically, the framework is designed to be tool-neutral—it defines the logical structure of testing processes without mandating specific tools, enabling organizations to integrate their existing toolchains within the structured framework. The framework addresses test planning, test design, test execution, and results evaluation as distinct phases with defined inputs and outputs, creating the traceability chain that TISAX evaluators and ISO/SAE 21434 auditors expect to see.

Implications for Taiwan's Automotive Supply Chain

For Taiwanese automotive component manufacturers navigating ISO/SAE 21434 compliance and TISAX certification, this research illuminates a commonly overlooked compliance gap. Many organizations invest heavily in TARA methodology and CSMS documentation while treating the testing and verification phase as a downstream execution task rather than a process design challenge.

Under UNECE WP.29 R155, vehicle manufacturers are required to demonstrate that their CSMS extends through the supply chain, which means Tier 1 and Tier 2 suppliers must produce auditable testing evidence—not merely assurances of security. Taiwanese suppliers that rely on unstructured manual testing face elevated risk of audit findings during TISAX evaluations, particularly at Assessment Level 2 (AL2) and above, where evaluators examine the depth and systematicity of testing documentation.

It is important to acknowledge the paper's methodological boundaries: the framework is conceptual and has not been validated against large-scale industry implementations. Taiwan's SME-heavy automotive supply chain also faces higher relative costs for test automation tool procurement and specialized personnel. Winners Consulting Services Co. Ltd. therefore recommends treating this framework as a reference architecture for process design rather than a direct implementation blueprint.

How Winners Consulting Services Co. Ltd. Helps Taiwan Suppliers Build Auditable Testing Mechanisms

Winners Consulting Services Co. Ltd. (積穗科研股份有限公司) supports Taiwan's automotive supply chain in achieving TISAX certification, implementing ISO/SAE 21434, and meeting UNECE WP.29 vehicle cybersecurity regulatory requirements. Based on the insights from this research, we recommend the following action steps:

1. Testing Process Gap Assessment (Months 1–2): Systematically map current testing activities against ISO/SAE 21434 Chapters 10–12 verification and validation requirements. Identify testing phases lacking documented input/output traceability and prioritize remediation.
2. Structured Test Process Documentation (Months 3–5): Design test plan templates aligned with V-model logic for each development stage, ensuring testing activities produce auditable evidence chains. Integrate with vulnerability and incident handling procedures to create a closed-loop security assurance mechanism.
3. Test Automation Feasibility Evaluation (Month 6 onwards): Assess the priority and cost-effectiveness of automated testing tools—including penetration testing frameworks and fuzz testing tools—scaled to organizational size and product complexity, with a focus on satisfying TISAX requirements for test reproducibility.

Frequently Asked Questions (English)

What does ISO/SAE 21434 specifically require for cybersecurity testing, and how should Taiwan suppliers prepare?

ISO/SAE 21434 Chapters 10 through 12 require organizations to conduct verification and validation of cybersecurity properties, including documented test plans, defined test methods, and traceable test results. The standard intentionally does not prescribe specific testing tools or execution methods, which is the gap this paper addresses. Taiwanese suppliers should prioritize establishing written testing process procedures with clear input/output documentation at each testing phase. Starting this process at least 6 months before a TISAX assessment or OEM audit is strongly recommended to avoid last-minute remediation efforts.

Does TISAX evaluation review cybersecurity testing documentation, and what are the consequences of deficiencies?

Yes. TISAX assessments, particularly at Assessment Level 2 (AL2) and AL3, include review of cybersecurity testing process documentation. Evaluators expect to see systematic testing plans, test case designs, and results reports that demonstrate traceability to security requirements. Unstructured or undocumented testing activities frequently generate Findings during TISAX evaluations, requiring corrective action within a defined remediation window—typically 3 to 6 months. Organizations that address testing documentation proactively typically complete assessments with fewer Findings and shorter overall timelines.

How does UNECE WP.29 R155 extend cybersecurity testing requirements to component suppliers?

UNECE WP.29 R155 requires vehicle manufacturers to establish Cybersecurity Management Systems (CSMS) and extend security requirements throughout their supply chains via contractual mechanisms. For component suppliers, OEM customers typically require auditable testing evidence demonstrating that delivered components meet agreed cybersecurity specifications. Taiwanese component suppliers that cannot produce structured testing reports risk failing OEM supplier audits, which can directly impact order qualification. Winners Consulting recommends proactively requesting clarification from OEM customers on testing documentation format and depth requirements to design compliant processes accordingly.

What resources and timeline are realistically required to build a structured automotive cybersecurity testing mechanism?

Based on Winners Consulting's experience supporting Taiwan's automotive supply chain, establishing structured cybersecurity testing processes from a baseline of informal practices typically requires 4 to 8 months, depending on existing testing infrastructure and product complexity. Tool procurement costs vary significantly by solution. However, process design and personnel training are often the more critical investments for SMEs. A phased approach—structured manual testing in Year 1, followed by selective test automation in Year 2—allows organizations to control initial investment while progressively building test capabilities aligned with ISO/SAE 21434 and TISAX requirements.

Why engage Winners Consulting Services Co. Ltd. for Automotive Cybersecurity (AUTO) advisory?

Winners Consulting Services Co. Ltd. brings specialized expertise across the full ISO/SAE 21434 lifecycle—from concept-phase TARA through to production verification and TISAX certification preparation. Our advisors understand the resource constraints and OEM audit pressures specific to Taiwan's SME-heavy automotive supply chain, enabling us to design compliance pathways that are both rigorous and proportionate. We support clients in establishing auditable cybersecurity management mechanisms within 7 to 12 months, minimizing rework risk during TISAX evaluations and OEM supplier qualifications.

日本語版 / Japanese Version

積穗科研股份有限公司（Winners Consulting Services Co. Ltd.）は、現代自動車のデジタル化が急速に進むなか、ADAS（先進運転支援システム）やV2X通信の複雑化に伴い、従来の手動サイバーセキュリティテストではISO/SAE 21434が要求する検証・妥当性確認のニーズに対応しきれなくなっていると指摘します。Hamazaryan、Karagiannis、Kraxbergerによる2021年IEEE VTC掲載論文は、自動車サイバーセキュリティのテストプロセスを構造化・自動化するための実践的フレームワークを初めて体系的に提示しており、TISAXおよびUNECE WP.29への適合を目指す台湾自動車部品メーカーにとって重要な示唆を含んでいます。