Mutual recognition of national military airworthiness authorities: A streamlined assessment process

,


Thank
The RM outputs RMIT R

Introduction
Having to comply with a number of disparate military regulatory requirements imposes "unsustainable" costs on the European Defence industry [1].In response, the Military AirWorthiness Authorities (MAWA) forum was established in 2008 by the European Defence Agency (EDA), with the aim of harmonizing national military airworthiness regulations This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/bync/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
within the European Union (EU) [2].The long term goal is to establish a single military airworthiness regulatory system for all EU countries [3].Achieving this goal requires [3]: 1.A common regulatory framework; 2. a common certification processes; 3. a common approach to organizational approvals; 4. common certification/design codes; 5. a common approach to the preservation of airworthiness; 6. arrangements for mutual recognition; and, 7. the formation of a European Military Joint Airworthiness Authorities Organization.Largely mirroring the civil aviation (namely the European Aviation Safety Agency (EASA)) approach and structure, it was agreed that a common set of European Military Airworthiness Requirements (EMARs), Acceptable Means of Compliance (AMC), and Guidance Material (GM) should be developed, accepted, and implemented into national regulation by all EDA members [2].Recently, the EMARs Parts 1, 21, 66, 145, and 147 (EMAR Part M, still under development), and the European Military Airworthiness Document -Recognition (EMAD-R), have been released.The EMARs are largely consistent with the EASA framework, and allow for military specific requirements.The EMAD-R details an agreed process for the recognition of other Military Airworthiness Authorities (MAAs), in terms of authority, certification, and approval of products and organizations, based on the EMARs [4].
Mutual recognition facilitates gains and improved efficiencies in all MAA interfaces with other MAAs, and their regulatory frameworks.The current global military focus for recognition is technical.Significant work on mutual recognition is being undertaken by the EDA, through the EMAD-R process, and within ASIC through an Airworthiness Working Group, which comprises of representatives from the United States Department of Defence (US DoD), United Kingdom Ministry of Defence (UK MoD), Canadian Department of National Defence, and Australian and New Zealand Departments of Defence.The EDA has finalized the first mutual recognition for Part 145 requirements (Maintenance Organizations) of the Airbus A400M between the UK and France.Similarly, ASIC have adopted the EMAD-R process, with the first recognition activity carried out between the UK MAA and United States Army for mutual recognition, primarily for type certification purposes (as part of the AH-64D Apache helicopter acquisition by the UK), as well as for personnel safety flying on US Army/UK MoD operated aircraft [5].
There is one primary difference between the approaches for mutual recognition being pursued by the EDA and ASIC.This is the reason for the EMAD-R process not being directly transferrable, although it was adopted by the ASIC [6].The EMAD-R mutual recognition relies on the consistent application of, or known compliance with, the EMARs.No such consistent airworthiness application is likely to occur within the ASIC, nor with MAAs outside the EU.Even further, it is unlikely that, at least in the near future, non-EU MAAs will develop compliance with the EMARs.This is in part motivated by significant cost (in regulation, education, oversight, and enforcement changes), with as yet limited non-EU motivation.Greater global implementation of the EMARs will provide added motivation.For this reason, an alternate method of recognition, or a framework for supporting the process described in the EMAD-R, is necessary.The motivation for this initiative is validated by simply examining the level of complexity and non-coherence across the different MAAs around the world [6].
This paper highlights the effective utilization of the Product-Behavior-Process (PBP) Bow-Tie model, proposed by Purton et al. [7] for filling the gap for non-EMAR countries.The PBP Bow-Tie model [7] is a specific, modified application of the conventional bow-tie [8][9][10][11][12][13][14][15][16], which is normally used for risk management and accident analysis.Importantly, no other assessment models or methodologies for regulatory frameworks have been identified in the literature, making this derivation and application unique.The PBP Bow-Tie model provides a structure for the systematic representation and analysis of disparate airworthiness regulatory systems.The model facilitates the identification of test points within the regulatory framework, which are to be answered by each MAA.The test point answers are used as a basis for comparing, and eventually assessing, two (or more) different regulatory frameworks.This assessment is supported through a framework assuming that technical integrity is assured by three key components: Product, Behavior, and Process (PBP) integrity.
A baseline assessment can be formed that supports the EMAD-R process, making it suitable for all airworthiness bodies, civil or military.A detailed explanation of the model, and the rationale behind it is provided in [7]; however, a summary is offered in the following section.The recognition process between the Australian Defence Force (ADF) and New Zealand Defence Force (NZDF) is assessed, and demonstrated as an exemplar case.

Derivation of the PBP Bow-Tie model
The basic layout of the PBP Bow-Tie model structure is illustrated in Fig. 1, and may be explained briefly as follows.
Conventional Bow-Tie models in aviation safety are utilized to assess the barriers in place, to reduce the likelihood, or reduce the consequence of a potential hazard, triggered by a specific event [8][9][10][11][12][13][14][15][16].In the PBP Bow-Tie model, the primary hazard or "top event" is the loss of technical integrity [7].Technical integrity can be maintained through assuring product, behaviour, and process (PBP) integrity [7,17].These three elements establish a set of threat lines (or causal scenarios) potentially leading to a loss of technical integrity, and in turn, potential loss events.
Barriers can be put in place to reduce the likelihood of any of the threat lines eventuating.Within the technical item lifecycle, there are three distinct activities: design, production, and maintenance.Barriers can be grouped in relation to these three activities.Barriers can also be grouped as preventative measures, i.e. those put in place to reduce the likelihood of losing technical integrity (grouped on the left-hand side of the PBP model, Fig. 1), and reactive measures (grouped on the right-hand side of the PBP model, Fig. 1), which aim to reduce the likelihood of consequential outcomes, given a loss of technical integrity.The proactive management of airworthiness places an emphasis on the preventative barriers; whilst, reactive management would have a stronger emphasis on reductive barriers.
Each barrier represents a specific regulatory process or "test point" that is undertaken within each of the highlevel technical lifecycle activities.For example, a test point may be that there is a standard set for aircraft maintenance processes.An attestation is made at each of these test points, verifying that there has been a judgment of acceptability at each test point, before the regulatory process proceeds to the next activity in the technical lifecycle.Threat paths describe those situations where there have been a series of incorrect attestations.This is shown in Fig. 2.
The PBP Bow-Tie model test points are scored, utilizing an independence metric.The independence categories utilized to distinguish the different levels for attestations within a regulatory framework are presented in Table 1.Independence was selected as the appropriate metric, due to the influence organizational and environmental factors have on human decision-making, which can lead to direct errors, the introduction of latent errors (the work of James Reason [18][19][20]) or safety drift (the work of Rasmussen [21,22]).For this reason, independent regulators have been established in fields in which there is required to be a primacy on safety (aviation, medicine, and food hygiene, for example).
Measuring each test point identifies where there is regulator interaction, and therefore identifies where there is a primacy on safety.The independence categories, when applied to each test point within the PBP Bow-Tie model, facilitate a picture (Iris plot) of an individual regulatory framework.Iris plots provide a concise means of visualizing the different factors that contribute to the technical regulatory framework, as well as facilitating direct comparison between different technical regulatory frameworks.In this case, the focus is on the technical regulations; but the framework and metrics are transferrable to any regulatory framework or process, which can be either operational or technical.The implementation of the model in a two-party recognition exercise is presented in the next part of this paper.

Implementation of the PBP Bow-Tie model
The PBP Bow-Tie assessment process is illustrated in the flowchart shown in Fig. 3.The process of assessment is efficient, when carried out with a knowledgeable member within the military airworthiness regulator, who can identify the organisations and appropriate attestation levels.Conducting the assessment is the first part of developing a comparison picture that is capable of informing recognition.The two outputs that enable the comparison are the Iris plots, detailed in this section, and the gathered evidence, detailed in Section 4.
The PBP Bow-Tie model was developed to facilitate recognition outside of the EU.It has arisen due to a requirement for known compliance with the EMARs within the EMAD-R process.Due to geographical proximity, the ADF and NZDF (two of the ASIC nations) established a recognition agreement to serve two purposes.These were firstly, progress recognition within ASIC, following the UK MAA and US Army recognition activity; and secondly, to establish the suitability of the PBP Bow-Tie as a method of comparing regulatory frameworks for countries not on the EMARs.This assessment provides a methodology to assess, and then compare, high-level regulatory frameworks.Different elements of the airworthiness framework can be assessed, by scoping the assessment to design (or

certification), production, or maintenance (operations).
The implementation of the model for the ADF and the NZDF comparison produced two Iris plots, which are presented in Fig. 4 and Fig. 5 respectively.As described in [7], the visualization of the output from the PBP Bow Tie model can be represented graphically, using Iris plots.The Iris plot can be constructed based on different groupings of the data, either by integrity threat line, or by technical   Each radial corresponds to a test point within a technical lifecycle activity (e.g.design, manufacture, maintain), and the length of the radial is the independence score assigned to that test point.The radials are shaded, to further assist visualization of the independence.As per the scale of Table 1, attestations made by a partially independent regulator (internal), or a fully independent (external/legislative) regulator, correspond to an independence score of greater than three.This is illustrated on the Iris plot by scores greater than the purple ring, which is aligned at three.Analysis of the Iris plots for the ADF and NZDF highlight areas of difference; for instance, as can be seen in Fig. 4, the ADF has strong process control within aircraft and aeronautical product design and maintenance, while there is limited oversight of production activities.From Fig. 5, it can be observed that the NZDF relies on the Civil Aviation Authority for oversight of the Defence Industry maintenance organizations, and like the ADF, has limited oversight of production activities.
A systematic basis for comparison between the two regulatory frameworks is needed, in order to enable interagency recognition.An inter-agency recognition process should address any identified differences between these two frameworks.Test point scores with differences of two (judged as significantly different) or greater within their scores, or alternatively, only one country, indicate regulator interaction for a test point.

Comparison of Regulatory Frameworks
A brief overview of the differences between Defence and Defence Industry is provided in the following sections.These differences are not outlined to prove that one regulatory framework is better than the other, but to identify where there are areas to be addressed, as part of the establishment of a process of mutual recognition between the ADF and NZDF.A detailed comparison of the Iris plots generated for the two regulatory frameworks is performed.This comparison is first made based on the activities undertaken by the Defence regulator (Fig. 6), and then on the activities outsourced to the Defence Industry (Fig. 7).

Regulatory framework differences for Defence Organizations
From Fig. 6 it can be observed that there a number of differences in the test points scores for the regulatory interaction by the ADF and NZDF.The ADF issues organizational approvals, and conducts compliance with regulations, and conformance to processes (indicated by the series of scores of four for design and maintenance process).The NZDF does not issue organization approvals, and only conducts conformance to the NZDF accepted engineering and maintenance publications.The ADF does not offer regulatory controls for supply; while, the NZDF defines the product acceptability requirements for supplied product (indicated by the regulator interaction at TP1.6 (supply in design)) and TP3.6 ((supply in maintenance) for the NZDF).The NZDF sets the required training standards for maintainers as a regulatory function, but does not make a formal attestation of acceptability.The ADF does not set the required standards through regulations, but does require a formal attestation of acceptability (indicated by Behaviour attestations for maintenance).The amount of effort and time to generate the Iris plots is significantly less, than the work required through the existing process of conducting on-site interviews and assessments.The Iris plots are effectively a quick and easy-to-use tool, to identify differences between regulatory frameworks up-front, providing a streamlined optimized approach to mutual recognition projects.

Regulatory differences for contracted organizations
From Fig. 7 it can be observed that there a number of differences in the test points scores for the regulatory activities undertaken by contracted external organizations, on behalf of the ADF and NZDF.The NZDF relies on the New Zealand Civil Aviation Authority for oversight of design and maintenance organizations, which is subject to requirements for expositions [indicated by scores of five (external regulator)].Both the design and maintenance organizations can still be subject to audits, to confirm there are no contraventions to the intent of the NZDF engineering and maintenance documents.The ADF treat Defence and the Defence Industry the same, with respect to exposition requirements.The NZDF require similar product acceptability requirements of the Defence Industry for supplied product.The ADF and NZDF are comparable for production oversight, with neither providing regulatory interaction for production; they both rely on other organizations to oversee production on their behalf.
A more detailed test-point to test-point comparison can be carried out; that is, each segment of the Iris plot is symmetrically compared (i.e.TP1.4 (right-hand side) is compared to TP1.4I (left-hand side)), for the ADF and NZDF.
As discussed, a difference in the respective independence scores of greater than two (i.e.three, one), or where the scores dictate that one service provides a regulatory attestation, and the other does not (i.e.four, three), warrant particular interest.Importantly, a greater independence does not indicate a better system, but the fact that there is a difference that warrants further examination.Due to the level of fidelity of the detailed comparison and defence security restrictions, the full analysis is not presented in this paper.

Iris plot limitations
There are points of interest extracted during the PBP Bow-Tie model interrogation of the test points that are not displayed visually in the Iris plot.For example, with regard to delegations of authority, one may note that most scores of three are from delegated authority; where a three indicates either a distinct management organization, or a person exercising a delegation of authority from the airworthiness authority.Both the ADF and NZDF utilize formal delegated authority from the Technical Airworthiness Authority (TAA), for design and maintenance decisions and attestations.The ADF has Design Acceptance Representatives (DARs), and the NZDF has Delegate Engineering Authority (DEA) issued to key personnel.Moreover, it has been shown that the Iris plot has no way of identifying the different organizations involved in the regulatory framework attestations.These limitations, however, can be covered during recognition dialogue, and analysis of the Iris plots.

Conclusion
The PBP Bow-Tie model implementation has demonstrated a method of comparing technical airworthiness regulatory frameworks for design, production, and maintenance.The comparison has identified areas of difference, and provided reasons and motivations for these differences, so that they can be understood, and accepted, or further investigated.It has provided a solid foundation for the next steps in the recognition process.
There is a degree of uncertainty surrounding military airworthiness regulatory frameworks, especially when making comparisons between different regulatory frameworks.This uncertainty is compounded, when each of the Nations making the assessment are trying to assure that in recognizing the regulatory framework of another, they are not introducing any risks into their own regulatory framework.Therefore, in initiating recognition dialogue, a method of comparing technical airworthiness regulatory frameworks that have differing regulatory structure and policy is paramount.The PBP Bow-Tie model provides a generic and systematic mechanism for representing disparate regulatory frameworks.This model provides the common framework necessary for comparison.The output can be represented graphically on an Iris plot, which facilitates examination of the interaction of the regulator with the regulated entities.It provides a method of identifying attestation differences, and allowing for systematic and holistic examination of the technical lifecycle.The Iris plots also create a platform for dialogue that promotes recognition.It is acknowledged that it is difficult to quantitatively address the suitability of the PBP Bow-Tie models; however, the use of the models in identifying the areas to focus on in this recognition effort has demonstrated the qualitative benefits of the PBP Bow-Tie assessments.
The PBP Bow-Tie model assessment is flexible, and with appropriate education on its derivation, along with the Iris plot illustrations, is capable of concisely conveying information.It can be utilized to provide a platform for inter-agency (mutual) recognition of regulatory findings and processes.It is also a method for self-examination, highlighting areas in which there is little regulator interaction.Both the ADF and NZDF utilized the PBP Bow-Tie model for this, and it provided a platform for successful recognition between the two countries, overcoming the unknown compliance with the EMARs, and providing for wider recognition, than the limited recognition activities already carried out using the EDA established EMAD-R process.The PBP Bow-Tie assessment has subsequently been included in the ASIC.In addition, this process is able to streamline the standard mutual recognition, and will inform all future recognitions, based on this successful demonstration of the ability of the assessment models to identify areas for recognition within the EMAD-R process, currently undertaken by the MAAs, both in terms of time, and effort.

Fig. 1 .
Fig. 1.Composition of the bow-tie framework, overlapped with the technical integrity definition and technical item lifecycle.This paper is focused on preventative barriers, shown on the left-hand side of this figure.

Fig. 2 .
Fig. 2. Breakdown of the steps required for attestations of acceptability, during the lifecycle of an item.

Fig. 3 .
Fig. 3. Flowchart of the PBP Bow-Tie assessment process for regulatory framework comparison.

Fig. 3 .
Fig. 3. Flowchart of the PBP Bow-Tie assessment process for regulatory framework comparison.

Fig. 4 .
Fig. 4. The ADF Iris plot symmetrical comparison grouped by technical activity.This Iris plot indicates a largely symmetrical regulatory framework.

Fig. 5 .
Fig. 5.The NZDF Iris plot symmetrical comparison grouped by technical activity.This Iris plot indicates that the NZDF regulatory framework provides different regulatory requirements for Defence and Defence Industry.
lifecycle activity.The result is two distinct plots of the results of the application of the PBP Bow Tie.Each plot displays the same information, but can reveal different subtleties within a regulatory framework.The first representation of the model output is on the technical lifecycle activity.In this paper, the assessment is extended to compare regulatory framework intricacies for Defence, and Defence Industry[7].It is important to distinguish these two sectors of military aviation, since each regulatory framework integrates them distinctly.In Fig.6and Fig.7the PBP Bow-Tie model scores for Defence are shown on the right-hand side, and for the Defence Industry (design, production, or maintenance performed under contract), are shown on the left-hand side.

Fig. 6 .
Fig. 6.The ADF and NZDF technical regulatory framework comparison for Defence.The symmetrical differences indicate areas requiring examination, during recognition efforts.

Fig. 7 .
Fig. 7.The ADF and NZDF technical regulatory framework comparison for Defence Industry.The symmetrical differences indicate areas requiring examination, during recognition efforts.

Table 1 .
Independence metric for the PBP Bow-Tie model assessment.
Scores greater than three indicate regulator interaction.