Section 20 - Quality Assurance - Operational and System Reviews

Quality Assurance (QA) is required by regulation for some certificate holders and is an integral part of SMS. There are actually two different levels of quality assurance you need to consider. The first is operational QA, which verifies that all activities are conducted in accordance with regulatory and company requirements, typically reflected in the appropriate control manual (MCM/ MPM for maintenance, and COM for flight operations). The second is system QA, or system review, which evaluates the overall effectiveness of the company SMS and the interaction of the individual processes within the company.

As a basic principle, most management system models follow a Plan, Do, Check, and Act (PDCA) Cycle of Continuous Improvement. All of the individual processes in a company are planned (PLAN), performed as planned (DO), reviewed for effectiveness (CHECK), and modified as necessary (ACT) to ensure that they are safe, effective and efficient.

All staff needs to know what they are supposed to do and how they are supposed to do it. This can only be achieved through effective communication (training, SOPs, work instructions, etc.) between the planners and the doers. The principle of Checking is to see what really happens on the job. Are company procedures adequate? Do all staff understand the procedures? Are the procedures being adhered to? If it is found that the actual practice is different than the instructions then you have two options. You can make sure staff follow the instructions or you can change the instructions to make them more suitable for the required task. Both options require some more planning and implementation, thus continuing the cycle. This Continuous Improvement Loop process should be used for assessing every activity within a company including: operations, maintenance, training, finance and administration.

In very small organizations where the planners and the doers are the same people or are working closely together, the main method of communication will probably be verbal. In this case, the Checking is occurring on a daily basis. Remember the key is to keep it simple and appropriate for the size of the organization.

One way to ensure internal coordination is through the use of control manuals. Most operators and maintenance organizations ensure policy and procedures are carried out properly by providing written guidance in the form of standard operating procedures, rules of conduct or maintenance instructions. These are usually placed in an Operations Manual or Maintenance Policy Manual. The policies and procedures relating to safety management systems would typically be expressed as a component of these manuals but may be contained in a separate SMS manual if the company prefers.

At the system level, the PDCA continuous improvement cycle operates due to the commitment expressed by the management through the SMS policy. It focuses on the SMS processes such as the hazard identification and risk assessment, reporting, training and documentation. This ensures that the system is operating the way it is supposed to and if it is not, what changes (Plan) need to be made and implemented (Do) in order to correct the deficiency.

SMS requires all of the individual processes within an organization to have effective communications linkages so that the safety ramifications of processes in one area are clearly understood within all of the other areas in the company. So part of the system assessment is to see how effective the communication would be between different areas. For example, in a larger operation: Are a pilot’s concerns about his or her lack of currency on a particular aircraft (due to scheduling issues) being considered by the dispatch and shared with the training department and the Chief Pilot? Again, in the case of very small organizations this is very easy to evaluate, as the person responsible for the different processes will usually by the same person.

Self-Assessment

Whether you call it a self-assessment, review or audit, the process is essentially the same. But remember that the operational audit and the systems audit are looking at different things. This process could be completed by a group or by an individual who is already familiar with your workplace and its procedures. The number of people involved will depend on the size and complexity of your organization. Some of the best observations will come from those who are performing the procedures and, by making the employees a large part of this process, they are also kept informed and involved. This is an area where the non-punitive reporting processes within a company are essential to the success of this review.

How often should you do this? Again, it will depend on several factors such as the size, complexity and type of operation but realistically, given the planning and time it requires, once or twice a year should work well provided it is done thoroughly. There may be prescribed frequency of review periods in the CARs. In the section on Hazard Identification and Risk Management we mentioned certain events during which the risk of mistakes is higher or where additional oversight is needed and so if these events occur, an additional self-assessment would be warranted.

The self-assessment should be planned so as to cover the entire workplace. In a small operation, this will likely take several hours, while for a larger operation the assessment could be conducted in different areas of the workplace on different days. An effective assessment will review all parts of the operation to identify strengths, weaknesses and areas of risk and will follow a carefully prepared checklist. This assessment may be done in small components over a period of time to reduce the effect on operations.

In analyzing procedures, the person or persons conducting the assessment will need to be very aware that people may do a specific procedure; in the way they were trained, or in the way they thought it should be done or even just in ‘the way it’s always been done’. The objective is to take a critical look at every part of the operation. Actual work practices will be compared with ideal, safe working procedures and at every point in the checklist you need to ask “in what way could human error make this item a link in an accident chain?”

Your checklists will have to be tailored to your operation’s size and environment and should be updated regularly. As a guide, Transport Canada audit checklists can be obtained from the TC website or from your principal inspector. Sample checklists for the SMS assessment can be found in the toolkit. You can use or modify these to build one relevant to your organization. A comments section will allow documentation of weaknesses and areas of risk, as well as of items not specifically included in the checklist.

In a very small organization, the assessment could be included as part of an annual or end of season company meeting. This may take the form of a review of the activities of the previous year to highlight both positive and negative aspects of company performance. A list of activities or jobs that the company has performed could be presented and problem areas that occurred would be discussed with the staff as a group and changes to procedures would be developed to prevent re-occurrence. In this way, the lessons learned from these cases could easily be shared and distributed throughout the company. This would provide an opportunity to close the continuous improvement loop by reviewing planned actions and making appropriate changes to procedures.

In order to satisfy the requirement for documentation in this area, a record of this meeting including the planned agenda and resulting actions should be kept in company records.

As with routine hazard identification, an effective self-assessment has accurate and complete documentation. A written report summarizing the results is recommended and should be provided to the Accountable Executive. This will provide the information needed for a management review.

Good documentation makes the results accessible to everyone in the organization and eventually will allow the results of several audits to be compared to provide trend analysis.

The section on “Risk Management” above discussed using risk analysis to set priorities for corrective action, and the importance of giving everyone feedback on the hazards identified as well as on the steps being taken to minimize or eliminate them. Exactly the same process is used for the follow-up to a self-assessment.

Above all, this assessment, by being in-depth and all-inclusive, is a golden opportunity to find and fix safety weaknesses which otherwise might be overlooked. Make it work for you.