2002 Local Client Satisfaction Survey

The content of this page is wider than 640 pixels

• Introduction
• Methodology
• Questionnaire Design
• Sampling Strategy
• Data Collection Operations
• Data Weighting
• Data Processing
• Data Analysis
• Limitations of this Research
• Satisfaction, priorities, delivery
• Client Satisfaction
• Priorities for Improvement
• Preferences for Delivery of the Application
• Conclusions and recommendations
• Service Improvement
• Local Measurement
• Appendix A
• Questionnaire

Introduction

In the context of the global Government of Canada initiative aimed at improving service to the public, the Passport Office has decided to initiate a client satisfaction measurement program. This report presents the analysis of the 2002 Passport Office local client satisfaction surveys. It is based on self-administered surveys conducted at each location of the Passport Office.

Assignment

The assignment included the collection of local satisfaction data from clients and the production of individual office scorecards. These tasks comprised the following steps:

• finalizing, pre-testing and adjusting the questionnaire based on work conducted in previous assignments;
• developing the sampling approach;
• managing the survey data and implementing editing rules where necessary; constructing a complete and fully documented data set;
• developing a structure for the local reports and applying it to each participating point of service;
• preparing an overall final report on the project

Structure of the report

The study methodology is presented in Chapter 2. Conclusions regarding client satisfaction, priorities for improvement and preferred application delivery method are presented in Chapter 3. Chapter 4 summarizes conclusions and recommendations for action.

Methodology

This research is based on a self-administered survey of clients. The following aspects of the methodology are discussed: questionnaire design, sampling strategy, data collection operations, data weighting, data processing, data analysis and limitations of the study.

Questionnaire Design

The questionnaire was developed by Circum Network Inc. based on results of previous research. By design, the questionnaire had to fit on one page. Previous survey results indicated that a limited set of service dimensions are of particular importance in the depiction of client satisfaction. They are:

1. the five key drivers of satisfaction noted in the Treasury Board "Policy Framework for Service Improvement in the Government of Canada"
• the competence of the staff;
• the courtesy of the staff;
• the fairness of the application process;
• the time it took to produce the passport;
• the passport as a travel document (also in c).
2. the key generic priority for improvement identified earlier
• the waiting time at the office;
3. the key determinants of overall satisfaction
• our responsiveness to your needs and the reliability of our service;
• our communications with you;
• access to our services;
• costs and payment options;
• the requirements to obtain a passport;
4. an overall assessment
• our service overall.

The questionnaire was built on these indicators. It also contained a question regarding the key improvement priority and the preferred method of delivery of the passport application. Clients were asked for further qualitative comments.

This questionnaire was pretested with 22 Passport Office clients in situ before the full fledged implementation of the field work. The pretest uncovered no significant issue with the questionnaire form. Several clients commented that they do not tend to participate in such surveys. The questionnaire is presented in Appendix A.

Sampling Strategy

The client population was defined as all recipients of a new passport who were returned their passport during the period starting on June 10, 2002 and ending when 1,000 survey forms were distributed; the last office to complete the process closed it on August 30, 2002. Completed questionnaires returned by mail were accepted until November 1, 2001.

The client population was stratified according to the office. Each office was to deliver 1,000 forms, at a rate which would stagger the sampling over the entire survey period. Based on 2001 work statistics, the proportion of clients who needed to be sampled was calculated for each office, by the contract manager. For example, if an office had produced 10,000 passports over that period in 2001, it had to sample 10% of clients; this was accomplished by including the survey form in every passport which ended by a certain digit, chosen at random. If 20% of clients had to be sampled, two ending digits were randomly selected.

Data Collection Operations

Local offices were responsible for the implementation of data collection procedures. No record of compliance with procedures could be maintained.

At the end of the survey period, the vast majority of the 30,000 questionnaire forms had been distributed to clients. Some offices appear to have fallen short of the target of 1,000 deliveries.

At the end of the data collection period, 6,903 questionnaires had been returned ? fewer than the 300 per office that were originally expected. Exhibit 2.1 provides the detail of the number of questionnaires returned, by office. Note that the National Processing Service was not expected to be included in the study, but was.

EXHIBIT 2.1 : Questionnaires returned, by office

Data Weighting

Ex post facto weights were required to ensure that the relative share of clientele of each Passport Office location was respected in the final data. Weights were based on passport production during the survey period; they varied between 0,1 (Thunderbay) and 9,6 (North York). The sample stratification design effect was not accounted for in inferential statistics calculations because the study focussed on local satisfaction measurements where no design effect occurs (by definition since points of service were the stratifying factor).

Data Processing

Survey data were managed using SPSS. Data were edited to ensure conformity to established response categories. Filtering logic instructions were developed to ensure that the reported data conformed to the logic of the questionnaire. The data were weighted according to the issuing office.

Data Analysis

Data analysis was done using basic stubs-and-banners crosstabs developed in SPSS. Percentage-based differences were tested on a percentage-versus-complement basis using two-tailed binomial distributions. Differences between means were tested using two-tailed t-tests.

Based on the full sample of 6,903 responses, the maximum sampling error is estimated at ±1.2 percentage points in the worst, complete-sample case (for a proportion of 50%, at a confidence level of 95%, not accounting for a stratification design effect, without correction for finite population). Sampling errors are wider for sub-samples. Exhibit 2.2 presents some typical sampling margin of error values.

Sampling margins of error for various sample sizes

Limitations of This Research

The results of this research are based on a large sample of 6,903 Passport Office clients; however, at the local level, sample sizes are significantly smaller, varying between 53 (North York) and 331 (Frederiction). Care must be taken to analyse survey results in the context of the sample base used.

If each office distributed 1,000 survey forms, then response rates vary in the same way that local sample sizes do: the local response rates would run from a low of 5% (North York) to a high of 33% (Fredericton). The overall response rate for the study would be 23%. This is a typical response rate for a self-administered questionnaire. If non-respondents share the attitudinal profile of respondents, this response level raises no inconvenience. However, it is not possible to assert the extent of correspondence between respondents and non-respondents. In the absence of evidence otherwise, we have assumed that no particular bias exists in the sample of respondents.

Satisfaction, priorities, delivery

This chapter deals with the three themes of the local survey of clients: client satisfaction, priorities for improvement and preferences regarding the delivery of applications.

Client Satisfaction

Measurement issues

The 2001 national survey of clients measured client satisfaction on a five-point scale and converted the results to a scale from 0 to 10. The Passport Office obtained a score of 8.2 on that scale.

In another study conducted for Treasury Board of Canada Secretariat, Circum Network Inc. proposed to that departments and agencies should standardize on the use of a weighted satisfaction score where the top-box of a five-point scale bears the value 100 and the bottom-box, a value of 0. Based on this methodology, the Passport Office satisfaction score was 82 in 2001.

The 2001 national survey of clients proposed to use a customer satisfaction index (CSI) aggregating various aspects of client satisfaction. This was seen as a way to take into account the multidimensional aspect of the concept of client satisfaction. However, three considerations have emerged since this report was presented:

• the proposed CSI included client reactions to the requirements to obtain a passport, to cost and payment options and to the passport as a travel document; in the context of a local client satisfaction survey, since these aspects of service are not within the control of local authorities, it seems unfair to include them as heavily into the measurement;
• the risks of relying on insensitive CSI measures have been outlined repeatedly in the literature; a more prudent, albeit more time- consuming, approach is to analyse each of the key drivers of client satisfaction individually;
• the value of single-item measurement (as opposed to scaled measurement using responses from several items) has be clarified in the literature.

Consequently, this report focusses its attention on each of the key dimensions of satisfaction and on the overall satisfaction judgment expressed by clients themselves (rather than on a somewhat artificial CSI).

EXHIBIT 3.1 : Client satisfaction results

Exhibit 3.1 summarises the results obtained regarding client satisfaction.
Exhibit 3.2 aggregates all responses returned.

EXHIBIT 3.2 : Overall results

• As in the 2001 national telephone survey, the 2002 local self-administered surveys produced a satisfaction score of 82.
• Staff competence and courtesy were most appreciated by clients (scores of 89 and 88).
• Least satisfying were cost and payment options (61 points) followed by the waiting time at the office (62 points).

EXHIBIT 3.2 : Overall satisfaction, by region

Dimension of satisfaction	Satisfaction score
Service overall	82
JWE	84+
JWC	76-
JWO	84+
JWW	81-

• Offices in the Eastern region and in Ontario received the higher satisfaction scores. This is due, most notably, to client perceptions related to:
• in the East, staff courtesy and competence, the time taken to produce the passport, the responsiveness and reliability of the service, communications with clients and access to services;
• in Ontario, the time taken to produce the passport, waiting time at the office (mainly) and the responsiveness and reliability of the service.
• Offices in the Central and Western regions received the lower satisfaction scores. This is due, most notably, to client perceptions regarding:
• in the Central region, the time taken to produce the passport(8), the responsiveness and reliability of the service, communications with the clients and staff competence;
• in the Western region, waiting time at the office and access to services.
• Exhibit 3.1 provides detailed results for each office.

Priorities for Improvement

Clients were asked to select one priority for improvement; some selected more than one. Exhibit 3.3 depicts the choices made by survey participants.

EXHIBIT 3.3 : Priorities for improvement

• Overall, waiting time at the office was seen as the first priority for improvement of client service by 33% of clients; this is followed by 24% who selected cost and payment options ? on the basis of the comments made by clients, the issue is "cost", not "payment options". The time taken to produce the passport is a distant third, with 11% of clients.
• While waiting time at the office was the number one priority for improvement across Canada, it is important to note that it was selected as such in only 10 of the 30 offices; these offices happen to be high volume in general; also, clients tend to lump heavily under the waiting time priority when it appears to be a local issue.
• With only two exceptions, at the local level, when clients did not select waiting time at the office as the first priority for improvement, they focussed on the cost of the passport. The exceptions are the National Processing Service (mail service) where production time was identified as a priority and the London office where access ranked first for improvement.

EXHIBIT 3.4 : Preferred method of delivery of the passport

Conclusions and recommendations

Service Improvement

The local client satisfaction survey has identified two clear priorities for improvement:

• overall, waiting time at the office is the first issue, but it concerns 10 offices out of 30;
• second overall ? but most significant in a majority of offices ? is the cost issue.

On this basis, the following actions are recommended:

• the development and implementation of an action plan to address the office waiting time in ten offices identified in this study, namely: St-Laurent, Richmond, Surrey, Ottawa, Hamilton, Calgary, North York, Montréal, Toronto and Vancouver;
• the development and testing of a communication plan to explain to clients the components of the price of the passport and the reasons for the current pricing; this could go a long way to addressing the "cost and payment options" issue identified as the first priority for improvement in 18 offices.

Local Measurement

As a first attempt at measuring client satisfaction at the local level within the Passport Office, this project has demonstrated that:

• local measurement is feasible;
• local measurement provides useful information for service improvement;
• satisfaction and priorities for improvement do vary from one point of service to another.

The observation of the unfolding of this project leads us to the following recommendations:

• the implementation of the next iteration of the local service satisfaction survey should aim at accelerating the process by shortening each phase of the study ? planning, data collection, data capture and reporting;
• the questionnaire should remain the same (with the possible deletion of the question on preferred method of application delivery) to ensure comparability of results;
• the Passport Office should plan to conduct its local client satisfaction survey again in 2004 ? thereby providing sufficient time for local offices to plan, organize and execute their service improvement strategy.