Directorates
·	Director General's Office
·	Data Development
·	Strategic Evaluations
·	Partnership and Continuous Evaluations
Publications
·	Evaluation Reports
·	Evaluation Briefs
·	Labour Market Development Agreement Series
·	Lessons Learned Series
·	Evaluation Tool Kit Series
·	Evaluation Forum Newsletter
·	From Theory To Practice

3.1 The RBA Client Data File

As part of the synthesis of the Mid-Term reviews, the research team compared the information obtained from the reviews with information derived from the client data that is uploaded by the RBA-holders to HRDC. It should be noted that the data file and uploading systems are in the developmental phase. The purpose of the comparison was a attempt a preliminary data validation exercise. Additional work must be completed in the areas of data integrity, reliability and validity as the system matures.

The HRDC data file analyzed for these comparisons is the RBA Client Data File developed and maintained by Information Services at HRDC-NHQ. The RBA data file consists of client data uploaded from the RBA-holders and is currently under development. As of 30 April 1998, over 13,000 RBA client records have been uploaded, and another 3,000 records have been received at NHQ but have not been uploaded into the file. Some RBAs have yet to upload any data to HRDC-NHQ.

Once data is received, it is matched with the BNOP file (Benefits and Overpayments) to ascertain the employment status of each EI client. The primary indicators of unpaid EI benefits, and unpaid social assistance are then calculated. (See Section 4. for a discussion of these indicators.) National-, regional- and RBA-level results are made available to all RBA-holders. The current analyzes were completed using the RBA Client Data File for those client records uploaded as of 30 April 1998.

3.2 Methodology

The team conducted two levels of analyzes with the data from the RBA Client Data File. The first set of analyzes was conducted to determine the level of completeness for number of clients or participants in RBA. That is, are the data contained in the RBA Client Data File an accurate reflection of the clients described in the Mid-Term review reports? This analysis will aid in the assessment of the accuracy of the primary indicators calculated using the RBA Client Data File.

The second level of analysis the team conducted was on the completeness of individual client records. That is, for the client records that are being successfully uploaded, do they contain data in the mandatory data fields? This level of analysis is connected to the first in that if there are very few client records, then an accurate analysis of indicators is not possible. Conversely, if there are many client records but they are relatively empty (many missing variables), then the usefulness of the data is also severely hampered.

3.3 Results

The following tables and text contain the results obtained by the team.

3.3.1 Analysis 1: How many client records are missing?

As can be expected in a complex data collection system under development, it appears that not all client records are being uploaded. As reported in Table 3.1, of the 15 RBAs included in the Mid-Term review process, 10 had successfully uploaded some client data. With one exception, the number of participants contained in the client data file for each RBA is lower than the number of participants reported in the Mid-Term review. If the data system was fully operational, we would expect the opposite, since data collection for the Mid-Term review process was completed several months prior to the data upload used for these analyzes.

Table 3.1 provides estimates of missing client records, based on the Mid-Term review information.

It should be noted that in most cases, these are likely underestimates , given the gap between Mid-Term reviews and data uploading. That is, it should be assumed that the RBA-holders have continued to provide interventions to new clients during the period between the Mid-Term reviews and the most recent data upload.

With this amount of missing data, the validity of the values calculated for the primary indicators (Unpaid EI and Unpaid SA) becomes questionable. With the exception of the Federation of Saskatchewan Indians RBA, the values for these indicators appear to be underestimates for the majority of RBA-holders participating in the Mid-Term review process. Particular caution should be paid to the primary indicator data of those RBA-holders that have uploaded a substantial amount of data (60% or more of expected cases). The tendency is to assume that once some data has been uploaded the data set is complete or representative of the RBA.

3.3.2 Analysis 2: What variables are missing?

Although a substantial portion of the records appear to be not yet uploaded to the RBA Client Data File, those records that have been received are for the most part complete. In order to calculate the primary indicators, the essential information is a client Social Insurance Number (SIN), type of client (EI, SA or other), number of SA clients employed, and intervention start and end dates. Other data that Mid-Term review RBAs are consistently providing are:

• tombstone data;
• intervention type;
• source of funding for intervention; and
• intervention outcome.

Social Insurance Numbers

As the SIN is the unique identifier for the matching link with the other databases (e.g., BNOP), this piece of data is crucial to success of the system. A perusal of the data revealed that most SINs were in the valid range (9-digits; no letters). Looking at the results of the calculations for unpaid EI benefits, it appears that the SINs were matched successfully between the BNOP and RBA Client Data File. If an RBA-holder indicated that it had 50 EI clients, then the calculations showed that n=50 for the calculations of EI Unpaid. It is assumed that if it had not been possible to match the SIN with the BNOP for all 50 cases, then this would have been indicated in the calculation outcome (i.e., n<50).

Type of Client

For the RBA-holders who participated in the Mid-Term review and successfully uploaded client data, the type of client was reported for almost all clients.¹⁷ As this is crucial to determining the primary indicators of EI and SA unpaid, it is a positive sign to see this variable relatively complete. Table 3.2 reports the numbers for the EI/SA split for each of the RBAs. This raises the question of whether any RBA clients are neither EI or SA clients? If that is the case (as reported in one Mid-Term review), why are these client records not being uploaded? They may not be calculated into any of the primary indicators; however, these interventions are RBA activities and perhaps should be considered when conducting uploads of client data.

Social Assistance Clients Employed

This variable requires a follow-up process by the RBA-holder to ascertain whether an SA client has found employment post intervention. It is difficult to assess the accuracy of the data collected on this variable without examining individual data collection systems at the RBA level. For the RBAs included in the Mid-Term review, there is an indication that almost all have some sort of follow-up data collection process in place. As reported in Table 3.2, except for one, all RBAs that had uploaded data had some Unpaid SA calculated. This illustrates the fact that they are reporting that some SA clients have found employment post intervention. For the RBA that showed no SA savings, it could be the case that none of the SA clients had found employment post intervention. Another interpretation is that the RBA has not yet set up a follow-up data collection process.

Intervention Start and End Dates

The client records uploaded by the RBAs participating in the Mid-Term review contain start and end dates for most interventions. This permits more accurate calculations of Unpaid EI and SA and Displacement (SA clients receiving RBA support, rather than SA while training).

Other Variables

Other main variables, such as Client Name, Address, Birthdate and Sex, are fairly complete. One variable that is only reported approximately 15% of the time is Telephone Number. For the purposes of calculation of primary indicators, it is not necessary. However, for the follow-up with clients in any evaluation work, this is an important piece of information that, although not mandatory, is extremely useful for conducting surveys for outcome or impact evaluations.

3.4 Summary

The RBA client data collection system is still under development. The Client Data File is incomplete. Using data obtained from the RBA Mid-Term review process, the team found that the RBA Client Data File contains data from only 10 of the 15 participating RBAs. Of those eight RBAs that have successfully uploaded data, only one RBA has managed to upload more client records than those reported in its Mid-Term review. This suggests that the RBA Client Data File should still be considered in the developmental phase. As a result, information derived from this file, including calculations of primary indicators, should be considered underestimates and not as being necessarily representative of all RBA activities in relation to the method of calculating the indicators. Whether this truly reflects the results of the RBAs is another question, which is explored in Section 4.

View Table 3.2

Footnotes

17	Since this review was conducted, HRDC has further refined the information being uploaded with respect to the type of client (reflecting the type of income assistance being received).