Illuminating Pathways

Applicants for child welfare positions are accustomed to providing basic information about their educational background and previous job experience. Once an applicant is hired, an agency will typically establish an employee record containing information such as start date, position number, and starting salary. However, in most agencies, data about job applicants is not saved or is stored in a different system than the one that maintains active employee data, resulting in a lost opportunity to use these data for strategic decision-making to improve recruitment and selection. In addition, most agencies do not routinely update their employee records to maintain a current complete profile of their child welfare workforce, reflecting the full range of key demographic, educational background, experience, licensure, and expertise variables. This represents another lost opportunity and hampers the agency’s ability to produce useful insights about its workforce, monitor underlying trends, and inform succession planning.

Child welfare agencies, and their pre-service education partners such as Title IV-E Master in Social Work stipend programs, lack guidance on the types of information to collect and track, how to maintain it, as well as how to analyze and share findings in impactful ways that can inform education, training, and human resources planning efforts. The purpose of this document is to provide such guidance. We outline what employee information would be helpful for agencies to collect data about, how best to measure each type of variable, when in the employee lifecycle this data should be collected, how often data should be inventoried and updated, and provide some examples of how this type of data could be used to analyze and illuminate educational pathways to the child welfare workforce.

How Should Agencies Collect Data?

1. Assess and audit their current data collection standards, including:
   • what types of employee information they already have stored
   • how that information is collected
   • how often that information is updated
   
   This will help agencies figure out what is needed to fill the gaps. Most agencies collect some background information from employees at hire, but it may be important for agencies to consider adding additional questions to HR documents given at hire to show a more comprehensive scope of individuals’ educational and experiential backgrounds.
2. Collaborate with their pre-service education partners to create parallel standards for how comprehensive data are collected about prospective and current students and alumni, who apply, intern, and then work for the agency during their payback period. Aligning the collection and measurement of the prospective and current workforce is crucial for ensuring that child welfare agencies can have a more complete understanding of the entire employee lifecycle.
3. Streamline the data collection process through the use of AI technology. AI programs are often used by organizations to pull information from applicants’ resumes to auto-populate online employment applications, and this same process could be used to pull information from newly hired employees’ resumes and applications to auto-populate the education and experience survey or information management system. During the onboarding process, employees could then review the information that was pulled for accuracy, make any necessary changes, and fill in the remaining items that were not covered in their resume or application, allowing them to make minor updates rather than having to complete the information in its entirety. This same process could also be utilized for current employees that were hired prior to the use of these new data collection methods.
4. Consider having prospective (e.g., interns) and current employees update their information at least yearly, so the agency can be aware of prospective and current employees’ up-to-date experience, education, training, or other credentials. This could also be done on an ad hoc basis with employees recording new training and qualifications as they acquire them, or could be tied into ongoing, regularly occurring agency processes, such yearly performance reviews.
5. Use information management software to report and update the data, which should include individual profile sections that can be easily updated by employees or students. Investing in an information management system is highly recommended to help streamline and automate the process of collecting and storing large amounts of employee information. If an agency does not have an information management system or is unable to collect all the information they need through such a system, they may then have to resort to using survey methodology to keep tabs on the credentials of their current and future workforce and to supplement their current data collection methods.
6. Make sure that employees are aware of how their data will be used, stored, and kept confidential. To achieve buy-in and encourage participation in this process, agencies should clearly communicate the benefits of data collection to their employees and consider providing incentives for timely and accurate data submission. It may be helpful for agencies to decide on a specified time of year when employees are tasked with updating their data. During this time period, frequent reminders could be sent out about updating information until employees complete this task. Agencies could also consider carving out protected time during a regular meeting to be used for the purposes of updating information.
7. Regularly assess the effectiveness of their data collection and management systems and adjust based on feedback and the changing information needs of the agency.

Measurement Guide

Measurement will go as smoothly as possible if the agency uses terms that employees and candidates would know. For example, we recommend documenting information on employees’ work roles within the agency and list some common roles that individuals in child welfare have (e.g., intake, investigation, foster care, adoption). However, agencies should make sure to tailor this language to the roles that are common in their specific agency to make measuring them as clear as possible.

Variable Guide

In each of the sections below, we outline variables, suggestions for measuring them, and recommended timepoints for data to be collected.

Lived Experience

Learning from individuals with lived experience with the child welfare system is increasingly important for informing child welfare systems’ policies, practices, and programs. Individuals with lived experience have experience that goes beyond what can be taught and learned through formal education and training. Because of this, agencies may want to consider asking employees to self-identify whether they have lived experience in the system, if they feel comfortable doing so. This information could then be leveraged to better include these employees in decision-making and program improvement in the agency.

However, it is important to give people with lived experience the option to volunteer this information or not and to choose how involved they want to be, so as not to exploit these individuals. Their perspectives and expertise are incredibly valuable, but agencies should avoid the burden of extra work and fairly compensate these individuals for time spent in providing their input.

Core functions to look for when selecting information management software:

Incorporating all these capabilities will ensure that important data are able to be captured during the application process, among current employees at all stages in the employee lifecycle, and for specific information related to training, such as dates, scores received, and the type of training completed.

• Applicant Tracking
• Human Resource Operations
• Learning Management Functions

Data storage is likely to look different depending on the size of the agency and the resources they have available. For smaller agencies, the most appropriate data storage may consist of a series of Excel sheets that are manually updated. Larger agencies may want to consider investing in an information management system.

Educational Partners of Child Welfare Agencies

Educational partners of child welfare agencies should employ student information systems that capture many of the education and experience related variables recommended above during the pre-hire period.

Agencies with educational or training institution partners should develop aligned collection and sharing protocols and data storage solutions that provide easy matching of individuals’ records between pre-service and in-service agencies. For example, agencies should be able to match data collected and shared by educational partners with that of current employees so they can clearly and reliably document the path of their workforce from pre- to in-service.

Agencies should provide for individuals and the teams they are part of to have access to the data they need to conduct analyses, derive insights, and share with stakeholders across the agency. Accomplishing these goals while maintaining security and compliance requires a combination of clear governance, well-defined access protocols, and supportive technology. Here’s
how to achieve this balance:

1. Role-Based Access Control should be used to assign permissions based on users' roles within the agency. This approach ensures that individuals only have access to the data necessary for their specific tasks.
2. Centralized Data Access Management system (e.g., single sign-on access, data request process, automated access records) should be implemented to streamline and monitor data access.
3. Data Catalog and Documentation (e.g., metadata, self-service options, and training) that describes the datasets available within the agency and their uses. This helps teams understand what data exists and how to access it.
4. Interdepartmental Data Sharing Agreements and protocols (e.g., guidelines, practices, use cases, and oversight) should be established for sharing data across departments and teams. The siloing of data within departments or teams is a common roadblock for any large organization.
5. Data Security and Encryption measures (e.g., secure file transfer protocols) should
   protect sensitive and confidential data through encryption and secure transfer
   protocols.
6. Review Access Regularly to ensure that current permissions are still appropriate
   and that those in new positions (e.g., hired, promoted, or transferred) have gained
   access to the data they need.
7. Data Access Training and Awareness should be provided regularly to ensure that all
   individuals understand the data policies and best practices for data access.
8. User-friendly Data Tools and Platforms (e.g., tools for surveys, learning management, and
   data visualization) should support access that is intuitive and user-friendly, enabling
   individuals and teams to efficiently locate and use the data they need.

By following these best practices, agencies can ensure that individuals and teams within them have the right level of access to the data they need while protecting sensitive information and maintaining security. Sharing with partners external to an agency (e.g., educational institutions) should also follow many of these practices to the extent they are relevant to how external partners share data with a child welfare agency or to the extent external partners can or need to conduct analysis of educational pathways on their own.

1. Simple Analysis

Once agencies have data collected, analyses can be conducted in a number of different ways, from looking at simple counts and percentages to performing more complex inferential statistical analyses. At its most basic, data can be used to determine counts of the types of individuals in the workforce.

These counts can then be turned into percentages in order to identify what proportion of the workforce falls into a certain category. Agencies can use these counts and percentages to look at the makeup of the workforce and at patterns in employee background information over time to see how the makeup of the workforce is changing.

Examples

Simple Education Data (counts):

• The number of individuals with each type of bachelor’s degree to assess the makeup of their workforce
• How many employees are coming from each local educational institution in order to determine the result of their recruitment and training efforts and adjust recruitment plans as needed

Simple Education Data (percentages):

• The number of individuals with each type of bachelor’s degree to assess the makeup of their workforce
• How many employees are coming from each local educational institution in order to determine the result of their recruitment and training efforts and adjust recruitment plans as needed

2. Complex Analysis

More complex analyses could also be conducted. For example, a chi-square test is a form of statistical analysis commonly used when examining the relationship between two categorical variables.

Examples of categorical dependent variables that could be tested with this method include:

• Whether an individual is hired/not hired,
• Promoted/not promoted
• Left the agency/remains at the agency.

Data can also be analyzed using t-tests, a form of statistical analysis commonly used to examine mean differences between two groups.

Examples

Chi-Square Tests:

• Whether individuals from a certain university are more likely to be hired than individuals from another university
• Whether participation in mentorship relates to being promoted
• Whether employees who did an internship at the agency before being hired full-time are more likely to stay

T-Test:

• Whether those who are promoted tend to complete more voluntary training sessions than those who are not promoted
• Whether those who are hired tend to have a greater number of years of previous experience in child welfare than those who are not hired

Data can also be analyzed using t-tests, a form of statistical analysis commonly used to examine mean differences between two groups.

How Data Could Be Interpreted and Used to Draw Conclusions

Examples of Data Use

Interpretation of data will necessarily depend upon the variables examined and the analyses conducted.

Using Title IV-E and Retention Data:

Agencies could analyze what percentage of individuals hired from a Title IV-E stipend program still remain at the agency after two years. If an agency analyzes this data and observes that those who come from Title IV-E programs tend to leave within the first couple of years, the agency may want to consider directing their recruitment efforts elsewhere or strategizing about how to improve retention. For example, the agency might want to consider what other college majors might be relevant to the field of child welfare and recruit from that talent pool instead, seeing if
that results in better performance and retention.

Using Training Data:

An analysis involving a simple count or average of the number of training hours that employees have taken over the past year might be used to determine whether more training is needed. If an agency sees that the average number of hours employees spent in training is extremely low, this finding could be used to suggest that an increased focus on training is needed.

Examples of Data Interpretation

When interpreting results from inferential statistics, like chi-square tests and t-tests, agencies should critically examine findings and unexpected patterns.

Analyzing Previous Education Data with Chi-Square Analysis:

For instance, if a chi-square analysis shows that individuals from University X are more likely to be hired than individuals from University Y, an agency might want to consider why this finding would have occurred. It may be that University X has a better program and tends to turn out more prepared employees than University Y, in which case the agency will likely want to continue investing effort in recruiting from University X.

In another scenario, it is possible that these results could have occurred not because individuals from University X are better qualified, but simply because the agency invested more recruiting effort into University X, which resulted in more hires. In this case, the agency might want to consider increasing their recruitment from University Y and building up a talent pipeline from that school. Using analyses to understand patterns like these can help agencies improve their recruiting and hiring practices and ensure fair opportunities for all candidates.

Interpreting Data

When examining real-world data like the preceding examples, results are often up for interpretation. Because these data are not collected in a controlled environment, it is possible that the same findings would have occurred for several different reasons. Additionally, once an agency arrives at possible reasons for the results, there may be numerous paths they could go down and decisions to be made about how they can use the results in an actionable way. Because of the complexities involved, it may be helpful for agencies to gather a diverse group of stakeholders who can examine findings, discuss possible reasons behind them, and determine how to use the findings to move the agency forward. Using a diverse group will allow a variety of perspectives to be heard, helping agencies look at the findings from a greater number of angles, contributing to more thorough interpretation of results and greater ideation about how results can be applied to improve functioning at the agency.

Sharing data-derived insights effectively within an organization, and with external partners, is crucial for informed decision-making, fostering collaboration, and driving change. By combining these strategies, agencies can foster a more data-driven culture, ensuring that insights are shared effectively within the organization and with external partners, empowering everyone to make informed decisions and forge change.

1. Tailoring insights to key audiences such as leaders, managers, supervisors, or external partners.
2. Focusing on actionable insights by sharing numbers while explaining what they mean and how they impact the organization.
3. Ensuring data accuracy and providing transparency before sharing is crucial for maintaining credibility and buy-in from key stakeholders.
4. Making data accessible through diverse communication strategies such as:
   • data visualization and dashboards (examples on the next slide);
   • presentations (e.g., storytelling);
   • collaboration tools (e.g., cloud-based data sharing tools, and communication tools like Slack or Teams);
   • newsletters and social media posts (e.g., LinkedIn);
   • data portals (e.g., analysis and visualization platforms like R, Knime, Tableau, or Power BI);
   • workshops and trainings (e.g., regarding data literacy, analysis, reporting, and security);
   • infographics (e.g., Canva, Visme, Stencil, or Adobe);
   • small in-person meetings;
   • and data documentation and knowledge bases (e.g., wikis).‍
5. Encourage data-driven decision-making through supporting an organizational culture that empowers individuals and teams to use data as a natural and low-friction part of their daily work.