November 2024
Natural Thinkers process and outcome evaluation: final report
November 2024
Natural Thinkers process and outcome evaluation: final report
1.0 Background
In 2015, the National Lottery Community Fund began supporting a programme known as A Better Start. It was a national ten-year programme. A Better Start was designed to enhance the support and services available to families and children living in five partnership areas across England.
Partnerships have been developing and testing ways to improve children’s diet and nutrition, social and emotional development, and speech, language and communication. The overarching aim has been to ensure children in these areas have the best start in life.
The partnership funded in South London, Lambeth Early Action Partnership (LEAP), has been developed as a collective impact initiative. LEAP has brought together services and activities to work towards shared goals to improve outcomes for very young children.
This report looks specifically at the work LEAP been doing to enhance communication, language and literacy development.
Evidence demonstrates the importance of early language development and communication. Children with strong communication skills are better able to interact with others, make friends and have healthy interpersonal relationships. Conversely, children with ongoing language difficulties are at risk of poor academic performance1.
Without effective help, a third of children with speech, language and communication difficulties will need treatment for mental health issues later in life2. Children with poor vocabulary at age five are twice as likely to experience periods of unemployment as adults3.
The Tickell Review into the impact of the early years foundation stage (EYFS) on children’s learning and development emphasised that children’s learning is a result of interacting with people and their environment from birth4. More specifically, recent research has suggested that interacting with the natural environment can be particularly beneficial for language development.
Nature-based playgrounds can provide a richer conversational setting for language use than the non-nature-based playgrounds5. When children play in natural environments, their range of potential experiences increases. This can encourage the kind of imaginative and creative play that has been shown to foster language skills6.
Lambeth Council developed Natural Thinkers in 2012 for early years settings and schools. Positive early results led to it being included in the Lambeth Early Action Partnership’s (LEAP) communication and language development (CLD) strand of work. Natural Thinkers aims to give children the opportunity to explore and be social in an environment they may find more calming.
This can promote positive social interaction, which helps children develop the communication skills that enable them to better express themselves.
The programme works with childcare settings and children’s centres. It aims to improve their outdoor environments through, training, support visits, development plans, network meetings and setting accreditation. Settings start by attending a 1-day training session, where practitioners learn about the Natural Thinkers’ framework of 10 commitments. At the end of the first session practitioners should be able to:
- Identify the benefits of nature to children’s wellbeing
- Give examples of how they might demonstrate the 10 Natural Thinkers’ commitments
- Give examples of potential hazards and suggest actions to keep risk low
- Plan for activities using natural resources outside
- Plan activities to the seasons
Trained practitioners support staff in their setting to identify what they need to work on to be able to meet the 10 commitments. Two further half-day sessions focus on practical activities designed to continue developing each commitment. Natural Thinkers trainers visit settings between training sessions to provide ongoing support.
The LEAP programme has given each setting £500 funding to support their development plans. Trainers have worked with settings to ensure that it was spent on the right priorities to support the programme. When a setting feels ready, they can request an accreditation visit to see if they meet all 10 commitments. Successful settings receive Natural Thinker setting status.
LEAP has rolled out the programme in Lambeth to complement the work its speech and language team do to provide universal and targeted support for children’s communication development. Natural Thinkers was also delivered as weekly Stay and Play sessions in children’s centres.
As the LEAP programme draws to a planned close, the National Children’s Bureau is exploring opportunities for future scaling-up of Natural Thinkers. As part of that process, this short report reviews available evidence on the delivery and impact of the intervention. It describes what has been delivered; how well services have been received by early years practitioners and parents; and what evidence exists to support the case that Natural Thinkers has had a positive impact on the development of the communication and language skills in pre-school children.
2.0 An Evaluation Framework
2.1 Theory of change models for Natural Thinkers
A theory of change (ToC) describes how and why a programme or intervention is expected to bring about change. Building a ToC, or logic model, is a widely recommended tool for evaluating the delivery and impact of programmes and interventions7, and a good ToC provides a very useful framework for evaluation. It describes what needs to be measured to establish how an intervention has been delivered (process), and the impact it has had on service users (outcomes).
We began developing the evaluation framework for Natural Thinkers using three different ToC models. The first two are described in a report from New Philanthropy Capital (NPC). The third was produced in partnership with service providers and practitioners by the RTK Ltd.
The first NPC model (see Figure 1 below) describes how changes in the way that Natural Thinkers children experience outdoor environments led them to learn new skills that support improvements in communication and language.
The second describes how training for settings, practitioners and parents enhances the way they use outdoor spaces to support children’s learning (see Figure 2).
The third model was created to identify the best measures to evaluate the delivery and impact of the Natural Thinkers programme (Figure 3).
In the next section, we describe how we took elements from these three models to create an evaluation framework.
2.2 Implications for an evaluation framework
An evaluation framework pulls together the ideas, conditions, and assumptions that determine how a programme is delivered and how its impact is assessed8.
The brief was to review existing Natural Thinkers documentation to address questions of the programme’s feasibility and effectiveness:
- Was the programme implemented as intended?
- How did practitioners and parents respond to the programme?
- Did the programme contribute to intended outcomes for practitioners, parents and children in the short, medium and long term?
The three ToC models provided the means to address each of these questions. Our first job was to amalgamate key elements from them into a single theory of change (see Figure 4).
Lines one to five of the model focus on outputs and outcomes relevant to settings, practitioners and parents. The remaining lines pertain to child outcomes.
2.3 Explicit, testable hypotheses
The great value of a clear, explicit ToC lies in its capacity to provide a practical evaluation tool9. A well-crafted ToC can be used to specify a set of testable hypotheses about each step in the change process. Furthermore, it helps identify the data sources needed to assess programme achievements against service delivery objectives, and to establish whether programme participants changed in ways hypothesised.
We used the newly created single ToC to create a set of hypotheses about Natural Thinkers that could be tested empirically through the analysis of data collected by LEAP.
The amalgamated ToC and accompanying hypotheses are set out in Figure 5.
3.0 Natural Thinkers grey literature and data availability
In this section we describe documentation relating to the delivery and evaluation of the Natural Thinkers programme.
Documentation covers three broad areas: supporting documentation that describes (i) an inventory of the data collected by providers and practitioners; (ii) process evaluations (such as details of how the programme has been delivered); and (iii) impact evaluations that attempt to answer questions about how the programme has influenced language and cognitive development of the children involved.
3.1 Natural Thinkers supporting documentation
Monitoring, Evaluation and Learning Framework
Sets out the data that services are asked to collect, along with guidance and tools to support the process.
It describes LEAP’s approach to using ToCs as a service delivery monitoring and evaluation tool. It also provides details of LEAP’s Shared Measurement System (SMS): the tools used to measure short, medium and long-term service outcomes. This comprehensive and robust approach to data collection and measurement promotes a more consistent approach to data collection across all LEAP services.
Its stated objective is to enable providers to use data as a means of demonstrating how services are working to improve outcomes for families. It details how LEAP analysts synthesise service-level data using their Data Integration Platform, and then share their analysis with practitioners in the form of quarterly service reports.
LEAP Quarterly Narrative Reports for the Natural Thinkers Service
Provide context to data provided by practitioners. They are made up of two sections:
Section 1: Asks practitioners to reflect on activities over the quarter, providing comments on user demographics, levels of engagement with the programme, feedback and outcome data.
Section 2: Asks practitioners for comments on delivery opportunities or successes; issues and risks (barriers to achieving the outputs specified in the programme ToC); and corrective action (planned actions designed to address the impact of potential issues and risks on delivery).
3.2 Data: service delivery and associated outcomes
Summary of quarterly service reports, January 2018 – July 2022
Examines delivery performance, key characteristics of families engaged, and the impact of the service on child outcomes – using quarterly data and the service ToC.
Practitioner feedback and additional data
Includes a practitioner training questionnaire template and a feedback form.
Family Questionnaire
Covers family feedback on Natural Thinkers ‘Stay and Play’ sessions.
3.3 Data: links between programme elements and child outcomes
Pre- and post- Leuven scale data and other LEAP measures
Natural Thinkers used the Leuven Involvement Scale for Young Children (LIS-YS)10 as a key outcome measure of how the programme had an impact. The scale assesses children on two dimensions: wellbeing and involvement.
Research has shown wellbeing in children to be indicative of how they are developing emotionally. A state of wellbeing results in a fair amount of self-confidence and self-esteem11.
Involvement is associated with educational progress. Children assessed as involved in activities are judged to be intrinsically motivated to carry on. These children experience satisfaction, enjoy themselves and participate enthusiastically12.
Initial evaluation report (2017, Lambeth Public Health Team and Vaga Associates)
Provides an evaluation of the Lambeth Food Flagship Borough Programme (2014-2016) of which the Natural Thinkers project was a component. Description of individual projects is limited to an overview to set the context for what was assessed. Data is predominantly made up of stakeholder perspectives.
The report concluded practitioners had actively engaged with the Natural Thinkers programme. It was positive about the impact it had on both children and early years settings. The report recommended, based on its findings, that investment in Natural Thinkers should continue. More specifically, it suggested that investment in the business-development dimensions of how Natural Thinkers had the potential to become a sustainable social enterprise.
Communication and Language Development (CLD) Services Evaluation (2024, National Children’s Bureau)
Summarises the process-and-impact evaluation of LEAP’s communication and language development services, of which the Natural Thinkers programme is a complement part.
The evaluation set out to improve our understanding of how LEAP’s bundle of services might make a difference to a workforce under pressure. This bundle was designed to drive improvements in child outcomes across pre-school settings. The focus of the evaluation was very much on service delivery and practitioner working rather than on child outcomes.
The evaluation report found that:
- LEAP services were implemented with moderate success.
- Natural Thinkers delivered the amount of training intended.
- Some settings struggled to engage with training because of:
– Capacity issues. Settings operating “at ratio but only just” sometimes found it difficult to release staff for training.
– High staff turnover. It was challenging to ever reach a position where the whole setting was trained.
– Covid-19. Poor Ofsted ratings and other unexpected events that led some to pause or stop their engagement with LEAP. - Where settings did experience difficulties, strategies that helped included:
– taking a flexible approach to training that gave providers more latitude around timing, format and location of training
– encouraging managers to send new staff on training on arrival, arrange cover whenever possible, and weave training attendance into formal appraisal processes - Long-term, in-context and personalised support was identified as a key mechanism of impact. Practitioners rated training, support and project resources positively.
- Providers thought training and support improved the promotion of CLD, identifying speech, language and communication needs in children, and delivering targeted support.
- Child outcomes showed signs of improvement over time. A measure used by Natural Thinkers to identify changes in children’s emotional well-being and involvement, the Leuven Scale13, provided evidence of positive change in both.
4.0 Narrative synthesis
In this section we indicate how the available evidence connects to test the hypotheses developed from the programme ToC.
This report did not set out to conduct rigorous process or impact evaluations. Its purpose is to highlight where available data sources could be used to test the hypotheses set out in the evaluation framework described in Figure 5.
The remainder of the section considers what the information available to the review tells us about the Natural Thinkers programme: how it has worked, the mechanisms by which it purports to make an impact on communication and language development, and for which children it has produced the greatest changes. Thirdly, it highlights where gaps in current data make it difficult to test hypotheses, and finally discuss lessons that can be drawn relevant to any future roll-out of the programme.
4.1 Indicative synthesis of available findings
The list of ten hypotheses derived from the programme ToC (see Figure 5) provides the framework within which we can offer some indicative findings.
1. Delivery
Indicative findings:
- QSR reports show around 700 children in LEAP settings reached by Natural Thinkers
- Data for 2020/2021 show 100% of parents rating Natural Thinkers sessions as ‘good’ or ‘very good’
- Partial data for 2020/2021 show 85% of parents reported having increased their knowledge and understanding about the benefits of learning outdoors
2. Training
Indicative findings:
- 118 practitioners provided feedback on training sessions
- 100% rated training positively
- 84% felt highly or very confident in planning and organising Natural Thinkers activities
3. Engagement
Indicative findings:
- 260 parents attended Natural Thinkers Stay and Play groups in the period 2020/2022
- 140 parents attended 4 sessions offered at Natural Thinkers stay and play sessions
- 92.5% of parents report having learnt something new because of attending the service (agree/strongly agree)
4. Stakeholders
Indicative findings14:
- 260 LEAP children under 4yrs engaged in Natural Thinkers activities at children’s centres Stay and Play 2020/2021
- 72% to 92% of practitioners reported positive experiences of CLD programmes
- Settings struggled to engage parents – around one-third of practitioners felt they improved support they offered to parents around their child’s outdoor learning and play
5. Support
Indicative findings:
- Due to settings facing continued staffing issues, attendance has been inconsistent
- 100% of practitioners attending rated training positively (strongly agree/agree)
- 96% of practitioners would recommend Natural Thinkers to other settings or to a friend or colleague
6. Home environment
Indicative findings:
- Narrative Report cited ‘majority’ of parents felt they learnt something from NT sessions
- Practitioners positive about the impact that improvements to their knowledge, confidence and practice had had on children and families
- Home Learning Environments improved for 0-3 years olds (though this was not statistically significant) and scores for 3–5-year-olds increased significantly15
7. Communication
Indicative findings:
- Parents commented on positive aspects of allowing children to explore freely with minimal interaction
- Practitioner observations that Natural Thinkers helped children to “find their voice (and) build their confidence with language”
- For children with delayed language development, their WellComm score for their age increased by an average of 1.5 points between their first and most recent assessment
8. Reasoning
Indicative findings:
- 100% of parents strongly agree/agree that overall, the experience was positive and that their child/children enjoy the session
- Overall, average involvement scores improve by 0.62 points from children’s first to last assessment
- WellComm scores, a measure of speech, language and communication (SLC) attainment, increased by an average of 0.8 points between children’s first and most recent assessment
9. Relationships
Indicative findings:
- Parental feedback highlighted benefits of allowing children to explore freely with minimal interaction
- Practitioners noted huge benefit in Natural Thinkers, identifying that outdoor activities had helped children to become “more outgoing and speak more”
10. Long-term impact
Indicative findings:
- Leuven wellbeing scores by an average of 0.55 points between children’s initial and final Natural Thinkers assessment
- Leuven involvement scores increased by an average of 0.62 points over the same period
- Children from non-English speaking households had higher well-being scores at the beginning and end of the programme compared to children who came from English-speaking households
4.2 How the intervention works, why and for whom? Indicative findings
The indicative evidence suggests the Natural Thinkers programme works largely as hypothesised in the evaluation framework described in Figure 6 of this report. Table 1 below provides a summary.
The programme provides practitioners with training, support and resources. Natural Thinkers does not work directly with parents. Practitioners then run sessions with parents and children in their early years settings or children’s centres. Feedback suggests parents rate these sessions very positively. They learn new skills and gain confidence in supporting their children interact with elements of their outdoor environment.
Natural Thinkers training can be successfully delivered in early years settings. The training appears both feasible and well-reviewed by practitioners, who feel highly confident after having attended a session to implement this programme in their settings.
Practitioners were uniformly positive about the impact that improvements to their knowledge, confidence and practice had had on children and families. Practitioners would also recommend the programme to their personal and professional networks. Practitioners observed that Natural Thinkers helped children to “find their voice (and) build their confidence with language” and helped children to become “more outgoing and speak more”.
Practitioners reported that, in early years settings, it was hard to engage with parents. However, one third did report that they were able to provide improved support to parents around their child’s outdoor learning and play.
Evidence regarding changes in parental and child behaviour comes largely from Informal observations by practitioners. Their feedback suggests parents respond positively to practitioner support, and quickly learn how to support their children’s learning.
This, in turn, leads to observed changes in the way children explore their outdoor experiences in ways that encourage them to interact with other children. Observed changes in language development and wellbeing are consistent with empirical data. Scores on measures including the WellComm measure and Leuven scales suggest children who have participated in the Natural Thinkers programme improve their communication and language skills, exhibit elevated levels of wellbeing, and become more involved in activities, including interaction with their peers.
For children with delayed language development in early years settings, their WellComm score for their age increased by an average of 1.5 points between their first and most recent assessment. Whilst direct causality cannot be linked to the NT programme specifically, it can be considered a significant enabling factor.
The NCB’s CLD evaluation did make the exploration of child outcomes a priority (p. 63). However, it did report evidence that children from households speaking a non-English language had higher wellbeing scores at the beginning and end of the programme compared to children who came from English-speaking households.
Similarly, evidence from the Natural Thinkers quarterly-service-reports suggests children with special educational needs (SEN) benefitted significantly from participating in the Natural Thinkers programme. The proportion of children with high wellbeing scores rose from 33% to 66% from initial to final assessments16.
The report found no differences in involvement between children from English and non-English-speaking households at the start of their engagement. Over the course of the Natural Thinkers programme, both groups improved their involvement scores. Improvement appeared to be better for children from non-English-speaking households. On second assessment, they had significantly higher scores than English-speaking children.
4.3 Gaps in current data
The data reviewed provided a very clear and comprehensive picture of service delivery. The Natural Thinkers delivery team collected comprehensive and robust data on a regular basis. Moreover, they used that data to monitor key aspects of service delivery in a way that enabled them to identify potential risks and issues as they arose.
Based on the data available, it would be reasonable to conclude that the use of data by the Natural Thinkers team to support and optimise service delivery has been exemplary.
Data for testing links between specific programme elements and child outcomes has relied on established and validated child outcome measures, including the WellComm measures and Leuven scales. These measures have good reliability and validity. That makes them appropriate tools for the job they have been used for.
Additional data could help uncover more information about underlying mechanisms by which observed change has been brought about. Specifically, where practicable, it may be worth considering whether structured observation of parent-child interactions could help develop understanding.
4.4 Lessons for future roll out of the Natural Thinkers programme
Regarding specific lessons for future roll out, this review suggests three:
- The value of taking time to develop theories of change for the programme is self-evident. They have served to identify data collection procedures that have played a key part in ensuring service delivery and risk management has been optimised. They have also enabled development of testable hypotheses that will enhance any future evaluations of programme roll-out.
- Relationships with early years settings are critical. These need to take account of the special pressures that the sector currently faces. Staffing shortages combined with potential growth in demand for places (driven by changes in government policy) can make it difficult for settings to find the time and resources to engage with new initiatives. This is true even for those interventions with demonstrable efficacy such as Natural Thinkers.
- Future funders of Natural Thinkers programmes can be reassured that, on current evidence, delivery can and will be optimised. At the same time, funders and practitioners need to accept there will always be limitations on the capacity to demonstrate specific causal relationships between one intervention and an outcome such as communication and language development. Child development happens in the context of a complex system of multiple influences. It may be that the best we can do when it comes to evidencing impact on child outcomes is to show how a programme like Natural Thinkers can justifiably be thought of as one significant contributor amongst others.
5.0 Appendix
| File name | Content Description | |
|---|---|---|
| 1 | NCB Natural Thinkers – Theory of Change 230331(1) | Natural Thinkers Theory of Change produced by the NCB |
| 2 | NT Practitioners Feedback Graph | Natural Thinkers – LEAP Practitioners’ Training Feedback Data |
| 3 | NT Practitioners Questionnaire Graph | Natural Thinkers Practitioner Questionnaire Data evaluating the NT programme |
| 4 | NT S&P Families Feedback Graph | Natural Thinkers Stay and Play LEAP Families Feedback Data |
| 5 | 201920 Natural Thinkers Service Plan | Service Plan for Financial Year 2019/2020 |
| 6 | 202021 Outputs Table for Natural Thinkers Delivery | Key Outputs/Activities for Natural Thinkers for FY 2020/21 |
| 7 | A.2 Natural Thinkers MEL | Monitoring, Evaluation and Learning (MEL) Framework setting out the data LEAP requires services to collect and submit. |
| 8 | A.2 Natural Thinkers ToC | Natural Thinkers Theory of Change |
| 9 | CLD Natural Thinkers TOC narrative 30.09.2020 | Accompanying narrative for Natural Thinkers Theory of Change |
| 10 | NAS User to end 2324Q3 | Anonymised biographical data on 2845 NT service users that registered between 2020 and 2023 |
| 11 | NAT User Data ALL YEARS | Anonymised biographical data on 717 NT service users that registered between 2016 and 2023 |
| 12 | Outcomes data | Natural Thinkers outcomes data from early years settings used family questionnaire and Leuven scale scores for 3286 children |
| 13 | QSR Natural Thinkers ALL YEARS | Natural Thinkers Quarterly Service Report 2018/19 Q4 – 2022/23 Q1, January 2018 – July 2022 |
| 14 | QSR Natural Thinkers Stay and Play ALL YEARS | Natural Thinkers Stay and Play Quarterly Service Report 2021/22 Q1 – 2022/23 Q3 April 2021 – January 2023 |
| 15 | 2.NT – Well-being and involvement summary sheet NEW 24.06.2019 | Summary sheet template for collection of Leuven Scale scores |
| 16 | A.2 Family Feedback Form Natural Thinkers | LEAP Family Feedback Form Template – Natural Thinkers Stay and Play |
| 17 | A.2 LEAP Practitioners Training Feedback Form Natural Thinkers | LEAP Practitioners’ Training Feedback Form Template – Natural Thinkers |
| 18 | A.2 Natural Thinkers – Practitioners Questionnaire | Natural Thinkers – Practitioner Post Training Questionnaire |
| 19 | A.2 Natural Thinkers Family Questionnaire | Natural Thinkers – Family Feedback on the NT programme Questionnaire |
| 20 | Leuven Scale | Description of Leuven Scales, the process-orientated child monitoring system for young children that LEAP has used as a child outcome measure. |
| 21 | Quarterly Narrative Reports | LEAP Quarterly Project Performance Monitoring Forms (Qualitative Reports) 2017 – 2023 |