How a collective impact initiative contributed to outcomes of young children in Lambeth

Programme-wide analysis of Lambeth Early Action Partnership using linked population datasets (Gemma Luck and Dakota Langhals, August 2024)

1.0 Key messages

* This finding is an outlier when triangulated with all our other research on the programme.¹ It is inconsistent with our findings for 2.5-year-old children.

While the data shows no significant differences between the reception-age groups, we are unable to determine the impact that LEAP had within the reception-age children who had engaged with its services. For example, it is likely that many LEAP-engaged families had greater needs than their peers, so a child who reached the end of reception may not show significant differences in their outcomes to those who did not engage with LEAP, but they may have improved upon their initial baseline level.

Data that would enable a before-and-after comparison is not available for population-level datasets. It is therefore impossible to say if we have improved outcomes in the group of children engaged with LEAP services, particularly SED and targeted services.

The assessment approaches for 2.5-year-olds and reception-age children are substantially different. We are therefore unable to use one as a baseline for the other. This prevents us from reaching a more meaningful understanding of the outcomes for 5-year-olds who engaged with LEAP.

Furthermore, we must consider the gap between LEAP provision, which ended before a child turned four, and the outcomes measured at the end of reception. Families in greater need of provisions that LEAP provided would have had persisting needs beyond LEAP, which will have likely been exacerbated by external factors such as Covid-19 and the cost-of-living crisis.

Pointedly, our wider research reveals that outcomes for parents engaged with LEAP services have improved.² The areas of improvement include parental mental health and wellbeing, and parental knowledge, skills and behaviour.

** These findings suggest that higher levels of need at the family level may have persisted beyond LEAP provision, which ended before a child turned four, despite engagement with services.

2.0 Background

Lambeth Early Action Partnership (LEAP) has delivered services in a focused area of Lambeth, South London. It is one of five local partnerships which has made up A Better Start – a national 10-year (2015-2025) programme funded by the National Lottery Community Fund. LEAP’s aim has been to improve the life chances of babies, very young children, and families.

We have funded and improved more than 20 local services to meet the needs of families from pregnancy all the way through the early years of childhood. We have given thousands of children in the LEAP area aged 0-3 a better start in life.

LEAP is a ‘collective impact initiative’, which means that all our services and activities have linked together and worked towards shared goals to improve outcomes for very young children. These goals were outlined in LEAP’s theory of change (ToC).³

Our ToC prioritised the changes we wanted to achieve with children directly. Evidence suggests that the areas of children’s development which LEAP focused on can have a significant impact on their long-term life chances, and outcomes, and are crucial to reducing health inequalities. Two of these areas are communication and language (CLD) and social and emotional development (SED).

LEAP has operated in an inner-London community where 68% of children live in ‘very deprived’ neighbourhoods (the 20% of the borough with the greatest levels of deprivation affecting children). We’ve delivered services in parts of Lambeth where young children experience greater inequalities than children in the rest of the borough.

LEAP is nearing the end of its 10-year funding. It’s an important time to complete analysis exploring the relationships between family engagement with our services and outcomes in early childhood.

This work is preceded by our second Annual Learning Report (ALR),⁴ which explores the population represented in the health visiting dataset who have engaged with LEAP services. There was a particular focus on LEAP’s priority populations of families who live in areas of greater deprivation, and those from ethnic minority groups. The ALR found that during the lifetime of the programme, LEAP has engaged an estimated two thirds of children under 5-years-old living in the LEAP areas. This engagement was found to represent the demography of the local population well.

Until recently, LEAP has not had access to many administrative datasets in a linkable format that enable analysis of child outcomes in the early years outlined in our ToC. The current research uses well-established administrative datasets to understand how key LEAP child outcomes may differ in comparison with non-service participants.

3.0 The outcomes

3.1 Ages and Stages – 3 questionnaire (ASQ-3)

The ASQ-3 is a developmental screening tool. It age-appropriately assesses child development in children between the ages of one month to five-and-a-half years. It is designed for use by early educators and health care professionals and captures parents’ knowledge of their children.

The tool screens five areas of development: communication, gross motor, fine motor, problem solving, and personal-social. Domain level scores also can be combined to provide a picture of overall development.

In Lambeth, health visitors administer the ASQ-3 as part of each 2.5-year-old’s health review.

The current research focuses on the communication and personal-social areas of the ASQ-3, as well as overall development across all five areas as detailed below.

3.2 Early Years Foundation Stage Profile (EYFSP)

The EYFSP is a statutory assessment of child development for children aged four and five years old. It is conducted by teachers at the end of the reception year.

Children are assessed against 17 Early Learning Goals (ELG) across seven Areas of Learning (AOL). The assessment identifies whether children have reached at least expected levels of development in each ELG.​

Children have a Good Level of Development (GLD) if they have at least the expected level of development in all ELGs in the Communication and Language (CLD) AOL, Personal, Social, & Emotional Development (PSED) AOL, Physical Development AOL, Mathematics AOL, and Literacy AOL.

The current research focuses on the CLD and PSED areas of learning from the EYFSP, as well as overall development using the GLD measure.

3.3 National Child Measurement Programme (NCMP)

The National Child Measurement Programme (NCMP) measures the height and weight of children in reception (aged four and five) and year 6 (aged 10 and 11), to assess overweight and obesity levels in children within primary schools. Data is collected and processed by local authorities.

The NCMP is a consented programme. Before the programme starts each school year, local authorities write to the parents and carers of all children eligible for measurement to inform them of the programme. Parents or carers can choose to withdraw their child from the process by responding to the letter.

Measured children are classified as underweight, healthy weight, overweight, or very overweight relative to a reference cohort of children for their age and sex.​ While a clinical classification is also available, we are interested in the relative classification as it also flags children who are more at risk of worse weight outcomes later in life.​

The current research focuses on measurements taken for reception class children.

4.0 Analytical approach

A paper outlining the data linkage, methodology and analytical approach is available to download on LEAP’s website.⁵

4.1 Child development at 2.5 years – ASQ-3

The Lambeth health visiting service is offered to all Lambeth-resident families with children aged 0-5. Information about each assessment or visit offered to families is recorded on Carenotes, the Lambeth health visiting service’s administrative system at Guy’s and St Thomas’ Trust (GSTT).

A dataset, meeting LEAP’s requirements, was extracted from Carenotes, pseudonymised and shared with LEAP in September 2023. The dataset includes data for all ASQ-3 assessments for children at the 2.5-year health review.

4.2 Child development at 4-5 years – Early Years Foundation Stage Profile

Information about all EYFSP assessments carried out in Lambeth state-funded primary schools is collected and processed by Lambeth Council via internal administrative systems. A dataset, meeting LEAP’s requirements for the EYFSP, was extracted, pseudonymised and shared with LEAP in February 2024 by Lambeth Council.

4.3 Child weight at 4-5 years – National Child Measurement Programme

In Lambeth, the NCMP is commissioned by Lambeth Council. Information about all NCMP measurements carried out in Lambeth state-funded schools for Lambeth resident children is collected and processed by GSTT and then securely transferred to Lambeth. A dataset, meeting LEAP’s requirements for the NCMP, was pseudonymised and shared with LEAP in March 2024 by Lambeth Council.

4.4 Generalisable approach across all three projects

LEAP-commissioned services routinely collect data about LEAP service participants and how they engage with LEAP services. The services share the findings with LEAP’s data team at regular intervals (quarterly for most services, termly for communication and language development services).

Data shared with LEAP is pseudonymised, has personal identifiable information removed and is uploaded to LEAP’s data integration platform, ‘the platform’. While processed, the platform links participants across services, creating a unique ID for each individual. This enables identification of the same individual across each service. A full description of the platform can be found in appendix 4 of our first Annual Learning Report.⁴

A final linked dataset for analysis was extracted for each project. This included all relevant assessment outcomes for children who live in the LEAP area, their user data, and any engagement data as a result of either their direct engagement with LEAP services or a family member’s engagement. This engagement with LEAP services was restricted to any engagement that happened prior to an agreed cut-off date, and for an agreed list of relevant services.

5.0 Data and Results

5.1 Ages and Stages – 3 questionnaire (ASQ-3)

5.1.1 The data

The ASQ-3 analysis used a cross-sectional secondary analysis aiming to test whether ASQ-3 developmental assessment scores of children aged 2.5-years from the LEAP area differ between those whose families engaged with relevant LEAP services and those with no relevant LEAP engagement. This was analysed using an adjusted logistic regression model controlling for confounding factors – as is illustrated in Table 1 below*.

A dataset linking LEAP service data to Lambeth health visiting data via LEAP’s Data Integration Platform was used. The final linked dataset included 1,853 children aged ~2.5 years with an ASQ-3 assessment record between January 2019 – September 2023 and who lived in the LEAP area. A total of 668 children from the LEAP area had family engagement recorded with a relevant LEAP service.

ASQ-3 assessment scores in the communication and language and personal-social development domains, as well as overall, were converted to binary outcome categories for the logistic regression. A flag was created to identify engagement with LEAP services relevant to ASQ-3 outcomes at 2.5 years, at least three months prior to their ASQ-3 assessment.

* Confounding factors included in the final analysis were ethnicity, local child specific area deprivation (Income Deprivation Affecting Children Index (IDACI) 2019;⁶ re-scaled to give relative deprivation quintiles within Lambeth, rather than national quintiles), sex, and healthy child partnership (HCP) level of support.

5.1.2 Findings

Descriptive analysis

Demographics for children from the LEAP area:

* (where ethnicity was recorded)

Outcomes for children from the LEAP area (as demonstrated in Figure 1 below):

Statistical analysis

Primary analysis of the ASQ-3 developmental outcome categories found that:

A secondary analysis explored the relationship between the ASQ-3 developmental outcome categories, and the number of services families engaged with. It found that children who had family engagement recorded for two or more relevant services were most likely to reach expected levels of development overall compared with one service or no engagement (p<0.05).

5.2 Early Years Foundation Stage Profile

5.2.1 The data

A dataset linking LEAP service data to EYFSP assessment records via LEAP’s Data Integration Platform was used. The final linked dataset included 952 children with an EYFSP assessment record in 2022 or 2023 and who lived in the LEAP area. Of those children, 406 had family engagement recorded with a relevant LEAP service.

This dataset facilitated a cross-sectional secondary analysis of EYFSP outcomes. The relationship between family engagement with LEAP services and developmental outcomes was assessed using an adjusted-logistic-regression model controlling for confounding factors.*

* The confounding factors included in analysis were ethnicity, sex, local child specific area deprivation (IDACI; re-scaled to give relative deprivation quintiles within Lambeth, rather than national quintiles), year of assessment, eligibility for free school meals,⁷ and an indicator for English as an additional language.

5.2.2 Findings

Descriptive analysis

Demographics for children from the LEAP area:

Outcomes for children from the LEAP area:

Statistical analysis

Our model found that children with family engagement with any relevant LEAP services were as likely to reach an overall good level of development, expected levels of development in communication and language development, and expected levels of development in personal, social and emotional development at the end of reception compared to their non-engaged peers when adjusting for confounding factors. Output from the statistical model can be seen in table 2.

None of the outcome domains show statistically significant differences between engaged and non-engaged children when considering engagement with the full list of relevant LEAP services. However, statistically significant differences do emerge when considering engagement with specific subsets of LEAP services:

A secondary analysis was also conducted exploring whether engagement with multiple services differed substantially from engagement with just one service, or with zero services.

Results were somewhat inconsistent, possibly reflecting relatively low numbers of children whose families engaged with multiple services. However, our model did find evidence that positive associations with universal service engagement may be particularly concentrated among those using two or more services. Additionally, our model found evidence that poorer outcomes in communication and language development may be present for children engaging with two or more of any relevant services.

5.3 National Child Measurement Programme

The NCMP dataset had a smaller cohort of children. The final linked dataset for the NCMP analysis included only 147 children from the LEAP area with a relevant LEAP engagement record. We found that the adjusted model does not identify any factors (ethnicity, deprivation, gender, HCP status) as significantly impacting child weight, and the models themselves had poor predictive power for NCMP outcomes.  Because of this, we have not continued with the NCMP analysis.

6.0 Limitations

This work comes with limitations that must be considered.

The process of linking children comes with a margin of error. It is possible that not all children who have a parent who engaged with LEAP were linked to that parent. Therefore, some children might have been incorrectly classified as non-engaged.

The approach to linking service users to individuals in administrative datasets differed for EYFSP. This was due to a lack of identifiers common to LEAP’s typical approach. This resulted in a lower rate of children linked between our engagement data and the EYFSP dataset. Therefore, some children might have been incorrectly classified as non-engaged.

Data linkage limitations are underpinned by the lack of a nationally recognised single unique identifier for children. LEAP developed an approach to consistently identify individuals as part of the development of the platform, to enable data linkage. However, as we have identified in the limitations of this project, this presents some challenges.

We were unable to account for a family’s depth of engagement with LEAP beyond a measure of multiple-service use. Consequently, the engagement flags may capture families that were only transiently engaged with services, including those who wouldn’t be expected to have been greatly impacted in less programmatic analyses.

Causal inference is limited by the cross-sectional nature of this work. We were unable to measure the changes to a child’s outcome before and after engagement with LEAP compared to a randomised control group. Rather, we can speak of associations between service engagement and developmental outcomes.

The limitations of our approach are clearest when considering what engagement says about a family. On one hand, we may expect that engaged families would have better outcomes than very similar families that do not engage. On the other hand, engagement may, in some instances, reflect acute levels of need. This could potentially lower baseline expectations for developmental outcomes relative to families that do not engage.

Alternative statistical approaches that better identify baseline levels of need, and that more precisely identify the programme’s contribution to outcomes over time, would paint a more complete picture. However, data quality and collection are key barriers to implementing such approaches. Outcomes are only collected at one point in time. We have only been able to access child-level data for Lambeth. It makes more sophisticated designs, such as differences in differences, or those that employ synthetic controls, not possible.

We do not have a satisfactory way to account for possible distortions caused by the Covid-19 pandemic.

There are possible ceiling effects to consider with the ASQ analysis, with consistently very high percentages of children reaching expected development at population level.

7.0 Summary of findings and conclusion – what does this mean in the context of the LEAP programme?

Very young children from families who engaged with LEAP services were more likely to reach expected levels of development across key LEAP outcome domains

The 2.5-year-olds whose families engaged with relevant LEAP services were 40% more likely to reach expected levels of overall development, than children who had not engaged with LEAP. They were 70% more likely to reach expected levels of communication and language development and 50% more likely for personal, social and emotional development.

Further analysis found that the more services a child’s family engaged with, the greater the odds of reaching at least expected levels of development at 2.5 years-old across all three outcome measures.

The findings are promising. Even more so when you consider the background factors that were in play:

The same findings were not reflected for children at the end of reception

For children from the LEAP area reaching the end of reception, there were no statistically significant differences between children whose families engaged with relevant LEAP services and their peers who did not.

In contrast, when considering sole engagement with LEAP services that specialise in social and emotional development, or with LEAP’s targeted services, engaged children were less likely to reach expected levels of development. On the other hand, children with family engagement in universal services were more likely to reach expected levels of development in the overall measure.

The likelihood that engagement with LEAP has had no positive association with child outcomes at reception, or even a negative one, is slim given the results we are seeing across all evaluation of LEAP at both service and programme level.

There is therefore a need to consider several factors when seeking to understand why results for 5-year-olds are inconsistent with the results we see for 2.5-year-olds.

Data challenges

A possible inflation of engaged families, flagged in the not-engaged group, could have weakened the engaged group. This is because of the alternative linking approach with EYFSP due to a lack of consistent identifier. The data linkage limitations are underpinned by the lack of a nationally recognised single unique identifier for children.

The time-lag between potential engagement with fully operational LEAP services, and a two-year gap in schools data-collection due to the Covid-19 pandemic, resulted in a smaller cohort for EYFSP analysis.

Greater need for services and persistent socioeconomic adversity

There is evidence of selection affecting outcomes.

Supplementary analysis showing strong negative associations among Serious Emotional Development (SED) services and targeted services suggests these may attract families with more acute or obvious need. Conversely, universal services present the most positive results, which by design may attract families with less acute or obvious need. These findings suggest that higher levels of need at the family level may persist despite engagement with services.

LEAP has worked with children under four years old. Many services in this analysis operate at the earlier end of the life course, including maternity and early infancy.

The wider environment for young families can be incredibly challenging for LEAP’s priority population. This is particularly true of previously mentioned factors, such as the cost-of-living crisis and a changed socio-political landscape since LEAP began. Both have exacerbated prolonged experience of poverty. Beyond the time when LEAP worked with families, people in the LEAP area continue to navigate life in the context of these challenges.⁹

Furthermore, the children included in this analysis had just emerged from Covid-19 restrictions. Lockdowns and restrictions on face-to-face service delivery may have dissipated the effect of LEAP services. It is well evidenced that the Covid-19 pandemic did not impact children equally.¹⁰ It is therefore likely that the pandemic undermined LEAP’s work to reduce inequalities gaps in outcomes for reception-age children.

Reflections

The data available for children at reception shows no significant differences for children whether they engaged with LEAP or not. This is inconsistent with our findings for 2.5-year-old children. However, none of our wider research validates this finding. Data that would enable a before-and-after comparison is not available for population-level datasets.

It is therefore impossible to say if we have improved outcomes in the group of children engaged with LEAP services, particularly SED and targeted services. Pointedly, our wider research reveals that outcomes for parents engaged with LEAP services have improved. The areas of improvement include parental mental health and wellbeing, and parental knowledge, skills and behaviour.

We must also consider the socioeconomic environment for families engaged with LEAP. It is possible that families with more acute needs, who engage with more targeted services, see these needs persist beyond the age for LEAP provision.

The results seen here for 2.5-year-olds are notable and positive for The LEAP programme, which has offered a holistic suite of services to local families in the LEAP area. However, the current analysis has not been able to demonstrate the same positive results for children at the end of reception who previously engaged with LEAP.

As a test-and-learn initiative, there remains a lot to understand about how place-based, collective-impact programmes for children can link and effectively use good-quality data for evaluation in the context of widening inequality, a global pandemic which disproportionately affected children living in more disadvantaged areas, and substantial social vulnerability.

8.0 Further reading

Documents (PDF):

1. Methodology
   A paper supporting the programme-wide analysis of Lambeth Early Action Partnership using linked population datasets on child outcomes

2. Analytical output
   A slide deck presenting charts and tables generated during analyses using the Ages and Stages Questionnaire (ASQ-3) and the Early Years Foundation Stage Profile (EYFSP).

Posters (PDF):

3. Ages and Stages Questionnaire (ASQ-3)

4. Early Years Foundation Stage Profile (EYFSP)

References