How can we evaluate learning culture?
I recently joined Armakuni as their Continuous Learning Lead. What that means in precise terms is still in flux, but my current working definition looks like this:
My goal is to help organisations create and sustain a continuous learning culture, capable of supporting lasting transformational change.
Which raises several questions…
- What exactly is a culture of continuous learning?
- How does one measure it?
- Can it be created and managed?
Disclaimer: I am coming at this from an academic perspective as opposed to an experiential one; I have no previous commercial experience of DevOps, management or cloud transformation. As such, I have no strong opinions for or against established approaches. Having suffered my share of pedagogical flights of fancy in academia, my aim is to remain as objective as possible, and to look for ideas which are simple and testable. Inevitably I will need to veer into the abstract at times, but I also intend to learn as much as I can about what my colleagues do and what they experience in practice, so I am better able to connect what I do with what they need.
In this article I will share the first stages in the process I am going through to gain a clearer picture of what continuous learning culture is, both in theory and in practice. I will also explain how and why I intend to evaluate the learning culture at Armakuni with a view to helping it develop its collective learning capabilities.
If you are less interested in the theoretical basis for the methods I’ll be using, you can skip the next couple of sections and head straight to ‘Planning the evaluation’!
What is `continuous learning culture’?
In the DevOps and agile communities, continuous learning has been recognised as a key factor that contributes to organisational performance.
The cycle of continuous improvement and learning is what sets successful companies apart, enabling them to innovate, get ahead of the competition — and win. (Forsgren et al., 2018)
The Scaled Agile Framework (SAFe)* describes continuous learning culture as a set of values and practices that encourage individuals — and the enterprise as a whole — to continually increase knowledge, competence, performance, and innovation (SAFe, 2021 March 22). It is one of 7 core competencies of the Lean Enterprise, as defined in SAFe 5.0, representing the stepping stones to achieving business agility.
SAFe bases its transformation model on the 5 disciplines described by Senge (1990):
Senge’s 5 disciplines seemingly fall within the broader context of collective learning theory — as in, the processes by which networks of learners synthesise and evolve shared knowledge, competencies and attitudes. The first stage in my process will therefore be to gain a better understanding of collective learning processes.
*Please note, this is not an endorsement of SAFe. I am merely noticing the connection with Senge’s 5 disciplines. You may unclench your fists.
Collective learning: an overview
Collective learning has been variously defined but is generally considered a dynamic and cumulative process involving social interaction, leveraging of relational synergies and development of shared understanding (Garavan & McCarthy, 2008). Understanding collective learning is said to require a multilevel perspective, whereby processes and outcomes are viewed differently at the different levels. Arrow and Cook (2012) have conceptualised individuals, groups and organisations as a nested hierarchy of learning systems, within which learning may be both directed (driven by explicit goals) and undirected (learners come to associate certain actions or behaviours with more positive outcomes.)
Processes
Collective learning processes may focus at the individual and collective (group or organisation) levels. Within these broader categories, processes can be strategic — resulting in measurable outcomes — or visionary — focusing on fostering behaviours it is assumed will make the organisation better able to respond to external changes (Garavan & McCarthy, 2008).
Considered at the organisational level, collective learning processes typically focus on the role of culture, routines and shared practices; while at the group level, collective learning involves attention to information which results in the development of shared mental models and an increased ability to satisfy common goals (Arrow and Cook 2012).
At the individual level, collective learning concerns the thoughts and behaviours of individuals. At this level, organisational strategies will tend to focus on personality and professional competencies with the assumption that these affect an individual’s ability to contribute to collective learning at the group and organisational levels.
However, without considering the wider contexts in which individual learning occurs, and how individual learning contributes to collective learning goals, there is potential for the misalignment of goals. A learning and development framework which aligns and facilitates growth across all 3 levels is therefore desirable, and will be another of my goals at Armakuni (and no doubt the focus of future posts).
Outcomes
Outcomes of collective learning processes have often been conceptualised as either cognitive (i.e. learning characterised by acquisition of knowledge, processing of information, and synthesis of new ideas and concepts) or behavioural (i.e. observable and measurable behaviours), and processes therefore differ depending where there is more emphasis. Examples of cognitive learning processes include active inquiry, critical reflection and collaborative learning. Behavioural processes will tend to focus on developing capabilities that it is assumed will lead to performance improvements.
Summary
Understanding how different learning systems behave and interact is thought to be important for the successful implementation of management strategies designed to foster continuous learning culture, and is consistent with Senge’s ideas around systems thinking. However, the question has also been raised about whether organisational learning can be designed, or whether it is an organic process. I will assume, for now, that both possibilities are true — to an extent. I’ll assume that, while managers can put in place certain structures to encourage and facilitate collective learning, it is a complex set of processes that comes with the possibility of undesirable outcomes. For example, Garavan and McCarthy are among others to point out the importance of psychological safety and alignment of individual and organisational goals for the avoidance of negative consequences of misguided collective learning strategies.
Aside: Psychological safety is said to reflect employees’ beliefs about the safety of interpersonal risk-taking (Klinger and Bajestani, 2009) and is a widely discussed topic in relation to continuous learning culture. In a psychologically safe environment, an employee is more likely to take risks because they have less fear of negative repercussions. It is often a key factor considered in relation to team information sharing and innovativeness, as well as in the creation of diverse and inclusive environments which are, in turn, critical for effective collaboration and group learning (Edmondson, 2009).
Planning the evaluation
The next stage in this process will be to evaluate the learning culture at Armakuni. I will attempt this at the individual and organisational levels. At the individual level I will be using a combination of surveys and one-to-ones, the format for which I shall go on to describe in this article. At the organisational level I shall perform an organisational learning audit in which I will attempt to map out the company’s collective learning processes.
The objectives behind the evaluation are as follows:
- To understand how people are learning in the company, whether they are gaining from it, and whether there are things management might do to improve learning opportunities, and/or to improve the effectiveness of the learning that is taking place.
- To understand how the learning is contributing to individual, group and organisational learning goals, and whether there is a need for greater alignment in any of these areas.
- To generate ideas around how we might integrate support for continuous learning with a new employee progression framework.
In the next section I will explain the methods I’ll be using in the evaluation.
Survey methodology
The methodology I’ll be using for the survey draws inspiration from Garvin et al. (2008), Bauer et al. (2007) and Dillon (2021). It will assess participants in 4 areas: social relationships, quality of tasks, support for learning, and attitudes to learning. In each area there will be 5 variables mapped to 5 Likert-type questions, the rationale for which I will summarise below. It felt important to me that the survey would not take participants more than a few minutes to complete. This is so that it can be repeated at intervals, and potentially incorporated into the one-to-ones where it may form the basis of a discussion around their learning needs.
The variables and their associated question statements are listed below. It is noted that most of the variables are not really possible to assess reliably through a single question statement, but it is hoped that the survey will still provide a useful snapshot from which more detail may be drawn out through the one-to-ones.
One-to-ones
The one-to-ones will give me the opportunity to probe more deeply into the different ways people learn and their attitudes to learning. In these conversations I will ask questions about the concrete processes by which people learn, and what challenges they face when learning. These conversations will be organic and confidential, although I expect to capture anonymised qualitative data in the process which will be used internally to inform next steps. I’d also like to leave the door open for people to articulate learning goals or challenges they’d like support with, and these may provide the agenda for follow-up coaching sessions for those who want it.
Some examples of questions I might ask are given below, some of which are taken from Dillon (2021). These are intended to stimulate further discussion rather than dictating a format. The duration of the one-to-ones should be no more than 15 minutes, hence the questions will help to give focus to the discussion.
Organisational learning audit
As I carry out the one-to-ones, I will concurrently be mapping the concrete learning activities and processes occurring in the organisation against the 4 quadrant collective learning model depicted earlier. This should give me an overview of the learning ‘landscape’ at Armakuni. It will also help give context to the discussions I have with individuals.
A second element to the organisational learning audit will be an attempt to map the networks and communication channels within the organisation which are implicitly or explicitly linked to learning. The rationale behind this exercise comes from the emphasis many theorists have placed on communication as a conceptually related process to learning (e.g. Pace, 2002). The assumption is that, by analysing the different ways that knowledge is shared across the organisation, we may be better able to identify weaknesses and opportunities for improvement. The process will essentially involve listing the different ways learning-related information is communicated and between whom, and performing a SWOT analysis. I will invite a group of 5–7 employees of different roles and career levels to participate in the SWOT analysis.
Findings?
The findings from this learning culture evaluation will be the focus of a follow-up article, in which I shall also reflect on the effectiveness of this process. It is still very much in the experimental stages, and I expect to refine it in subsequent iterations. Comments and suggestions are welcome!
References
- Forsgren N, Humble J, Kim G (2018) Accelerate. IT Revolution.
- Scaled Agile (2021, March 22) Continuous Learning Culture
- Senge, Peter. The Fifth Discipline: The Art and Practice of the Learning Organization. Penguin Random House LLC.
- Garavan T, Mccarthy A (2008). Collective Learning Processes and Human Resource Development. Advances in Developing Human Resources. 10. 451–471. 10.1177/1523422308320473.
- Arrow H, Cook J (2012) Configuring and reconfiguring groups as complex learning systems. In Work Group Learning: Understanding, Improving and Assessing How Groups Learn in Organizations (pp. 45–72). Taylor and Francis. https://doi.org/10.4324/9780203809747
- Klinger R, Forghani Bajestani M (2018). Psychological Safety. 10.1093/OBO/9780199846740–0154.
- Edmondson A, Roloff K (2009) Leveraging Diversity Through Psychological Safety
- Anna Dillon (2021, March 21), How to assess learning culture
- Bauer J, Rehrl M, Harteis C (2007). Measurement of learning culture: a motivational approach. ttps://doi.org/10.1371/journal.pone.0237978
- Garvin D, Edmondson A, Gino F (2008) Is Yours a Learning Organisation. Harvard Business Review. 86. 109–16, 134.
- Chick, N. (2013). Metacognition. Vanderbilt University Center for Teaching.
- Craig k, Hale D, Grainger C, Stewart M (2020) Evaluating metacognitive self-reports: systematic reviews of the value of self-report in metacognitive research. Metacognition Learning 15, 155–213 (2020). https://doi.org/10.1007/s11409-020-09222-y
- Pace R (2002). The Organizational Learning Audit. Management Communication Quarterly, 15(3), 458–465. https://doi.org/10.1177/0893318902153008