CASE STUDY 3 – EDUCATIONAL FRAMEWORK
Introduction
The final case study departs from the previous two. Whereas the case studies presented so far have concentrated on technical-related competencies that were aimed at professional engineers, the final one has a very different target audience. The area of interest for this case study is education, in particular the assessment of schoolchildren and their performance over a school term. The approach taken to generate the following framework and associated assessments was exactly the approach described in this book – that of UCAM.
The background of this case study comes from the world of education and is mainly concerned with assessing schoolchildren during their education. Currently, teachers are required to produce a ‘school report’ card for each pupil that is updated at the end of every school term. These school report cards require that each pupil is given a rating, on a scale of between 0 and 4 (five levels in total) in a number of areas that have a single-word description, for example: ‘Focus’, ‘Task completion’ and so on. These single-word descriptions were grouped into three broad categories that were: ‘Effort’, ‘Behaviour’ and ‘Attitude’.
At the time the work was started, the main problem was a perception that there was little guidance provided as to how to interpret these single-word statements, making assessment very difficult and unrepeatable from a teacher’s point of view. Also, from the pupil’s point of view, they had no real idea how they were being assessed, which meant that they did not know which aspects of their schooldays were being assessed at any point in time.
On an anecdotal level, there was also the idea that there was an imaginary ‘line’ that should not be crossed by pupils. Many people, including pupils, teachers and parents, used this concept of a line. Although this ‘line’ was a conceptual one, apparently some pupils knew when they had crossed it and were in trouble. Therefore, a secondary requirement of the exercise was actually to see if this ‘line’ could be defined in some way. This was to prove a challenging case study, but one that was sufficiently different to test fully the approach and core processes defined as part of UCAM.
Framework definition process
The framework definition process, in this case, used the school educational material as its starting point. There was no formal standard for pupil’s competencies as such, but there was plenty of material made available to the teachers from the school and from the local education authorities, about what was expected. With this in mind, the following framework was generated.
Figure 5.10 shows that there is the key concept of a ‘Competency’ that describes the general skills required by each pupil. These competencies were grouped into three ‘Competency Area(s)’, each of which consisted of four ‘Competency(ies)’.
The three ‘Competency Area(s)’ that were defined were:
‘Effort’, which describes how the pupil approaches a particular piece of work in terms of how much they are able to focus, understand and tackle work;
‘Behaviour’, which describes the way that the pupil interacts with fellow pupils, gets involved with lessons and their attitude towards equipment;
‘Attitude’, which describes their personal attitude in terms of the way they view themselves and their own responsibilities.
Figure 5.10 Framework definition for the educational framework
Each of these competency areas had four ‘Competency(ies)’ associated with it. The competencies that were identified for the ‘Effort’ competency area were:
‘Focus’, which describes how well a pupil can focus on a single task and relate this to other tasks and other interests in the outside world;
‘Task Completion’, which describes not only the pupil’s attitude to getting work finished, but also assesses how well they can look back over their completed work and learn from what they have done;
‘Problem Solving’, which describes the pupil’s attitude to how they tackle a particular problem, how and when they will seek help and how they work as part of a team to solve problems;
‘Organisation’, which covers areas as simple as having a tidy desk, up to organising their work within the wider context of the National Curriculum.
The competencies that were identified for the ‘Behaviour’ competency area were:
‘Interaction with others’, which is mainly concerned with the pupil’s attitude towards others in the class and how their own behaviour may impact others;
‘Behaviour decisions’, which describes how much responsibility the pupils take based on their own behaviour, as well as assessing how they react to being instructed to behave in a particular way;
‘Practical risk’, which assesses how the pupil considers the impact of their behaviour in terms of personal and group safety, including equipment and tools;
‘Class discussion’, which describes the pupil’s behaviour when holding a group discussion in a classroom environment, how willing they are to get involved and how much they contribute to the discussion.
The competencies that were identified for the ‘Attitude’ competency area were:
‘Responsibility’, which is the basic attitude towards learning and work;
‘View of learning’, which describes the pupil’s attitude to the value of learning and how useful it is to their lives;
‘Personal challenge’, which is largely associated with assessing a pupil’s motivation towards work and how they can set personal goals;
‘Approach’, which describes how the pupil’s view their entire ethos on how they work, including their change motivations dependent on the subject matter.
Each one of these competencies is held at one of five ‘Level(s)’ that were defined as follows:
Level 4 – excellent;
Level 3 – good;
Level 2 – satisfactory;
Level 1 – poor;
Level 0 – unacceptable.
For each competency, there was a single ‘Indicator’ defined in the form of a statement that could be applied to the pupil to which they could answer ‘yes or no’ or ‘true or false’.
Framework population process
The framework population process is mainly concerned with defining the applicable competency set and defining the evidence type for the assessments. In terms of the applicable competency set, this is very straightforward as there is nothing to do. The applicable competency set is a subset of all the available competencies for assessment. In the case of this particular application, the competency set was created especially for the work at hand, therefore, it is the applicable competency set.
In terms of the evidence types that would be accepted and their associated timeliness, again this was quite straightforward. There are only two evidence types that need to be considered here:
The observations of the teachers. As the assessment of a pupil is a very personal thing, it can only be carried out accurately based on the teacher’s knowledge, understanding and personal relationship with the pupil. Therefore, the only evidence that can be accepted here for a third-party (teacher) assessment is the observation and opinion of the actual teacher. It could be argued that the results of academic and educational tests could also be used as an input, but this was discounted on the basis that test results are a different type of assessment altogether, and that the two sets of results, both test and assessment, should be used together to provide a complete picture of each pupil.
The self-opinion of each pupil. This is only applicable in the case of a pupil performing a self-assessment, but it was seen to be a very good way to get pupils to understand the whole assessment process.
At this point, it is usual to look at the timeliness of each of these evidence types. Again, this is very simple as the timeliness is restricted to a single school term. This timeliness is the same for both evidence types.
The completed applicable competency set is shown in Table 5.5.
| EFFORT | ||||
|---|---|---|---|---|
| Level 4 | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion |
| Level 3 | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion |
| Level 2 | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion |
| THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! | ||||
| Level 1 | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion |
| Level 0 | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion | Teacher observation, self-opinion |
| Focus | Task Completion | Problem Solving | Organisation | |
Table 5.5 shows the applicable competency set for the educational framework. In fact, this is not the complete applicable competency set, but just one-third of it as it only shows the ‘Effort’ competency area. This has been done purely for space-saving reasons as the evidence types are the same for each competency at each level, so there is little additional value in showing all the diagrams.
What is particularly interesting about this table, however, is the concept of ‘the line’. The line here represents the boundary between what is acceptable and what isn’t and provides a minimum performance level for all pupils. As will be seen in the next section, the way that the indicator statements above and below the line are treated is slightly different but, otherwise, the concept of ‘the line’ is a useful one that could be applied to any competency framework to show the minimum acceptable criteria for a competency profile.
Assessment set-up process
The assessment set-up process is concerned with defining the set of people to be assessed, which in this case was based on classes in the school. This is also the process where the assessment documentation is assembled and configured. Again, due to the simplicity of the previous framework population process, the documentation for this process is both simpler and easier to produce.
As there was only a single indicator defined for each competency at each level, it is actually possible to show these statements in the same format as the applicable competency set, with the indicator statements being printed in each cell, rather than the evidence types. This made the documentation far simpler and far easier on the eye than the usual volume of tables that is needed for assessments.
Tables 5.6, 5.7 and 5.8 show the assessment sheets that were generated.
| EFFORT | ||||
|---|---|---|---|---|
| Level 4 | I sometimes spend time outside lessons following up subjects which have interested me | While completing a task I look for ways I can extend my understanding | I work with others to overcome problems together. I am specific about the help I need, and I work on other areas while waiting for help | My organisation extends beyond the specific lesson to include other areas of the curriculum and life |
| Level 3 | I focus on what I have been asked to do throughout the lesson and attempt to link this to other things I am learning | I complete tasks, and think back over what I have learned without prompting | I listen carefully to any instructions and read information to see if I can find an answer before asking for help | My work is organised logically, and I tidy away anything which needs to go away |
| Level 2 | I focus on achieving what I have been asked to do for most of the lesson | I always complete what has been asked of me | I make sure I know what my problem is before asking for help | My work is in my file and I always tidy my equipment away properly |
| THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! | ||||
| Level 1 | I often only focus on a task when reminded by the teacher | I often don’t completely finish a task, or I do the experiment but don’t write down the results | I make some effort to understand what the problem is if I am stuck | I often lose work, and try to get away with not doing my share of the tidying up |
| Level 0 | I only focus on a task under one-to-one adult supervision and with threats | I only work under one-to-one adult supervision and with constant prompting | Immediately I encounter a problem, I ’down tools‘ and may become disruptive | I refuse to keep my own work together, or help to tidy up |
| Focus | Task Completion | Problem Solving | Organisation | |
It can be seen here that the indicators are shown in the cells of the table. Each indicator is in the form of a statement that can be agreed with or disagreed with, making the assessment simple.
| BEHAVIOUR | ||||
|---|---|---|---|---|
| Level 4 | The way I behave has a positive impact on those around me in terms of learning outcomes | I take full responsibility for my own effective behaviour | I work safely and accurately with full awareness of risk and help others to do the same | I act in ways which enable others to take part in discussions as well as contributing my ideas |
| Level 3 | I always treat and think of others as I would wish to be treated myself | I choose behaviours which help me to learn (for example, listen carefully during class teaching) | I always work safely and effectively within a group | I listen carefully to everyone’s views and think about the points which are being discussed |
| Level 2 | I am polite to both adults and classmates | My behaviour does not stop others learning | I use equipment carefully and work safely round others | I mostly operate within the rules for class discussions |
| THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! | ||||
| Level 1 | I sometimes say negative things about people (for example, people trying to learn are ‘sad’ or ‘losers’) | I usually wait for an adult to point out inappropriate behaviour before I control it | I behave thoughtlessly in ways which could be dangerous for others, or could damage equipment | I often thoughtlessly disrupt class discussions, by talking across others, for example |
| Level 0 | I am deliberately rude or aggressive to staff or pupils | If an adult asks me to behave in a particular way, I deliberately ignore them | I deliberately damage equipment and books and act dangerously around others | I deliberately interrupt discussions in a calculated and negative way |
| Interaction with others | Behaviour decisions | Practical Risk | Class discussion | |
Again, the assessment sheet shown here has the same structure as the previous one, but this time Table 5.9 refers only to the ‘Behaviour’ competency area.
| ATTITUDE | ||||
|---|---|---|---|---|
| Level 4 | I take responsibility for progressing my own learning and try to help others as well | I apply lessons learned about learning to all areas of the curriculum | I like to challenge myself, and I know that if I do the best I can, I will always get a feeling of achievement | I look for an interesting way in to any topic I am offered and try to interest others as well |
| Level 3 | I work to improve the areas which will help me to learn | I have an open and enquiring mind and am ready to learn anything | I start with the idea that I will be able to complete a task, and I like the feeling I get when I have mastered something that was difficult | I feel tired at the end of a lesson, but I feel pleased that I have accomplished all that I can |
| Level 2 | I accept that my attitude to what I am learning will affect the outcome | I am open to suggestions about how to improve my learning | I am more likely to persevere when I have a natural interest in the subject | I always make an attempt to understand a topic, but I make more effort for a topic I like |
| THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! | ||||
| Level 1 | I don’t feel like its my responsibility to change | I don’t have a view of learning | I give up easily on a task and hope someone else will do it for me | I make little effort to understand what has been asked of me |
| Level 0 | I blame my lack of progress on anyone or anything but myself | I argue that learning is worthless | I know I won’t be able to do the task, so I refuse to try | I refuse to think about what I am supposed to be doing |
| Responsibility | View of learning | Personal Challenge | Approach | |
Table 5.8 shows the final of the three competency assessment sheets that were produced. Again, the indicators take the form of a statement that has to be either agreed with or disagreed with, and this is where a subtle but important variation comes in. The assessment is carried out according to the following rules:
In order to pass the assessment successfully, the pupils must answer in the negative for the ‘Level 0’ and ‘Level 1’ indicator statements. Therefore, when asked if the statement ‘I blame my lack of progress on anyone or anything but myself’ (Level 0, ‘Responsibility’), then the pupil must answer ‘no’ rather than ‘yes’ to pass this level.
In order to pass the assessment successfully the pupils must answer in the positive for the ‘Level 2’, ‘Level 3’ and ‘Level 4’ indicator statements. Therefore, when asked if the statement ‘I accept that my attitude to what I am learning will affect the outcome’ (Level 2, ‘Responsibility’), then the pupil must answer ‘yes’ rather than ‘no’ to pass this level.
It is necessary for the pupil to achieve a pass in each level to achieve ‘Level 4’, which is the maximum.
Assessment process
The assessment sheets shown in the previous section were used as a basis for the actual assessments themselves. These assessments could be carried out relatively quickly, compared with the assessments in the previous sections. This was due to a number of reasons:
There was only a single indicator per competency, making the whole process of assessment far quicker, as there was a fraction of the number of indicators when compared with the other case studies in this chapter.
The answers to the indicator statements were based on the observations of the teachers and, because teachers spend a long time with their pupils, there is a very strong relationship between teacher and pupil, when compared to the assessors and assessees in the previous case studies. Indeed, in most cases, the assessors have never met the assessees before and have no previous or established relationship with them.
Due to the high number of students in a group, it is quite easy for the teacher to remember all of the indicator statements, negating the need to wade through swathes of documentation, as in the previous case studies.
Due to the simplicity of the assessment sheets, they could be very simply marked up during the assessments to produce the profile.
The profiles for the assessments would, therefore, look similar to the one shown below.
Table 5.9 represents a typical profile. This is a very powerful tool as it allows the teacher to see very quickly, with the use of ‘the line’, whether or not a pupil is on track with their studies. Also, by carrying out these assessments on a term-by-term basis, it is possible to see how the pupil’s competencies have evolved over the school year and, indeed, over a number of school years. An additional benefit is that pupils were given these assessment sheets which gave them an insight as to how they would be assessed, which led to an increase in the morale of many of the pupils.
| EFFORT | ||||
|---|---|---|---|---|
| Level 4 | I sometimes spend time outside lessons following up subjects which have interested me | While completing a task I look for ways I can extend my understanding | I work with others to overcome problems together. I am specific about the help I need, and work on other areas while waiting for help | My organisation extends beyond the specific lesson to include other areas of the curriculum and life |
| Level 3 | I focus on what I have been asked to do throughout the lesson and attempt to link this to other things I am learning | I complete tasks, and think back over what I have learned without prompting | I listen carefully to any instructions and read information to see if I can find an answer before asking for help | My work is organised logically, and I tidy away anything which needs to go away |
| Level 2 | I focus on achieving what I have been asked to do for most of the lesson | I always complete what has been asked of me | I make sure I know what my problem is before asking for help | My work is in my file and I always tidy my equipment away properly |
| THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! THE LINE! | ||||
| Level 1 | I often only focus on a task when reminded by the teacher | I often don’t completely finish a task, or I do the experiment but don’t write down the results | I make some effort to understand what the problem is if I am stuck | I often lose work, and try to get away with not doing my share of the tidying up |
| Level 0 | I only focus on a task under one-to-one adult supervision and with threats | I only work under one-to-one adult supervision and with constant prompting | Immediately I encounter a problem, I ‘down tools’ and may become disruptive | I refuse to keep my own work together, or help to tidy up |
| Focus | Task Completion | Problem Solving | Organisation | |