THE ASSESSMENT SET-UP PROCESS
This section describes the ‘Assessment Set-up’ process introduced in ‘The UCAM Processes’ section above and executed in order to set up an assessment or set of assessments.
Requirements for the Assessment Set-up process
The requirements for the ‘Assessment Set-up’ process are given below in Figure 4.13, which highlights the relevant requirements on the diagram introduced in Figure 4.1 above.
Figure 4.13 Requirements for the Assessment Set-up process
As can be seen in Figure 4.13, the main requirement for the ‘Assessment Set-up’ process is to ‘Set up assessment’. The process must enable assessments to be set up to meet the requirements to be able to ‘Assess competency for accreditation’, ‘Assess competency for education’, ‘Assess competency for recruitment’, ‘Assess competency for staff appraisal’ and ‘Assess competency for self-assessment’. The activities that are needed in the process to meet these requirements, and the artefacts that form inputs to and outputs from the process are described in the following section.
Contents of the Assessment Set-up process
The contents of the ‘Assessment Set-up’ process are shown in Figure 4.14, annotated to show whether the various artefacts are inputs or outputs from the process.
Figure 4.14 shows that there are three activities that have to be carried out in the ‘Assessment Set-up’ process, namely ‘define competency scope’, ‘identify assessment needs’ and ‘review’. The process takes the ‘Populated Applicable Competency Set’ output from the ‘Framework Population’ process and uses it to generate the ‘Competency Scope(s)’ for the assessments. The way that the ‘Assessment Set-up’ process is carried out is shown in the following section.
Figure 4.14 The contents of the Assessment Set-up process
Carrying out the Assessment Set-up process
Figure 4.15 shows how the ‘Assessment Set-up’ process is carried out. The ‘soft boxes’ (rectangles with rounded corners) represent the various activities that have to be carried out and correspond to those shown in the bottom compartment in Figure 4.14. The vertical divisions (swim lanes) indicate which stakeholder role is responsible for carrying out which activity, and correspond to one or more of the stakeholder roles identified in Figure 4.2. The small rectangles containing arrows (known as ‘pins’) show inputs to and outputs from the various activities, with the name of the artefact flowing into or out of the activity shown on the line connecting the pins. These artefacts correspond to those found in the middle compartment in Figure 4.14.
Figure 4.15 shows that the ‘Assessment Set-up’ process begins with the ‘Assessment Definer’ carrying out the ‘identify assessment needs’ activity. This is conducted in order to decide just why the assessment or set of assessments is to be conducted and generates the ‘Assessment Criteria’ containing these reasons.
The ‘Assessment Definer’ then takes these ‘Assessment Criteria’ along with the ‘Populated Applicable Competency Set’ (which the diagram shows is an input to this process) and uses them as inputs to the ‘define competency scope’ activity. Knowing why the assessment is to be carried out (the ‘Assessment Criteria’) and the competencies against which an assessment can be conducted (the ‘Populated Applicable Competency Set’) enables the ‘Competency Scope’ to be defined. This scope details the exact competencies against which the assessment will be performed, along with the levels to which each competency will be assessed. Such scopes are often produced for a defined role within an organisation and it is not uncommon for a library of such reusable scopes to be produced by an organisation over time as the process is run a number of times. Eventually, it may be possible to dispense with this process (or, at least, run through it very quickly) once an organisation has such a library of previously defined scopes from which they can pick.
Figure 4.15 Carrying out the Assessment Set-up process
The ‘Competency Scope’ feeds into the ‘review’ activity, which is carried out by the ‘Reviewer’ in order to check that everything is OK with the defined ‘Competency Scope’. The results of the review activity are documented in the ‘Review Results’ artefact. If there are problems with the ‘Competency Scope’, then the process starts again with the ‘identify assessment needs’ activity. If there are no problems, then the ‘Assessment Set-up’ process terminates, having successfully determined the criteria for the assessments and generated the scopes against which the assessments will be carried out. The various artefacts of the ‘Assessment Set-up’ process and their relationships are discussed further in the following section.
Artefacts of the Assessment Set-up process
The main output from the ‘Assessment Set-up’ process is the ‘Competency Scope’ (or scopes) that is to be used as the basis of a set of competency assessments. The artefacts used and produced by the process, along with their relationships, are shown in Figure 4.16.
Figure 4.16 Relationships between the artefacts of the Assessment Set-up process
While the main output of the ‘Assessment Set-up’ process is the ‘Competency Scope’, this cannot be produced without understanding the reasons for conducting the assessment. These reasons are captured in the ‘Assessment Criteria’ artefact. This is specified as a formal process artefact deliberately to force the documentation of the criteria to be produced, rather than allowing them to be something that is just thought about and never written down, an all-too-common approach which often leads to consideration of the reasons for an assessment to be skipped, resulting in ill-defined assessments and consequent problems in defining an appropriate scope.
The ‘Competency Scope’, generated based on these ‘Assessment Criteria’ is a subset of the ‘Populated Applicable Competency Set’ described above in the section on the ‘Framework Population’ process. No assessment can practically assess against every possible competency applicable to an organisation (or even part of an organisation) and so the ‘Competency Scope’ details just those competencies against which competency will be assessed. It consists of a number of references to competencies from the ‘Populated Applicable Competency Set’. Such a reference is represented in the diagram as a ‘Competency Reference’. In addition, the ‘Level’ to which a competency is to be assessed must also be defined together with the ‘Evidence Type’ that will be accepted when assessing that competency at the indicated level.
Finally, the ‘Review Results’ artefact is simply a record of the outcome of the ‘review’ activity that is carried out at the end of the process to ensure that all the generated artefacts, and, in particular, the ‘Competency Scope’, are fit for purpose.
An example of a ‘Competency Scope’, first seen in Chapter 3, is given in Figure 4.17.
Figure 4.17 Example ‘Competency Scope’
Figure 4.17 shows an example ‘Competency Scope’ for a requirements engineer role within an organisation. A number of competencies, grouped into competency area, are shown along the bottom of the diagram, here intended to show all the competencies from the organisation’s ‘Populated Applicable Competency Set’. The vertical axis indicates the various values of ‘Level’ that have been defined, with the contents of each cell defining the ‘Evidence Type’ for a given competency at a given level. The ‘Competency Scope’ itself is indicated by the shaded area. Thus, ‘Validation’ is to be assessed to ‘Supervised practitioner’ level only and ‘Life-cycle process definition’ is not to be assessed at all.
In practice, the number of competencies in a ‘Populated Applicable Competency Set’ is likely to be large and so all the competencies from the set are not all likely to be shown on a ‘Competency Scope’. Also, all the competencies shown in Figure 4.17 are taken from the INCOSE Systems Engineering Competencies Framework but, as has been explained earlier, the whole point of the UCAM is that competencies from a number of different frameworks could, and in reality would, make up the ‘Populated Applicable Competency Set’ and any ‘Competency Scope’ generated from it.
Summary of the Assessment Set-up process
The ‘Assessment Set-up’ process exists to establish the ‘Assessment Criteria’ and ‘Competency Scope(s)’ for an assessment or set of assessments. The ‘Competency Scope’ is one of the main inputs to the ‘Assessment’ process, defining the competencies that will be assessed and the levels to which each will be assessed.
Discussion on the Assessment Set-up process
Not all assessments are carried out for the same reason and therefore the purpose (or purposes) of an assessment should be clearly defined. This is the role of the ‘Assessment Criteria’. This set of ‘Assessment Criteria’ is what defines the ‘Competency Scope’ against which the assessments will be conducted. Such a competency scope should be well-defined, stating clearly the competencies that are to be assessed and the levels to which they will be assessed, as well as being practical in terms of assessment time and resources. Experience has shown that conducting an assessment with seven indicators to an average of level three can easily take up to three hours. When the preparation time and the time to analyse results is included, together with the fact that to conduct an assessment effectively requires two assessors (see the section on the ‘Assessment’ process below), this time can easily amount to a man-day of effort per assessment, so knowing the reasons for and scope of an assessment is vital in order to ensure that the time is well spent.