Skills Education

Using Bing Bot to assist with VET assessment tool development 25.3.23

Following on from my earlier video where Bing Bot and I got into a bit of an argument over how many elements a unit of competency has, here's a video that went more to plan. (Video below)

The plan being: to show how Bing Bot can be used as a tool to generate ideas for VET assessment tool development.

I deliberately picked a unit of competency where not much 'general' information exists like there might be in a BSB unit on leadership and management skills or a SIR unit on customer service skills.

I gave Bing Bot the: 

Context - "You are an instructional designer writing assessment tools for vocational education and training in Australia"

This specifies how I'm expecting the results to be framed

Task within the context - "You must write about saw doctoring, specifically the unit FWPSAW3247 - Replace tungsten tips"

This primes the AI to understand why I'm asking the question

Action - "Research this"

This tells the AI what to do

Then I told it to get information on the dimensions of competency so it would understand my next instruction. Which was:

"Give me ideas for competency based assessment tasks for this unit that incorporate the dimensions of competency"

Within a conversation, the AI (Bing Bot or ChatGPT) will remember the context of what's already been discussed which means you don't have to keep prompting it on the same thing.

Bing Bot came back with this output as ideas:

"Some possible ideas for competency based assessment tasks for the unit FWPSAW3247 - Replace tungsten tips that incorporate the dimensions of competency are:

1. A practical task where the learner has to replace tungsten tips on two different circular saw blades according to workplace procedures and manufacturer instructions. The learner has to demonstrate task skills by performing the task to the required standard, task management skills by selecting and using appropriate tools and equipment, contingency management skills by identifying and resolving any faults or defects, and job/role environment skills by following health and safety and environmental protection practices.
   
   
2. A written or oral test where the learner has to answer questions about the principles and techniques of replacing tungsten tips, such as the types and characteristics of tungsten tips, the brazing material and technique, the tip position and side clearance, and the quality standards and objectives. The learner has to demonstrate task skills by applying their knowledge to the task, task management skills by organising their answers logically and clearly, contingency management skills by explaining how they would deal with irregularities or breakdowns in routine, and job/role environment skills by referring to relevant legislation, regulations and standards.
   
   
3. A portfolio of evidence where the learner has to provide documentation related to replacing tungsten tips, such as work orders, risk assessments, waste disposal records, quality checklists, and feedback reports. The learner has to demonstrate task skills by producing accurate and complete documentation, task management skills by integrating documentation with work processes, contingency management skills by showing how they responded to feedback or complaints, and job/role environment skills by complying with workplace policies and procedures."

As an assessment developer, you should now review the output to determine:

• It's suitability
• Whether a task based on these ideas will collect sufficient, valid evidence
• How you can build out any viable ideas into actual assessment instruments

I'd also recommend confirming with an SME (subject matter expert) whether the ideas are sound.  Then you can build out the tool with its tasks and benchmarks, and again, have the SME review to confirm accuracy and relevance to industry requirements. And of course, always get the tool validated pre-use to confirm all unit requirements are covered and the tool is fit-for-purpose.

Some notes on the above:

In idea 1, Bing Bot came back with a suggestion for a practical task where ".. the learner has to replace tungsten tips on two different circular saw blades according to workplace procedures and manufacturer instructions..."

This is great because although the unit does not specify 'two', it does use plural 'blades'  and the definition of competency in the Standards for RTOs 2015 states that competency is "...the consistent application of knowledge and skill to the standard of performance required in the workplace. It embodies the ability to transfer and apply skills and knowledge to new situations and environments." 'Consistent' meaning "acting or done in the same way over time, especially so as to be fair or accurate" which means more than once.

It also says this task should be done "according to workplace procedures and manufacturer instructions" - which again, is great as this means the task must align to components in the performance evidence and underpinning knowledge evidence requirements.

In idea 2, Bing Bot suggested a written OR oral test - great considering a) reasonable adjustments that might apply in certain cases, b) oral questioning can circumvent academic integrity issues linked to use of AI for completing written tests

In idea 3, a portfolio of evidence covers performance over time and in different circumstances - which is aligned to the actual definition of competency

In all, there are some good starting points suggested and sound rationale provided for how the suggested tasks go on to incorporate the dimensions of competency.

Dimensions of competency

Remember how I asked Bing Bot to research the dimensions of competency? Well, in each idea suggestion, Bing Bot gives an explanation of how the assessment task idea incorporates the dimensions of competency.  

In VET, dimensions of competency are implicit in the expected outcomes of a unit. They allow for the following aspects of performance:

Task skills

• Skills to satisfactorily undertake and complete work tasks to standard
  
  

Task management skills

• Skills to plan and coordinate completion of different tasks on the job
  
  

Contingency management skills

• Skills to handle problems or issues that may come up in the performance of duties
  
  

Job role or environment skills

• Skills to follow workplace protocols and complete work in an environment that may have interruptions and other personalities as factors to account for

Note that dimensions of competency are not included as written statements within a unit of competency, but they are nonetheless, aspects of job performance. (Taken from the Big Book of kNOw, 2022).

Finally, I asked Bing Bot to give me an image of what a worn, damaged or broken tip looks like (the very thing the student is required to work with in this unit).

The AI came back with an explanation of what a 'worn, damaged or broken tip' is and had a go at creating an image for me:

While the image did not seem to be in context of a circular saw blade used in a timber mill, you can see the quality of the image production and perhaps the possibility of getting an image generated for other, difficult-to-find visual examples.

I could have further prompted the AI to research and/or produce images in the context of a timber saw mill but decided to call it quits at that point.

- M. Charlton

Top