Using GenAI to analyze scenarios against a framework

“How do we share this stuff? I know we used to use blogs to share this practice. How do we find out about practice like this now?”

The question came from Brian Lamb in the Q&A session following my recent presentation on how Open Education can support the development of digital literacies. So this post is to pick up Brian’s call to action and blog my practice and share. In this case, it is how I used ChatGPT to prepare my recent presentation on how open education can support the development of digital literacies for educators and students.

I am trying to be more intentional with how I experiment with GenAI tools and I am working on incorporating it more into my regular workflow to better understand the limitations and capabilities of the technology.

When I started planning my Digital Learning Strategy Forum presentation I knew I wanted to try to be explicit in connecting open education practices to the 8 competencies in the digital literacy framework, and I wondered if ChatGPT could help me make those connections.

I started by uploading the framework and then asked ChatGPT to use that framework to analyze the 2 different scenarios I had chosen to use as my examples in my presentation – a student open pedagogy activity where students create and edit Wikipedia content, and an educator specific one where an instructor uses an open textbook as a source to create ancillary materials with GenAI.

My first prompt prepared ChatGPT for what I wanted to do.

I am going to provide you a framework of competencies and skills that has been developed to define digital literacy skills for post-secondary educators. I want to then provide you with some scenarios of activities that educators might do with students. I would like you to analyze the activities I give you against the framework and determine what elements of the activity match the skills and competencies in the framework.

To which ChatGPT responded:

That sounds like an excellent approach! Please share the framework of competencies and skills, and then provide the scenarios. I’ll help analyze how each activity aligns with the specified skills and competencies.

“That sounds like an excellent approach!” You gotta love how working with GenAI always feels like an exercise in positive affirmation. As an aside, if you want to try a fun little activity that will make you feel real good about yourself, ask NotebookLM to create a podcast about you based on your LinkedIn profile.

After I uploaded the framework & ChatGPT told me it was ready for the first scenario to analyze I added the first scenario I wanted it to analyze. I wanted to use a real-world example, so I went to the openly licensed WikiEDU site and found an openly licensed example of a higher education focused student Wikipedia activity from Melony Shemberger, Ed.D. is Associate Professor of Journalism and Mass Communication at Murray State University that I fed into ChatGPT.

ChatGPT gave me a lengthy response. I reviewed it and thought it was quite accurate and made good connections between the framework and the activity explicit. It provided solid rationale of competencies students would develop using this activity that all aligned very well to the competencies in the framework.

I edited the responses and picked a few that I could use as examples to build the slide for my presentation.

The prompt for the second scenario where an instructors uses an open textbook to develop ancillary materials was not quite as detailed as the Wikipedia activity so that may explain why the ChatGPT response was not quite as accurate. Here was my prompt:

This one is focused on developing the digital literacy skills of an instructor. An instructor wants to use an open textbook published by OpenStax. However they think the reading level is a bit too technical for the level of their first year students, so they want to use a generative AI tool such as ChatGPT to rewrite portions of the textbook to make it more accessible for students. They also wish to use generative AI to create a glossary list of terms for their students, and to create a series of self-assessment questions and answers that students could use to self-assess their own knowledge. Because the textbook is released with a Creative Commons license, the instructor feels comfortable running the textbook through a generative AI tool without fear of violating the copyright of the textbook authors. Using this scenario, can you analyze the digital skills an instructor might develop against the digital literacy framework I gave you earlier?

ChatGPT’s response was not quite as accurate as the first scenario. In this case, ChatGPT rewrote the competency headings rather than using the competency headings in the digital framework, or combined multiple headings. As opposed to the first scenario where ChatGPT stayed true to the competency headings, ChatGPT listed the competencies for this scenario as;

• Information Literacy and Content Curation – this one mixes 2 competencies together. Well, 1.5 competencies as it forgets to include Creation as part of the “Creation & Curation” competency.
• Technical Literacy and Tool Competency – Tool Competency isn’t identified as a core competency in the framework
• Digital Communication and Design Skills – Design Skills isn’t a core competency and, like Information Literacy & Content Curation, this combination of competencies fails to include Community with Communication.
• Ethical and Legal Awareness – good
• Critical Thinking and Reflective Practice – Reflective practice isn’t listed as a core competency
• Pedagogical Adaptability and Responsiveness – neither is a competency heading

Now, while these may seem minor deviations in how competencies are named and grouped, it does show that ChatGPT has not stuck strictly to the framework and has instead taken a bit of liberty with how it groups and identifies the competencies. If you needed to have a very accurate mapping of a scenario against a framework, ChatGPT failed in that regard.

However, even though there are some liberties taken here, I think ChatGPT has actually come up with some very interesting competencies that could be useful additions to the competency framework. For example, that last one around Pedagogical Adaptability and Responsiveness included 2 bullet points that I thought were really good.

• Responsive to Student Needs: Recognizing that the textbook may be too technical for first-year students highlights the instructor’s adaptability to meet students at their level. By leveraging AI to create more accessible materials, the instructor is actively enhancing the learning experience based on the needs of the cohort.
• Innovative Use of Technology for Personalized Learning: Creating tailored resources, like a glossary and self-assessment questions, shows a commitment to personalized learning pathways, which can help students learn at their own pace and assess their understanding independently.

When you are making a competency framework for educators, these seem like good suggestions that, with a bit of massaging, could be powerful digital learning competencies for educators and, in particular, open educators looking to incorporate some open practices into their own teaching & learning practice.

In the end, ChatGPT’s suggestions took a bit more work on my part to develop a good mapping slide for the presentation.

All in all, ChatGPT did provide some good suggestions and gave me a good starting point on the key themes I could use to frame my talk. And, in this case, the suggestions that were not in perfect alignment with the framework actually have provided me with some new considerations that I can bring back to the Digital Literacy working group for consideration for future refinements to the framework.