Post by Joe, Data Science Unit, Welsh Government

Darllenwch y dudalen hon yn Gymraeg

The Data Science Unit has recently been exploring potential uses of Generative AI. One project has developed a proof-of-concept tool that uses Generative AI to assist in editing text content. The aim of the tool was to:

• support content designers in the Welsh Government by providing them a better draft to work from; and,
• help everyone produce better content (we do not have enough content designers for all our content!)

The tool could save the content designers time, focus their skills on the more challenging problems and improve our written content.

What we did

The purpose of the current version of the tool is to:

• improve the readability of content by using shorter and simpler words and sentences;
• improve the accessibility of content by following Easy Read guidance on text (find out more about Easy Read from the UK Association for Accessible Formats);
• edit the structure of content so that it is easy to understand, has a logical flow and uses headings, bullet points and shorter descriptions to convey information; and,
• provide consistency to ensure style guidance is followed consistently across a range of content, platforms, audiences and formats.

So far, we have tested 3 different Large Language Models (LLMs). You may have heard about, or even used, LLMs like Chat GPT or Microsoft CoPilot. We are experimenting with similar models in a test environment so that we can control how creative the model can be, how much text it can generate and test how the model responds to being provided extra information.

On this project, for example, we have given LLMs a role using prompts (instructions) like:

• “You are a content designer responsible for editing content to make it easier to understand and to read”; or,
• “You are a content critic and, as part of your role, you give feedback on the accessibility of text information”.

Our tests give the LLMs a role and specific guidance on how to design content effectively. We provide a first draft of some content and then ask the model to edit it using Easy Read and Welsh Government guidance. This includes things like:

• avoid passive voice;
• group similar topics together;
• avoid double negatives; and,
• use only one idea per sentence.

What we found

The model returns an updated draft which fixes accessibility issues and simplifies the original content. A comparison of the original draft and the AI revised version is given below.

Figure 1: Example of first draft content before and after AI edit:

To understand whether the draft has been simplified we used Flesch Kincaid Grade Level (FKGL). FKGL measures the approximate reading grade level of a piece of text and is often used to indicate how difficult text is to understand. From our experiments, the model successfully reduced the reading age of content on 7 out of 8 examples. In some of these cases the model performed as well as, or even better than, human content designers. FKGL is a big step towards evidencing how difficult a piece of text is to read but simplification is just one part of good content design.

We also want to assess if the content has been grouped into sections for a coherent flow, and whether all the relevant information is included. For this reason, we have chosen to also assess the content manually with data scientists and content designers to get human feedback on the work.

Manual reviews have shown that the model has successfully performed some of the tasks with which it has been prompted, like removing certain expressions. One example of this is changing “carbon monoxide is a silent killer” to “carbon monoxide is a dangerous gas that can’t be seen or smelled. It can be deadly”.

Alongside this, we have discovered some limitations:

• content designers found that the LLM sometimes missed some of the logical flow of content and opted for sections that seemed disjointed; and,
• specific Welsh Government style guidance is sometimes disregarded by the model.

We also know that the LLM is unable to go back and ask for clarification or further information in ways that human content designers can do. It demonstrates how important it is for content designers to oversee the content edited by AI and be involved in the design process.

We aim to deploy the proof-of-concept with some updates to address the feedback above as an assistive tool in future.

We hope to follow this blog up in future with more information and results from our work using LLMs to assist in content design.