How to Prepare Your SOPs and Documentation for AI
Without the right foundation, we may find that the AI structures we try to build will come crumbling down.
As AI becomes more and more prevalent, I get a lot of questions about how it will impact learning, training, and performance support.
At this point, I don’t think there is anyone who doesn’t recognize that AI will have a dramatic impact with both positive and negative consequences.
The question is, how do we adopt a strategy that will do the following:
Identify areas where AI can make an impact
Help AI perform at its best
Mitigate as much as possible the risks of AI
Key Concepts & Defining the Scope
When it comes to AI in learning, training, and documentation, it is crucial to understand a few key concepts.
I’ll explain in terms of OpenAI and ChatGPT since that is what most people are familiar with.
GPT 4 has been trained on public knowledge as of April 2023
This includes both public domain and copyrighted content
It has NOT been trained on your company’s private, domain-specific knowledge
GPT 4 is only as good as the information it has access to
When GPT 4 doesn’t have adequate information to form a response, it tends to make up answers instead of telling you that it doesn’t know
For the purposes of this article, I am going to narrow the scope of the type of learning, training, and performance support I am talking about to Operational Knowledge.
Operational knowledge includes your SOPs, policies, FAQs, best practices, troubleshooting procedures, workarounds, and everything else about how your business works.
Garbage In, Garbage Out
It is very likely that GPT 4 has not been trained on your operational knowledge since that information is private. If you want to use GPT 4 to power empower your employees to find answers quickly or to empower other AI applications to apply your SOPs/policies to workflows then you need to feed that knowledge to the AI engine.
A perfect example is customer support bots. I interact with these bots on a regular basis. Very rarely do these AI-enhanced chatbots actually help me solve a problem. They provide vague or even incorrect answers.
But that isn’t the fault of the AI.
When I can’t find a solution to the problem using the AI chatbot I start searching the documentation. Sure enough, the documentation that is supposed to solve my problem is either out of date or doesn’t exist.
I just had this experience yesterday. We are using new accounting software with an AI-enabled help interface. I didn’t know how to perform a function in the software, so I asked the chatbot. It gave me a quick answer, but when I tried to follow the instructions, I was completely lost. None of the menus it was describing to me existed.
So then I searched their documentation. Sure enough, the documentation existed but was outdated.
A human agent might see that outdated documentation, recognize that it was outdated, and guide me the right way. But at the same time, a human agent might not be aware that the documentation had changed around a specific procedure, rely on their faulty memory, and give me outdated information.
Both the AI and the human work from their memory. The documentation you feed the AI constitutes its memory.
Garbage in, garbage out.
ChatGPT can make a great demo. Upload your 50-page SOP manual and then start asking ChatGPT questions. ChatGPT will pull out answers and insights. It is amazing.
But when you get into really using this in a work environment, challenges can crop up.
How do you protect against hallucinations?
How do you keep ChatGPT’s source knowledge up to date?
How do you ensure that employees only have access to the information that they should see and not sensitive information they shouldn’t see?
Some people erroneously believe that ChatGPT will write their documentation for them. ChatGPT can be a great documentation accelerator. We are currently using it to onboard new clients, and it is saving massive amounts of time.
But ChatGPT won’t be able to invent knowledge that hasn’t been documented clearly and thoroughly.
The Future of Documentation
As AI proliferates, documented knowledge will become more and more important. As you develop your strategy, you need to answer these questions:
How do we ensure comprehensive documentation for AI tools?
How can we maintain the currency and accuracy of our content?
How do we avoid documentation conflicts that could mislead AI?
In conclusion, while AI's potential in knowledge management, training, learning, and performance support is undeniable, harnessing this potential requires a meticulous approach to knowledge management and documentation.
Companies that operate at the Guide Stage or higher on the Knowledge Ops Maturity Model will be uniquely positioned to take full advantage of all that AI has to offer. Why? Because they will have the right information to feed the AI.
If you would like to discuss any of these concepts, please don’t hesitate to reach out.
The future is very exciting, but if we don’t lay the right foundation we may find that the AI structures we try to build will come crumbling down.