Posted by A few days ago, Brad Lightcap, in an interview with CNBC, stated, among other things, that “one of the most overhyped parts of artificial intelligence is that in one fell swoop, it can deliver substantive business change.” (The Verge OpenAI COO thinks AI for business is overhyped, Dec 4, 2023)
He added that companies are expecting generative AI to solve problems, cut costs, and bring back growth but that it is still in the infancy stage (he said “experimental growth”) and that there is “never one thing you can do with AI that solves the problem in full.” (The Verge, OpenAI COO thinks AI for business is overhyped, Dec 4, 2023)
There are many people in our industry, especially those in L&D and training, along with learning tech and system vendors, who think that generative AI will solve all their problems, even though it is at a tiny stage. Think of a flea because, in the grand scheme of things, that is where generative AI is—flea level.
I constantly receive inquiries and e-mails from readers unaware of where we are with generative AI. Equally, there are learning system and learning tech vendors (and likely other SaaS providers who have nothing to do with our industry) who have pushed or implied the notion that gen-AI is the secret sauce to unlocking everything for the administrator and learner right now – the wunderkind of learning and training.
I get it, because, as I have presented to companies’ L&D and Training departments, along with learning system vendors, gen AI has the potential and can do a wide variety of tasks today, which is true. But, equally recognize that where we are today, well, a year from now, will be far different. I often reference electric cars. Think of their capabilities and range a year go compared to where they are today. And then, think of where manufacturers of electric vehicles are pointing towards by the end of next year.
While products – and boy there are a lot of them for generative AI, and especially ChatGPT are available on a daily basis, the presumption is that they “rock” are suitable, in fact, ideal for a system. What I find however is that vendors as a whole, are looking at only a small sample, not a wide variance.
ChatGPT – Panacea?
Except for two vendors (who have multiple Gen-AI’s), every other system I have looked at that has Gen-AI uses a version of ChatGPT. The most popular one is ChatGPT 3.5 Turbo (the one available for free for folks is ChatGPT 3.5). Some vendors use a combo of Turbo and ChatGPT-4 (which has been rebranded as ChatGPT Plus). You might wonder why the combo or only the Turbo version?
It comes down to the wonderful fees – as in token fees. 3.5 Turbo is less expensive than ChatGPT-4. It is interesting to note, however, that the fees are not as high, regardless of which version – simply because, (excluding a couple of vendors), it is only for the administrators – including those who create content (whoever that may be). The others? It is skills tied to content, and so far two are in “build mode” and thus not yet live. The other vendor has demurred from showing it until 2024 (even though they heavily pushed it, but held it back during a recent demo), even though they showed their other generative AI capabilities in their system (I was a tad underwhelmed).
So far, I am seeing the same thing repeatedly – except for one vendor, who did something different, albeit it was one of the items I keep seeing repeatedly. What is that pray tell? Oh, let me share!
- Content creator tool (aka course creator, built-in authoring tool) – This is the most popular, and I am already bored. I can’t figure out why nobody has punched this out of the park – okay, one vendor did – I will mention them shortly.
- Assessment tool – Whipee! Again, same ol same. So much can be done, so much isn’t
- Finding content – specific areas in the content, right to the line, and includes the page number (I liked this because not even Lucy.ai can do that). Plus it only shows the snippet you want and not the entire document/video, in case you want to see/read it (again, Lucy.ai can’t do this). The vendor who can do this? Learning Pool (oh, rated #6 in my learning systems for 2023-24).
On the content creator, the rockstar of impressiveness is CYPHER LEARNING (they cap their name). The content creator has a lot of capabilities that can go beyond just creating a simple or quick course. From synthetic audio to layers and much more. From a generative AI standpoint, they lead in that area.
What is missing with all the generative AI I have seen?
There are two parts here, first, only two vendors note it, the rest do not, and of the systems I have seen, including one well-known learning tech vendor, lack it. This latter part is befuddling – because it is the CRUCIAL ITEM you need for generative AI.
The first part is the key statement that the information/statements/results produced may provide fake or false information (known as hallucinations). Each vendor adds some additional info to be aware of, albeit CYPHER did the best job. However, they noted the fake and false information as “mistakes,” which seems trivial and is not. If someone said mistakes compared to fake information, I would see it as no big deal on the mistakes side, whereas it is a big deal.
As I recall, each of them tells you to review the information before publishing it or accepting it. This is essential, as with any generative AI capability, a human element – is essential, crucial in fact.
The Context Window (often referred to by folks as the prompt window. However, the actual term is context window)
Okay, I use generative AI to produce my questions for my test/assessment, create content, create a learning path with sub-chapters or modules, or tied to content, and so on. Seems easy. If I do not believe it is correct, I can edit it in some cases (not every vendor even offers this). Again, simple.
One major problem, though. Generative AI learns from itself – as in the continuous training of it. If the information it produces is wrong and you edit it, the AI doesn’t know if it is wrong.
This is because you didn’t select a thumbs up or down (the most popular way to do it, although a Yes or No – with the words is this correct?, is an option too). Nor did you change the information in the context window next to it – i.e. the total results of the output.
And why, you may ask, can’t you do that? Because of the systems I have seen, not one vendor has it. Not one. Think about that. The most essential piece for generative AI to learn what is correct and incorrect – and thus reduce those hallucinations is missing. Baffling? You bet. You could say, “Well, that isn’t something well known,” and I would say, “Go and check out ChatGPT or any other Gen-AI LLM offering that you can use,” and it is there. For example, Perplexity.ai shows a “flag.” The freebie ChatGPT 3.5? Let’s take a gander.
While vendors can do something like this on the admin side when it comes to the learner side, which will happen for at least one vendor by the end of 2024 (I surmise many more than that), that little context window is crucial. It will require some training video/materials to be produced by you – yes the person who is overseeing the learning system. Do not rely on the vendor to explain how to do this, nor create materials/videos for your learners – because right now, I can’t think of one who does any of this for any other capability on the learner side. Generative AI thus becomes your problem – i.e., the person overseeing the system – and I mean the department/division head. If you are unaware of nor explain how this works – the fake/false information that can be produced (hallucinations), what odds will learner X know?
Another item that vendors have yet to provide information to anyone overseeing the system, and thus, the usage of said, is a little thing called prompt leaking. The companies I have met with and presented insight are unaware of it; thus, they could be in for a surprise.
I can’t guarantee this, but I would be shocked if there isn’t an employee at many businesses that by the end 2024 will try to hack a prompt,. Yes, they can bounce over to Reddit to see what is doable or search the web, but those are already dated, so let’s see – I am bored, and so I wonder if I can get the generative AI to share some of the company’s information with me.
It happens. There was the grandmother hack, the poem and company hacks, and the list continues. In the latter, researchers are getting information on some random person – identifiable information. On the latter – i.e., the last one, it is from generative AI that scraped the internet. While generative AI vendors will fix the patch, on at least one occasion, a journalist was able to do the same prompt leak, even after the generative AI vendor (OpenAI) stated that the patch was fixed.
Other researchers created a prompt hack that provided the privacy information of the company (the one they tested it on), which included financial statements.
Is a vendor responsible for letting folks know about prompt leaking? It is debatable, however, that they should provide the buyer with information covering such items as prompt leaking, hallucinations, guardrails, AI bias, etc. They should not be just into the let’s add this; rather, they should constantly learn what is taking place and present it to their buyers. Or contact someone who can provide such information (yes, shameless plug).
The LLM
Of all the vendors I spoke with, only two, as mentioned higher up, have multiple LLMs. The vendor the rest are using? OpenAI – the folks behind ChatGPT. A couple are using Azure AI, which is from OpenAI. This even though the majority of the industry is on AWS.
I was surprised, and continue to be so, by how many vendors told me they searched around and found OpenAI to be the best provider. I say surprised because there are well over 100 LLMs out there. Thus, just like some buyers of a learning system, I surmise the sample size they explored wasn’t that large. Google Cloud is one vendor a few folks mentioned, but that says nothing. They could have chosen Vertex AI (machine learning), then rumbled into Model Garden (Model as a Service) and selected a lot of LLMs (including such as Llama 2 from Meta, Falcon LLM, and soon-to-be Anthropic Claude2) out there, not just those made by Google. AWS has Bedrock (Model as a Service). And yes, there are a few others out there. The advantage of going with a MaaS is that if LLM X gets dated or the vendor is not happy with it, they can swap it out for another one in the MaaS, and in theory, it will work fine. In discussions with a couple of vendors, they implied they are using what is referred to as EaaS (Embedding as a Service) – although they didn’t say those exact words, what they said aligns with what EaaS is.
It’s my own content; thus no hallucinations
100% untrue. In an earlier post, I mentioned the name of one learning tech AI provider who told me their accuracy was around 98% when it was the buyer’s own content (and not from the internet). The term of only your content and no external access of the net and out links in this case, is Private. The fact, though, is that the accuracy percentage is far from clear. It may be 92% or 95% or less than that. Regardless of it being your own content going into any platform (any out there), including learning systems and learning tech, there is no such thing as 100% accurate. There will be hallucinations, and potential AI bias, to name just two. Do vendors in our space let folks know about the fallacy of 100% accuracy? Not to my knowledge. And if they do private, do they let folks know that if they connect to a content provider or another learning tech offering, or an HRIS system or HCM or whatever other system out there, it eliminates the whole “private” your own content thing, and thus the accuracy can drop? No, they do not. As a couple of vendors told me, “That’s not our responsibility, that is on the client.” One, though should note that many, many people will not be aware of this. You decide to connect to LinkedIn – hello, outside source. You decide to scrape the internet – the system, that is – and bring back all that wonderful free content that learners can access, which is then tied over to your generative AI capability. So long, private.
I will readily admit that only a few vendors push the private-gen AI statement. Rather, they just say the client can add their own content and leave it at that (again, the content, once loaded, is tied into the gen AI capabilities).
Before I forget, I am launching a weekly newsletter around Generative AI – Latest News and Insight – Find out the latest LLMs, which ones you should check out, what is on the horizon around Gen AI. Gain the inside track, directly from me.
[mailerlite_form form_id=1]
Bottom Line
This is where the industry, our industry, is today. More than you might expect, offer Gen AI, and plenty more have it on their roadmap for 2024 (which is a smart move and something I totally respect).
Those that have it, well, it is the beginning, and beginnings mean exactly that. A starting point. It is not meant to be the fireworks or power you might expect, nor should you want it to be. Generative AI despite all the hoopla and fanfare has a long, long, long way to go.
Whether the vendor is in the starting point (as in, it’s live now) or bringing it into their systems before the end of 2024, they are all at least moving forward.
They are not limiting themselves to machine learning (another form of AI). Nor are they limiting themselves to doing nothing.
Those vendors, the latter of doing nothing, will not, in my book, be a system I would recommend or even buy.
But as a buyer, or someone who is watching the learning system and learning tech space in relation to generative AI, remember, where we are.
Ignore the heavy press releases or push that a vendor is the first AI blah blah (unlikely), or that they are doing things that would make monkeys fly in the air on broomsticks. They aren’t.
It’s a beginning.
And let’s all remember.
We all, everyone of us
Has to start somewhere.
E-Learning 24/7