Posted by Generative-AI which I have written about many times (and will in the future too, as it impacts our space), continues to be “hot.” The challenges include “quality,” “hallucinations,” “bias,” and “output.” There are a lot of products already out there for someone to choose from, and in this analysis I focused on the whole PDF (and it can include other documents depending on the platform) and the usage of text-based generative AI.
GPT-4 which is the “best,” LLM from OpenAI, now offers plugins (it is still in beta). Since it offers this option, I had to go and hit the test to find out which one does the best job, which one(s) are so-so and if there were any that totally failed with hallucinations? I then selected two standalones (i.e. they are not in the GPT-4 plugins) to see how they stood up.
Since ChatGPT is popular, although there are some early indications that the “novelty,” may be wearing off, it is still very hot with your employees (Congrats!). As such, if someone was to spring for the fee-based GPT 3.5 or GPT-4 (the only one with the plugins), and decided to just go and upload their PDFs, and then extract – well, they might be surprised by the findings. And if you as a learning system vendor is thinking or using GPT Model, and GPT-4 and are exploring adding the PDF extraction angle, then you are in for a real treat.
With the standalones, the ones I tried out are 100% free. One even has a chrome extension (shudders). Anyone who has access to a browser and is searching for a PDF AI offering can find these and many more.
The one – and it is a big one – to remember is that at least for now, when you upload your PDF and seek those summaries, tap into the premise of an answer engine (which reallly is it), that it is 100% text. You won’t see the PDF itself on one side for example, and the text on the other side – i.e. the answers to your questions. You only see the retorts.
The two standalones I tried though, did show the actual PDF on one side of the screen, and then the responses to my questions on the other side, with one offering such suggestions to select to start. Each of them then highlighted the text on the PDF to correspond to the answer, and included the page it came from. Sounds great, but surprise – each ran into issues with that – the highlighted text didn’t align to the question, and twice the page to which is corresponds to (based on the question) was wrong. If someone isn’t paying attention to this, they are going to assume that it is 100% correct, and therein lies the problem with generative-ai in general – it is not infallible.
On the Gen-AI PDF options (all products BTW you can find on the web), the results were all over the map. One of them, based on a question I asked produced a response that was 100% wrong – a total hallucination. Not one item was correct. And it wasn’t a fluke. When I went with a different question, it provided a response that was never mentioned in the PDF – as though it just picked it out of the blue.
The Vendors – With GPT-4 (I used as plugins)
- AI PDF – The web site, you can do nothing with it. So, no need to add the link.
- AskYourPDF
- ChatwithPDF
- MixerBox Chat PDF
- Link Reader – The web site of Link Reader is now called something else, like Sider or something like that. But Link Reader is readily available as a plugin, and you can find YouTube videos on how to use it.
Of that list, under “Popular Plugins” – AI PDF, AskYourPDF and ChatwithPDF where in the top list (along with others, that aren’t PDF) – thus think of it, as just popular in general.
The Test PDF
I went with my Learning System Buyers Guide – 18 pages full of very useful information. Easy PDF to upload, and then start right in.
The Questions
I tried to stay consistent here by may have rephrased it, with the same goal. For Link Reader once I saw the first hallucination, I then had to go another step to see if it was a “one-off” or if it happened again, with a very specific and different question – and yeah, it happened again.
What were the questions?
- Summarize or provide a summary of the Learning Systems Buyers Guide
- What are the key items around pricing?
- What are some pricing examples?
With Link Reader – whose big push is to find links that you can click and go to the web, within your documents – and PDFs, I had to really expand out – so I asked what helpful articles are in the buyers guide? There are 17 with links in the guide. Link Reader, the first time around provided seven, of which only two actually existed; and the second time around with the same question – five were provided, only two were in the PDF. It also noted that there were no links in the PDF. I went with a third question asking, “What Skills Based Measurement Platform does he recommend?” – The response was 100% hallucination – more on that later.
What I learned from each of the plugins (excluding Link Reader) was that their responses were not consistent – thus while one provided this type of summary, another went a different way. That’s fine as long as the information is relevant and on-point.
AI PDF
What is great about any of these, okay, I am referring to TextChat responses, is that you do not have to pose your inquiry as a question. You can just ask as a statement. The key though is if you want specifics, you have to go specific; asking for a summary is general – wanting to know pricing around AABD is specific.
Q: Summarize the Learning System Buyers Guide
AI PDF response:
Here’s a summary of the “Learning Systems Buyers Guide” by Craig Weiss:
- The guide provides insights and knowledge about various learning systems, aiming to help those with limited or zero knowledge about these systems make informed purchasing decisions (Page 2).
- The guide covers topics such as due diligence, pricing, and the importance of seeing a demo before purchasing a learning system. It also emphasizes the importance of understanding industry terminology and the fact that there are no refunds in the industry (Page 2).
- The guide discusses the concept of “Learning System Types” and explains that a Learning System is any system or platform where learning/training (via e-learning, aka online learning) is the core of the solution (Page 3-5).
- The guide provides a list of articles that will help those new to the learning system space or those who want to know more (Page 6).
- The guide emphasizes the importance of understanding the target audience of a learning system. It breaks down the market share into three categories: Combo (both employees and customers), Customer Focused, and Employee Focused (Page 7-8).
- The guide discusses pricing models in the industry, particularly the popular “Monthly Active Users” model. It explains how this model works and what to watch out for (Page 9-12).
- The guide recommends a three-year contract as the ideal duration. It argues that one year is too short to assess ROI, while anything longer than three years is risky due to potential changes in the vendor’s situation (Page 13).
- The guide concludes by discussing the importance of having a clear use case and business case when choosing a learning system (Page 14).
AI PDF for the plugins, was the only one which provided the page number next to the response. Plus, it was the only one to go chronically number ordered, rather than page four, then page two and so on.
That said, the first bullet point with the page number is wrong. The AI misinterpreted what I actually wrote, which was about buyer guides in general – that you find. The actual text
“I love buyer’s guides. There is something about gaining insight and knowledge around various products or offerings when you either have limited or zero-knowledge about these items, but you want to purchase one at some point.”
Not once does it refer to my guide, but I saw time and time again, various AI PDF programs saying otherwise. Something to recognize when using these offerings – in that if the writer does some generalities, the program can take it as meaning the specific document you are summarizing.
On the page 13, the only reference to the number of years is the last year. The other years and the explanation is is on the previous page. As for the clear use and business case, that starts on page 13.
The summary as a whole is just pulling down a header or extracting a line or two of text. Because you can’t see the PDF at the same time – screen wise, you would need to have both the GPT-4 window open and the PDF one too. Again, this is a downside of using a Plugin with the GPT-4 direct location. If it was to be in a learning system, well, uh, a finagling would need to be done to get it all on the same screen.
When I asked the question around key pricing information, overall, it did a solid job. I found the response to work for anyone who just wanted the highlights if you will. I went one step further by asking for some specific price examples (which is in the PDF). It responded with two examples, then added some more information, which had nothing to do with specific examples.
If you go to AI PDF web site, you can’t test it out there – the site is in beta mode. Thus, it is GPT-4 or nothing to use AI PDF.
AskYourPDF
I had high hopes for AskYourPDF, because it seemed so tantalizing.
Summarize the Buyers Guide oh great AskYourPDF:
The “Learning Systems Buyer’s Guide” by Craig Weiss is a comprehensive resource designed to help individuals and organizations make informed decisions when purchasing learning systems. Here’s a summary of the guide:
- Purpose: The guide aims to provide insight and knowledge about various learning systems, especially for those who have limited or zero knowledge about these items but are interested in purchasing one.
- Due Diligence: The guide emphasizes the importance of due diligence. It advises buyers to always check out the learning system, ask the right questions, and see a demo. It also warns against assuming that vendors understand industry terminology.
- Misconceptions: The guide highlights some common misconceptions, such as the belief that everyone already has a learning system, that everyone knows e-learning, and that talent management is the same thing as learning.
- Pricing: The guide notes that pricing can be high for enterprise and large enterprise buyers, especially when procurement is involved. It also discusses the concept of “bundled” and “range/band pricing.”
- Formal Proposal: The guide advises buyers to seek a formal proposal from vendors, which should include information about the vendor, a listing of some clients, features/functionality they want to highlight, the benefits of selecting them, and the formal pricing proposal.
- RFP/RFI Templates: The guide provides two RFP/RFI templates, one for employees and the other for customer learning/education. These templates can be used to find the right learning system for your needs.
- Helpful Articles: The guide provides a list of articles that can help those who are new to the learning system space or those who want to know more. These articles cover topics like learning systems with employees, learning system contracts, decision criteria before buying a learning system, LMS or LXP, and more.
If you start saying, “Hey, there are some similarities,” you would be correct. Both were wrong on “Purpose” to a slight degree. Yes the first part is correct, but the latter doesn’t exist on page two or anywhere for that matter. The rest is spot on. As with AI PDF it offered to answer any additional questions.
What I found most interesting was when I asked for the highlights, it nearly hit the same responses as summary, but with a couple of key differences, the biggest being “RFP Process” which would definitely be of interest to someone. It should have appeared as one of the summary items.
Between AI PDF and AskyourPDF, I liked the latter with the dark bold topic headers but missed the page numbers (although considering AI missed on a couple). Equally for me, the summary with AskYourPDF was better with the summary around specific items that I think many folks would find of interest.
Summary Winner – It was very close, but the bold definitely stood out and the headers; AskYourPDF wins.
Key Pricing Information
AskYourPDF presented six items – with some details. The six included the MAU (Monthly Active User) model, the important of no refunds, “up to”, and pricing difference between Enterprise and High Enterprise. I liked that it included the whole “up to” angle, because it is very relevant, and yes, MAU is by far the biggest takeaway here.
AI PDF also presented six items with a bit more details depending on which of the six. The Enterprise response and never telling your budget were the same as AskYourPDF. MAU was noted by presented differently. Proposal slid in here. Again, with AI PDF page numbers were included.
Key Pricing Information Winner – A close one. I’m calling Tie.
On the provide some pricing examples, AskYourPDF failed. None of the specific examples which had number calculations were presented; rather it was more of insight, which should have been included with the pricing information, albeit two actually appeared in the “pricing information.”
Between AI PDF and AskYourPDF, AI PDF wins this round.
MixerBox Chat PDF
I really hate nag screens, especially when they appear as text at the bottom of each response. MixerBox did this on every response. It was brutal. I refuse to present the information, but did you know that there are more plugins available at MixerBox?
MixerBox Summarize the guide!
The “Learning System Buyers Guide” provides valuable insights into the learning system industry. Here are some key points:
- The guide discusses the importance of understanding the vendor’s offerings, including their client list, marketing strategies, highlighted features and functionalities, benefits of choosing them, and their pricing proposal.
- It emphasizes the importance of customer support in the industry, suggesting that vendors should provide twice-a-year data on response times, follow-up, and the number of support calls.
- The guide introduces the concept of “Monthly Active Users” (MAUs), a popular pricing model in the industry. It explains that MAUs are typically based on one person who logs into the system at least once in a 12-month period.
- The guide provides a breakdown of the usage of learning systems, stating that the high range is up to 79% employees vs. 21% customers, with the usual range falling in the 55-65% mark towards employees. Some vendors in this category include Absorb, Learn Amp, Cornerstone, and Fuse.
Please note that this is a brief summary, and the document contains more detailed information. Also, if you’re interested in more tools like this, you might want to check out other plugins by MixerBox in the plugin store. One of them is the ‘MixerBox WebSearchG’ plugin, which lets you browse the internet effortlessly within the ChatGPT interface! For more information, you can visit the website of MixerBox’s plugins: MixerBox Plugins.
Alright, you saw the pitch of the plugin, but wait there’s more – I refuse to show the pitch on other responses.
Overall, the summary failed. An 18-page document, that has info on 17 of those pages, and you give me four points? At least they didn’t give the “lack of experience,” result.
Key Pricing Information
MixerBox did a bit of a better job here – with bolded headers (which it didn’t have for the summary) covering Bundled and Range/Band Pricing. But it missed with zero information around MAU – which is the key dominator in how vendors pitch the pricing aspect. I’m not sure I would put a Proposal here, but AI PDF did the same thing.
On the key pricing examples, MixerBox presented the same two items on Bundled and Range, and then added MAU, but again slide proposals here. The odd thing is that in the document it says, “here are some pricing examples,” and shows numbers. AskYourPDF figured this out, the others didn’t. Yes, you can argue that MB provided examples – by showing the model types, but model types and key pricing info to me are different things.
Current Tally
Mixer failed with summary, did so-so with pricing information, and so-so (I will give into the types), on examples. In other words, of the three, they are dead last. Trust me, they won’t be when this is done.
ChatWithPDF
Summarize CWP
The “Learning System Buyer’s Guide” provides comprehensive insights and advice on purchasing learning systems. Here’s a summary:
- Introduction: The guide is designed to provide knowledge about various learning systems to those with limited or no prior knowledge. It emphasizes the importance of due diligence, asking the right questions, and never assuming that the vendor knows the industry terminology.
- Learning Systems: The guide categorizes learning systems into three main categories based on their target audience: Employee Focused, Customer Focused, and Combo (both employees and customers). It emphasizes the importance of understanding the focus of a learning system as it can explain why they have certain functionalities and capabilities.
- Pricing: The guide discusses the industry’s pricing strategies, including “bundled” and “range/band pricing.” It also introduces the concept of Monthly Active Users (MAU) pricing, where an active user is defined as a person who logs into the system for that month or logs into the system and takes a piece of content.
- Contract Duration: It advises against contracts longer than three years due to the risk of the vendor being bought out or making changes to the system that may not align with the buyer’s needs.
- Use Case and Business Case: The guide emphasizes the importance of having a clear use case and business case when selecting a learning system. It encourages buyers to provide detailed scenarios of what an end-user will do in the system.
- Proposal from Vendor: When seeking a proposal from a vendor, it should be itemized, include setup pricing and support details, and match the agreed duration (e.g., three years).
- Negotiation: The guide advises never to reveal your budget to the vendor. The initial price quoted is often a street price or retail, and there’s usually room for negotiation.
The guide is a comprehensive resource for anyone looking to purchase a learning system, providing detailed insights into the industry’s practices and what to look out for when selecting a vendor.
Once again, the “The guide is designed to provide knowledge about various learning systems to those with limited or no prior knowledge,” retort appears. To be honest, when I asked for highlights, it was listed a couple that should have been here in the summary. The extraction of lingo as you can see with nearly all of them is my vernacular.
If CWP had used those couple of items found in highlights which could slide into summary, they would lead. That said, still promising.
Key Pricing Information
CWP started strong with the Bundled and MAU, but then went off the rails with the Proposal. There was no mention around Range/Band which would have been relevant far more so, that say negotiation (also presented).
Pricing Examples
CWP listed three and went into a deep dive with MAU far more than the other vendors. Bundled provided the information, although not the specific number example. The last item just didn’t fit here – negotiation.
The Tally – Going into the Final Stretch, it is neck and neck with AI PDF and AskYourPDF in the lead, ChatWithYourPDF just slightly behind and MixerBox being a nag, dead last.
Link Reader
I always talk about hallucinations, and sometimes I see eye rolls as if there isn’t a reason to press this as much as I do. Link Reader validates the whole hallucinations statement.
Link Summarize The Guide
I readily admit that Link started out strong – far better than the others (except the page listed one, only because it provided the page numbers)
The document you provided is a comprehensive guide titled “Learning Systems Buyer’s Guide” by Craig Weiss, who is the CEO and Lead Analyst of The Craig Weiss Group. The guide is intended to provide insights and knowledge about various products or offerings in the learning systems space, especially for those who have limited or zero knowledge about these items but are considering purchasing one.Here are some key points from the guide:
- Caveats for Buyers: The guide emphasizes the importance of due diligence when purchasing a learning system. It advises buyers to thoroughly check out the learning system, ask the right questions, and remember that there are no refunds in the industry.
- Learning System Types: The guide explains different types of learning systems, including Learning Management Systems (LMS), Learning Experience Platforms (LXP), Coaching Platforms, Mentoring Platforms, Learning Suites, and Talent Development Platforms (TXP).
- Pricing: The guide discusses the pricing models in the industry, particularly the Monthly Active Users (MAU) model, which is the most popular pricing model in the industry. It also warns about vendors who might attach a percentage increase in their contract each year under the guise of “inflation or some similar angle.”
- Contracts: The guide recommends a three-year contract as the most appropriate duration. It argues that one year is too short to assess ROI, while contracts longer than three years are risky due to potential changes in the vendor’s business.
- Use Case and Business Case: The guide distinguishes between a use case, which is what the organization wants to accomplish with the learning system, and a business case, which is what the business wants to accomplish with training/learning.
- Demo: The guide emphasizes the importance of asking the right questions during the demo and understanding the vendor’s customer support and service.
The guide also provides links to several articles that can help those new to the learning system space or those who want to know more. The articles cover a range of topics, from decision criteria before buying a learning system to turning training into a profit center.
When you compare this to all the others, you can clearly see the differences, plus some similarities. Overall, though, I felt it did the best job of all of them with this first part.
Link Reader pushes the whole “web links” err links to go the web thing, so I had to bounce right over to test it (and yes step away from the other items).
In the guide, it lists 17 different sites that you can click and go to the article/post. When I asked Link what the helpful articles were (using the exact wordage for that section), the first time it showed seven. The Power of the LMS Sandbox, Learning System Comparison Guide were just two of many that does not exist in my entire guide. One they noted, I gave it credit even though it is not under the helpful articles section but is a link to my Template. Thus of the seven, two were in the helpful articles, and one was pressented elsewhere in the guide. I should add that before it provided this information, it listed this statement:
“The guide provides links to several articles that can help those new to the learning system space or those who want to know more. Here are the articles mentioned:”
Since four do not exist, I think you can see where this is going.
I went ahead and redid it to see if the same thing would happen or if it would correct itself. Nope. Five were presented, including my personal favorites – the comparison and sandbox. Just to verify that there wasn’t an article that perhaps the AI was seeing as an article, I did a search for “sandbox” in the guide. It appears a whopping ZERO times. Once again though the AI saw FindANLMS which notes the text “comparison” of learning systems – as an actual helpful article. Still the results were not promising.
I then went one further step and asked Link Reader the following:
“What is the one skills-based measurement tool he recommends?”
Link Reader gave me a false answer – it said “Skillboard.” However, in the document, under the specific statement of the skills-based measurement platform I recommend it Skills Base. Skillboard never appears in the entire guide.
I didn’t go ahead and seek our pricing summary and examples, because once I experienced the multi-hallucinations, I decided, what’s the point?
Bottom Line
When I compared the standalones – ChatPDF and PDF.AI I ran into similar issues with page numbers being off and missed highlight areas. One produced information that seemed to come out of the blue. That said, because the PDF was visible on the left side, and the highlighting was either Blue (ChatPDF) or Yellow (PDF.AI) they did I think a better job, as a whole. Neither was perfect, and neither would be ones that I would rely on 100% as being exact without verifying everything as I go thru it.
And therein lies the rub.
You must go thru all of them – every one of them to verify that the information is correct. The human element must exist, and I can tell you, a lot of people ignore that and just go with the flow.
In my test, Link Reader generated information that came out of outer space, and with the recommendation, being false. MixerBox, just failed across the board, ChatWithYourPDF, AskYourPDF and AI PDF had hits and misses.
For right now, think of Generative-AI PDF products as more of an answer engine.
With you giving them
A helpful hand. And extra check.
To validate.
Right from Wrong.
E-Learning 24/7