The first “aha” moment for large language models (GPT, Claude, Gemini) is passing, and it’s becoming clear that they are stuck in a trap of mediocrity. What are the reasons for this, and how can we overcome it?
Reasons for mediocrity
In machine learning theory, the “no free lunch theorem” states that there is no single “best model” for every possible problem.
Although modern language models were trained on “good answer examples” created by specialists, these examples covered a wide range of fields without prioritizing any specific one. Therefore, the model is trained to provide the best average answer across all domains, rather than the absolute best answer for a specific field.
An AI trained this way becomes a victim of its own design – if you try to “please everyone all the time,” you are almost by definition destined to be mediocre.
A model meant for every use case must also be very “average” in its language – always moral, inclusive, and sticking to the rules. Since new and interesting things usually happen at the boundaries, it is difficult for a language model to argue about them.
How to help language models move beyond mediocrity
To achieve high-quality results with a language model, there are two main paths: A. teaching with examples and B. creating a custom solution.
Teaching the language model using examples
When language models are trained, they are shown examples of questions and correct answers, and math is used to make the AI mimic the response pattern.
If you want to reach a high level with AI, you must do the same – provide examples of ideal answers and then ask the model to mimic them. How do you do it?
The first step is gathering questions and ideal answers. The rule here is: the more examples, the better, but remember that 10 excellent examples are better than 20 mediocre ones.
The easiest way is to provide sample questions and ideal answers as a response guide for the model to use. Technically, this is called “inference time learning,” which directs the model to activate a specific part of its logical ability.
Doing this well requires some learning and knowledge of the tools, but it’s nothing that a thorough training course couldn’t fix.
If you can gather 50 or more examples, it’s worth creating your own fine-tuned model. In fine-tuning, a special version of the AI is trained to automatically generate answers in a way that matches your examples. To do this, explore the options on your chosen AI platform or ask a development partner for help.
Creating custom solutions
If you want to fully automate a task or if the task is too complex/large for existing tools, you need to create a custom solution.
For our clients, we’ve seen the need for custom solutions in situations where a language model needs to read the company’s entire document database to provide answers, or when they want to fully automate a workflow (e.g., extracting info from emails).
Custom solutions have three main advantages over off-the-shelf tools:
- Quality can be taken to a very high level,
- Automated tests can be created for them, and
- They can be connected to various IT systems (the AI reads data from one system and automatically writes the result to another).
Summary
In conclusion, because of how they are trained, modern language models are “mediocrity machines” and can disappoint professional users. However, this can be overcome by teaching the AI with examples to respond the way we want, or by developing a custom solution designed for a specific use case.
