The White House recently convened a group of executives of leading artificial intelligence firms to discuss safety concerns. FTC chair Lina Khan, meanwhile, wrote an op-ed in The New York Times calling for preemptive regulation due to “several risks” stemming from AI being dominated by “large incumbent technology firms.”

But calls for quick regulation are based largely on narrow views of current implementations, such as ChatGPT, and uses of the revolutionary large language models (LLMs), such as GPT-4, that power them. Some kind of regulation may turn out to be necessary — and some questions, like those related to ownership and use of intellectual property, surely demand answers. But we believe regulation at this early stage risks foreclosing the real economic and social advances that AI might bring. And responding to Chair Khan’s concern, we believe that regulation now is also likely to lock in the current AI leaders, reducing potential competition rather than increasing it.

LLMs and Their Limitations

Ever since the research lab OpenAI stunned the world with its release of ChatGPT in November, large language models have captured the public imagination. Their potential is enormous. But there are metaphorical asterisks: they have a tendency to make things up or to introduce biases while sounding confident and authoritative. Policymakers worry that this “botsplaining,” or “hallucinating” as professionals call it, is creating trust and safety issues, fueling calls for regulation.

The push for regulation has two problems. First, it’s based partly on misconceptions of how LLMs operate, and second, it focuses on LLMs as if ChatGPT is the way we’ll always use artificial intelligence. Dealing with trust, safety and bias is important, but we need to address them in ways that don’t block future productive uses of these new tools or, paradoxically, prevent the organic development of AIs that better avoid the problems in the first place.

A key misconception is that the way to address hallucination and bias is to improve the underlying datasets used to “train” the models or to increase the number of parameters in the models. That will help, but it is costly in terms of data and the additional computational power needed, and experts disagree on how much more progress we can make simply by doing what’s done now, only bigger.

Layers

The solution to these problems, and the way LLMs are likely to contribute in the near term to economic growth, is to think of large language models as inputs into a process rather than a straightforward query-and-response interface. LLMs are bad at separating fact from fiction, but good at identifying the important part of a query. After all, that’s what they’re trained to do.

It is this insight that motivates a multi-layered approach to getting more useful responses from LLMs. In a multi-layered approach, the user prompts the LLM the same way they do now. But instead of immediately returning an answer to the prompt, the LLM’s response is used to search a database of “embeddings” — a way of storing the semantics, not just the raw words, of a text. The results are then appended to the original prompt, and the LLM produces a new answer.

Why is this approach better? Consider a common LLM hallucination today. When asked for research papers on a particular topic, the LLM often simply invents them. The titles might sound like real papers, and the claimed authors are scholars whose work is in the general field, but the papers don’t actually exist.

A multi-layered approach can handle this problem. It takes the user’s query and performs a search of the research literature. A key point is that it searches over a well-defined body of texts. A tool for medical research might search over a collection of medical research. A tool to aid lawyers could search over collections of laws, case histories and law reviews. It could then can summarize the actual literature or provide a list of actual citations based on its search.

Those aren’t just theoretical examples. A tool called Elicit from a non-profit group called Ought shows the potential. Built on top of multiple language models, Elicit performs deeper searches of the academic literature than tools like Google Scholar alone can. According to its website, Elicit aspires not only to find research papers, but also to write a literature review from them. For economists, that’s akin to automating the prestigious Journal of Economic Literature so that it can produce a top-flight review of the research on any topic on demand.

Other groups, like the open source Llamaindex, are working on ways to make this process simple for anybody to do themselves. Just add the corpus of text you want to work with and go.

A similar approach is to allow the LLM to query an external application directly. In this vein, OpenAI is introducing ChatGPT plug-ins, which allow developers to describe their data to ChatGPT, which can then return the information, either directly to the user or for further processing.

Using models as pieces of larger system has other advantages, as well. AI developers and researchers will tell you that LLM models are essentially black boxes — once started, they don’t leave obvious tracks to follow — making it difficult to determine sources of bias. But because multilayer systems may break problems into smaller parts, it becomes easier to determine the sources of bias and address them.

A Pause May Refresh

None of this means regulation is ultimately unnecessary. But it does mean we should take a beat.

We don’t really know how LLMs will be used. People have adopted it faster than any other technology in history, and each week seems to bring new advances and changes. Regulation risks locking in current uses, which may ultimately bear little resemblance to how these tools end up contributing to society — or being made redundant before it can even be implemented.

A thoughtful approach to AI policy should focus how the technology will evolve, rather than on what is currently possible. Rather than being distracted by attempts to create the perfect chatbot, at this early stage of AI integration policymakers should seek a deeper understanding of how the technology works and what potential benefits and problems will develop as we gain experience with it.