On human and machine intelligence

Prelude

Since this blog is mainly about machine learning (ML), I thought it would be appropriate to start with a general discussion about various kinds of ML algorithms and their relation to human intelligence. Don’t worry, I am not a psychologist, nor a neurologist. My knowledge about actual human intelligence is quite limited. What I am referring to as intelligence here has a much more practical meaning. I hate titles in the form “X is all you need” too, which makes having chosen the following title even more ironic:

Human intelligence is all you need!

Consider the following qualitative plot (and beware of any qualitative plots, even the correct ones are always misleading).

Notice that the term intelligence is within quotation marks, as the usage here refers to something else than what you might be used to. On the vertical axis, human intelligence describes the amount of “work”, the human has to conduct to achieve good results with the corresponding methods. Although I am only considering supervised algorithms here, similar arguments can be made about other ML algorithms from various domains.

For example, consider the generalized linear model (GLM) and ist special case the linear regression. One needs to extract good features, and has to have some knowledge about the distribution of the target variable as well as its relationship to input features, thus the model family. Although these models are ranked quite low on the horizontal model capacity axis, this does not mean they are bad models. It merely means that the number of different functions are smaller, which are representable with a fixed size of these models. Actually, given sufficient human or user ‘intelligence’, i.e. correct assumptions, they are the best models and often achieve high and even the best ranks on kaggle competitions (see for example here or here). Notice that although the models are linear, some features are polynomial or sinusoidal (i.e. Fourier), which makes these solutions non-linear, but in any case, these simple models can beat more complicated ones, given enough human input.

However, these models sometimes require superhuman ‘intelligence’, as the perfect features or the model family may not be as clear or easy to derive. In this case, we can use the more ‘intelligent’ models, i.e. ones with a higher capacity. However, the more capacity a model has, the more data it will often need. The models in the second category. These models seek to reduce the required human ‘intelligence’, specifically the exact model family and to some extent features, by some smoothness or local convexity assumptions about the underlying function. These models tend to scale with data to some extent, but they suffer from having a small data set, compared to the linear models.

Going up the model capacity scale, we see that the required human ‘intelligence’ decreases along with the assumptions. Multi layer perceptrons (MLP) drops most of the assumptions and increase the capacity greatly compared to the previous group of methods. Nevertheless, the requirements about data as well as other difficulties in training increase also greatly. To overcome some of these problems, various inductive biases are used in structures. Residual networks (ResNet) were revolutionary for their times and they allowed training much deeper models. The skip connections are still used in many architectures. Moreover, long-short term memory (LSTM) and convolutional neural networks (CNN) exploit the locality assumptions about their inputs; they assume that the features that are close to one another are more related to each other. This allows them to learn more meaningful representations compared to feed-forward architectures. Nonetheless, all of these architectures are closely related to MLPs, as all latent dimensions are combined with each others.

Finally, there are transformers, which deserve their own category. Most of the current work about large language models use transformers. Their success influenced other domains such as vision to try them out too. They drop the inductive biases, introduced by humans through LSTMs and CNNs. Instead, they try to learn their own inductive biases through the attention mechanism; they neither assume every input feature to be related to every other one as in feed-forward networks, nor they require a locality assumption between the related and relevant inputs. Thus, they reduce the human input compared to previous methods and instead gain the intelligence in a more self-supervised way, even for supervised tasks. However, they require quite a lot of data. This is less of a problem for the language domain, but more of a problem for other domains such as computer vision or tabular data sets. Not only the amount of data, but also the amount of compute makes them quite expensive endeavors. Nonetheless, they often achieve state-of-the-art results for those who can afford to use them. Moreover, they seem to enable a better scaling compared to the previous frameworks.

So what?

Are the transformers the solution to all our problems? Should we abandon linear models, or kernel-based methods entirely? In this post, I tried to convince you otherwise. All of these models have their place and all of them are valuable. I hope to have shown that using the qualitative plot above. There is also an argument about the infamous “No-free-lunch” theorem, but most of the modern researchers seem to have (to some extent rightfully) a problem with this argument, so I tried to refrain from it. Nevertheless, knowing about all of these approaches and choosing the most appropriate one should yield the best results and I hope to have given you a rough guide from one perspective about how you can eliminate the inappropriate ones. I hope I could also motivate you to learn about the ones that you have been ignoring until now. Finally, please listen to Yannic, my favorite youtuber, talk about why the transformers are valuable but also are not the final solution to all our scaling problems, which also summarizes my thoughts quite well.

If you find any typo or other mistake, please feel free to contact me. It would make me very happy to know that someone else is reading these. See the about page for contact information.