Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning Quotes

AI is going to bring a new renaissance for humanity, a new form of enlightenment, if you want, because AI is going to amplify everybody's intelligence.

I don't like to call [human intelligence] AGI because human intelligence is not general at all.

[Can a commercial entity] produce Wikipedia? No. Wikipedia is crowdsourced because it works. So it's going to be the same for AI systems, they're going to have to be trained, or at least fine-tuned, with the help of everyone around the world. And people will only do this if they can contribute to a widely-available open platform.

Every reasonable ML technique has some sort of mathematical guarantee. For example, neural nets have a finite VC dimension, hence they are consistent and have generalization bounds... every single bound is terrible and useless in practice. As long as your method minimizes some sort of objective function and has a finite capacity (or is properly regularized), you are on solid theoretical grounds.

Many of the papers that make it passed the review process are [good but boring] papers that bring an improvement to a well-established technique... Truly innovative papers rarely make it, largely because reviewers are unlikely to understand the point or foresee the potential of it.

The analogy I've been using is the fact that perhaps an equivalent event in the history of humanity to what might be provided by Generalization of AI assistant is the invention of the printing press. It made everybody smarter.

The direction of history is that the more data we get, the more our methods rely on learning. Ultimately, the task use learning end to end. That's what happened for speech, handwriting, and object recognition. It's bound to happen for NLP.

[AI progres is very dependent on Moore's law.] The one thing that allowed big progress in computer vision with ConvNets is the availability of GPUs with performance over 1 Tflops.

Don't get fooled by people who claim to have a solution to Artificial General Intelligence, who claim to have AI systems that work "just like the human brain", or who claim to have figured out how the brain works (well, except if it's Geoff Hinton making the claim). Ask them what error rate they get on MNIST or ImageNet.

The vast majority of human knowledge is not expressed in text... LLMs do not have that, because they don't have access to it. And so they can make really stupid mistakes. That’s where hallucinations come from.

[Large language models] require enormous amounts of data to reach a level of intelligence that is not that great in the end. And they can't really reason. They can't plan anything other than things they’ve been trained on. So they're not a road towards what people call “AGI.”

My problem with sticking too close to nature is that it's like "cargo-cult" science... I don't use neural nets because they look like the brain. I use them because they are a convenient way to construct parameterized non-linear functions with good properties. But I did get inspiration from the architecture of the visual cortex to build convolutional nets.

I try to stay away from all methods that require sampling. I must have an allergy of some sort. That said, I am neither Bayesian nor anti-Bayesian... I think Bayesian methods are really cool conceptually in some cases... but I really don't have much faith in things like non-parametric Bayesian methods...