Never start with a chiplet design; go for the biggest chip you can imagine and build it. - Jensen Huang

If you were building a chip company and you were taping out a chip, the tapeout of a chip is around $100 million, just the tapeout. Not to mention the tools, which are probably another $100 million, and not to mention all the engineers, all the systems you’re bringing up, things like that. In order to build one of our chips, it’s a few billion dollars. And we’re just one chip company. There’s a whole bunch of chip companies. When they tape out a chip it’s no less than $25 million. Writing, developing a large language model–taping out a chip these days, what the software industry is learning is that building these large language models is kind of like taping out a chip.

The computing fabric that compute connects processors needs to be quite high speed. The faster the processors, the greater need for high speed computing fabrics and so it’s a matter of scale and the effectiveness of the scale. For example, if you want to increase to 1000 processors, the linearity of that scale up would be less linear and it would plateau earlier if the interconnects were slower and so that’s basically the trade-off. It’s just a matter of how far can you scale and what is the effectiveness of the scaling, the linearity of the scaling.

Everyone is a programmer now. You just have to say something to the computer.