How to pronounce“autoregressive”in English

That's called autoregressive prediction, which is why those LLMs should be called autoregressive LLMs, but we just call them at LLMs.

We talked about models being autoregressive.

That is not to say that autoregressive LLMs are not useful, they're certainly useful.

And it's not going to be autoregressive prediction of tokens, it's going to be more something akin to inference of latent variables

We feed that to basically an autoregressive decoder, which can be very simple, that turns this into a text that expresses this thought.

And they figured out that a completely convolutional model can do autoregressive modeling as long as you do masked convolutions.

Do the same thing as sort of the autoregressive LLMs do, but for video.

- I would love to sort of linger on your skepticism around autoregressive LLMs.

So does it make sense to you that autoregressive LLMs are able to be so damn good?

And that's what you can build an autoregressive LLM from.

but the blueprint of those systems will be extremely different from autoregressive LLMs.

- Abandon autoregressive generation.

- No, there's one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive ones, but including the BERT style bidirectional ones,

So because of the autoregressive prediction, every time an LLM produces a token or a word,

text diffusion models might use transformer architectures, it's not an autoregressive transformer.

And like, you know, our autoregressive models and diffusion models ended up winning.

And then you can decode it into whatever you want in the raw input space using autoregressive or diffusion doesn't matter.

And then there's a special case of it, which is the autoregressive trick where you constrain the system to not elaborate a representation of the text

But again, the same intuition you're applying to saying that autoregressive LLMs cannot have a deep understanding of the world,

But they do have trade-offs, and nothing has yet replaced the autoregressive transformer as the state-of-the-art model.

But if we talk about the state-of-the-art, it's pretty much still the transformer architecture, autoregressive, derived from GPT-2 essentially.

- Yeah, so pre-training, I'm using some of the same words to really get the message across, is you're doing what is called autoregressive prediction

- Yeah, and now he's saying something else which is, he's saying autoregressive models might be a dead end.

Because if I were to take your wisdom and intuition at face value, I would say there's no way autoregressive LLMs