But it's becoming increasingly clear that LLMs based on the transformer model will never be able to scale their context much further than the current frontier, due mainly to context rot. Taking advantage of greater context will require architectural breakthroughs.
Will it though? The human mind can hold less context at any one time than even a mediocre LLM. The problem isn't architecture. It's capturing context. Most of it is in a bunch of people's heads and encoded in the physical world. Once it's digitized and accessible through search, RAG, or whatever, the LLM will be able to use it effectively.
