Perhaps, in some way, adding memory (along with the equivalent of 'dreaming') will solve most of those problems.

LLM need to accumulate based on new knowledge, put emphasis on most recently learned material and feed it to itself in many different ways. Perhaps too, some meta-token based parallel learning mechanism is needed whereby grammars are learned from examples so that a weight is given to the grammar as well as the direct token match. When direct token match fails, the grammar is used and when both fail (very low probability predictors) then the input is memorized to be ‘dreamt’ about later.