In this discussion session, I would like to identify critical components for a holistic approach to learning in neural networks together with other participants. The focus will be on how we can combine the best of the two worlds: offline training for creating the initial knowledge base and online training for adaptation and extension of this knowledge, without catastrophic forgetting.