|Tytuł||Reducing Catastrophic Forgetting With Learning on Synthetic Data|
|Publication Type||Conference Paper|
|Autorzy||Masarczyk W, Tautkute I|
|Conference Name||Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops|
Catastrophic forgetting is a problem caused by neural networks' inability to learn data in sequence. After learning two tasks in sequence, performance on the first one drops significantly. This is a serious disadvantage that prevents many deep learning applications to real-life problems where not all object classes are known beforehand; or change in data requires adjustments to the model. To reduce this problem we investigate the use of synthetic data, namely we answer a question: Is it possible to generate such data synthetically which learned in sequence does not result in catastrophic forgetting? We propose a method to generate such data in two-step optimisation process via meta-gradients. Our experimental results on Split-MNIST dataset show that training a model on such synthetic data in sequence does not result in catastrophic forgetting. We also show that our method of generating data is robust to different learning scenarios.