In my day to day work, I train models on data using R packages that have no extension for Bayesian priors. I will generally have a large dataset to start off with, and add new data as needed.
Any time I want to update the model, I have to train the entire thing from scratch.
Are there ways of mitigating the considerable and slowly-increasing time cost of re-training everything from scratch, when I am unable to use Bayesian priors in my model?
A couple of approaches have occurred to me. Model training generally allows for initial weights/parameters to be specified. Setting the initial weights to the weights of the previous model may be a start, but presumably you need to include the previous data, or else the model will move from the old weights to capture only the new data.
Does training old + new data using initial weights trained from old data decrease the training time appreciably? Are there any other practical considerations for dealing with this type of situation?