Research direction: Intelligent Model Weight transfer between LLMs [R]
热度趋势
趋势数据积累中
百分比基于当前可用热度信号,而非评论数或独立用户人数。
Few days ago I feel like I need to get started with researching about LLMs. One thing which strikes the most in my mind, how we can reduce the time required for pre-training an LLM model to just few minutes. Right now the most efficient method that we have is knowledge distillation, which still takes time in response generation by the teacher model from prompts, backpropagation and training, to adjust the weights of student model making it to mimic the teacher model.
What if there is any way where we can adjust the model weights of an untrained model so that it becomes mathematically the same function as of the trained model.I want to figure out if there any such algorithm exist which would perform simple mathematical operations on the untrained model such that it becomes mathematically same function as the trained model.
If this become successful there is no need of training under distillation process or any conventional process, just few math operations on the untrained model, and then it's done, which would be taking few minutes. I need guidance and collaboration for someone who is working in this direction.