I developed a 270 million parameter language model entirely from scratch as an independent research project

推荐理由

这条记录涉及生成能力或端侧推理进展，适合跟踪模型效率、部署门槛和应用机会。

AI 摘要

该语言模型是一个拥有2.7亿参数的独立研究项目，其架构基于定制的Transformer。模型采用了旋转位置嵌入、RMSNorm、SwiGLU前馈层和分组查询注意力等技术。此外，该模型还包含一个为本地推理优化的自回归解码器。

The model is built on a custom Transformer architecture featuring Rotary Positional Embeddings, RMSNorm, SwiGLU feed forward layers, grouped query attention, and an efficient autoregressive decoder optimized for local inference.

Here is the Huggingface Spaces Demo link - https://huggingface.co/spaces/pranavupadhyaya52/WikiSmartBot

For anyone interested in the pretraining notebook, I've shared the link here - https://colab.research.google.com/drive/1cxRLxUPX_mT4nst-0xGdhctEdqdIlMDb?usp=sharing

主题标签模型发布端侧推理

原始关键词#independent#developed#parameter#entirely#language#research

查看原文reddit.com

单一来源，暂无交叉验证