Skip to content
#

lm

Here are 106 public repositories matching this topic...

ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.

  • Updated Sep 18, 2025
  • Python

Add this topic to your repo

To associate your repository with the lm topic, visit your repo's landing page and select "manage topics."

Learn more