cross-posted from: https://lemmy.intai.tech/post/72919

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts - Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

Related Article: https://lemmy.intai.tech/post/72922

  • @shotgun_crab
    link
    English
    4
    edit-2
    1 year ago

    Thread reader link is down (Twitter API limits I guess). Is there any backups?

    • manitcorOP
      link
      fedilink
      English
      4
      edit-2
      1 year ago

      its just a bit of a rant based on this, source article is still here and kicking. https://lemmy.intai.tech/post/72922

      also check out the recent Gerorge Hotz, Freidman podcast.

      its been kind of an open secret for weeks now, people are digging to try and prove it out.

      • @shotgun_crab
        link
        English
        1
        edit-2
        1 year ago

        Oh I see. Thanks, it’s interesting stuff