Microsoft Research has upped the game with an even smaller model. phi-1 is a transformer based model with just 1.3 billion parameters.

cross-posted from: https://lemmy.world/post/422633

I wonder if high quality datasets is the future as opposed to using internet scraped data that might produce lower quality output. Either way, neat model!