Open Source GPT-NEO & GPT-J

Picture from Unsplash

GPT-3 is the Deep Learning model that was presented a few months ago and that amazed the world with its ability to generate text, translate, perform arithmetic operations, and most importantly, perform tasks for which it has not been specifically trained.

Now, a Discord community, EleutherAI, has trained several models of GPT-3. And they have come up with two very interesting versions that we will explore here:

GPT-Neo

As stated on their web:

GPT⁠-⁠Neo is the code name for a family of transformer-based language models loosely styled around the GPT architecture. Our primary goal is to train an equivalent model to the full-sized GPT⁠-⁠3 and make it available to the public under an open licence.

GPT⁠-⁠Neo is an implementation of model & data-parallel GPT⁠-⁠2 and GPT⁠-⁠3-like models, utilizing Mesh Tensorflow for distributed support. This codebase is designed for TPUs. It should also work on GPUs, though we do not recommend this hardware configuration.

GPT-Neo currently has two versions, one with 1.3 billion parameters and one with 2.7 billion.

GPT-J-6B

In parallel, a version of GPT, called J-6B, has been developed that performs like the 6 billion parameter version of GPT-3. This would result in the closest open-source model to GPT-3 performance.

In fact, it can be tested on the following web site

The tests yield such interesting results as:

Q: If you were the moon, how would it feel to land a spaceship on you?

A: That’s the task (or, rather, failure) which NASA’s next big moon mission will attempt. NASA’s Lunar Atmosphere and Dust Environment Explorer (LADEE) mission launched on an Atlas V rocket from Cape Canaveral Air Force Station in Florida at 7:03 AM EDT on Tuesday, with a planned landing at 9:03 AM EDT. (For that keeping score, the landing was pushed back several times and is now scheduled for about 2 PM EDT.)

“Despite the challenging conditions, the team at NASA’s Ames Research Center did a great job in designing, building, and testing LADEE,” said Ames Research Center Director Dr. Gary Glick. “LADEE is now on the surface of the moon and about to begin a…