You can now train ChatGPT on your own documents via the API

OpenAI announced Tuesday that it has fine-tuned GPT-3.5 Turbo, the artificial intelligence model on which the free version of ChatGPT is based, using its API. OpenAI claims that in certain scenarios, the fine-tuned model can perform as well as GPT-4 at a lower cost.

Fine-tuning in artificial intelligence refers to the process of taking a pre-trained neural network (e.g., GPT-3.5 Turbo) and further training it on a different data set (e.g., your user data) that is typically smaller and possibly related to a specific task. This process builds on the knowledge gained by the model during the initial training phase and refines it for a specific application.

So essentially, fine-tuning trains GPT-3.5 Turbo to work with user-generated content such as project documentation or any other written reference. This can come in handy if you want to build a GPT-3.5-based AI assistant that is familiar with your product or service, but has no knowledge of it in its training data (which, remember, was taken from the web until September 2021).

"Since the release of GPT-3.5 Turbo, developers and companies have been asking for the ability to customize the model to create a unique and differentiated experience for their users," OpenAI wrote in its promotional blog. "With this launch, developers can now perform fine-tuning to make this model better for their tasks."

While GPT-4, a more powerful relative of GPT-3.5, is well known as a generalist that adapts to many subjects, it is slower and more expensive to execute. OpenAI offers fine-tuning 3.5 as a way to get GPT-4-like performance in a specific area of expertise at a lower cost and faster execution time. "Early tests have shown that a fine-tuned version of GPT-3.5 Turbo can match or even exceed the basic GPT-4 level capabilities in some narrow tasks," they write.

In addition, OpenAI claims that fine-tuned models provide "improved controllability," meaning better adherence to instructions; "robust output formatting," which improves a model's ability to consistently output text in a format such as API calls or JSON; and "custom tone," which can give a chatbot a personalized flavor or character.

OpenAI claims that fine-tuning allows users to reduce the size of hints and save money on OpenAI's API calls, which are paid per token. "Early testers reduced the size of hints by up to 90% by fine-tuning the instructions in the model itself," OpenAI says. Right now, the context length for fine-tuning is set at 4,000 tokens, but OpenAI says fine-tuning will be extended to a 16,000-token model "later this fall."

You have to pay for using your own data

By now you're probably wondering how to use your own data to train GPT-3.5 and how much it costs. On its blog, OpenAI describes a simplified process that shows setting up a system hint with the API, uploading files to OpenAI for training, and creating a fine-tuning job using the curl command-line tool to query the API's web address. Once the fine-tuning process is complete, OpenAI says the customized model is immediately available for use with the same rate restrictions as the base model. More information can be found in the official OpenAI documentation.

Of course, you have to pay for all this, and the price is divided into the cost of training and the cost of use. GPT-3.5 training costs $0.008 per 1,000 tokens. In the usage phase, API access costs $0.012 per 1,000 tokens for text input and $0.016 per 1,000 tokens for text output.

By comparison, the basic 4k model GPT-3.5 Turbo costs $0.0015 per 1,000 tokens of input and $0.002 per 1,000 tokens of output, so fine-tuning the model costs about eight times as much. And while the GPT-4 model with 8K context is also cheaper at $0.03 per 1,000 tokens of input and $0.06 per 1,000 tokens of output, OpenAI still claims that money can be saved by reducing the need for hints in the fine-tuned model. It's a stretch, but in narrow cases it may be applicable.

Even if it costs money to train GPT-3.5 to work with custom documents, it might be worth it for some people - if you can keep the model from making stuff up. Customizing documents is one thing, but trusting the accuracy and reliability of GPT-3.5 Turbo results in a production environment is quite another. GPT-3.5 is well known for its propensity to misrepresent information.

Regarding data privacy, OpenAI notes that, like all of its APIs, data sent to the fine-tuning API is not used by OpenAI (or anyone else) to train AI models. Interestingly, OpenAI will send all fine-tuning client training data through GPT-4 for moderation purposes using its recently announced moderation API. This may explain some of the cost of using the fine-tuning service.

And if 3.5 isn't enough for you, OpenAI claims that fine tuning for GPT-4 will happen this fall. From our experience, GPT-4 doesn't give out bugs as much, but fine tuning this model (or the 8 models rumored to work together under the hood) will likely cost a lot more.