ChatGPT is learning new tricks. OpenAI introduced Code Interpreter with GPT-4

Last week OpenAI made an important announcement that breathes new air into ChatGPT’s sails. Developers introduced the Code Interpreter extension, available to all ChatGPT Plus members.

This plugin allows the chatbot to work with uploads from users, which gives it entirely new capabilities. Beyond just its database, now ChatGPT can get information from your images, your videos, your uploaded documents and PDFs and data tables. The chatbot can now make sense of a whole new slate of formats, including CVS and JSON.

You can give the AI precise instructions on how to handle your uploads, in natural language, just like you’ve been doing so far.

Data analysis

ChatGPT has the capacity to analyze and understand data. It will perform complex data transformations, statistical analysis, and visualization. It looks at graphs, creates graphs from tables or the other way around. More importantly, it finds connections within that data. This new feature is where the “AI can see patterns at scale” feature shines.

You can practically talk to your CVS and get insights into the composition and patterns of datasets.

Practical examples of how this works in the wild:

• Clean up your spreadsheets and reorganize them without going through the tables manually. Visualize data: generate graphs based on your CVS and JSON tables, plot data history. Find patterns in your data and discuss them with ChatGPT. You can identify trends and perform detailed analyses.
• You can perform an in-depth SEO analysis of your data and ask ChatGPT to find you patterns. It might notice some areas of your website fare better than others, it can notice seasonal trends or dead ends. The biggest and most impressive trait is that you can introduce the data, and ASK ChatGPT questions in natural language.
• Provide the Code Interpreter with map coordinates, and it will generate dynamic visual representations. ChatGPT can now create an animated GIF that will show you changes over time, differences between locations or PoV navigation. For example, if you have data about SAT exams in different regions, the Code Interpreter can generate a GIF that helps you visualize the rate of success over time and regions. Each frame of the GIF could represent an increase or decrease in results for different years, and connect it with investments.

Computer vision

Computer vision trains computers to interpret and understand the visual world. You can give ChatGPT images and videos and it will recognize faces, detect and recognize objects, reconstruct scenes. This gives a whole new world of features to ChatGPT.

You can now tell it to interpret and comment on an image, tag the different faces in a video, and detect and count objects. ChatGPT’s computer vision is not yet 20:20. You can ask it to describe an image, and it will be a hit-and-miss. It even misses the mark on colors or counting objects. When we asked what it sees in one picture, ChatGPT filled in many of its blanks with hallucinations. It saw a bicycle when there was none, it listed colors that weren’t there and described sentiments that weren’t apparent. It kinda reminded me of my ex telling me what he thought I wanted to hear. On a separate note, my ex was 30, ChatGPT is a few months old. Give it time.

Practical examples of how this works in the wild (once it’s fully trained):

• Well-trained computer vision can help video editors sort through rough cuts. When you have over 10 hours of video from a conference and you need to find those 3 interviews where the host didn’t stutter, you learn to appreciate it.
• It can help you process galleries, and sort and browse photographic data. Think of the last time you wanted to filter your search for images only containing your office building, on three separate HDDs.
• Once AI learns how to look at elements of images separately, the next step is editing them separately. Would you like to find all pictures of your family and make them all look like they’re not having a miserable time together? Fret not. If AIs can identify faces, they might actually get a good photo of your teenage daughter looking excited.

Document reading and analysis

The bot now has more information to work with, beyond just talking to you about whatever is in its database. Upload your own documents, and, within that conversation, ChatGPT will have extra data. Computer visualization also grants it OCR capabilities, so it will turn any images into text.

It becomes a lot easier to train ChatGPT on your own data. Considering the bot is frozen in 2019, this is more than just an anticipated feature. In a professional setup, this changes ChatGPT from a (very cool) toy to a customized work partner.

Practical examples of how this works in the wild:

• Training the model on your own data means the chatbot becomes whatever you want it to become. It might turn it into a professional writer that understands your business better. It might mean it does research together with you or works within the framework you want it to. It might mean it co-writes a novel, within a certain world construction.

Creating software

There are multiple users who already created small experimental game samples in under 20 minutes. From ideation to debugging, ChatGPT was prompted to go through the entire process. ChatGPT will use your own libraries, so you can add images, graphic models, samples, and pieces of game mechanics.

One mind-blowing prompt comes from Kris Kashtanova. Here is what he writes, word by word :

“If something doesn't work ask GPT-4 to fix it (you can copy an error and paste in GPT-4) like you would ask a human programmer.” This puts ChatGPT in an internal dialogue with itself to fix the code, which is a whole new level of creativity.

In closing

ChatGPT is becoming more powerful in a professional setting. At the moment, half the people I know are “planning to test it someday” and “heard it was pretty cool.” The other half is using it as a glorified search engine with a nice personality. There are very few people who do deep dives and explore what AI can do. I have a feeling they’re the ones who are going to survive the robot uprising.