Can ChatGPT help us write FlexMeasures?

As you might have noticed, Generative AI is entering the scene, and next to answering questions or compiling information, it might also be used to write code.

Can ChatGPT help us write FlexMeasures, our opensource EMS for smart energy flexibility? In this post, you can see how it works and what my experience was. It might some day become a tool in our belt, but I’m careful.

Here is the notes to help you understand what I did, what ChatGPT did ― and what it didn’t do:

• First, I decided I want ChatGPT to add to an existing code base. That is where 98% of the programming work actually happens ― for instance, adding a new feature. I had also tried it to write a new piece of code from scratch (which was interesting, as well, but took some nudging, let me know in the comments if you want to see that, as well).
• I also chose a piece of code for which there should be many similar examples out there ― a REST API. AI lives on data, and this way I’m giving it a task which has data on the web to help it.
• I already have two endpoints, and I wanted ChatGPT to write one more: PATCH, for editing records. In the end, it wrote two more (PATCH and DELETE)! Delete is actually the fourth missing one, so that is awesome!
• You can see how I tell it what to do, add my code and then tell it where to create new code.
• Then, I change the setting for maximum length. This is a longer piece of code, so I needed him to think bigger 🙂 I keep the temperature at 0, as I had some bad experience with its playfulness in this example task.
• However, at first, ChatGPT crashes. Being an experienced programmer, I simply try again – and it works!
• At the 2-minute mark, you can see the resulting code (in green)!
• The created documentation and boilerplate code would save me some boring work. And it gets some stuff correct, for instance that PATCH needs a write permission (the existing code only had read permissions). But that’s just boilerplate. The logic is another thing …
• ChatGPT could not guess how I actually access records in the database. This is due to such context in existing programs usually being hard to infer. You have to read other files, in other packages, maybe even tests to find out how the data layer really works. The context of ChatGPT is simply what has been written above and what similar projects or people have written elsewhere. So ChatGPT opted to simply write “# TODO: Implement this“. At least it was honest here. I guess if I had turned up the temperature, I would have gotten wild ideas how FlexMeasures supposedly does this. Wild, but almost certainly wrong.
• Finally, I’ve used the code-davinci:002 model here. OpenAI just discontinued that. GPT-3.5 is the new model. I might try that another day.

For completeness, what about the case where we ask ChatGPT to write an API from scratch? That is actually where it shines more, setting up pieces of code directly.

Of course, I want code that fits with the rest of my project. I know what libraries I want to use, for instance.
In this video below, you see the elaborate prompt I created. The result is very cool (even though I’d need to fix some things, like the data type of account_roles, that should be a List).

Anyways, keep these two main points in mind:

1.  Prompt creation takes considerable time, as well
2. I don’t start new projects that often – this is for me an edge case (for some IT agency who works a lot of projects, it might be different)
3. You get one large file – if organizing code across files is what you want (usually you do, for instance to separate data access from logic, think MVC), then there is some work left. I can imagine that future versions are better at this, maybe.

Let me know if this was interesting or helpful. Certainly, these tools can play a role in the life of us programmers soonish. But it’s not as obvious as some people may think.

Also, who knows what OpenAI will do with this, how much they charge for it, what the IP situation of the datasets being used etc etc. Interesting times ahead!