How to use a custom GPT to debug legacy systems

A few months ago, an ancient system I am responsible for failed. It's an old Java server that sits in front of an Android app. The Java server behaves as a state machine, which is controlled by the app clients. The clients do not have access to control everything about the condition, but it does have a third subsystem which is aptly called "the pilot".

The source code for the pilot is long-gone, and since it controls much of what happens, it can be challenging, to say the least, to debug every time an error occurs.

The error that had occurred was that the app's camera function had suddenly started failing randomly. A mistake we thought we had encountered before. The challenge last time was that the camera function depended on receiving a message from the pilot, but the pilot's IP address had changed, because in this system all IPs are obviously hardcoded (and at the same time dynamically allocated).

A quick inspection showed that this time all IPs were correct.

The code hadn't been updated in a long time, so it was unlikely that there was a new bug in the software itself.

Debugging further in this system is extremely challenging. The challenge lies in the fact that nobody knows the system's source code particularly well, and some of the source code is, as mentioned, inaccessible. On the positive side, the server logs a lot. Very much. The logs are not particularly easy to read, as they largely only log which commands have been sent to the Java server, and possibly which keys in the state have changed:

Combined with the Android Logcat logs for the app, this gives a fairly complete picture of what is happening at all times. But there is a lot going on, and it is difficult to track the state of both the server and every client. That makes it challenging to pinpoint exactly what is going wrong when an issue occurs. While working on the camera bug, however, I had an idea: What if I train a custom GPT on all the logs from the server and Logcat, along with larger parts of the available source code?

I went to chat.openai.com, and created a new custom GPT. I named it after the project's name, and gave it the following instructions:

Before uploading any project material, confirm that your organization permits the selected AI workspace and that its access, retention, and data-handling controls meet the project's requirements. Remove API keys, tokens, credentials, personal identifiers, customer data, and any other confidential information from source code and logs. If that cannot be done safely, do not upload the material.

After completing those checks for this project, I uploaded a selection of relevant, sanitized source code files to the GPT's "Knowledge" base.

With this as a starting point, I began to question the GPT about my specific problem. I told the GPT what happened, what I expected to happen, and informed of the previous similar errors we had had (which turned out not to be the problem this time).

I uploaded a large set of sanitized Logcat and server log files to the approved workspace. After digging through these files, the GPT highlighted a portion of the logs that I had overlooked. It indicated a problem with a camera library coupled with revoked permissions on Android. The library was replaced with some simple homegrown code, partly generated with ChatGPT, and the error disappeared.

All this was done in about an hour after I started setting up the GPT. I'm guessing that without this assistance I would have spent a good many more hours trying to figure out what the problem was.