First Impressions of ChatGPT

I wasn't prepared for ChatGPT. OpenAI released it on November 30th, it hit a million users in five days, and I signed up expecting the usual chatbot experience. Ask it something, get a mediocre response, close the tab. That is not what happened.

Not a Chatbot

I've used chatbots before. Client service bots that redirect you to a FAQ. Voice assistants that mishear every third word. The category had trained me to expect a narrow tool pretending to be smarter than it is.

ChatGPT doesn't feel like that. Ask it to explain how a bank's net interest margin works, and it gives you a coherent, accurate explanation. Ask it to write a Python function to parse a CSV, and it writes working code. Ask it to draft a generic professional message, and the output reads coherently rather than mechanically.

The range is what got me. This isn't a tool that does one thing well. It handles writing, code, analysis, summarization, and conversation, all in the same interface. I kept testing different prompts expecting to find the ceiling, and it kept being competent.

Where It Breaks

It's not perfect. I caught it confidently stating things that were wrong, with the same assured tone it uses when it is right. There is no signal for when it is guessing versus when it knows. That is a real problem. If you don't already know the subject, you can't tell when it is making something up.

It also doesn't know anything recent. The training data has a cutoff, so it can't tell you what the Fed did last month or what is in the latest regulatory guidance. For any work where current information matters, that is a significant limitation.

But the mistakes don't undo the impression. Even with the errors, the baseline quality is well beyond anything I've seen from a language model.

The Banking Angle

Where this might eventually fit in banking is the obvious next question, even though my engagement with the technology stays on personal time, on personal projects, until policy and governance frameworks catch up. There is no production AI use in a regulated context without that infrastructure, and that infrastructure does not exist yet.

Speculating about eventual fit, banking runs on text. Compliance documents, client correspondence, regulatory filings, policy manuals, loan files. A tool that can read, summarize, and draft text at this level is a capability any bank will eventually want to figure out how to use safely.

Client service is one obvious area. Routine inquiries about balances, rates, and onboarding are the kinds of questions that come up constantly at any bank. A model that can handle those conversations naturally, instead of the robotic chatbot patterns that frustrate everyone, could eventually change how front-line work is structured.

Internal productivity is the other. Drafting documents, summarizing notes, generating first versions of routine writing. From personal exploration, the time savings on these kinds of tasks are real, and the leverage on routine writing would be meaningful once policy permits.

What I Don't Know

I have more questions than answers right now. What accuracy bar would let a model handle a client-facing interaction safely? How does this fit into the model risk and vendor management frameworks banks already use? How will the regulatory side think about this kind of tool as it matures?

And the bigger question. If this is what GPT-3.5 can do, and it is free, what does the next version look like? The pace feels fast. Faster than I expected, and faster than the operational infrastructure for using it responsibly is going to catch up to.

I don't know where this goes. But I know it is not a toy. Something shifted, and I think we are going to look back at this month as the beginning of something big.