9 posts tagged with "Data Analysis"

Most of the conversation about data strategy at a community bank ends up in the wrong place. People want to talk about tools, vendors, dashboards, the latest architecture. The harder problem is sitting one floor up. The people making decisions do not need more data. They need a small set of things they can act on, and they need it consistently.

All the attention right now is on large language models. But there's a whole category of machine learning that's been quietly useful for years and doesn't require a GPU or a billion parameters. Clustering is one of those techniques. The idea is simple. Given a pile of client data, can an algorithm find natural groups without being told what to look for? I wanted to try it.

Exploring transaction data with pandas gave me a baseline. Counting things, grouping things, building a sense of what normal looks like. That was useful, but it left me staring at tables of numbers. What I wanted was to see the shape of what's happening over time.

I wanted to try the same data tools on transaction-level records. Mortgage lending data is useful, but the day-to-day activity of a bank lives in its transactions. Deposits, withdrawals, transfers, loan payments. Every client interaction with their account generates a record.

With Jupyter, pandas, and matplotlib set up, let's put those tools to work on real data.

Any real work with data starts with actually looking at it. Before reaching for a heavyweight analytics platform, the fastest way to get your hands on a dataset is a notebook running on your own machine. That personal workbench is where I do most of my exploring, and Jupyter is what powers it.

In the previous posts, I covered transactional databases for real-time operations and data warehouses for historical analytics. Both work with structured data. But what happens when you need to store log files, images, sensor readings, or raw event streams? That's where data lakes come in, and understanding them has changed how I think about data architecture.

In the previous post, I looked at transactional databases that power real-time operations. But I've learned the hard way that running complex analytical queries against production databases is a bad idea. It slows down the application and puts operational data at risk. Data warehouses solve this by providing a separate system optimized for analysis rather than transactions.

I keep running into the same mistake when looking at data: two things happen together, so one must be causing the other. It's a natural assumption, and it's often wrong.