GuruGraffiti Writing On The Walls Of The Internet
To recap from Part 1, the goals of this rabbit hole project were to use DuckDB’s fast processing engine alongside an existing Visual FoxPro 9 (VFP) application, to take advantage of DuckDB’s import/query speed on large datasets (100M+ rows, ~2GB) while keeping VFP as the primary application layer. The primary technical obstacle was that DuckDB only ships a 64-bit ODBC driver, and VFP9 only supports 32-bit ODBC which ruled out a direct database connection.
This project was exhausting and exhilarating at the same time. It would have been easy for me to end this journal at Part 9 and walk away, but there were a lot of take-aways from this journey I wanted to document for my future self and for others who are interested in doing their own similar investigations. I also wanted to talk a little about what I learned working with an AI agent like Claude.
Let’s start with some context:
First, I wanted to find a way to use DuckDB with Visual Foxpro even though there was no obvious way to make that happen. I have been programming with VFP almost exclusively for the past 25+ years and I can tell you I would never have attempted this experiment if I did not have the confidence I’ve built up in VFP. I have never had anything I wanted to really do with VFP that wasn’t possible. It is simply and amazing product and one of the nicest features is it allows me to use VFP independently or as a front-end to other tools. For many years, I’ve been building SQL queries for databases like SQL, MySQL, Oracle, PostgreSQL, FirebirdSQL, etc, etc, and linking to these systems via ODBC drivers. I know the fundamentals of the process VFP->ODBC->database are not only functional but also very efficient. The only part that was a real challenge in making DuckDB and VFP work together was in finding a way to getting around the missing 32-bit ODBC driver that I normally depend upon. However, for whatever reason, I never doubted there was a way to make DuckDB work with VFP, but I couldn’t guarantee the solution would be efficient and fast. When you are faced with a difficult task as a programmer, you have to believe beyond a shadow of a doubt there is a way to solve the problem regardless of your programming language.
Second, because I was going to be working with a tool like DuckDB and I didn’t know the documentation for DuckDB as well as I knew the documentation for VFP, I decided I would expand the scope of the project to include working with an AI tool even though my previous experiences with AI left me with rather negative feelings about AI. I have to admit the decision to work with Claude.AI was probably the deciding factor in making this project a success. The fact Claude was designed with a deep understanding of programming and programming languages made all the difference in the world. If I had been working with some other AI, I would have likely thrown up my hands fairly early and given up on this whole effort. It is definitely true: success often comes down to the choice of the tools you use.
Initially, I was only planning to utilize Claude.AI as a reference or research assistant to help me with the syntax associated with DuckDB, but as I continued to work with Claude, the relationship changed. I half-jokingly referred to Claude as my “new wingman” and yet by the end of this journey, Claude earned that title by providing a bit more than just references to the documentation or syntax advice. Claude became another programmer with a completely different skill-set looking over my shoulder as I navigated through the various hurdles I faced throughout the project. As I began to become more acquainted with how to phrase my questions, I got more and more out of the answers from Claude. Sometimes, the answers led me down a dead-end blind alley. But sometimes, the answers would make me think in a way I hadn’t before. In fact, the solution that finally helped me achieve the goal of the project came after a conversation with Claude. But how that happened actually bears further discussion.
You see, your ego can be the best motivator toward finding a solution regardless of your profession. I mentioned earlier having the utmost faith in Visual Foxpro, but you might also have faith and trust in yourself that you can find a solution or you never will. However, as important as it is to have an ego about your work, it can also be your Achilles heel because it can get in the way of accepting or even looking for answers from outside sources. For example, if you are afraid of asking dumb questions from your peers or your betters, then you may never ask the right question that leads you to the correct answer. I’ve been guilty of this in the past and I suspect a lot of other people have too. Being afraid of being ridiculed for not knowing something is a road-block to success. Being afraid of failure is the only thing I know of that can stop you faster.
Working with Claude.AI, I found it easy to ask dumb questions or to admit I didn’t understand the answer I got the first time. Claude patiently answered every question I posed almost giving too much information. I found it was hard sometimes to not be distracted by the depth of some of the answers because I was really enjoying learning some of the behind the scenes details, but eventually, I learned how to take what I needed and move on.
I also mentioned in a previous part of the story that it is worth starting a new conversation from time to time, restate the goal and the progress you’ve made so far, to clear the air and keep Claude.AI focused on the solution and weed out all of the dead-ends we had discovered along the way. I can’t tell you when that moment will come for you but I can say it came for me when my gut told me it was time to clear the decks and start over. I only did this a couple of times during the project but if I had it to do over again, I would probably have done it at least once more. You don’t want to lose your progress, but you also don’t want to leave Claude with too much context. (I’m actually curious if other programmers do this as a general rule or just me.)
In the end, I feel like this project was a success beyond my wildest expectations. I had no idea what the true capabilities were nor how really fast DuckDB, SQLite and VFP could be when they were working together, so I couldn’t really imagine what success looked like. Importing more than a million records per second and then retrieving a large quantity of records from the imported data in just a couple of seconds absolutely blew me away. But the most surprising result of all was how well I was able to adapt to working with Claude.AI. As I’ve noted, my previously results with other AI systems were not fun and definitely not successful. I could not have been happier with the fact I didn’t have re-write the code and/or logic Claude suggested just to make it work. The ideas were solid, grounded in real-world programming and the commands were accurate for all of the tools used in the project.
So, you may be curious at this point, am I still the AI skeptic I claimed to be at the beginning of this project?
In terms of relying solely on AI to write code for projects, then my answer is yes, I’m still the same old skeptic. However, in terms of using AI as a peer where the programmer and the AI work side-by-side filling in the gaps in each other’s knowledge, then I would say my skepticism is gone. I can confidently tell you I will work with Claude again and in fact, I’ve already had Claude review some Oracle queries to help me optimize them to get faster results.
I’m looking forward to the next real-world project where I’ll get to apply what I’ve learned along the way to “Making FoxPro Ducky” and in the meantime, I’ll be looking for the next cool database tool which comes along.
I’ll probably give Claude a call when that happens.
