#OnTapToday — This is Part 2 of a series. Part 1 introduced the Agentic Data Foundry: the idea that data engineers should stop writing transformation code and start describing intent — letting AI agents handle the SQL. If you haven’t read that, it’s worth a few minutes. This post asks the question that every engineer eventually …
Agentic Data Foundry: Stop Building Pipes, Start Describing the Future
#OnTapToday — a concept that’s been consuming my brain for the better part of a year. I want to talk about why the way we’ve been doing data engineering is fundamentally broken, and what I think comes next. I’m going to say something that might ruffle some feathers: we’ve been doing expensive plumbing and calling it …
Fun Refactoring Python Code
I recently came across a coding challenge. Well, let’s say we needed to handle several combinations of filtering data in a #Pandas Dataframe leveraging the #Streamlit selectbox widget. Not a difficult lift per se. The fun was where it started and where it ended up. Let’s get started. Scenario I have 4 columns of data …
Snowflake Snowpark JAVA UDF with JAR – Streamlit bonus
Recently, I was asked the following question: “How can I manage my Snowflake JAVA UDFs without needing to re-create the UDF every time I make a change to the logic? And by the way can you show me how to invoke that from Streamlit?” The first thing that popped in my head was, “Streamlit is a …
Data Warehouse Migration with Snowflake VSC and Flyway
Wow! It has been a year since my last blog post. This post was originally going to cover getting #Flyway set up with Snowflake. Then a couple of days ago, my friend Kent Graziano tweeted a link on getting Visual Studio Code (VSC) connected to Snowflake. Considering I had just come off a lengthy project …
Let DBeaver Build Your ERD from a Snowflake Data Warehouse
I was recently responding to a question about DBeaver and the ability to automatically generate an ERD. The following post will walk you through the process. ;TLDR Looking for an easy way to create one of these? Let’s create 3 simple tables. A parent table with a primary key constraint and two additional …
Python Abort trap: 6 fix after Catalina update
A couple of days ago, I update my MacOS to Catalina and immediately after that, all the Snowflake scripts I wrote in #Python started returning an Abort trap: 6 error. My initial thought was something must be a compatibility issue with either the python connector or new default terminal shell zsh. Fortunately I was wrong, …
Snowflake Migration using StreamSets
#OnTapToday, using @StreamSets to migrate data from your source database to Snowflake. In this post, I will walk through the process of configuring StreamSets to migrate data from my on prem DB to Snowflake. WHY You need to get some data from a source database to the Snowflake data warehouse. There were a few options …
The Winds of Change
I was asked a while ago by a twitter follower this question. “One year ago to now, how has your use of technology changed in your job role?” Interestingly enough, I celebrated 1 year @snowflakeDB on April 16, 2019. I think now is a good time to answer that question. In a nutshell, a year …
Connect to Snowflake Data Warehouse with GO
In this installment, I am going to walk through the process of connecting GO to the Snowflake Data Warehouse Service (DWaaS). This tutorial requires that you have a Snowflake account. You can sign up here for a 30 day/$400.00 trial. Download and Install the ODBC Driver Once you have your account set up in Snowflake, …
