Second of all, I'm looking forward to going through this because this is what I wanted to tinker with when it came to metrics. I'll have to follow up later, but it's my bedtime!
I built this functionality using Python first. Then I kept adding more modules for various pieces of data I wanted and the pipeline got more complicated.
Then I started transformations, and re-organized the data into database tables with proper relationships so that I can use SQL to make queries.
At that point I needed to make the pipeline more robust for errors, so I used Prefect tool to enable caching and retries.
I wrote a recent article about Prefect and Dagster, these open source orchestration tools.
A simple script has grown into a full data pipeline.
Let's see what happens with these Chrome browser extensions. I'm just getting hang of building them. I'm not very good at using JavaScript, but learning some new tricks every day.
And that learning and sharing is what keeps me excited about this.
Love it but I’m curious why you didn’t extend the earnings extension
Hi Lee, I'm using GraphQL APIs that are not documented. In this extension I have to call two separate APIs to get the fields I wanted.
Also, earnings extension was designed to pivot the earnings by month and I have not found a way to get monthly claps and responses yet.
In the future it might make sense to consolidate these into one, but it's easier to debug smaller modules like these two I've created so far.
Both are great. Thank you.
First of all, love the rebrand!
Second of all, I'm looking forward to going through this because this is what I wanted to tinker with when it came to metrics. I'll have to follow up later, but it's my bedtime!
Thank you Vince,
I built this functionality using Python first. Then I kept adding more modules for various pieces of data I wanted and the pipeline got more complicated.
Then I started transformations, and re-organized the data into database tables with proper relationships so that I can use SQL to make queries.
At that point I needed to make the pipeline more robust for errors, so I used Prefect tool to enable caching and retries.
I wrote a recent article about Prefect and Dagster, these open source orchestration tools.
A simple script has grown into a full data pipeline.
Let's see what happens with these Chrome browser extensions. I'm just getting hang of building them. I'm not very good at using JavaScript, but learning some new tricks every day.
And that learning and sharing is what keeps me excited about this.
I think that’s exactly what is supposed to happen with any cool project, right?
I’ve been doing something similar for my work, and it’s ended up becoming a key part of what makes my daily grind a lot of fun.
C++ was my first real tackle, but Python is my go-to these days. I have no experience with webscraping, but your progress sounds compelling!