Alex Marshall

Backbone case study

In late 2021, seed-stage supply chain SaaS startup Backbone contacted Kaleidoscope Data about up-leveling their analytics offerings, including offering a paid "Data Warehouse" service to their customers and starting a data practice. Kaleidoscope Data worked with the CTO and engineering team to build out a scalable analytics infrastructure, grow their

Integrate many data sources into your data warehouse with DBT

We walk through our DBT project structure and how it helps us transform data from many point-of-sale (POS) systems into a retail analytics data warehouse. * Background * File Structure * Base Tables * Staging Tables * Core Tables * Inventory * Conclusion Background For the last several years, we have been building and maintaining a retail

Continuous API Data ELT into BigQuery using Cloud Run

ETL/ELT frameworks can make working on big data pipelines a lot easier, as compared to writing extraction and transformation scripts from scratch. On the other hand, for very lightweight and standalone data extraction tasks, it's hard to beat a quick script run serverless on your cloud provider of choice.

Data Transformation architecture for your customer-facing Data Warehouse

Most data folks understand a "Data Warehouse" as an internal tool for reporting and analytics across different teams in your organization. However, it can also be used externally - by customers who want more granular access to their data (often your largest customers). Charging customers for this data access can

Multi-Tenancy in BigQuery

Background Multi-tenancy in BigQuery can be accomplished in several ways. This article by the folks at Google has many helpful tips for data isolation and security. The main idea is isolating tenant data in BigQuery datasets - all under a single client data project - rather than creating a new