109 items, newest first
12 min LISTEN podcast episodeThe Databricks incident where the most senior engineer never touched the keyboardThursday, just after two in the afternoon. The first message doesn't come from an alert, it comes from a person in...Aug 2026
12 min LISTEN podcast episode9 Databricks decisions that reveal your real seniority (and one that only looks senior)Three in the morning. The nightly job that loads the orders table died halfway. Half the tasks are green, and the rerun...Aug 2026Think Like Seniors
The 4 Questions That Decide a Databricks Architecture ReviewThey score what you rejected, not what you built (6 minutes)Aug 2026Delta & Table Design
PAIDWhat Actually Happens When You Write to a Delta TableWhere orphan files and MERGE conflicts start (9 minutes)Aug 2026Delta & Table Design
11 min LISTEN podcast episodeThe Databricks job market in 2026: there are two of them, and only one answersOne engineer tells you hiring is back and recruiters are warm again. The next one is sixty applications deep and...Aug 2026Interview Like Seniors
Databricks Promotion Committees Don't Grade the CodeThe signals behind a $50k senior jump (6-min read)Aug 2026Think Like Seniors
PAIDHow Databricks Genie Answers 84.5% of Questions RightThe context layer behind it, in plain English (6 minutes)Aug 2026Unity Catalog & Platform
11 min LISTEN podcast episodeDatabricks broadcast joins: when the memo beats the meeting (and when it kills a task)Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data,...Aug 2026Spark Performance & Compute
PAID9 Data Modeling Questions That Win $200k Databricks OffersWhy 3NF loses and query patterns winAug 2026Delta & Table Design
12 min LISTEN podcast episodeBuild the platform or use the platform: the two Databricks data engineer tracks nobody namesTwo Databricks engineers sit at adjacent desks. Same title, same pay, same words in the ladder document. Watch a year...Aug 2026Think Like Seniors
PAIDHow Senior Engineers Actually Choose DLT vs WorkflowsThe compose pattern most teams miss (6 min read)Aug 2026Pipelines & Data Quality
PAIDWhat Actually Happens When You Cache a DataFrameIt is lazy, shared, and can be evicted mid-queryJul 2026Spark Performance & Compute
10 Cost Questions Every Senior Databricks Engineer Should NailWhat juniors miss and interviewers quietly gradeJul 2026Spark Performance & Compute
11 min LISTEN podcast episodeWhy Unity Catalog exists: the Databricks governance chaos it was built to endYou inherited Unity Catalog already switched on. You learned the catalogs, the schemas, the grant statements, and never...Jul 2026Unity Catalog & Platform
PAIDMore Data Quality Checks Make Your Pipeline Less ReliableThe case for deleting half your data quality checksJul 2026Pipelines & Data Quality
11 min LISTEN podcast episodeHow to get credit for Databricks platform work without becoming the on-call martyrYour pipeline hasn't failed in months, and nobody noticed. Then it breaks at 2am, you fix it in twenty minutes, and...Jul 2026Think Like Seniors
How Unity Catalog Permissions Actually Work in DatabricksWhy SELECT alone still returns permission denied (6 min)Jul 2026Unity Catalog & Platform
PAIDWhy the Best Coder Rarely Makes Staff EngineerWhat the promotion committee measures instead (6 minutes)Jul 2026Think Like Seniors
Auto Loader vs Structured Streaming Is a False ChoiceThe real decision is your source, not the engineJul 2026Pipelines & Data Quality
10 min LISTEN podcast episodeWhy Databricks Built Delta Lake When Parquet Was Already Good EnoughA nightly job is halfway through writing a batch when the cluster dies. Nobody runs a rollback, because there's nothing...Jul 2026Delta & Table Design
PAIDHow Photon Actually Works Under the HoodThe four workloads where its premium buys you nothingJul 2026Spark Performance & Compute
8 min LISTEN podcast episodeThe Databricks pipeline that ran green for a year and was wrong the whole timeA revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole...Jul 2026Pipelines & Data Quality
PAIDWhat Senior Engineers Check First in a Databricks Cost ReviewThe investigation order that cuts spend without breaking prodJul 2026Spark Performance & Compute
PAIDStop Tuning Spark Configs. Fix Your Data ModelThe 90% of slowness no executor setting can touchJul 2026Delta & Table Design
11 min LISTEN podcast episodeHow Photon actually makes your Databricks queries faster (and when it silently doesn't)Two engineers run the same SQL on the same Delta table. Same data, same cluster size, copy-pasted code. Alex goes to...Jun 2026Spark Performance & Compute
PAIDWhy Delta Lake Exists and What Plain Parquet Couldn't Do The four failures that killed the raw data lakeJun 2026Delta & Table Design
PAID10 Unity Catalog Questions Every $180k Databricks Interview AsksThe answers that pass the $180k governance screenJun 2026Unity Catalog & Platform
12 min LISTEN podcast episodeThe Databricks interview round nobody studies for (and almost everybody fails)Picture the debrief room after a Databricks loop. Two candidates went through that day. On paper, a coin flip: SQL...Jun 2026Interview Like Seniors
PAIDWhat Senior Databricks Engineers Do That Mid-Levels Don'tThe habits promotion committees actually citeJun 2026Think Like Seniors
PAIDYour Spark Cluster Is a Restaurant Kitchen at Rush HourThe mental model that makes every bottleneck obviousJun 2026Spark Performance & Compute
PAIDBuild a Production-Grade Databricks Genie Space - With a Real Accuracy BenchmarkA free hands-on lab: tune instructions, certified queries, and Unity Catalog functions until Genie is measurably accurate - on Databricks Free Edition.Jun 2026Unity Catalog & Platform
11 min LISTEN podcast episodeThe Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)Your Spark job has been running for forty minutes. The dashboard shows your cluster isn't even busy. So you do the...Jun 2026
PAIDHow Databricks Autoscaling Actually Decides to Add NodesThe one signal it watches, and why a maxed cluster liesJun 2026Spark Performance & Compute
PAIDWhy Deleting Data From Delta Tables Makes Them BiggerDeletion vectors, tombstones, and the VACUUM step that frees storageJun 2026Delta & Table Design
12 min LISTEN podcast episodeYour Databricks data quality framework is a Yeti: everyone talks about it, nobody has seen it workAn architecture review. A platform team is presenting their data quality setup, and honestly, it's impressive....Jun 2026
PAIDYour Slowest Spark Job Is Your Cheapest LessonScaling the cluster throws away the answer it just gave youJun 2026Spark Performance & Compute
11 min LISTEN podcast episodeWhy senior Databricks engineers write less code than mid-level onesTwo engineers, same team, both five years in. Last quarter Mark shipped forty-seven pull requests across three...Jun 2026
PAID7 Databricks Scenarios That Reveal If You're Actually SeniorThe reflex that costs you the title under pressureMay 2026Think Like Seniors
PAIDWhy a Bigger Cluster Won't Fix Your Spark OOMAn OOM is one bucket overflowing, not total RAMMay 2026Spark Performance & Compute
12 min LISTEN podcast episode4 habits that quietly turn your Databricks Delta Lake into a swampYou built the table right. Well-partitioned, documented, fast enough that the row count came back before you finished...May 2026
PAIDThree Medallion Layers Is Cargo Cult on a Small TeamMay 2026Delta & Table Design
18 min LISTEN podcast episodeLiquid Clustering vs Z-Ordering: 4 questions that decideYou open your Databricks workspace. Two Delta tables. Same size, same downstream BI workload. Table A was partitioned...May 2026
PAIDI built a job board only for Databricks data engineers650+ verified roles across 26 countries. Direct apply links. A free Monday digest matched to your level. Because generic boards bury the only filter that matters.May 2026
PAID6 Steps to Cut 30-50% of Databricks Cluster WasteThe audit takes an afternoon, the savings last quartersMay 2026Spark Performance & Compute
23 min LISTEN podcast episodeThe compounding curve: why some Databricks engineers' salaries grow 5x faster than othersYear one. Two new juniors join the same Databricks platform org. Same starting salary, same skills, same desk. Year...May 2026
The Paid Subscriber Reading List17 deep-dives available to Pro subscribers: interviews at every salary band, Spark internals, and the career moves between $150k and $250k.May 2026
PAIDHow Databricks Engineers Disqualify Themselves From Promotion in 30 SecondsWhat managers actually hear when you read your Jira board (3 minutes)May 2026Think Like Seniors
PAIDHow One Candidate Beat 99 Others for a $200k Databricks Data Engineer RoleThe exact answers that survived each of 4 interview roundsMay 2026Interview Like Seniors
17 min LISTEN podcast episodeThe 90/9/1 rule of Databricks performance work - how to triage Spark optimization in 60 secondsYour team is three weeks into a Databricks performance push. Broadcast hints in PRs. AQE flags toggled like christmas...May 2026
PAIDThe Skew Question That Kills $200k InterviewsHow seniors triage salting, AQE, and broadcast in 60 secondsApr 2026Spark Performance & Compute
19 min LISTEN podcast episodeThe Databricks data engineer in 2026 - the four shifts that just changed your jobYou scroll past the cancelled junior req, the "serverless first" line on your director's planning slide, and the third...Apr 2026
PAIDThe System Design Interview That Separates $200k Offers from Polite RejectionsHow two Databricks engineers approached the same system design question with opposite strategies, and what their contrast reveals about the single skill that determines who gets senior-level offers.Apr 2026Interview Like Seniors
14 min LISTEN podcast episode9 Behaviors Quietly Killing Your Promotion To Senior Databricks Data EngineerMid-level is a down escalator. It looks like flat ground. You feel productive, your tickets close on Friday, your...Apr 2026
PAIDWhat 3 Years of Databricks Experience Actually Means4 things hiring managers scan for instead of tenureApr 2026Interview Like Seniors
PAID11 Resources I Built for Databricks Engineers Last MonthPractice tools, coding exercises, a podcast, and moreApr 2026
16 min LISTEN podcast episodeThe Dashboard Theater: What Databricks Engineers Build That Nobody OpensYou check the usage logs on a dashboard you spent two weeks building. Zero views. Not low views. Zero. The stakeholder...Apr 2026
PAIDWhy 3NF Is Killing Your Databricks DashboardsStop normalizing your analytical tables. The textbook was written for a different database.Apr 2026Delta & Table Design
PAID7 Questions That Expose How You Debug Slow PipelinesApr 2026Spark Performance & Compute
16 min LISTEN podcast episodeThe Preparation Gap: What Interviewers Actually Evaluate in Databricks Data EngineersShe could explain shuffle hash join versus sort merge join. She knew when Adaptive Query Execution kicks in. She had...Apr 2026
PAIDHow Liquid Clustering Actually Beats Partitioning + Z-OrderThe decision tree for every 2TB+ Delta Lake tableApr 2026Delta & Table Design
PAIDHow to Handle Schema Evolution Questions in $200k Databricks Data Engineer InterviewsThe follow-up questions that separate mid-level from senior answersMar 2026Interview Like Seniors
15 min LISTEN podcast episodeThe Invisible Engineer: Why Your Best Work Gets the Least RecognitionShe kept a terabyte-scale pipeline running for six months without a single incident. Not one page, not one late...Mar 2026
PAIDWhy Two Correct Answers Get Different Salary BandsA 3-round Databricks interview breakdown, side by sideMar 2026Interview Like Seniors
PAID6 Data Quality Checks I Build Into Every Databricks PipelineThe silent failures that broke executive dashboards for 5 daysMar 2026Pipelines & Data Quality
PAIDHow Delta Lake Achieves ACID Using Only JSON FilesThe _delta_log commit protocol in 10 minutesMar 2026Delta & Table Design
PAID7 Reasons Databricks Data Engineer Offers Disappear Before You Get ThemNone of them are about your Spark skills (5 min read)Mar 2026Interview Like Seniors
Free Hands-On Databricks Labs5 open-source projects you can run on Databricks Free Edition — from streaming pipelines to certification prepMar 2026
PAID10 Questions That Win Senior Data Engineering OffersThe dual-purpose framework from 200+ interviews (6 min read)Mar 2026Interview Like Seniors
Build a Real-Time Fraud Monitoring System: Master Structured Streaming & WatermarksSubtitle: A free hands-on fintech project: Learn deduplication, stream-static joins, and windowed rules engines on Databricks.Mar 2026Pipelines & Data Quality
PAIDHow One SQL Line Becomes 4,000 Parallel TasksThe complete guide to Spark’s execution engine - from SQL to parallel tasks, and how to find bottlenecks when things get slow.Feb 2026Spark Performance & Compute- Learning PathsCert Prep: Databricks Data Engineer AssociateHow to pass the certification - official courses, study guide, practice exams, and free hands-on labsFeb 2026Learning Paths
- Learning PathsCert Prep: Databricks Data Engineer ProfessionalHow to pass the certification - official courses, hands-on labs, and exam prep for the advanced Professional examFeb 2026Learning Paths
- Learning PathsHow to Get a Job as a Junior Databricks Data Engineer - Learning PathA curated learning path from first accreditation to Associate certification, with courses, hands-on projects, and foundational skills. Comment your ideas below!Feb 2026Learning Paths
- Learning PathsHow to Get a Job as a Mid-Level Databricks Data Engineer - Learning PathA curated learning path with professional certifications, advanced projects, interview prep, and specialization skills. Comment your ideas below!Feb 2026Learning Paths
- Interview Like SeniorsPAID10 Interview Questions for Senior Databricks Data Engineer Roles ($180-220k)Where Technical Skills Meet Leadership ThinkingFeb 2026Interview Like Seniors
PAIDCode Reviews at Scale: What Fortune 500 Companies Actually Do (And Why Your Quick LGTM Is Fine)The 80/20 framework that focuses your attention on schema changes, resource sizing, and failure paths - while safely speeding up the restFeb 2026Think Like Seniors- Interview Like SeniorsPAID10 Interview Questions for Junior Databricks Data Engineer Roles ($80-100k)The Fundamentals That Get You HiredFeb 2026Interview Like Seniors
PAIDSpark Performance Interview Questions: What Interviewers Actually EvaluateAn interviewer's insider view on the diagnostic framework that wins $200k+ Databricks Data Engineering offers.Feb 2026Interview Like Seniors
PAIDHow Liquid Clustering Actually Works in DatabricksWhy automatic data organization replaces weekly OPTIMIZE jobs, eliminates partition explosions, and lets you change clustering keys without rewriting data.Jan 2026Delta & Table Design
PAIDThe Databricks Hiring Manager's Guide: Why Certifications Don't MatterThe 3 specific signals that separate senior engineers from the stack of 100 identical resumes.Jan 2026Interview Like Seniors
PAIDThe Databricks Compute Selection Guide: Jobs, All-Purpose, SQL Warehouses, and ServerlessWhy scheduled jobs on All-Purpose clusters are bleeding thousands of dollars a month - and how to fix it in 15 minutesJan 2026Spark Performance & Compute
PAIDThe Databricks Debugging Maturity Ladder: Junior to PrincipalHow engineers progress through trial-and-error, analysis paralysis, and over-engineering before earning the wisdom to find the one bottleneck that mattersJan 2026Think Like Seniors
PAIDThe $150k vs $250k Databricks Data Engineer: Why Technical Skills Aren’t EnoughThe 18-month roadmap from "my work should speak for itself" to strategic visibility, high-impact projects, and data-driven compensation growthJan 2026Think Like Seniors
PAIDThe Spark Cluster Parallelism Guide: Why 32 Nodes Can Be Slower Than 4The task-to-core ratio that separates $600/month waste from systematic diagnosis - with the complete decision framework for repartition vs. scaleDec 2025Spark Performance & Compute
PAIDPartition Pruning vs Data Skipping: A Databricks Interview Deep DiveHow Spark's optimization layers work at directory, file, and row levels - and why understanding the difference wins offersDec 2025Spark Performance & Compute
PAIDUnderstanding Spark Shuffle: The Complete Architecture GuideWhy your 10-minute job became 2 hours, how data skew causes 100GB → 500GB inflation, and the three-phase mechanism behind every GROUP BYDec 2025Spark Performance & Compute
PAIDThe Databricks Technical Debt Navigation Guide: When to Ship and When to PerfectThe pragmatic judgment framework that senior engineers use to balance velocity and quality - with 8 real scenarios, decision templates, and a 30-day practice planDec 2025Think Like Seniors
PAIDInside the Delta Lake Transaction Log: From Write to Time TravelHow 2KB JSON files control your 10TB table, and why OPTIMIZE temporarily doubles your storage before making queries fasterDec 2025Delta & Table Design- Pipelines & Data QualityPAIDReal-Time Pricing Optimization with Databricks: An E-Commerce Case StudyHow streaming data pipelines, Delta Lake, and MLflow turn customer purchases into data-driven pricing decisions in under 3 hoursDec 2025Pipelines & Data Quality
- Interview Like SeniorsPAIDDatabricks Data Engineer Interview: Answering the Project Failure QuestionThe ownership framework that transforms a deflective answer into a $200k+ offer-winning response with quantified impactDec 2025Interview Like Seniors
- Interview Like SeniorsPAID10 Spark Performance Questions for $180k+ Databricks Data Engineer RolesThe Offer-Winning Answer GuideDec 2025Interview Like Seniors
- Pipelines & Data QualityPAIDDatabricks Auto Loader Deep Dive: A File's 7-Step Journey from S3 to DeltaMaster schema inference, file notifications, and checkpointing to troubleshoot ingestion failures and optimize your pipelines.Dec 2025Pipelines & Data Quality
PAID10 Popular Interview Questions for Mid-Level Databricks Data Engineer Roles ($120-150k) Where "I Know The Syntax" Stops WorkingDec 2025Interview Like Seniors
PAIDDatabricks CDC Interview: The 200GB/Day Pipeline with SCD Type 2Why custom MERGE logic is the trap, and how AUTO CDC + Liquid Clustering is the senior answer that wins FAANG offersDec 2025Pipelines & Data Quality
PAIDWhy You Don't Need to Write "Perfect" Spark CodeHow the Catalyst Optimizer and AQE fix your query logic in real-time (explained via a road trip).Dec 2025Spark Performance & Compute
PAID𝗛𝗼𝘄 𝗦𝗽𝗮𝗿𝗸 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗪𝗼𝗿𝗸𝘀A simple educational dialogueDec 2025Spark Performance & Compute
PAIDThe Optimization Trap: Junior vs. Senior Data EngineersOne burns money, one builds ROI. Why your response to "make it faster" defines your career level.Dec 2025Think Like Seniors
PAIDJunior vs Senior: Databricks "MERGE INTO" Code ReviewHow seniors review for reliability, not syntax.Nov 2025Delta & Table Design
PAIDDatabricks Data Engineer Interview: Explain Lazy EvaluationWhy knowing just “transformations don’t execute immediately” won’t cut it - and how Catalyst’s logical plans and pushdown optimizations make all the differenceNov 2025Spark Performance & Compute
PAIDDatabricks Data Engineer Interview: A Broadcast Join Case StudyWhy a 50MB table can cause a driver OOM crash, and how concurrency is the real culprit.Nov 2025Spark Performance & Compute
PAIDRight-Sizing Your Databricks Cluster: A 500 GB Case StudyA Practical Guide to Shuffle Partitions, Executor Sizing, and Handling Data Skew.Nov 2025Spark Performance & Compute- Delta & Table DesignPAID10 Delta Lake Questions for $150k+ Databricks Data Engineer RolesThe Offer-Winning Answer GuideNov 2025Delta & Table Design
PAIDThe Databricks Schema Management Guide: When to Evolve and When to EnforceThe strategic pattern that turns schema evolution from a 40-hour debugging disaster into controlled change managementNov 2025Delta & Table Design- Delta & Table DesignPAIDCracking the Databricks System Design Interview: The 2TB ChallengeThe architectural thinking pattern that turns a basic pipeline answer into a staff-level response - with real numbers and governance includedNov 2025Delta & Table Design
PAIDData Modeling in Databricks: The Complete GuideFrom stakeholder interviews to production deployment - the 20% of knowledge that delivers 80% of resultsNov 2025Delta & Table Design- ArticlesPAIDA free hands-on project: learn PySpark and pass the Databricks certificationBuild a production-grade Medallion pipeline that covers 80% of certification topics - on Databricks Free EditionNov 2025
- Think Like SeniorsPAIDEveryone thinks being a Databricks Data Engineer is “sexy”.Let me break the myth...Nov 2025Think Like Seniors
- Delta & Table DesignPAIDThe Delta Table Optimization Guide: Partitioning, Z-Ordering, and Liquid ClusteringThe 20% of knowledge that solves 80% of optimization decisionsNov 2025Delta & Table Design
- Think Like SeniorsPAIDWhat’s the difference between a $100k, $200k, and $400k+ Databricks Data Engineer? Asked this in a community with engineers at all levels.Nov 2025Think Like Seniors
Read all 74 paid articles free for 14 days
74 deep dives are paid-subscriber only, plus every article once it is 2 weeks old. The Substack trial runs 14 days, long enough to read the ones you came for and decide.
Start the free Substack trial →Substack asks for your email first, then applies the trial. You are not charged until it ends. Or subscribe free for the newest 2 weeks of posts.