https://bricksnotes.com/blog Practical writing on Databricks, Spark, Delta Lake, and the craft of data engineering. en Thu, 03 Sep 2026 02:11:09 GMT https://bricksnotes.com/blog/data-engineering-faster-decisions https://bricksnotes.com/blog/data-engineering-faster-decisions Thu, 03 Sep 2026 02:11:09 GMT BricksNotes Your pipeline runs in four minutes and the decision still waits for Monday. A practical look at decision latency, designing backwards from the decision, and what changes when the consumer is an agent. data engineering decisions databricks https://bricksnotes.com/blog/databricks-small-file-problem-optimize-vacuum https://bricksnotes.com/blog/databricks-small-file-problem-optimize-vacuum Wed, 02 Sep 2026 03:10:00 GMT BricksNotes A Delta table that was fast in week one became slow by month three. The data had not grown much. The file count had. Here is what the small file problem is, why streaming and frequent merges cause it, and how OPTIMIZE and VACUUM keep a table healthy. databricks delta lake performance fundamentals https://bricksnotes.com/blog/delta-lake-time-travel-explained https://bricksnotes.com/blog/delta-lake-time-travel-explained Tue, 01 Sep 2026 03:38:53 GMT BricksNotes Delta Lake keeps every version of your table. Learn how to query older versions with VERSION AS OF, read table history, and undo bad writes with RESTORE, with simple examples that run in Databricks Free Edition. Databricks Delta Lake time travel data engineering https://bricksnotes.com/blog/databricks-pipeline-observability-alerts-runbook https://bricksnotes.com/blog/databricks-pipeline-observability-alerts-runbook Mon, 31 Aug 2026 01:51:24 GMT BricksNotes A green run is a weak signal. Here are the four questions your pipeline should answer on its own, with freshness checks, run metrics, Delta constraints and alerts you can build in Free Edition. Databricks monitoring data-quality pipelines https://bricksnotes.com/blog/databricks-electric-acquisition-lakebase-agents https://bricksnotes.com/blog/databricks-electric-acquisition-lakebase-agents Sun, 30 Aug 2026 02:25:52 GMT BricksNotes Databricks acquiring Electric is a bet that agents need data close to where they run. Here is what it means for Lakebase, and what it changes for data engineers. Databricks ai-agents lakebase architecture https://bricksnotes.com/blog/what-is-lakebase-explained-simply https://bricksnotes.com/blog/what-is-lakebase-explained-simply Sat, 29 Aug 2026 02:56:00 GMT BricksNotes Lakebase is managed Postgres inside Databricks. Here is the idea behind it, where it fits, the use cases worth trying, and why agents make it more than a convenience feature. Databricks Lakebase Data + AI Architecture Agents https://bricksnotes.com/blog/databricks-query-optimization-execution-plan https://bricksnotes.com/blog/databricks-query-optimization-execution-plan Fri, 28 Aug 2026 01:42:09 GMT BricksNotes Two queries, identical results, wildly different cost. Here is what happens between the SQL you write and the answer you get, and the seven techniques that decide how much work the engine does. Databricks Spark Performance SQL Data Engineering https://bricksnotes.com/blog/databricks-broadcast-vs-shuffle-joins https://bricksnotes.com/blog/databricks-broadcast-vs-shuffle-joins Thu, 27 Aug 2026 02:09:50 GMT BricksNotes One line joined orders to customers, and an eleven minute job ran all night. Here is what a join really does on a cluster, and how to choose the right one. Databricks Spark Performance Data Engineering https://bricksnotes.com/blog/schema-changes-data-contracts-databricks https://bricksnotes.com/blog/schema-changes-data-contracts-databricks Wed, 26 Aug 2026 01:34:12 GMT BricksNotes Schema evolution is normal. Uncontrolled schema evolution is dangerous. The five change types, why a data contract does what a schema cannot, and six habits that make change visible, testable and safe in Databricks. Databricks Schema Evolution Data Quality Data Engineering https://bricksnotes.com/blog/databricks-duplicate-records-deduplication https://bricksnotes.com/blog/databricks-duplicate-records-deduplication Tue, 25 Aug 2026 00:34:08 GMT BricksNotes A pipeline can finish green and your numbers can still be wrong. Duplicates rarely break pipelines. They break business decisions. Here is how to handle them properly. Databricks Data Quality SQL Fundamentals https://bricksnotes.com/blog/databases-built-for-agents-not-just-developers https://bricksnotes.com/blog/databases-built-for-agents-not-just-developers Mon, 24 Aug 2026 00:53:14 GMT BricksNotes For forty years databases were designed around a human who could ask a teammate. Agents cannot. Here is how a data platform changes when machines are the main readers, and what to do about it in Databricks this week. AI Agents Unity Catalog Data Engineering Databricks https://bricksnotes.com/blog/databricks-data-skew-slow-tasks https://bricksnotes.com/blog/databricks-data-skew-slow-tasks Sun, 23 Aug 2026 03:34:15 GMT BricksNotes The code did not change. The data did. Here is how one heavy key turns a nine minute job into a fifty one minute one, and how to fix it. Databricks Spark Performance Joins https://bricksnotes.com/blog/csv-json-parquet-delta-which-format-to-use https://bricksnotes.com/blog/csv-json-parquet-delta-which-format-to-use Sat, 22 Aug 2026 15:16:16 GMT BricksNotes Editorial CSV, JSON, Parquet, or Delta? A practical comparison with runnable Databricks Free Edition examples, real size and speed differences, and a simple rule for each layer of your pipeline. Delta Lake Parquet File Formats Data Engineering https://bricksnotes.com/blog/bricksnotes-40000-to-100000-learners https://bricksnotes.com/blog/bricksnotes-40000-to-100000-learners Fri, 21 Aug 2026 23:57:32 GMT BricksNotes Editorial A hundred thousand data professionals have now learned with BricksNotes. Here is the honest story of how it happened, what we built along the way, and why we are not stopping. Community Milestone Data Engineering 2026 https://bricksnotes.com/blog/databricks-data-quality-checks https://bricksnotes.com/blog/databricks-data-quality-checks Fri, 21 Aug 2026 02:21:49 GMT BricksNotes Editorial A successful run tells you the code finished, not that the data is right. How to write quality rules, decide what happens to bad rows, and enforce them with Delta constraints and pipeline expectations. Databricks Data Quality Delta Lake Pipelines https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings https://bricksnotes.com/blog/built-first-data-engineering-project-two-evenings Thu, 20 Aug 2026 13:30:43 GMT BricksNotes Editorial A boundary instead of a study plan: two evenings, 100 messy orders, one gold table, and a pipeline that runs twice without breaking. Here is the full walkthrough you can repeat tonight in Databricks Free Edition. Databricks Data Engineering PySpark Getting Started 2026 https://bricksnotes.com/blog/databricks-late-arriving-data https://bricksnotes.com/blog/databricks-late-arriving-data Thu, 20 Aug 2026 01:50:50 GMT BricksNotes Editorial A phone was offline, a partner file was late, a queue backed up. Here is how to keep event time honest and stop numbers from drifting quietly. Databricks Streaming Delta Lake Data Quality https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream https://bricksnotes.com/blog/databricks-backfill-without-breaking-downstream Wed, 19 Aug 2026 01:51:14 GMT BricksNotes Editorial Reloading March means touching a table six dashboards already read. Here is the plan that makes a backfill boring instead of frightening. Databricks Delta Lake Backfill Pipelines https://bricksnotes.com/blog/agent-discovers-agent-applies https://bricksnotes.com/blog/agent-discovers-agent-applies Tue, 18 Aug 2026 13:38:17 GMT BrickNotes Editorial An agent found the table, read the schema, and proposed the merge in forty seconds. Everything it got right came from work a data engineer had already done. AI Agents Unity Catalog Data Quality Delta Lake https://bricksnotes.com/blog/schema-evolution-databricks-pipelines https://bricksnotes.com/blog/schema-evolution-databricks-pipelines Tue, 18 Aug 2026 01:46:27 GMT BrickNotes Editorial A source team added one column and renamed another, and the nightly job failed. Here is how schema change actually works in Databricks, and how to handle each case calmly. Databricks Delta Lake Schema Evolution Auto Loader https://bricksnotes.com/blog/how-to-become-ai-data-engineer-with-bricksnotes https://bricksnotes.com/blog/how-to-become-ai-data-engineer-with-bricksnotes Mon, 17 Aug 2026 00:43:50 GMT BrickNotes Editorial A complete path for a fresh data engineering aspirant. Free Edition setup, the lessons in order, the habits that matter, and how to grow into AI ready work. Career Beginners Learning Path AI Data Engineer https://bricksnotes.com/blog/auto-loader-vs-copy-into-databricks https://bricksnotes.com/blog/auto-loader-vs-copy-into-databricks Sun, 16 Aug 2026 04:27:00 GMT BrickNotes Editorial A daily CSV drop grew to seven hundred files and turned a five minute job into forty. Here is how COPY INTO and Auto Loader remember what they already loaded, and how to choose. Databricks Auto Loader Ingestion Delta Lake https://bricksnotes.com/blog/idempotent-pipelines-safe-reruns-databricks https://bricksnotes.com/blog/idempotent-pipelines-safe-reruns-databricks Sat, 15 Aug 2026 03:02:12 GMT BrickNotes Editorial A nightly load failed at 2am, got rerun at 7am, and quietly doubled a revenue number for a week. Here is what idempotency really means, and the four write patterns that make a Databricks pipeline safe to run twice. Databricks Delta Lake Pipelines Data Quality https://bricksnotes.com/blog/databricks-190-billion-journey https://bricksnotes.com/blog/databricks-190-billion-journey Fri, 14 Aug 2026 02:37:26 GMT BrickNotes Editorial Databricks closed a $5 billion round at a $190 billion valuation in August 2026. Here is the full thirteen-year story behind that number, from Spark at Berkeley to Delta Lake, Unity Catalog, Lakeflow and the agent era, and what it means for data engineers. Databricks Story Lakehouse Unity Catalog 2026 https://bricksnotes.com/blog/streaming-table-vs-materialized-view-databricks https://bricksnotes.com/blog/streaming-table-vs-materialized-view-databricks Thu, 13 Aug 2026 03:54:48 GMT BrickNotes Editorial A nightly silver job that used to finish in twenty minutes started running past breakfast. Nothing was wrong with the code. It was rebuilding the whole table every night. Here is how to decide between a streaming table and a materialized view in Databricks, with examples that run on Free Edition. Lakeflow Streaming Materialized Views Databricks https://bricksnotes.com/blog/delta-lake-zorder-vs-liquid-clustering https://bricksnotes.com/blog/delta-lake-zorder-vs-liquid-clustering Wed, 12 Aug 2026 03:14:25 GMT BrickNotes Editorial Your dashboard slowed down and the query never changed. Most Databricks performance problems are layout problems. A practical guide to partitioning, Z-ORDER, and liquid clustering in 2026. Delta Lake Performance Liquid Clustering Databricks https://bricksnotes.com/blog/context-engineering-skill-for-data-engineers https://bricksnotes.com/blog/context-engineering-skill-for-data-engineers Mon, 10 Aug 2026 12:41:12 GMT BrickNotes Editorial An agent answered a simple question confidently and wrongly. Nobody had decided what it was allowed to know. That decision is now a job, and data engineers are the natural owners of it. Context Engineering AI Agents Databricks Governance context-advantage https://bricksnotes.com/blog/agentic-data-stack-built-for-ai-agents https://bricksnotes.com/blog/agentic-data-stack-built-for-ai-agents Sat, 08 Aug 2026 15:50:08 GMT BrickNotes Editorial Data platforms assumed a human was always in the loop. Agents break that assumption. Here is what context, tool access, observability, cost, and approval look like now. AI Agents Databricks Governance Context Engineering https://bricksnotes.com/blog/databricks-lakemeter-cost-estimation https://bricksnotes.com/blog/databricks-lakemeter-cost-estimation Fri, 07 Aug 2026 01:39:08 GMT BrickNotes Editorial Someone asks what your pipeline will cost. Lakemeter, the open source tool from Databricks Labs, turns that shrug into an estimate you can defend. Databricks Cost Open Source https://bricksnotes.com/blog/databricks-unity-ai-gateway-ga-explained https://bricksnotes.com/blog/databricks-unity-ai-gateway-ga-explained Wed, 05 Aug 2026 19:48:57 GMT BrickNotes Editorial Databricks made Unity AI Gateway generally available on August 4, 2026. Here is what it actually does, why cost and control matter more than model choice, and what still belongs to you. Databricks AI Governance Data Engineering https://bricksnotes.com/blog/databricks-variant-data-type-explained https://bricksnotes.com/blog/databricks-variant-data-type-explained Wed, 05 Aug 2026 03:00:00 GMT BrickNotes Editorial Databricks Variant is now generally available. It is a native SQL type for semi-structured data that solves the old tradeoff between flexibility and query performance. Here is what it does, how Variant Shredding works, and how to start using it with real code. Databricks Delta Lake Data Engineering https://bricksnotes.com/blog/100000-data-professionals-bricksnotes-milestone https://bricksnotes.com/blog/100000-data-professionals-bricksnotes-milestone Fri, 31 Jul 2026 03:43:02 GMT BrickNotes Editorial More than 100,000 unique data professionals have visited BricksNotes, and over 25,000 practice exams have been completed across Associate, Professional, and Analyst certifications. What these numbers mean, and where we go next. Community Milestone https://bricksnotes.com/blog/databricks-genie-code-sql-migration-explained https://bricksnotes.com/blog/databricks-genie-code-sql-migration-explained Fri, 31 Jul 2026 03:26:56 GMT BricksNotes Editorial Databricks announced an agentic code converter in Genie Code that translates proprietary SQL dialects to ANSI SQL. Here is what it does, what it does not do, and why open SQL matters for your career. databricks sql-migration genie-code ansi-sql unity-catalog https://bricksnotes.com/blog/databricks-context-engineer-associate-explained https://bricksnotes.com/blog/databricks-context-engineer-associate-explained Thu, 30 Jul 2026 02:04:01 GMT BricksNotes Editorial Databricks has a new certification that is not about models. It is about the information you put around them. Here is what the Context Engineer Associate covers, and why data engineers are closer to it than they think. databricks certification context-engineering ai https://bricksnotes.com/blog/databricks-free-edition-playbook-2026 https://bricksnotes.com/blog/databricks-free-edition-playbook-2026 Tue, 28 Jul 2026 02:35:10 GMT BricksNotes Editorial A calm, practical seven-day path for learning Databricks on Free Edition in 2026. What you get, what you do not, and the exact steps to build real skills. Databricks Free Edition Learning https://bricksnotes.com/blog/databricks-previews-late-july-2026 https://bricksnotes.com/blog/databricks-previews-late-july-2026 Mon, 27 Jul 2026 13:40:34 GMT BricksNotes Editorial A calm read of the Databricks previews from mid to late July 2026. Genie Agent API, agents that can read files in Unity Catalog Volumes, Lakebridge, pipeline unit tests, and a few smaller wins. databricks previews genie lakeflow unity-catalog https://bricksnotes.com/blog/ai-data-engineer-roadmap-2026 https://bricksnotes.com/blog/ai-data-engineer-roadmap-2026 Sun, 26 Jul 2026 21:00:00 GMT BricksNotes Editorial A calm, stage-based roadmap to becoming an AI data engineer in 2026. Databricks foundation, lakehouse pipelines, AI applications, and agent-ready systems. ai-data-engineer career roadmap context-advantage databricks https://bricksnotes.com/blog/failed-databricks-associate-exam-twice-what-fixed-it https://bricksnotes.com/blog/failed-databricks-associate-exam-twice-what-fixed-it Sun, 26 Jul 2026 03:26:07 GMT BricksNotes Editorial Two failed attempts taught me more than any cheat sheet. Here is the calm, five-part routine and two-week plan that finally worked. certification associate-exam study-guide https://bricksnotes.com/blog/apache-spark-42-defining-truth-age-of-ai https://bricksnotes.com/blog/apache-spark-42-defining-truth-age-of-ai Sat, 25 Jul 2026 02:52:39 GMT BricksNotes Editorial Apache Spark 4.2 shipped in July 2026. Metric Views, mature Spark Connect, vector search in SQL, and Auto CDC change where truth lives, how agents talk to Spark, and how streaming catches up to reality. Spark Databricks AI https://bricksnotes.com/blog/design-beautiful-aibi-dashboards https://bricksnotes.com/blog/design-beautiful-aibi-dashboards Tue, 21 Jul 2026 02:42:59 GMT BricksNotes Editorial Themes, layout, and the small design habits that turn a Databricks AI/BI dashboard from noisy to trustworthy. https://bricksnotes.com/blog/agent-bricks-ga-ship-without-regret https://bricksnotes.com/blog/agent-bricks-ga-ship-without-regret Tue, 21 Jul 2026 02:39:25 GMT BricksNotes Editorial Agent Bricks is now generally available. Here is how to take an agent from demo to production without it going off the rails. https://bricksnotes.com/blog/lakebase-ga-do-you-still-need-postgres https://bricksnotes.com/blog/lakebase-ga-do-you-still-need-postgres Sun, 19 Jul 2026 02:03:59 GMT BricksNotes Editorial Lakebase is generally available. Do you still need a separate Postgres next to your lakehouse? A calm, honest answer. lakebase postgres unity-catalog architecture databricks 2026 https://bricksnotes.com/blog/data-foundation-for-ai-agents-databricks https://bricksnotes.com/blog/data-foundation-for-ai-agents-databricks Sat, 18 Jul 2026 04:05:07 GMT BricksNotes Editorial Before you plug an agent into your warehouse, ask four quiet questions about the data underneath it. ai-agents data-quality unity-catalog medallion databricks https://bricksnotes.com/blog/context-control-cost-choice-data-engineer-guide https://bricksnotes.com/blog/context-control-cost-choice-data-engineer-guide Fri, 17 Jul 2026 03:25:14 GMT BricksNotes Editorial The interesting question is not which model to pick. It is what the model sees, what it is allowed to do, what one answer costs, and how locked in you become. context-advantage ai-agents career databricks https://bricksnotes.com/blog/genie-one-mobile-what-it-means https://bricksnotes.com/blog/genie-one-mobile-what-it-means Thu, 16 Jul 2026 15:05:29 GMT BricksNotes Editorial Genie One is now a mobile app. A quiet shift in how business users meet their data, and a bigger one than it looks. databricks genie mobile ai unity-catalog https://bricksnotes.com/blog/databricks-data-analyst-associate-practice-exam-live https://bricksnotes.com/blog/databricks-data-analyst-associate-practice-exam-live Tue, 14 Jul 2026 19:35:35 GMT BricksNotes Editorial A free, full-length practice exam for the Databricks Data Analyst Associate certification is now live. Try it, miss it, review it, and try again. databricks certification data-analyst-associate free-edition https://bricksnotes.com/blog/what-data-engineers-really-do https://bricksnotes.com/blog/what-data-engineers-really-do Sun, 12 Jul 2026 04:33:18 GMT BricksNotes Editorial Data engineers help data move safely from where it is created to where it becomes useful. A calm, beginner-friendly guide to the real job. data-engineering data-pipelines beginners databricks https://bricksnotes.com/blog/free-edition-refresh-2026 https://bricksnotes.com/blog/free-edition-refresh-2026 Fri, 10 Jul 2026 03:13:04 GMT BricksNotes Editorial We reviewed every chapter for accuracy, tidied the writing, and updated the steps for how Databricks Free Edition works today, including serverless compute and the newer chapters on pipelines, Apps, and Genie. databricks free-edition update data-engineering https://bricksnotes.com/blog/innovation-data-ai-people-together https://bricksnotes.com/blog/innovation-data-ai-people-together Sun, 05 Jul 2026 23:37:13 GMT Team BricksNotes A retail forecast that failed, a hospital model, a fraud check, and a logistics team. Real stories about why innovation lives where data, AI, and people meet. data-and-ai innovation data-engineering https://bricksnotes.com/blog/databricks-builders-advantage-data-ai https://bricksnotes.com/blog/databricks-builders-advantage-data-ai Thu, 02 Jul 2026 12:55:18 GMT Team BricksNotes The people who lead Databricks are the same people who built the underlying technology. Here is why that matters, and why data and AI is a direction, not a trend. databricks data-and-ai leadership