Chen Hirsh's Data Engineering Blog

Chen Hirsh's Data Engineering Blog Blog

0

Databricks

Think Twice Before Deleting a User: Avoiding Ownership Chaos in Databricks

Deleting a user in Databricks might seem harmless—until workflows start failing, SQL queries break, and ownership chaos unfolds. In this post, I share a hard-learned lesson about Databricks ownership, how to prevent disruptions, and what to do if you’ve already made the mistake. Learn best practices for managing SQL objects, workflows, and user permissions to avoid unexpected failures. Because when it comes to user deletion in Databricks, thinking twice can save you from a major headache.

0

Data Engineering / Databricks / Python

Write data to one CSV file in Databricks

Exporting data to a CSV file in Databricks can sometimes result in multiple files, odd filenames, and unnecessary metadata—issues that aren’t ideal when sharing data externally. This guide explores two practical solutions: using Pandas for small datasets and leveraging Spark’s coalesce to consolidate partitions into a single, clean file. Learn how to choose the right approach for your use case and ensure your CSV exports are efficient, shareable, and hassle-free.

Databricks / Python

The Databricks Debugger

Exploring the Databricks Debugger: Writing flawless code on the first try is a dream, but debugging is a reality for most developers. In this post, I dive into the new Databricks code cell debugger, sharing my first impressions and tips for getting started with this powerful tool.