Databricks job clusters are stricter than interactive clusters
While converting a list to a dataframe, I got a type error, but only on a job cluster, not on an interactive cluster. How to get around that and why does it happen?
Tagged: Python
Write data to one CSV file in Databricks
Exporting data to a CSV file in Databricks can sometimes result in multiple files, odd filenames, and unnecessary metadata—issues that aren’t ideal when sharing data externally. This guide explores two practical solutions: using Pandas for small datasets and leveraging Spark’s coalesce to consolidate partitions into a single, clean file. Learn how to choose the right approach for your use case and ensure your CSV exports are efficient, shareable, and hassle-free.
The Databricks Debugger
Exploring the Databricks Debugger: Writing flawless code on the first try is a dream, but debugging is a reality for most developers. In this post, I dive into the new Databricks code cell debugger, sharing my first impressions and tips for getting started with this powerful tool.
How to enable system tables on Databricks
System tables on Databricks can help us monitor and manage our Data Warehouse. In this post I’ll show how to enable them and how to install the Jobs Dashboard based on system tables.
Working with Excel files in Databricks
Excel is one of the most common data file formats, and, as data engineers, we are required to read data from it on almost every project. Working in Databricks, you can read and write Excel files, but you need to pay attention to some pitfalls.
The mystery of the vanishing variable
A Python variable has vanished! Can you help the confused Data Engineer find out why?