How to enable system tables on Databricks
System tables on Databricks can help us monitor and manage our Data Warehouse. In this post I’ll show how to enable them and how to install the Jobs Dashboard based on system tables.
Category: Data Engineering
Cleaning Data with Spark
Cleaning data is a very common task for data professionals. In this post, I demonstrate a few common data cleaning task with spark Python and SQL.
Databricks workflows for-each – not quite there yet
Databricks recently added a for-each task to their workflow capability. How does it work and what are its limitations?
Working with Excel files in Databricks
Excel is one of the most common data file formats, and, as data engineers, we are required to read data from it on almost every project. Working in Databricks, you can read and write Excel files, but you need to pay attention to some pitfalls.
How to plan a successful Data Architecture
In the era of cloud computing, it’s really easy to create and change data services, so in each project we have architecture decisions to make, and each developer has to deal with these considerations.
This is a short summary of a meetup I gave about Data Architecture in the “Microsoft Data Engineers Club” community.