DA DI DE Data

Observations on Data Engineering

One of the most significant outcomes of increased processing power and adoption of the internet has been the explosion in ever growing volume of data. Available storage space has gone from minuscule kilobytes only a few decades ago to now having virtually unlimited cloud storage.

Data creation, processing and storage now employs a large portion of our global workforce. Chances are that either your industry or even your role has shifted towards dealing with more data.

We now produce data almost as naturally as an increase in disorder predicted by the second law of thermodynamics.

The title of the post is inspired by the “Be My Lover” song by La Bouche famous for its “la-da-di-da-dah” hook. Increase in data volume also drove further specialization into data engineering roles — many of those roles have titles like “DA, DI & DE”.

Let’s look at those data engineering related functions. The goal is not to list out clear boundaries between these functions as many overlap and differ between companies.

Database Administrator (DBA):

• General Role: Managing and maintaining database systems to ensure their performance, availability, and security.
• Skills: Database management systems (MySQL, PostgreSQL, Oracle), backup and recovery, performance tuning, and security.

Data (Pipeline) Engineer (DPE):

• General Role: Building…