Dattell report reveals most popular technologies for data engineers

Discover some of the most popular tools employers are expecting prospective data engineers to work and be familiar with, according to Dattell.

Demand for technology professionals has been rising after a brief downturn following the start of the COVID-19 pandemic. And one tech job with slots to fill throughout the industry is that of data engineer. Responsible for creating the infrastructure needed to consume, process and store vast amounts of information, data engineers are seeing an increase not just in job openings but in the tools that help them do their work.

Current data engineering job market

A report released June 29, 2022, by data engineering provider Dattell looks at the current market for this career. To compile its findings, Dattell collected and analyzed a huge amount of data itself, matching specific technologies with the number of job openings.

Eyeing the job market for data engineers, Dattell analyzed 340,000 different job postings. Among these, 35% were for positions in data orchestration, 30% for data storage, 29% for data visualization, and 6% for data processing.

Additionally, Dattell analyzed the languages used by data engineers. Python was the most popular, preferred by employers 38% of the time, followed by Java at 33% and SQL at 22%. Combined, the three of them account for 550,000 job openings.

SEE: Hiring Kit: Database engineer (TechRepublic Premium)

Moreover, compensation for data engineering jobs varied from $60,000 as a starting salary to as high as $180,000 for more advanced positions. The highest compensations were offered to those working with Kubernetes, Elasticsearch, PostgreSQL and Terraform, with salaries landing at $140,000 and higher. And while there are a healthy amount of openings for those working with Tableau and Power BI, most of them offer less than $100,000.

Data engineering technology trends

In its research, Dattell compiled a list of the top 20 most popular data engineering technologies, pointing to such tools as Microsoft Power BI, Terraform, Chef, Spark, Elasticsearch, Hadoop and Kafka. The products cited offer support in areas spanning data storage, data orchestration, data processing and data visualization, indicating that no single segment of data engineering dominates the rest.

Based on Dattell’s research, the five most popular data engineering tools are MongoDB, Tableau, Kubernetes, PostgreSQL and Ansible, covering data storage, data orchestration and data visualization technologies.

Among the tools themselves, Tableau and Kubernetes took the top spots for the greatest number of job openings. Other tools generating a high number of open jobs were Ansible, Hadoop, Terraform, Splunk, Power BI, MongoDB and PostgreSQL.

And among the data orchestration tools examined, Kubernetes is by far the leader, followed by Ansible. Though both products fall into the same space, each one is used differently. Kubernetes lets professionals manage and maintain container health, while Ansible allows them to deploy configuration changes and manage updates.

Rising popularity of free and open-source tools

With so many free products available, employers don’t see a great need to pay for data processing tools, according to Dattell. As a result, companies are looking for employees and consultants with expertise in free and open-source technologies, such as leading tools like Apache Spark and Apache Kafka.

For example, with data storage technologies, paid tools are preferred by 59% of employers, leaving a still hefty 39% who prefer free tools. Among the most popular data storage technologies analyzed by Dattell, MongoDB was the most popular, followed by PostgreSQL, both of which are free. Some products, such as Elasticsearch, come in both paid and free versions.