How I became an Analytics Engineer
The Analytics Engineer: A Comprehensive Introduction
Welcome, everyone, to a space designed to give clarity about a relatively new role in the data team – The Analytics Engineer. This title is fairly young, and there are limited resources available online. Therefore, in this conference session, we aim to explain the role of an Analytics Engineer, explore its significance in the business, and discuss how you too can pave your way to this ever-evolving role.
Understanding the Role of an Analytics Engineer
Several years ago, Spotify recognized a missing piece in their team structure. They needed an individual who could bridge the gap between understanding the business, software engineering practices, and technical know-how. Essentially, they required someone who lived in the intersection of deep business knowledge and robust technical skills. In addressing this need, they inadvertently birthed the role of an Analytics Engineer.
We are looking for an expert in building out analytical data layers to join the band and help drive a data-first culture across the ad supported business in Spotify.
The Analytics Engineer's role exists 'in-between' a Data Engineer and a Data Analyst. The Data Engineer's role involves preparing and maintaining the data infrastructure, while the Data Analyst uses this data to solve problems and answer business questions. The Analytics Engineer bridges the gap between these two roles, introducing good software engineering practices to the analysts' efforts and bringing a business mindset to the Data Engineering tasks.
Data Roles – A Closer Look
Data Engineer
These individuals focus on more technical aspects of data management. They handle data integrations, manage data pipelines and endpoints, and maintain the data warehouse.
Data Analyst
On the other hand, Data Analysts deep dive into business-oriented tasks. They answer business-related questions, work with business stakeholders, build dashboards, do forecasting, and handle reporting.
Analytics Engineer
This 'in-between' role ensures clean, transformed data is provided for analysis. Applying the best software engineering practices to data-related tasks, maintaining documentation and definitions, and training business users on using data tools are some of the essential responsibilities of an Analytics Engineer.
Tools of the Trade
Given the bridge-like nature of the role, Analytics Engineers utilize a spectrum of tools. Some of these include:
- Cloud data tools like Snowflake, BigQuery, and Redshift for data storage,
- ETL tools such as FiveTran, Talend, and Stitch for data loading,
- DBT or Dataform for Data modelling, and
- BI tools like Looker, Mode, and Tableau for data presentation.
The Journey to Becoming an Analytics Engineer
The role of an Analytics Engineer requires a unique blend of business acumen, technical prowess, and data management knowledge. This combination doesn't come overnight – it's a career journey, refined and tailored over time. Below is the career trajectory for an Analytics Engineer:
- Start with a strong understanding of databases concepts, SQL, and data pipelines, alongside insights into reporting and software engineering best practices.
- Experience in data warehousing, ETL development, and working with an ERP system gives valuable practical experience.
- Branch out into being an IT systems specialist or SQL developer to deepen the technical knowledge combined with business understanding.
- Finally, merging all the skills leads to the role of the Analytics Engineer, a perfect blend of technical and business knowledge.
In Conclusion
The Analytics Engineer is a career path with vast opportunities and immense potential. For those interested to pursue this path, focus on continually expanding your knowledge around databases, SQL, data pipelines, reporting, and software engineering best practices. With the advancing technology and growing businesses need to make data-driven decisions, the role of an Analytics Engineer will continue to become increasingly significant.
If this piece sparked your interest and you want to dive deeper into the topic, feel free to reach out or find additional resources in the GitHub repository linked here.
Video Transcription
Again, welcome everyone. Um I hope you're enjoying uh the conference. There's certainly a lot of like very interesting talks and I appreciate that you are attending to. So a little bit an introduction about myself. I am an analytics engineer as the title says.And um what I'm going to talk today is how I end up with this title basically because this is a fairly new uh title. I would say there's not much also online. Uh So, and I really like it and I want to share it and I want more people to know about it and what are analytics doing analytics engineers doing. Uh So you can also be one if you want. Um I'm not going to introduce myself a little bit more because we're going to go more into details into my career. So, but first, let's start what is exactly the analytics engineer role. So a few years ago in a Spotify, um they realized that they had, they were missing a person, basically, they were missing someone um that could get like to know the business, but also the software engineer and practices and the technical stuff because they were seeing that the analysts, they're very deep into business knowledge, but they lack of the computer science background basically.
And the data engineering, they have it, but they're very deep into the technical stuff. So they needed something in between. They needed like a connection and this is what they posted, they didn't know what they needed basically, but they needed something and this is what they put in their, in their show of opening. Basically. we are looking for an expert in building out analytical data layers to join the band and help drive and the data first culture across the ad supported business in Spotify. And um they invented a title kind of like data specialist, something like that. But basically, this is how they end up uh hiring their first analytics engineer for the company. So as I said, basically, they created this role where lights a little bit in between the data engineer and the data analyst. So if we think about the roles in the data team, most of us who already know what a data engineer does and what a data analyst does as well, and we know how the connection is and how they deal with the data. So data engineers are going to be preparing and maintaining the infrastructure that the data team is going to be using so very deep technical oriented and data analyst, they're going to be using that data to answer questions to solve problems.
So they are going to transform that data basically to translate it to English, to answer the questions that the business has. So they need to have a lot of business knowledge as well or domain knowledge. So here is where the analytics engineer comes in the line in between the data engineer and the data analyst is and the analytics engineer in between is a little bit blurry. So they are going to be analytics engineers that are going to be more on the data engineer side and analytics engineers that are going to be more on the data analyst side. But basically what the analytics engineer is going to do is going to introduce the good software uh engineering practices to the efforts of the data analyst and the data scientists as well. And it's going to bring an analytical and business um kind of mindset to the efforts of the data engineering if we go a little bit more deep into the responsibilities. So we see here, the data engineer is going to be taking care of the platform. So they're going to build custom data integrations, they're going to manage the the pipelines, uh they're going to develop and deploy the machine and learning and points.
Um And they're going to also take care of the data warehouse where the data analyst is going to do deep inside works. They are going to be answered questions about like how's the net revenue like uh what happened when a user does this, what are our users doing? Uh They're going to work with business stakeholders, they're going to be working with the business people that know nothing about data and tech and they could be going to be building uh dashboards doing forecasting reporting. So the analytics engineer is going to provide clean transform data for this analysis for the analyst. So the analyst doesn't have to do the technical work. Basically, they and they can do what they know best which is analyze the data. They're going to apply software engineering, best practices to the code. So version control um C I CD testing um and they're going to know they're going to know best how to write good code basically. So they're going to maintain this code uh using the best and applying the best practices, also maintain documentation and definitions and they're going to train the business users on how to use all these tools that they are maintaining. And I'm talking a lot about tools. So let's go a little bit deeper into that. So since the analytics engineer goes a little bit from the data engineer to the data analyst, there are a bunch of uh steps that the data goes through that the analytics engineer is going to be involved with.
So the first one is data storing and here we're going to talk about uh cloud data warehouses like a snowflake, Bigquery or Redshift. So the analytics engineer should be familiarized with this. Uh tools as well. Uh We're talking about data loading. So here we can talk about ETL tools like five turn lib or stitch. Then comes the data modeling. And they are now very cool tools like DVT or data form that kind of like started with the analytics engineer movement. And at the end they need to know also about data presentations. And here we are talking about B I tools like looker mode or or Tableau. Yes. So there's a question that says as the analytics engineer does the data preparation part for the data analyst. Yes, they do. Um So the, let's say um I'm going to give like a more clear example. So let's say that now the business needs a new KP I depending on a, a new kind of like data point. So the analytics engineer is going to take care of um making that data available. So of course, there's going to be a data engineering work that's going to maybe like get the back end events or something like that. But basically what I would do is I would try to build the pipeline in the ETL tools and I'm going to do the data modeling using in my case, I use DVT a lot and, and then I'm going, so these models are going to be available for the data analysts and they're the ones that are going to know what the business users want, right?
And I'm going to make this data available in the va tool as well. But at the end, the data analysts they can build on top of whatever I did. I clean the data, the data that I clean and I made available for, for them. So how did I get myself to the analytics engineer role? So this is my geographic journey. So as I said before, II, I come from Argentina and in there, I studied systems engineer and started working there as well. And at the moment, I'm now living and working in Berlin in Germany. So let's go a little bit more into details. And um I'm going to be telling exactly how the knowledge that I acquired over the years. They um I use them as a metic engineers today. So as I said, I started studying systems engineers in 2010 and I finished in 2015 um overall. So I learned a lot about computer science about managing it projects as well. And in there, there were several subjects that were about how to write code, how to use the right patterns. So things like the drive concept, for example, how to write clean code, basically how to make it maintainable. And um and uh I, I'll get to the question at the end that one. So and then there were also several subjects about uh databases, data warehouse, and information, data, information management.
So all of these are the subjects I would say they helped me the most to get where I am today because I learned the very good practices of a software software engineer and also learned all the basics for data management as well in there. I also started to work as a technical and functional implementer for an ER P company. So my work there was a little bit mixed. I wasn't fully technical, but I also wasn't fully on the business side. So I needed to learn a lot about payroll. I was specialized in payroll mostly, but I need to learn a lot about accounting about taxes as well. And then I needed to use my technical knowledge to write the queries that people would use to calculate payroll to calculate taxes as well. So and there I started to work with databases um for the first time aside of like the studies and I started with SQL server and relational databases. So it serve it, it helped me a lot to understand how business users think how the accountant thinks. For example, in this case and how I could apply my technical knowledge to help them do that work. After I graduated, I started working as a consultant, I started working as an er P consultant. So I continue with this mixed profile. So I never was going fully technical and never was going fully on the administration business kind of side.
Uh Again, I was using a lot of my knowledge and still forming myself learning more accounting, more about taxes and com combining those to help people to use their systems, to build reports, to understand what the data looked like when there was an issue. So I had to write a lot of SQL a lot of procedures I had to write even ETL um to bring the data from other systems to the er P system, for example. And I also learned a lot about management about project management as well because I had to do this was a very small company. So it helped me a lot to be one woman show basically in the projects. And then in 2018, I decided to go to Europe and travel and I end up living in Berlin and that's when I started working as an IT systems specialist. And this job, I thought it was perfect for me. But because again, I was going to be working with an ER P system in the finance department, helping accountants to do their jobs basically. So doing the reporting, integrating the er P without the tools. Uh But then soon I realized that I was a bit over with er P systems because they're all different, they all changed, they're all very customized. So at the end, it's always learning again and again how they work.
So that's when I move and I moved to SQL developer because I wanted to be for the first time, a little bit more technical and that's where I started working for the first time as well with data warehouses. I knew the theory, I hadn't started, but that's when I started building a data warehouse for real. It was an SQL server and I had to really, really get my game with dimensions and um and fact tables and what are data parts and how to get the data from other places. But at the same time in the company, they were doing an ER P implementation. So I started to work a little bit on the project and soon enough, they moved me to a complete new team to implement and do like the whole data migration of the er P. So once again, I couldn't get out from the er P side. Uh And I still had like this mixed profile once again, even though I tried to go a little bit technical and then it was good to me that this is what I wanted, I didn't want it to be 100% technical. I wanted it to be something in between and I wanted to be technical and not and not on the er P side, right?
So that's when I moved to analytics engineer and I really enjoy it because I get to know a lot of like the domain knowledge, I get to know a lot about the business but I don't have to deal with the business stakeholders all the time because I have the analysts and the data scientists doing that for me.
And there are also the date where I can just be there with my music and just working in my code and being deeply technical at the same time. And now um there are a lot of questions that I'm going to answer in this part of the, of the talk. So for example, what is the best way to prepare and practice SQL? Uh Can I know what are the professional shop qualifications? We should have to be an analytics engineer. What skills and tools uh do you think are important for becoming analytics engineer? Except data preparation and data exploration techniques? And if I use Python and R in my work, so these are the things that I feel are the most important ones to become an analytics engineers. So first of all databases concepts um learn a lot about what are databases, how they work, what are data warehouses, data modeling. So read about relational models C A date, for example, read about Kim Bolt, read about Inmon, read about data bolts. You don't need to know the very detail, but you at least need to see what is out there. And also the more you know, the more ideas you're going to get, right. Learn about SQL SQL is everything for me.
So this is, this is I don't personally I don't use Python or R but I do know about other analytics engineers. So for example, in Air B NBI know they use Python and they use Jupiter as well. So yes, I would recommend Python as well, but mainly SQL. So I do the whole data modeling in SQL. And I've been working with SQL for the past years. So you should know from basic queries to window functions. How what is the best way to prepare and practice SQL? Download anything like postcards, for example, my SQL create a database or get a database from somewhere. There are a bunch of open websites and start querying. There's also websites like hackerrank, I really, really like Hackerrank for example, and you can go there and you can write queries and those are similar to real life cases. So that is very good experience as well. Read about data pipelines, Etlelt. What are those things? What is the data pipeline again? You don't need to know like at the very detail, there are also a lot of tools but you need to understand the concepts, learn about reporting. So have a general knowledge on how to present the data using data visualization tools. And at the end, I also it's very important to know the good practices. So the software engineer and best practices for code below, what does the good code looks like? What should you do to get like good code and maintain code?
Um So I hope you, you enjoy it. I hope you, you understand a little bit more what my job is about. Uh you can find me in github. And in there, I have a repository where you can find these slides and in the slides, not only I put the links on the tool slide link before, but there's also this last slide where I put the link to a lot of um articles that I think are very interesting, especially the one, for example, at Spotify, where I took the, the, the quote at the beginning and at the end, you're going to find these analytics readings.
This is a repository that has a lot of links to a lot of useful resources to learn about all the concepts that I mentioned before. Do you have um any more questions? I think we are also uh good with time. So if you have any more questions, I can stay for a few more minutes. Uh um Anyways, I appreciate uh you attending to this talk and um I hope you continue enjoying the conference. Yes, it's uh data engineer is similar to analytics engineer, but the data engineer is going to be more technical focused. So they're going to care more about infrastructure. Basically. Great. Yeah, you can also, I also left my linkedin here. So um you can also reach out to me there, you can connect uh if you have questions or anything, I'm always uh willing to talk um and discuss and also share resources that I have if, if, if you need any uh if you want to get like deep your knowledge and something else. Thank you. Bye.