Since I was little, I’ve been obsessed with numbers, from counting the number of steps on a staircase to how many tiles were in my room. From a young age, I started learning math and figuring out how to average. I started taking probabilities and random data out of anything I could. When it came time to choose a university major, I decided to study computer engineering because, in addition to math, I also really like technology and knowing how things work.
When I began taking database classes, a new world opened up for me. I found that I was able to get any information I wanted based on facts. From there, I began to learn more about databases just as a hobby. I realized that it was very different to work with a database with thousands of records versus one with millions of records — that’s how I was introduced to big data.
A year ago, I had the opportunity to enter the data apprenticeship program at Wizeline. Our small group consisted of people who already had some skills related to data and the basics of programming.
During this eight-week program, we were dedicated to learning tools and concepts related to data, such as distributed systems, data warehouses, data lakes, orchestration, and data streaming. We also explored cloud computing platforms, and some of us even got certified in one of them, such as Azure, AWS, or GCP. During our guided sessions and mentoring, the company’s best engineers helped us solve any issues and taught us how to apply what we learned to real projects.
At the end, we did a project where we applied everything we learned and showed it to the entire company. You can learn more about the project in this blog from team members Laura, Mariana, and Andrea. Thanks to this great experience, we had the opportunity to become data engineers in the company and to be assigned to real projects with clients.
Since completing the apprenticeship program, I have worked on two data-related projects. I’m going to provide an overview of each.
Data visualization
For this project, we had a very old database set containing information related to cars. We had data such as model, brand, classification, gasoline type and consumption, performance, and CO2 emission.
To start, we performed an analysis of all the data to help us determine which data we wanted to show. We decided to explore the relationship between each brand’s CO2 emission and performance. To do so, we had to create a pipeline with Airflow to extract and clean the data, then transform it into a format that would help us process it. Next, we did an analysis using MySQL queries, which delivered some interesting insights. Finally, we created graphics to display our findings with the help of Python libraries like Pandas and Seaborn. This project was shown to the solution architect team to give them an idea of what the data team is capable of.
Salesforce migration
This data migration project, which is still ongoing, has been a great challenge for the whole team. The objective is to migrate data from Salesforce to an improved version of it. The main steps involved so far have included: extracting objects from Salesforce and several payment platforms, analyzing the data contained in each, using mapping to identify where the data is going, and transforming the data according to the needs of the new organization.
In the process, we’ve stored the data in a bucket in AWS and made queries with MySQL. For the data transformation, we’ve mainly created stored procedures to make a more automated process. We have also made some scripts with Python and Pandas to optimize the queries. The biggest challenge we’ve encountered is handling this volume of data. Due to the large volume, optimizing all of our processes has been key for this project.
Conclusion
In both of these projects, I have faced real problems related to data and used various tools that I’ve learned about in recent months.
I am very happy that I made the decision to enter the apprenticeship program, as it has helped me turn my hobby into what I do every day. I was able to achieve this with the help of the data team, allies, and, of course, my teammates. I am excited to continue learning and taking advantage of all the opportunities the company offers us. I’m also eager to find out what challenges will be in store for me in my next project.
