Developing a New Data Pipeline to Improve Supply & Demand Forecasting and Increase Productivity by 25%

Our client, an American multinational food manufacturing company, partnered with Wizeline to improve data capabilities for business analysis, leveraging technology, data science, cloud, and automation to enhance supply & demand forecasting and increase productivity by 25%. 

Read more to see how the Wizeline team implemented an optimized, automated workflow to streamline data processing and cut the process for end-to-end automated predictions from an unpredictable amount of time to an average of 1 hour.

Data is only helpful if transformed into timely insights that support related business decisions. Our client is focused on maximizing its data for strategic decision-making to advance its digital evolution. However, the data used during the decision-making process came from diverse, unconnected sources from a wide range of contributors and relied on manual processes, jeopardizing quality and accuracy. In addition, the software engineering processes were not optimized, siloing knowledge and not documenting methods properly. All of this impeded our customer’s ability to quickly provide business forecasts to the company’s decision-makers.

The Wizeline team created a new data pipeline, a crucial element to enable our customer to achieve its digital evolution by leveraging data analytics for decision-making. The pipeline aims to periodically show supply & demand forecasts to the company’s decision-makers. These forecasts help shape commercial strategies and inform essential business decisions.

To develop the pipeline, Wizeline implemented an optimized, automated workflow to streamline data processing by:

• Implementing Infrastructure as Code (IaC) to reduce maintenance and ease replication between AWS environments, making it possible to include the solution in the Disaster Recovery plan
• Using Github Actions to develop CI/CD automated processes
• Improving security through Docker image scanning
• Reducing the length and cost of the machine learning (ML) model training instances by replacing SageMaker notebooks with training jobs, following SageMaker’s best practices
• Reducing the ML training model complexity from n3 to n2
• Boosting code quality by implementing a code linter and refactoring it, increasing its robustness
• Accelerating new feature development through continuous agile testing to the code pipeline
• Executing mail alerts, allowing immediate visibility of issues
• Documenting every step of the process, including the use of containerized environments to speed up onboarding, knowledge transfer, and scalability

Technology Stack

• AWS (CloudFormation, Glue, SageMaker, Lambda, Athena, S3, ECR)
• Github and Github Actions
• Docker
• LGMB
• Python (Boto3, Sklearn, Diagrams (mingrammer Diagrams as Code) Pandas, Profiling, Pdoc, Pytest, Pylint, GreatExpectations 0.13.20)