SAS to Big Data Migration Platform
A strategic migration from SAS to a Dataiku, Hadoop and Spark platform with CI/CD across environments.
Business issue
Why I was brought into the project
The credit risk platform needed to migrate critical SAS pipelines to a modern Big Data stack while preserving production reliability and governance.
Context
The environment around the project
A credit risk team needed to migrate SAS pipelines to a Big Data platform.
Functional environment
The functional context covered credit risk data pipelines, governance, documentation and production delivery constraints.
Technical environment
The technical environment combined Dataiku, Python, PySpark, Hadoop, Hive, Big Data, SQL, SAS, Confluence and Agile practices.
Challenges
The migration involved critical pipelines, multiple execution environments, HDFS data design, CI/CD, user support and cross-functional coordination.
Solution
My contribution and its impact
My contribution to the project
I contributed to the migration plan, designed target HDFS structures, implemented optimized Dataiku and PySpark pipelines, and set up CI/CD across environments.
- Migration plan from SAS to Dataiku and Hadoop
- Optimized PySpark and Dataiku pipelines
- CI/CD process across development, staging and production
- Platform services for versioning, governance and documentation
Impact
The project helped move critical risk processing toward a more maintainable and industrialized platform while supporting users and enforcing development standards.
- Modernized credit risk pipeline execution
- Improved development and deployment practices
- Better governance and user support on the new platform
Impact metrics
Approach
How I structured the work
- Analyze SAS architecture and define the target Big Data and Dataiku structure.
- Develop optimized PySpark and Dataiku pipelines with industrialized scenarios.
- Set up CI/CD, documentation and user support practices across environments.
Takeaways
What I learned from this project
- Large platform migrations need governance, training and support in addition to pipeline development.
- CI/CD becomes a central reliability lever when multiple environments are involved.
- Modernizing legacy SAS assets requires understanding both old architecture and new platform constraints.