Looking for some advice on how your teams bridge the gap between Data Engineering (DE) and Data Science/Generative AI (GenAI) projects. Right now, my team’s primary focus is migrating ETL data to the cloud (almost finished!). Beyond migration, we build scalable pipelines and optimize/automate workflows.

While migrating to the cloud is great, sticking solely to this routine could limit our learning. That’s why I’m curious – how do your organizations involve DE teams in GenAI projects and research? After all, data scientists and ML engineers often have deeper knowledge in these areas.

To foster collaboration, we’ve created a DE4DS workspace (Proof-of-Concept) to shift reporting from traditional systems to the cloud. Additionally, we’re exploring ways to involve DE in GenAI research POCs.

What other suggestions do you have for integrating DE teams into the exciting world of AI?

Looking forward to hearing your thoughts and experiences!

Suggestions to integrate data engineering team for AI and Gen AI projects/research.