If AI builds the pipelines, what's left for data engineers?
I came across this interesting company, Zeit AI (YC S24), which recently raised €5M. They’re building an “autonomous data engineer” that can connect 600+ systems and automate much of the work involved in bringing data together.
The interesting part isn’t the technology itself.
With AI making coding and integration increasingly accessible, connecting systems may become much easier.
But connecting systems is only one part of data engineering.
Maintaining trustworthy definitions, permissions, lineage and all those messy edge cases in production is where enterprise data engineering earns its keep.
And this is what I find interesting about AI and data engineering.
If AI makes the building part easier, the value of the data engineer may shift towards understanding what should be built, why it should be built, and whether the result can actually be trusted.
The code may become easier to generate.
The judgment behind it may become even more valuable.