This conversation has been edited for length and clarity.
The 30-Second Splash
Ido: Most underrated skill in data?
Shachar: Communication.
Ido: London, Tel Aviv, or New York?
Shachar: London — that's where I am, so I have to say London.
Ido: One application that's not on the front screen of your phone?
Shachar: PayPal. It used to be. It's not anymore — and I think there's something to unpack there.
Ido: Is AI going to replace data engineering?
Shachar: I think it's going to enhance data.
Ido: That's the politically correct answer — but we'll dive deeper into that in a moment.
What do you actually do as a data advisor?
Ido: In two sentences — what does your day-to-day look like as an advisor today?
Shachar: There's no typical day; every day is different. But fundamentally, I've spent the past 20-plus years in data. I started as a DBA, moved into a team lead role early, and I've been at the intersection of data and leadership ever since. Over those years I saw a lot of companies that had plenty of data, good teams, and decent platforms — and yet nothing was working. No one was happy.
So in 2023 I took a step back and asked: why are we still failing with data as an industry? I realized two things. First, I have the playbook — I've spent my whole career fixing broken data teams. And second, I can have far more impact by helping CDOs and VPs of data deploy that playbook in their own organizations than by being a chief data officer at one company at a time. So that's what I do now: I work with data leaders to help them succeed and fix their teams.
Ido: Is there one playbook that makes data organizations work — one that hasn't changed over the years?
Shachar: It's not one playbook, but there are a handful of patterns — a few recurring reasons data teams fail — and those haven't changed. The technology has changed enormously, but the underlying reasons are the same: how do you connect data to the business? How do you focus on impact? How do you find your sponsors and champions, and create the right culture, incentives, and processes? There are a lot of small things data teams can do to elevate themselves, and a lot of teams miss them. I call them the missing ingredients of data. They're the common reasons I see data organizations fail over and over again.
What did data engineering look like 20 years ago?
Ido: You've been in data for more than 20 years. Take me back to the early days — where was the focus, and how has it evolved?
Shachar: When I started, I was a DBA on Oracle 9206 — to date myself a little. Back then we had servers in data centers, and we handled the operating system upgrades, the storage, the networking, the backups and recoveries, building and rebuilding indexes. All that platform and infrastructure work took an enormous toll on us.
The other thing is that we were essentially blocking software engineers from making changes to the databases. Every time they wanted to log a new piece of information, they had to come through us — we'd model the change, deploy it, make the schema changes. But that meant we had conversations first: what data do you want to keep, why, who's going to use it, and what for? Do we even need it? We made sense of the data before it ever went into the database.
In a way, that was great — it's the opposite of what happens today. The data was always modeled and always relevant to what people actually wanted to do. Software engineers didn't love working with me because I slowed them down, but once the data was in, it was completely useful. If I had to summarize the good old days in two points: infrastructure work ate a huge amount of our time, and the data was always modeled and thought through. Both are basically the opposite of what's happening now.
What broke when we moved to data lakes?
Ido: We shifted to faster, less dependent ways of creating data — but we broke the governance, the modeling, the structure. How did that change the role of the data engineer?
Shachar: If you walk the timeline: we had the good old days where everything was in relational databases — structured, highly governed, highly modeled, highly usable. Then Hadoop and open source came along, we decoupled compute from storage, and we said: store whatever you want, wherever you want, regardless of how you're going to use it. Make sense of it later instead of before.
That broke the contract. People assume it accelerated everything, but I'm not sure it did. It removed the dependency between software engineers and data teams, but everyone today is talking about data contracts and data governance — and we used to have those. There has always been a contract. We just gave up on it, and now we're trying to get it back.
The other consequence is that software engineers stopped thinking of themselves as data producers. They create a lot of data that's useless, unusable, or not high enough quality for what we need — and then we have to go back and fix it later, once it already exists.
Ido: It's funny to think that all that technological improvement didn't actually make the data team's job easier.
Shachar: No — in a way it made the role a lot more chaotic. In the past there was a contract, there were conversations, there was governance. Now you're handed data and you don't know what it is, what it means, what its quality is, whether it's complete or accurate. And you have to make sense of it. Data has become an afterthought for a lot of product teams, and data teams end up dealing with much lower-quality data than we had 20 years ago.
What two tips would you give a CDO today?
Ido: You work with a lot of companies on exactly this. What are two tips you'd give a CDO to make use of their data and not get stuck with low quality and no real value?
Shachar: Tip number one: it's not about technology. Whatever your problems are, technology is most likely not the root cause. It's usually people, process, or culture. A lot of CDOs and VPs of data come into an organization and immediately launch a modernization or replatforming project — moving off SQL Server onto Snowflake or Databricks. That's fine, but think about what actually changes as a result. If the migration succeeds, you're doing exactly the same things you do today, just on a different platform. The business doesn't see the change, and the business doesn't care.
So if I came into a new organization, even with an old, legacy platform, I'd focus on bringing business value first and then gradually, incrementally change the technology underneath — with a transition plan, rather than announcing a full-blown modernization project that eats half the team's time and is almost invisible to the business.
Ido: So: focus on the business, not the technology.
Shachar: Yes. And always be creating value.
Where do you see AI fitting in?
Ido: There's a new wave coming — it's called AI. How do you think it's going to help data organizations, and where are the opportunities?
Shachar: We all have opinions, and nobody actually knows what's going to happen. But here's what I think. First: it's never been easier to build the wrong things than it is today, because everybody's a builder and everybody can build anything. And from an analytics perspective, it's never been easier to ask the wrong questions and get the wrong answers. AI gives you the illusion that everything is possible — but there's often a good reason we weren't doing those things. That makes this a leadership problem as much as anything: thinking about why we're doing what we're doing, and what the purpose is.
Second: AI continues a trajectory we've been on for decades. In the 1950s, to make a computer do something you had to know exactly which CPU and architecture you were on and move bits between registers. Then C abstracted a lot of that away — you describe more of what you want and it gets translated down. Then C++ and Java let you describe the world in objects, closer to how we see it. Then Python, where a one-liner does an enormous amount. Each step moves further from how you want things done toward what you want done. When you talk to Claude, you tell it the end result you want — not how to get there. That's the shift we've been seeing for 60 or 70 years, taken to the extreme.
The interesting debate I keep having is this: should you even care how things are done, if the result is the one you wanted? A lot of people say they don't care what code Claude generates as long as it passes their tests. That's a valid question, and I don't know the answer. But my hope for data engineering is that we deal less and less with the specifics of how things happen, and move toward describing what we want — what data we want, what it looks like, how we'll use it.
Does that mean the business takes over the data work?
Ido: Let me take the opposite seat for a second. I agree that making a data organization succeed is about people, processes, and culture — and I think AI is a chance to shorten those processes and build a better culture. AI lets us shift right: give more technical responsibility to the business, to the people focused on impact rather than technology. The ones who create the data can close the loop and use it themselves, with less dependency and fewer handoffs.
Shachar: There are fewer handovers and fewer people involved — agreed. The only assumption baked into that is that the business operators and analysts closer to the work actually know how to manage data. And the real question is: in this new world, is managing data still a profession? I believe it is. Data is an asset that needs to be closely managed — there's still room for governance, management, and optimization. The assumption that business stakeholders know what they want, and know how to describe it in words, is something we'll need to validate.
Ido: We'll see. The future's going to be interesting.
The photographer analogy
Shachar: Enabling more people to do things is good — but let me give you an analogy. I have the latest iPhone. Its camera is probably orders of magnitude better than what professional photographers had 30 or 40 years ago, and I take hundreds of photos and videos a month. Does that make me a photographer? No. If I had a big event, would I hire a professional? For sure. They understand light, framing, how to create the perfect moment — things I don't. They take fewer pictures, and all of them are good, because they're professionals. I take hundreds, and one or two come out good.
The profession isn't going away. Enablement is good — it means more photos in the world, more of the world documented than ever. But it doesn't replace photographers. It's the same with data. Managing data is a profession; there's something real to know. More people can be involved in creating data, but I'm still not convinced data engineering is going anywhere.
What's your advice for people early in their data careers?
Ido: That's a perfect analogy for thinking about the future. For our early-career listeners — what would you recommend they do to navigate their careers?
Shachar: A few thoughts. First: find people who are strong, who care about you, and who you can learn from — and stay close to them. Always be learning. Later in your career you can optimize for the mission or the industry — medical tech, fintech, whatever. But early on, the most important thing is who you work with. Find a great team that challenges you and helps you grow.
Second: the world is always changing, and it always has. The work of a data engineer today is very different from 20 years ago. What doesn't change is that your job is to create value — to make data useful for the organization. I used to do that by controlling block sizes, defining partitions, and rebuilding indexes. Today it's modeling data and building pipelines and transformations. In the future it'll be something else again. You're just entering at one point in time — don't think it's all ending.
Third: there are incredible resources now — LinkedIn, YouTube, Substack, communities, meetups — but the signal-to-noise ratio is bad. A lot of people's experience or knowledge isn't the best for you to learn from. Be very selective about who you learn from. Don't follow everything blindly just because it's out there.
And the last thing: understanding the business has never been more important than it is today. Never. In the past you could get away with being a DBA who just rebuilt indexes and migrated data files between partitions and storage devices. Those jobs are disappearing, and the business and technology keep getting closer together. AI accelerates that — because as you move from how to what, if you don't understand the business, you don't understand the outcomes you're trying to drive or why you're doing any of it. The people who learn the business and know how to create value with data will have unlimited opportunity. The ones who don't, unfortunately, won't stay relevant for long.
Closing takeaways
Ido: A few things I'm leaving with:
- The reasons data teams fail haven't changed — they're about people, process, and culture, not technology. Tools come and go; the missing ingredients stay the same.
- The shift to "store now, model later" broke a contract we're now rebuilding under names like data contracts and governance. It didn't make the data engineer's job easier — it made it more chaotic.
- AI enhances data work rather than replacing it. It moves us from how to what — but enablement isn't expertise, and managing data is still a profession.
- Whatever changes, the mission doesn't: understand the business and create value with data. That's never mattered more than it does now.
Shachar: Thank you so much for having me. It's a pleasure.
If you enjoyed this conversation, follow the Data Splash podcast. We have a lot more coming. Until next time — keep your data flowing.
For the full episode:
Youtube: https://youtu.be/0JD4Uj34su8?si=CJp797T5gXzCESBc
Spotify: https://open.spotify.com/episode/4HkSTmA7tO1QkY9iKRUdpM?si=99BDajsgQEC0Ql95aUm4Hw
To follow the Data Splash:
Youtube: https://www.youtube.com/playlist?list=PLJboMtNQp-MtIK1v7fn6eA35Ylh2T85IN
Spotify: https://open.spotify.com/show/4MD8LSNbpzW46qeq3qptpr
