This conversation has been edited for length and clarity.
The 30-Second Splash
Ido: Snowflake or Databricks?
Prakash: Databricks.
Ido: Most underrated skill in data?
Prakash: Data modeling.
Ido: Dream vacation destination?
Prakash: Any place that has a cabin, a lake, and a mountain.
Ido: Is AI going to replace data engineering?
Prakash: Probably no.
How is data engineering structured at Atlassian?
Ido: Tell us a bit about Atlassian, how the data organization is structured, and the value you're getting from your data today.
Prakash: Most of your audience probably already knows Atlassian — we're the company behind Jira and Confluence, among other flagship products. Data engineering at Atlassian is huge. We're a sizable organization, and we support the company through three main pillars of data teams.
The first pillar is teams aligned to business functions — sales, marketing, customer support, finance, workplace, and so on. They focus on making business teams successful by providing analytics capabilities and helping teams make decisions at warp speed.
The second pillar is data engineering teams that work directly on our products — Jira, Confluence, and the rest. They build pipelines to understand how customers use our products. We run experimentation pipelines to support A/B tests, do feature usage analysis, sessionization, and study how customers navigate between products. We use that behavioral understanding to ship better products and improve the ones we have.
The third pillar cuts horizontally across business lines and products. They build our top-level metrics — including the ones we report to Wall Street.
We also have a data science organization that works alongside data engineering to build insights and analytics, and research teams that focus on customer and product research. Together, we're an ecosystem that supports the company's engineering and business teams to make sure Atlassian is set up for success.
Ido: So you have everything — AI, BI, external, internal, high-level dashboards, day-to-day operations.
Prakash: It's a pretty big gig. We call ourselves data-driven. Every decision we make about our product and our business uses data. So data engineers' jobs are very safe.
What was the original vision for using AI in data engineering?
Ido: A few months ago, you shared with me the idea of using AI to fundamentally change how data engineering works at Atlassian. How did you arrive at that idea?
Prakash: About six months ago, we had this vision: a lot of what data engineers do day-to-day is repeatable and automatable. We had already automated parts of our data engineering workflows through normal scripting and programming. But the real potential of AI became realizable for us only after we migrated from our old data lake to a new one.
That migration took multiple years. In the old world, we didn't have environment isolation — production, development, and staging all lived in the same place. We didn't have repeatable patterns for how data engineers built ETL pipelines. Everyone had their own way of doing it. It was a known problem for years.
The migration changed that. We standardized how data products get built. We organized them into bronze, silver, and gold layers. We introduced real environment isolation. And we made a big shift from "do it your way" ETL pipelines to declarative-style ETL pipelines. Now there's a uniform way of building data products across all our data engineering teams.
Once we had that uniformity, it created a unique opportunity. There were now very few patterns for building data products — everything was codified and YAMLified. That's when we could realistically bring AI automation on top.
About four or five months ago, I wrote a one-pager I called the AI4DE Vision Document. I did some industry research on where things were heading and how good LLMs were at understanding data workloads. I broke down the SDLC of data engineering into sub-capabilities and asked: where can we focus to get the biggest ROI from using LLMs to automate parts of the workflow?
The biggest motivation, honestly, is that data engineers are always underfunded. We're always behind. There's so much work to help the company build assets and drive analytics, and we're always short of people. The only way to scale and keep up with company speed was to automate using AI as a force multiplier.
Where did you see the biggest ROI?
Ido: When you looked at the different stages of the workflow, what stages did you identify, and where did you see the biggest ROI?
Prakash: We mapped the SDLC of data engineering and started with the first stage: requirements gathering — understanding what the business wants, the current data model, and what changes are needed. We considered investing here, but decided it wasn't where the biggest ROI was.
Where there is an existing data model — and we use dbt, Airflow, Databricks, that whole stack — we get inundated with incremental requests. Add a column, modify business logic, handle a new edge case. These small tickets were death by a thousand cuts across our DE teams.
So our first focus was: can we take those small tickets, deploy an AI agent with the right skills to understand the business ask from a JIRA ticket, make the code changes, and open a pull request? Data engineers can review, give it more information, and iterate. That was the first problem we tackled.
The second area was data ops — managing on-call, data failures, production support. We saw real ROI in building an on-call agent that could look at a job failure or a stakeholder question about data quality, investigate what's wrong, and proactively recommend fixes to the on-call engineer: "Here's what I found. Here's what I'd recommend. Want me to do it, or do you want to handle it yourself?" That was our second-biggest ROI bet.
The third area is making data AI-ready for agentic analytics. There's a lot of demand for self-serve. We can't build dashboards fast enough for the business to get insights. We wanted to invest in making the data AI-ready so people could ask questions themselves and self-serve in a trusted, reliable way.
Those three pockets — incremental development, on-call, and AI-ready data — were where we decided to go deep.
What does the state of automation look like today?
Ido: Where are you today? How much have you succeeded in automating, and what return are you seeing?
Prakash: It's a journey. We're about six months in. Initially, we didn't block dedicated time for AI-driven development — this was a coalition of the willing. We wanted to make sure business goals weren't being put on hold. Whatever people were already working on had to keep happening. Then engineers with extra capacity came together as a small central working group to build skills, test them, and evaluate what worked.
Now we're running workshops and training modules across the entire data engineering org to bring everyone in and show them how to use AI. Early on, everyone was experimenting in their own way — hundreds of different patterns. At some point it became hard to consolidate the efforts. With a small group now driving consistency, people are working with each other to improve the skills and apply them in day-to-day work across all three categories.
For category one — developing things faster — we're seeing it work reliably. Without being too aggressive, work that used to take five days to develop and test now often takes less than three. There's been a clear productivity improvement.
I'm careful about claiming "30 to 40 percent improvement." I look at it through total cost of ownership. When we use AI to write code, are we writing scalable code? Does AI-generated code break in production? Are we accelerating development but raising operational cost? Through that TCO lens, I still see a meaningful improvement, especially because we're investing in both pillars in parallel — using AI to intercept problems faster on the operational side, and using AI to develop ETL code faster on the build side. Compounding gains come from running both at once.
I can clearly say we're seeing 10%+ productivity gains, even at this early stage. It's only going to get better as our maturity improves.
Did this match the journey you described — start small, hit fragmentation, then consolidate?
Ido: Let me check I understood the journey: you picked three areas to focus on, let teams experiment freely, ended up with lots of different solutions and skills, then ran workshops to unify the patterns and solve the shared challenges. Today you're seeing real productivity gains, even if they're hard to attribute to a single task. Is that right?
Prakash: Yes, we definitely feel we're heading in the right direction. There were a lot of details in between — small things like, can we use AI to improve data quality on our tables by generating boilerplate data quality checks from existing metadata? We've taken many small steps across the SDLC and found a lot of areas where we've sped up the development process.
For self-serve, the biggest challenge right now is that data is extremely hard to query through LLMs. People ask questions in many different ways. Whether you use Genie, Cortex, Claude Code, or any other tool pointed at your data, the translation from natural language to SQL — including the semantic layer — isn't 100% accurate. BIRD-Bench shows top models around 84%. Humans are at 91%. Most other models — Databricks, Gemini, and so on — sit around 81-82%. There's still a real gap in how reliably you can self-serve.
But Atlassian is investing significantly in improving our context layer — better metadata, better lineage, better catalog maturity. We're heading toward a rich context layer that powers not only self-serve insights but also vertical and horizontal AI agents that can run workflows.
Will AI replace data engineers?
Ido: It always comes back to the foundations. Without good metadata, lineage, and curated datasets, you can't trust the output. So this work — making sure those things are in place — will always be data engineering's job. Coming back to your earlier answer: AI won't replace data engineers, but how will the role change?
Prakash: AI will definitely improve the value proposition of data engineers. Data engineers will probably stop writing full ETL pipelines — maybe they'll write small parts of them. They'll become more like architects, pushing the value proposition of the role toward enabling business value rather than being deep in the code.
That's how the role evolves. But data engineers remain crucial because someone needs to build the core foundations and fundamentals that unlock the power of AI in the first place.
What were the hardest technical and human challenges?
Ido: Can you share one or two technical challenges you solved during this journey?
Prakash: Before I get to the technical side, there's a human challenge worth naming. The pace of AI development is so fast that no one can keep up. So when you're driving a transformation like this, you have to ask: how do you deliver knowledge to people effectively? How do you upskill the workforce? You can't just tell them, "Go figure it out." You have to provide guided learning pathways and step-by-step examples of how to use AI tools in real day-to-day work. That step-by-step enablement is critical.
Atlassian has its own AI agent called Rovo. It's also a product we offer to customers on top of the Atlassian suite. We use Rovo daily to ask questions, draft pages, draw diagrams, and write code. It's a powerful tool that can connect to any LLM under the hood. But if you don't show people how to use it for a specific task, it becomes a dumpster fire of inconsistent usage.
On the technical side: hallucination is real. It's extremely difficult to get deterministic answers, no matter how many guardrails you apply, how good your instructions are, or how atomic you make your skills. When you orchestrate skills into an agent and ask it to do something, it will hallucinate three to four times out of ten. The job of any engineer working with AI today is to know when it's right and when it's hallucinating. You can't take its output at face value — you have to add validations on top.
The other thing I'd flag is organizational. Companies that don't encourage people to try new things will fall behind on AI adoption. Early on, we deliberately let people try AI freely, find new patterns, and explore different approaches, because we wanted to spark creativity. When you have many different pathways, you can see what works, what doesn't, and then consolidate.
Organizations that don't allow that experimentation will see a lagging effect on AI.
Where do you see Atlassian — and the industry — in 18 months?
Ido: Where do you see Atlassian a year, year and a half from now?
Prakash: I wish I had a crystal ball. From an industry standpoint, I think 2026 and 2027 are when we'll start to see real improvement in how LLMs work with data. Companies are pioneering this — building products to make data engineering and data analysis autonomous. That's exciting.
The bigger industry shift is that the data roles we've separated for years — data engineer, data analyst, analytics engineer, business analyst, data scientist — will start converging. They'll unify under one or two titles. Maybe "data practitioner" or "data expert," whatever the industry settles on.
The bar to learn what other roles do is much lower now. You can ask Databricks Genie or Cortex to write a linear regression on your data and get the full statistical analysis. The bar is shortening, and the roles will eventually converge. The momentum is clearly building in that direction. Every company I see right now is building tools that converge skill sets. Reliability is the open question, but eventually we'll get to a point where these tools are reliable and deterministic.
Ido: I see this convergence everywhere in the industry. My personal bet is that the new role will be something like "knowledge engineer" — because what you're really crafting is the knowledge of the organization. Data is a technical means; knowledge is the goal. Airbnb at the last Open Data Forum summit called it "full-stack data engineering." Different names, same direction.
Prakash: I've heard a few other titles at Gartner Summit and elsewhere — Context Engineer, Agent Evaluator, Agent Orchestrator. These roles emphasize being good at building custom agents to perform tasks using the skills of the underlying roles. The value proposition gets pushed upstream toward value creation rather than just doing the work.
The roles themselves won't disappear. But this shift will push everyone in data toward higher-value work — partnering with business teams to understand what they actually need to solve, then using the tools available to be 10x more productive.
For Atlassian specifically: I think we're heading in the same direction. First, we want to make our existing data engineers extremely productive — give them every tool they need to meet the company's needs today and tomorrow. Then we want to upskill our people so they understand business concepts deeply and can solve business challenges effectively, without having to worry about the data stack itself. We want to coach our team to be value creators, not just data builders.
Closing takeaways
Ido: Thank you so much for joining me, Prakash. A few takeaways I'm leaving with:
- Foundations come first. Without good architecture and metadata, you can't scale to AI.
- You have to experiment. Organizations that don't put AI in their teams' hands today will find themselves irrelevant in two years.
- AI isn't magic — and it's not plug-and-play. Especially in data, there's a lot of know-how required. You need to understand what works in your specific environment, and that takes time. So start early.
It's been amazing to see how early Atlassian took the leap and how far you've already come.
Prakash: Thank you for having me. It was really fun to nerd out on these concepts. I'm excited to see where this all goes — for Atlassian, for Upriver, and for the industry.
If you enjoyed this conversation, follow the Data Splash podcast. We have a lot more coming. Until next time — keep your data flowing.
For the full episode:
Youtube: https://www.youtube.com/watch?v=WZx5MU3rU98
Spotify: https://open.spotify.com/episode/79ofCOZfpy02rPCYHRJ0Pw
To follow the Data Splash:
Youtube: https://www.youtube.com/playlist?list=PLJboMtNQp-MtIK1v7fn6eA35Ylh2T85IN
Spotify: https://open.spotify.com/show/4MD8LSNbpzW46qeq3qptpr
