AI & the role of Data Engineer

The work before AI

Before AI entered the daily workflow, most of a data engineer's time went into work that produced no insight on its own. It cleared the way for someone else to produce one.

The recurring items looked like this:

• Writing ingestion code for another source system with a slightly different API and a slightly different idea of what a timestamp is
• Hand-mapping source fields to warehouse columns, then maintaining that mapping every time the source changed without telling anyone
• Rebuilding a derived table because the definition of a metric shifted upstream
• Reading logs at 7am to work out which scheduled job failed, and whether the failure mattered before the morning reports went out
• Writing the tests and documentation everyone agreed were important, and which were the first thing cut when a deadline moved

The friction was rarely the data modelling itself. It was the volume of small, exacting work sitting between a source system and a table anyone could trust.

Because that work was slow, a good deal of it never happened. The freshness check nobody wrote. The column nobody documented. The business rule that lived in one person's head and in a Slack thread from two years ago.

Where AI already changes the day to day

Several tasks in the data engineer role have moved for real, in the everyday routine rather than in theory.

1. Pipeline and transformation drafts. Given a source schema and a description of the target table, a model produces a first version of the ingestion job or the transformation. The draft is a starting point to be corrected, not a finished artefact, which is the point: the task changes from writing to reviewing, and reviewing a hundred lines is faster than typing them.

2. Failure triage. When a scheduled job fails, an assistant with access to the logs, the recent commits and the lineage graph can propose the likely cause instead of leaving someone to read stack traces in order. The engineer still decides whether to rerun, backfill, roll back or leave it alone until the source system is fixed.

3. Data quality tests. Generating a first set of not-null, uniqueness, referential and freshness checks from a schema takes minutes. This matters more than it sounds, because these tests were historically the work most likely to be skipped. DORA's guidance on data health makes the parallel explicit: in the same way code has automated tests, data needs automated validation for accuracy, completeness and timeliness.

4. Documentation, metadata and lineage. Producing a plain-language description of what a table contains, what a column really measures and which downstream models depend on it has gone from a task nobody had time for to a task that can run continuously. DORA measures this capability partly by asking teams how likely they are to get a definitive answer about a data point within an hour of searching, which is a fair description of what undocumented warehouses cost.

5. Query and cost work. Spotting the full table scan, the missing partition filter, the model rebuilding everything nightly when an incremental would do. Warehouse spend is a number a data engineer is accountable for, and finding the worst offenders is now a question rather than an investigation.

None of this is niche. In DORA's 2025 State of AI-assisted Software Development report, based on survey responses from nearly 5,000 technology professionals gathered in June and July 2025, 90% reported using AI at work, up 14.1% on the previous year.

What continues to be deeply human

None of the above touches the parts of the data engineer role that carry the consequences.

Deciding what the data means. Defining "active customer" or "completed order" is a negotiation between finance, product and operations before it is a modelling problem. A model can implement whichever definition it is handed. It cannot decide which one the company will be held to, or notice that two departments have been using the same word for different things since 2019. The consequences of that decision land downstream on whoever reports the number, which is a boundary examined in AI & the role of Data Analyst.

Choosing what to guarantee. Freshness, completeness and cost trade against each other. Committing that a table will be correct by 7am, and accepting what that commitment costs in compute and on-call load, is a promise made to other people. Promises are not a technical output.

Deciding what not to build. The cheapest pipeline is the one that never gets written. AI has lowered the cost of building things, which makes the discipline of declining more valuable rather than less. Every table that exists has to be maintained, explained and eventually deprecated by someone.

Negotiating upstream. Most data quality problems begin in a system owned by another team. Getting that team to stop changing the meaning of a field without notice is a matter of relationships and organisational standing, and it is often the work with the widest effect a data engineer does all quarter.

Owning the blast radius. A wrong join or filter rarely fails loudly. It produces a number that looks plausible and travels into dashboards, forecasts, board decks and, increasingly, into AI features that present it to customers. Someone has to be accountable for that number. Accountability does not transfer to a tool the way output does.

How seniority changes with AI

The clearest pattern is that AI compresses the time to a working pipeline without compressing the time to knowing which pipeline should exist.

That helps a junior engineer considerably. It shifts the constraint for everyone else. If a team can generate five times as many models, the limit stops being how fast anyone can write them and starts being how many someone can hold in their head at once and still understand how they fit together.

DORA's 2025 findings sit exactly on that seam. Comparing professionals with higher and lower AI adoption, the report associates AI adoption with higher software delivery throughput, a reversal of its 2024 finding, while still associating it with higher software delivery instability. More work moves through the system, and more of it needs immediate intervention afterwards. In data work, that intervention usually looks like a number that changed without anyone deciding it should.

Validation, in other words, has become the scarce skill rather than production. The same report found that 30% of respondents have little or no trust in the quality of AI-generated code, which DORA reads as a healthy sign of mature adoption rather than a problem to be trained away.

A senior data engineer is therefore doing less typing and more of something harder to measure: reviewing work they did not write, holding the model of how the warehouse fits together, and catching the thing a less experienced colleague did not know to check. Two of the seven capabilities in DORA's AI Capabilities Model, healthy data ecosystems and AI-accessible internal data, describe conditions that data engineers are largely the ones who create.

Risks of using AI without context

Bad data, faster. DORA's position on this is direct: where a data environment is fragmented or of low quality, AI accelerates the production of incorrect or irrelevant output. The failure mode is not that the model refuses to work. It is that it works confidently on top of a definition that was already wrong, and does so at volume.

Transformations that look right and are wrong. Generated SQL compiles, runs and returns rows. A join that silently duplicates rows on a one-to-many relationship, or a filter that drops the rows where a nullable field is null, produces a number rather than an error. Review has to be adversarial in a way that reviewing application code often does not.

Rationale that never gets recorded. When a rule can be generated in seconds, the reason behind it is easy to leave undocumented. KWAN's whitepaper on scaling calls the accumulated version of this problem Context Debt: the "why" behind past decisions gets lost, and teams end up reworking things nobody can explain. AI-generated pipelines are an efficient way to accumulate it, because the code exists but the argument for it never did.

Not knowing whether it worked. Measuring whether AI actually made a data team faster is harder than it looks. METR built a randomised controlled trial to measure exactly this for developers and has since redesigned the study, reporting that its follow-up data gives an unreliable signal and that self-reported speed-ups can be quite inaccurate. If a research non-profit with a controlled design finds this difficult, a quarterly dashboard of merged pull requests is not going to settle it either.

A thinner bench later. If the work junior engineers once used to build intuition about a warehouse is absorbed by AI, and nothing deliberate replaces it, the cost appears years later as fewer people able to tell a real anomaly from a broken pipeline. It is a slower version of the delivery risk covered in how to scale tech teams without increasing churn or delivery risk.

How KWAN thinks about this

KWAN builds AI-augmented data and engineering capacity, whether that is individual specialists joining an existing team or a dedicated team assembled around a product. The model stays people-first: every consultant works with a dedicated People Experience Partner, and seniority is treated as something built on purpose rather than something that appears once AI has absorbed the entry-level work.

These are the same questions that come up when KWAN builds a data team around a client's actual context rather than a generic role description: where AI takes the load off, and where a person still has to hold the meaning. That conversation is always open. Start it here.

Frequently asked questions

1. Will AI replace data engineers?

No. AI drafts pipeline code, tests and documentation, which removes much of the typing. Deciding what the data means, what a pipeline guarantees to other teams and who is accountable for a wrong number are not tasks that transfer to a tool.

2. Which data engineering tasks does AI handle best today?

First drafts of ingestion jobs and transformations, triaging failed scheduled jobs, generating data quality checks, producing documentation and lineage descriptions, and finding query patterns that waste warehouse compute.

3. Why does data quality matter more now than it did before AI?

Because AI systems consume the output of data pipelines as well as helping build them. DORA's research found that the positive effect of AI adoption on organisational performance depends on internal data being high quality, accessible and unified.

4. Does AI reduce what senior data engineers are responsible for?

No, it moves it. Producing transformations gets cheaper, so the constraint becomes review capacity and understanding how a growing set of pipelines fits together, which is senior work by definition.

5. What is the biggest risk of using AI without context in data engineering?

Generated transformations that run without error and are still wrong. They return a plausible number rather than a failure, and that number travels into dashboards and decisions before anyone questions it.

6. Is AI adoption in data engineering mostly about replacing tasks or changing them?

Changing them. The tasks that consumed the most hours, boilerplate, mapping, first-draft tests, shift towards AI. The tasks that carry the consequences, definitions, guarantees, accountability, stay where they were.