NVIDIA describes agents rewriting Spark UDFs for GPU execution
NVIDIA reported that a Spark UDF rewrite cut operator time from 274.55 hours to 12.78 hours on one 600-billion-row workload at an unnamed consumer technology company.
Event / data period: 2026-10-13
Published: · Updated:
NVIDIA described how user-defined functions (UDFs) in Apache Spark can force a GPU pipeline to fall back to CPU execution, adding data-transfer and runtime overhead. Its cuDF skills guide coding agents to test a UDF's behavior, rewrite it with Spark SQL, cuDF or custom CUDA, then measure equivalence and performance. The post cites an unnamed consumer technology company that converted more than 30 production UDFs using the workflow.
For one 600-billion-row workload, reported operator time was 274.55 hours with a CPU-based UDF and 12.78 hours with a cuDF GPU implementation; NVIDIA called this a 22-fold speedup. The post specifies 16 TB of input and 17.4 TB of output, but does not name the customer or provide an independent measurement. This result belongs to one workload and is not a performance guarantee for all Spark jobs.
Data
Operator time for a rewritten Spark UDF
Reported CPU and GPU implementation times for one 600-billion-row workload at an unnamed consumer technology company.
- CPU-based UDF operator time
- 274.55 hours
- 2026-10-13 · Observed
- cuDF GPU-accelerated UDF operator time
- 12.78 hours
- 2026-10-13 · Observed
Methodology
CPU and cuDF/RapidsUDF operator times for one production UDF, as reported in NVIDIA's 13 October 2026 post. The source describes a 600-billion-row workload with 16 TB of input; this is one example and should not be generalized to all Spark jobs.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Subject suggested from the title or summary: technologydeveloper.nvidia.com ↗