James Ding
Apr 09, 2026 16:48
Notion migrated from Spark on EMR to Ray, slicing embedding prices 80% and bettering question latency 10x. Uber and Salesforce shared comparable AI infrastructure wins.
Notion has slashed its AI embedding pipeline prices by greater than 80% after migrating from Apache Spark to Ray, the distributed computing framework backed by Anyscale. The productiveness software program firm additionally achieved 10x enhancements in question latency whereas consolidating three separate jobs per area into one.
The migration particulars emerged at Ray Day Seattle on April 9, 2026, the place ML engineers from Notion, Uber, Salesforce, and Apple shared hard-won classes about scaling AI infrastructure.
What Notion Truly Modified
Mickey Liu, a software program engineer on Notion’s search platform crew, walked by the overhaul. Their unique setup used a three-step Spark pipeline working on Amazon EMR: information chunking, third-party API requires embedding era, and writes to a vector retailer.
The ache factors had been predictable however extreme. Double compute prices. Third-party API fee limits throttling throughput. Debugging nightmares when failures occurred throughout instruments—driver and executor logs weren’t even continued in YARN.
The brand new structure streams Kafka information instantly right into a Ray cluster dealing with CPU chunking, GPU embedding era, and vector retailer writes in a single pipeline. No intermediate S3 handoffs. What began because the backend for a Q&A characteristic in 2023 now powers all of Notion AI and customized brokers.
Uber and Salesforce Report Related Features
Uber’s Peng Zhang detailed how their Michelangelo ML platform advanced from TensorFlow/Horovod to Ray with PyTorch. The standout transfer: separating CPU data-loading nodes from GPU coaching nodes in a heterogeneous cluster design. End result? GPU utilization jumped 20%, and coaching time dropped roughly 50% in choose pipelines.
Salesforce tackled a distinct beast—summarizing paperwork as much as 200,000 tokens lengthy (roughly a brief novel) with P95 latency beneath 15 seconds. Their crew used Ray to chunk paperwork and run parallel inference throughout a distributed actor pool with vLLM, then merge outcomes. They landed on 1-2 GPU information parallelism because the candy spot after working scaling experiments instantly on Ray.
Why This Issues Past These Firms
Robert Nishihara, Ray’s co-creator and Anyscale co-founder, opened the occasion by framing the core drawback: AI infrastructure retains getting more durable. Multimodal information processing, reinforcement studying workloads, and multi-node LLM inference are pushing current instruments previous their limits.
Each speaker landed on the identical conclusion from totally different angles—their earlier tooling ran out of highway.
Apple engineers Charlie Chen and Haocheng Bian highlighted basis mannequin coaching challenges: large unstructured information, billion-plus parameters, and sparse architectures like Combination of Specialists. Conventional engines fail as a result of information pipelines and coaching frameworks run in separate environments with no shared context.
What’s Subsequent
Ray Day Seattle kicked off Anyscale’s 2026 “Ray on the Highway” tour—eight cities throughout three international locations. The corporate can also be working invite-only buyer roundtables at every cease to preview their product roadmap.
For groups hitting comparable partitions with Spark or different distributed frameworks, Notion’s full technical writeup is obtainable on their engineering weblog beneath “Two Years of Vector Search at Notion.” The 80% price discount and 10x latency enchancment supply a concrete benchmark for anybody evaluating comparable migrations.
Picture supply: Shutterstock
