Home / Learn

Where can you run existing PySpark for less than Databricks?

If most of your jobs are small PySpark and Spark SQL workloads, you have three kinds of option. Stay on Databricks and move jobs to cheaper compute, move the same code to a serverless Spark service such as EMR Serverless, Dataproc Serverless or AWS Glue, or run it on a Spark Connect engine such as Databasin Sail that keeps the PySpark API but scales workers to zero. Rewriting small jobs for DuckDB or Polars saves the most, but it is a rewrite.

Databasin · November 10, 2026

Selected for running existing PySpark code with little or no change, ordered by how much of your current setup each keeps.

Quick picks

OptionBest forCode changes
Databricks jobs and serverless computeKeeping Databricks and paying less per jobNone; job configuration only
Amazon EMR ServerlessTeams on AWS with batch Spark jobsPackaging and job submission
Google Dataproc ServerlessTeams on Google Cloud with batch Spark jobsPackaging and job submission
AWS GlueETL jobs that fit Glue's managed Spark runtimeGlue job setup; optional Glue APIs
Spark on KubernetesPlatform teams that already run KubernetesImages, scheduling and operations
Databasin SailPySpark DataFrames and Spark SQL on an Azure lakehouseConnection string; check compatibility
DuckDB or PolarsJobs that fit on one machineA rewrite off the Spark API

Seven places to run the same jobs

Databricks jobs and serverless compute

Best for

Teams that want to stay on Databricks and cut the bill.

Setup

Move scheduled work off all-purpose clusters onto jobs or serverless compute and right-size it.

Scheduled jobs running on interactive all-purpose clusters are a common source of a climbing bill. Jobs compute is billed at a lower rate, and serverless options start and stop with the work. Reviewing cluster sizes, auto-termination and idle notebooks often recovers a meaningful share before anything moves.

Strengths

  • No code or platform migration.
  • Unity Catalog and governance stay as they are.
  • Changes can be tested one job at a time.

Limits

  • Still priced in DBUs plus cloud compute.
  • Savings depend on how jobs are configured today.
  • Small jobs still pay for cluster start-up.

Amazon EMR Serverless

Best for

AWS teams running batch PySpark without managing clusters.

Setup

Create an application, package dependencies and submit jobs through the API or an orchestrator.

EMR Serverless runs open-source Spark and bills for the resources a job uses while it runs. Existing PySpark code usually runs as it is; the work is in packaging, data access and replacing anything that depended on Databricks-specific features such as notebooks, dbutils or Delta features you used through Databricks.

Strengths

  • Open-source Spark, so PySpark code carries over.
  • No clusters to size or keep running.
  • Fits AWS data and identity setups.

Limits

  • Databricks utilities and notebooks need replacing.
  • Governance and catalogs are yours to set up.
  • AWS only.

Google Dataproc Serverless

Best for

Google Cloud teams running batch Spark.

Setup

Submit batches with dependencies; connect to BigQuery or Cloud Storage as needed.

Dataproc Serverless runs Spark batches and interactive sessions without cluster management, billed for the resources used. It is the closest equivalent to EMR Serverless on Google Cloud and carries the same migration work around Databricks-specific features.

Strengths

  • Open-source Spark runtime.
  • Close integration with BigQuery.
  • No cluster sizing.

Limits

  • Google Cloud only.
  • Notebooks, catalogs and access rules to rebuild.
  • Start-up time on each batch.

AWS Glue

Best for

ETL jobs that move and reshape data on a schedule.

Setup

Define jobs, connections and the Glue Data Catalog; choose worker types.

Glue is a managed Spark runtime aimed at ETL, billed per worker time. Plain PySpark jobs run, and Glue adds its own DynamicFrame APIs and crawlers. It suits pipelines more than interactive analysis or BI serving.

Strengths

  • Fully managed Spark for ETL.
  • Built-in catalog and crawlers.
  • Billed for job run time.

Limits

  • Less suited to interactive work.
  • Glue-specific APIs can create lock-in.
  • AWS only.

Spark on Kubernetes

Best for

Platform teams that already operate Kubernetes.

Setup

Build images, run the Spark operator or spark-submit, set node pools that scale down.

Apache Spark runs natively on Kubernetes. With node pools that scale down between runs, the compute cost can be low; the cost moves to the people who build images, tune resources and keep it running.

Strengths

  • Runs on any cloud or on premises.
  • Open-source end to end.
  • Fine control over resources.

Limits

  • You operate everything.
  • A system node usually stays on.
  • Governance and notebooks are separate projects.

Databasin Sail

Best for

PySpark and Spark SQL on a governed lakehouse in Azure, with BI and notebooks on the same tables.

Setup

Point Spark Connect clients at a Sail cluster; our team checks a job's functions against the compatibility list first.

Databasin Sail speaks the Spark Connect protocol, so PySpark 4.x DataFrames and spark.sql run without a rewrite. Queries execute on a Rust engine built on the open-source LakeSail project and Apache DataFusion rather than the JVM. Workers scale to zero when idle and start on the next request; cluster prices are listed in the Sail documentation and billed by the minute. Sail reads and writes Iceberg and Delta tables, and Tableau, Power BI and notebooks connect to the same cluster.

Strengths

  • Existing PySpark and Spark SQL carry over.
  • No JVM tuning; pick a size and run.
  • Idle workers cost nothing.

Limits

  • No Scala or Java, RDDs or pyspark.ml.
  • A small set of Spark functions differ.
  • Python UDFs on single-user clusters only; runs on Azure.

DuckDB or Polars

Best for

Jobs whose data fits on one machine.

Setup

Rewrite the transformation and schedule it in a container or function.

Many daily ETL jobs process gigabytes, not terabytes. DuckDB and Polars handle that on a single machine and cost very little to run. The catch is a rewrite off the Spark API and a single-machine ceiling for jobs that later grow.

Strengths

  • Lowest running cost for small data.
  • Simple to run anywhere.
  • Fast on a single machine.

Limits

  • Code must be rewritten.
  • Single-machine memory and compute.
  • No shared catalog or access rules by default.

How these options were selected

The guide assumes the code already exists and works. It weighs how much of that code survives the move, what the bill depends on afterwards, and who has to run the result. Raw speed was not a criterion: for small jobs, start-up time and idle cost usually matter more than throughput.

Code that carries over unchanged

35%

Cost for small and idle workloads

30%

Operations the team must take on

20%

Governance and BI on the same data

15%

Databricks · Compute · Accessed 2026

AWS · Amazon EMR Serverless · Accessed 2026

Google Cloud · Dataproc Serverless · Accessed 2026

AWS · AWS Glue · Accessed 2026

Apache Spark · Running Spark on Kubernetes · Accessed 2026

LakeSail · Sail · Accessed 2026

Databasin · Spark compatibility and clusters and sizes · 2026

What happens next

Bring a job your team runs every day. We can check its functions against Sail's compatibility list and run it with you on your own data.

Book a demo

What happens next

  1. Book a demo

    Tell us which jobs drive the bill and where they run today.

  2. Check one job

    We review its functions against the compatibility list and run it on Sail.

  3. Plan the move

    Decide which jobs move, which stay and how the tables are shared.

FAQ

Will my PySpark code run unchanged?

Plain DataFrame and Spark SQL code usually does on any open-source Spark runtime and on Spark Connect engines. Code that uses Databricks utilities, notebook magic, RDDs, Scala or Spark ML needs changes or a different target. Check each job's imports before choosing.

Is a rewrite to DuckDB worth it?

For a job that processes a few gigabytes on a schedule, often yes. For jobs that share tables with BI and other engines, or that may grow, keeping the Spark API avoids a second migration later.

Do I lose Unity Catalog if I move?

Its access rules and lineage stay in Databricks. Sail can read Unity Catalog Delta tables in place (in preview), and other options need their own catalog. Decide where governance lives before moving jobs.

Can I move only some jobs?

Yes. Open table formats let several engines read the same tables. Many teams keep heavy or ML workloads on Databricks and move small scheduled jobs elsewhere.

Explore more

Databasin Sail →

Shared compute for reports and notebooks on the same lakehouse tables.

Data platform costs →

Find the work behind the bill before replacing tools.

Databricks or Snowflake →

Choose a platform by the work your team does.

Bring a slow or costly job

We can look at the jobs behind your Databricks bill and test one on Sail with your team.

Book a demo