SparklyR on Databricks Serverless: A Working Setup and Real Measurements

I set out to write a conventional sparklyr optimization guide — cluster sizing, caching strategies, the usual. Then I tried to actually run the examples against a current Databricks workspace, and the ground had shifted under the topic: there were no clusters to size, half the caching APIs threw exceptions, and getting a working connection took a chain of undocumented version pins. This post is the field report. Every claim in it passed or failed in a live session on 2026-07-10.

The post is organized in five parts:

1. The landscape — why serverless-from-RStudio is now the sparklyr path
2. Setup — six steps from fresh install to working connection, with the exact error each step prevents
3. Limitations — what doesn't work on serverless, verified
4. Working patterns — the replacements that do work
5. Measurements — real numbers, and when they'd flip

Part 1: The Landscape

Three facts define the situation for R users on Databricks today:

1. New workspaces are serverless-only. Signing up through the AWS Marketplace provisions a serverless workspace: no classic compute plane, no cluster creation UI, no instance types. Getting a classic cluster would mean provisioning a traditional workspace against your own AWS account (cross-account IAM role, VPC, subnets).

2. Serverless has no R runtime. Serverless notebooks support Python and SQL only. Even on classic compute, R notebooks require a cluster in Dedicated access mode — on the Standard mode most policy-managed workspaces default to, any R cell fails with Your administrator has only allowed sql and python commands on this cluster.

3. Local RStudio + Databricks Connect is therefore the only sparklyr path on a default setup. The stack: your R code → sparklyr (R interface) → pysparklyr (bridge package managing a Python environment) → Databricks Connect (client protocol) → serverless compute.

Part 2: Setup

Everything below was verified on this configuration:

Each step below follows the same pattern: what to run, and the exact error you get if you skip or vary it.

Step 1: R packages

install.packages(c("sparklyr", "pysparklyr", "dplyr", "dbplyr",
                   "purrr", "lubridate", "ggplot2", "lobstr", "DBI"))

If you follow older guides instead: they say install.packages("pryr") for memory profiling, which now fails with "package 'pryr' is not available for this version of R" — pryr was archived from CRAN in January 2026. lobstr::mem_used() is the drop-in replacement.

Step 2: The Python environment

pysparklyr::install_databricks(version = "17.3")

This builds ~/.virtualenvs/r-sparklyr-databricks-17.3 (needs Python 3.12 on your PATH; takes a few minutes).

If you skip the version: serverless connections fail with A cluster 'version' is required, please provide one — with a classic cluster pysparklyr reads the runtime version from the cluster, but serverless gives it nothing to inspect.

If you pick 16.4 (the LTS a reasonable person would choose): NotImplementedError: Serverless mode is not yet supported in this version of Databricks Connect — despite documentation saying serverless works from 15.1. Use 17.3, the version Databricks' own serverless tutorials use.

Step 3: The pandas pin

reticulate::virtualenv_install("r-sparklyr-databricks-17.3", packages = "pandas<3")

Restart R afterwards.

If you skip it: the fresh environment gets pandas 3.x, whose Arrow-backed dtypes break pysparklyr's conversion of results into R. The symptom is baffling — the connection works, queries run, but every fetch dies with "Failed to fetch data: All columns in a tibble must be vectors." (A point-in-time bug as of mid-2026; presumably fixed upstream eventually.)

Step 4: Credentials

Create a Personal Access Token: avatar → Settings → Developer → Access tokens → Generate new token. If asked for scopes, pick all-apis with a short expiration. Then store both values in ~/.Renviron — never in a script:

usethis::edit_r_environ()   # creates ~/.Renviron and opens it

DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
DATABRICKS_TOKEN=<personal-access-token>

No quotes, no spaces around =, file must end with a newline (a missing final newline silently drops the last variable). Restart R — .Renviron is only read at startup — and verify with Sys.getenv("DATABRICKS_HOST").

Why all-apis and not a narrower scope: Databricks Connect drives a Spark Connect gRPC session, which none of the documented fine-grained scopes (sql, command-execution, unity-catalog, …) is stated to cover, so narrow tokens can fail at connect time with a 401/403. A PAT is always capped by your own workspace and Unity Catalog permissions — scopes limit API surface, not data access — so short expiry is the control that matters.

Step 5: Connect and smoke-test

library(sparklyr)
library(dplyr)
library(dbplyr)

sc <- spark_connect(
  method = "databricks_connect",
  serverless = TRUE,
  version = "17.3"     # must match Step 2
)

DBI::dbGetQuery(sc, "SELECT * FROM range(5)")   # expect 5 rows, id 0..4

If you smoke-test with sdf_len() instead: 'SparkSession' object has no attribute 'defaultMinPartitions' — it reaches into the legacy SparkContext API, which Spark Connect doesn't expose. dplyr verbs and SQL translate cleanly; older sdf_* helpers are hit-and-miss.

Step 6: Find your writable catalog

DBI::dbGetQuery(sc, "SELECT current_catalog(), current_schema()")
DBI::dbGetQuery(sc, "SHOW CATALOGS")

If you assume the catalog is main (as most guides do): SCHEMA_NOT_FOUND: The schema main.default cannot be found. In a fresh serverless workspace the writable default was workspace.default, alongside the read-only samples and system catalogs. You also need Unity Catalog privileges on it (USE CATALOG, USE SCHEMA, CREATE TABLE, MODIFY, SELECT) — automatic in a workspace you own.

Part 3: Limitations — What Doesn't Work on Serverless

Every row verified in a live session:

The common thread: serverless manages memory and partitioning itself and exposes no caching knobs, and Spark Connect exposes no SparkContext. Code that sticks to dplyr verbs and SQL — which is most well-written sparklyr code — runs unchanged.

Part 4: Working Patterns

Materialize to Delta (the compute() replacement)

Two helpers make it ergonomic:

catalog <- "workspace"   # from Step 6
schema  <- "default"
qname   <- function(name) paste(catalog, schema, name, sep = ".")

materialize <- function(x, name) {
  spark_write_table(x, qname(name), mode = "overwrite")
  tbl(sc, in_catalog(catalog, schema, name))
}
drop_table <- function(name) {
  DBI::dbExecute(sc, paste("DROP TABLE IF EXISTS", qname(name)))
}

Anywhere a classic-compute guide says compute("cached_result"), write materialize("cached_result"). It costs a storage write instead of a cache, but the result survives session restarts and other jobs can reuse it.

Iterative pipelines

The pattern carrying a real workload — iterative outlier trimming, with percentile_approx() standing in for R's untranslatable quantile():

raw_data <- spark_test_data %>% select(key_column = id, value)
data_cleaned <- raw_data

for (i in 1:5) {
  center <- data_cleaned %>%
    summarise(center = percentile_approx(value, 0.5)) %>%
    pull(center)

  cutoff <- data_cleaned %>%
    summarise(cutoff = percentile_approx(abs(value - center), 0.95)) %>%
    pull(cutoff)

  data_cleaned <- data_cleaned %>%
    filter(!is.na(key_column), abs(value - center) < cutoff) %>%
    materialize(paste0("clean_iteration_", i))

  # Drop the previous iteration — the Delta analog of tbl_uncache()
  if (i > 1) drop_table(paste0("clean_iteration_", i - 1))

  cat("iteration", i, "rows:", sdf_nrow(data_cleaned), "\n")
}

Measured on 1M rows, each pass trimming the expected ~5%:

One incidental R gotcha for anyone looping over dates: for (date in date_sequence) strips the Date class, so paste0("s3://bucket/date=", date) silently produces date=19601 (the epoch day) instead of date=2023-09-01. Iterate over as.character(date_sequence).

Moving data in

For test data and lookup tables, copy_to() works over Connect — with memory = FALSE, and with the pleasant surprise that Databricks Connect transfers via Arrow automatically (no arrow R package needed):

spark_test_data <- copy_to(sc, test_data, "test_data",
                           memory = FALSE, overwrite = TRUE)

What helps and what doesn't, since guides disagree:

Part 5: Measurements

The workload: 1M rows (categorical, numeric, date, logical columns), a filter → group_by → summarise pipeline, run three ways from local RStudio against serverless.

A second comparison, computing a small regional aggregate:

The headline insight is not "distributed wins" — it's why, and when it wouldn't. Over Databricks Connect, every collect() drags the full dataset across the internet to your laptop; that's the entire 12x gap in the second table. In a notebook on classic compute, the R session sits on the driver node, collect() is a fast in-datacenter copy, and at this scale Spark's coordination overhead can make early-collect competitive or even fastest. Same code, opposite conclusion depending on where the R session lives. Benchmark in your actual deployment topology — no rule of thumb survives a topology change.

The hybrid row quantifies the serverless caching trade-off: Delta materialization cost ~2.7x the fully distributed time (a storage write is not a memory cache) but still beat collecting into R by ~2x — and unlike a cache, the table persists across sessions.

Measurement mechanics: system.time() around a forced collect() for elapsed timings; lobstr::mem_used() before/after (with gc() for a clean baseline) for your local R session's memory, which on Connect is what fills when you collect. Executor-side behavior lives in the workspace's Spark UI.

The cost model

DBU counts aren't comparable across the two models. Published same-job comparisons range from ~30% cheaper to ~2x more expensive on serverless — measure your own workload. For experimentation, serverless has a nice property: spark_disconnect(sc) is all the teardown there is.

The Checklist

The complete skip-to-working recipe:

1. R 4.4+, Python 3.12 on PATH
2. install.packages(c("sparklyr", "pysparklyr", "dplyr", "dbplyr", "DBI", "lobstr")) — lobstr, not the archived pryr
3. pysparklyr::install_databricks(version = "17.3") — not 16.4, version pinned
4. reticulate::virtualenv_install("r-sparklyr-databricks-17.3", packages = "pandas<3")
5. PAT with all-apis scope, short expiry → ~/.Renviron (trailing newline, restart R)
6. spark_connect(method = "databricks_connect", serverless = TRUE, version = "17.3")
7. SHOW CATALOGS to find your writable catalog — don't assume main
8. copy_to(..., memory = FALSE); materialize() to Delta instead of compute(); plain joins instead of sdf_broadcast(); percentile_approx() instead of quantile()
9. Keep work distributed; collect() only summaries — every collected row crosses your internet connection

The broader lesson: the gap between what documentation, older guides, and LLM-generated examples describe and what a current default Databricks deployment actually does is wide and widening as the platform moves serverless-first. Verify against a live session — it's the only standard worth publishing.