How to Integrate Snowflake with Apache Spark

How do I integrate Snowflake with Apache Spark?

Configures the Snowflake Connector for Spark so you can read from and write to Snowflake directly from Apache Spark jobs.

Welcome to the Galaxy, Guardian!
You'll be receiving a confirmation email

Follow us on twitter :)

Oops! Something went wrong while submitting the form.

Description

Example H2

Example H3

Why integrate Snowflake with Spark?

Unified workloads let you crunch Snowflake data with Spark[4m[0m[F[2K[?25l

Why How to Integrate Snowflake with Apache Spark is important

How to Integrate Snowflake with Apache Spark Example Usage


# Join Spark DataFrames and write to Snowflake
orders_df = spark.read.format("snowflake").options(
  **base_options,
  dbtable="ORDERS"
).load()

items_df = spark.read.format("snowflake").options(
  **base_options,
  dbtable="ORDERITEMS"
).load()

revenue_df = orders_df.join(items_df, "id").groupBy("customer_id").sum("total_amount")

revenue_df.write.format("snowflake").options(
  **base_options,
  dbtable="CUSTOMER_REVENUE"
).mode("append").save()

How to Integrate Snowflake with Apache Spark Syntax


# Reading from Snowflake
df = (
  spark.read.format("snowflake")
    .option("url", "jdbc:snowflake://<account>.snowflakecomputing.com")
    .option("user", "SF_USER")
    .option("password", "SF_PASS")
    .option("dbtable", "ORDERS")
    .option("warehouse", "COMPUTE_WH")
    .option("sfDatabase", "ECOMMERCE")
    .option("sfSchema", "PUBLIC")
    .option("role", "ANALYST")
    .load()
)

# Writing to Snowflake
df.write.format("snowflake") \
  .option("url", "jdbc:snowflake://<account>.snowflakecomputing.com") \
  .option("user", "SF_USER") \
  .option("password", "SF_PASS") \
  .option("dbtable", "ORDERITEMS") \
  .option("warehouse", "COMPUTE_WH") \
  .option("sfDatabase", "ECOMMERCE") \
  .option("sfSchema", "PUBLIC") \
  .mode("overwrite") \
  .save()

Common Mistakes

Omitting the warehouse parameter. Without it, Snowflake picks the default warehouse, which may be suspended or underpowered. Always set .option("warehouse", "COMPUTE_WH") or your jobs will fail or throttle.
Forgetting to set sfSchema when writing. Spark will attempt to write to the PUBLIC schema, causing "object not found" errors if the target table lives elsewhere. Add .option("sfSchema", "YOUR_SCHEMA") to avoid confusion.

Frequently Asked Questions (FAQs)

Does the connector support pushdown?

Yes. Snowflake executes filters, projections, and aggregations serverF[0m[K[?25h

Want to learn about other SQL terms?

dbt tests

dbt tests are declarative assertions in dbt that automatically validate data quality during model builds.

dbt utils

dbt utils is a community-maintained dbt package that adds time-saving utility macros for analytics engineers, such as generating surrogate keys, safely unioning models, and automating date spines.

dbt Tutorial Glossary: Build, Test & Deploy SQL Models

dbt is an open-source framework that lets data teams transform raw warehouse tables into tested, documented, production-ready datasets using version-controlled SQL.

Trusted by top engineers on high-velocity teams