Running Apache Superset in Kubernetes: A Complete Guide

Running Apache Superset in Kubernetes

Learn why and how to deploy Apache Superset on Kubernetes for scalable, production-grade business-intelligence, including Helm-based installation, persistence, scaling, and common pitfalls.

What Is Apache Superset?

Apache Superset is a modern, open-source business-intelligence (BI) platform that lets users explore data and build interactive dashboards through a browser-based UI. It ships with a SQL editor, rich visualization library, metadata caching, and a robust role-based access control (RBAC) model.

Why Run Superset on Kubernetes?

Kubernetes (K8s) has become the de-facto standard for orchestrating containerized applications. Deploying Superset on K8s offers several advantages:

• Scalability: Horizontal Pod Autoscalers (HPAs) can add web or worker replicas on-demand.
• High availability: Self-healing ensures pods are rescheduled when nodes fail.
• Declarative infrastructure: All resources live in version-controlled manifests, supporting GitOps workflows.
• Seamless upgrades: Rolling updates reduce downtime during Superset version bumps.
• Cloud-agnostic: Run identical stacks on any managed or self-hosted cluster.

Superset Architecture Overview

A production Superset cluster typically contains five core components:

• Superset Web Server – Flask and Gunicorn handle HTTP traffic.
• Celery Worker – Executes async tasks like report generation.
• Celery Beat – Schedules periodic tasks.
• Metadata Database – Stores dashboards, charts, and user credentials (commonly Postgres).
• Message Broker & Cache – Redis is used for Celery queues and caching.

Kubernetes lets you containerize each piece, wire them together with Services, and expose the UI with an Ingress.

Deployment Strategies

1. Helm Chart (Recommended)

The apache/superset Helm chart maintained by the Superset community encapsulates best-practice manifests and sane defaults. Using Helm allows parameter overrides via values.yaml, version pinning, and one-command upgrades.

2. Raw YAML Manifests

If you need fine-grained control, craft your own Deployment, StatefulSet, Secret, and Ingress objects. This path requires more maintenance but offers maximum flexibility.

3. Kubernetes Operators

Some teams wrap Superset inside a custom operator to automate database migrations and configuration seeding. Operators add CRDs but can simplify Day-2 operations.

Step-by-Step Helm Installation

Prerequisites

• Kubernetes 1.23+ with access to create namespaces, PVCs, and Ingress
• Helm 3
• A provisioned Postgres database (CloudSQL, RDS, or on-cluster)
• A Redis instance (in-cluster or managed)
• An Ingress controller (NGINX, ALB, etc.)

1. Add the Helm repo

helm repo add superset https://apache.github.io/superset

2. Create a namespace

kubectl create ns superset

3. Create Secrets

kubectl -n superset create secret generic superset-secrets \
--from-literal=secret_key="$(openssl rand -hex 32)" \
--from-literal=db_pass="POSTGRES_PASSWORD"

Storing secrets in Secrets (or an external vault) avoids hard-coding sensitive values in values.yaml.

4. Craft values.yaml

replicaCount: 2
configOverrides:
SECRET_KEY: "{{ .Values.global.supersetSecretKey }}"
SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:{{ .Values.global.dbPass }}@postgres.example:5432/superset
REDIS_HOST: redis-master
service:
type: ClusterIP
ingress:
enabled: true
hosts:
- host: superset.example.com
paths: ["/"]
tls:
- secretName: superset-tls
hosts:
- superset.example.com

5. Install

helm upgrade --install superset superset/superset -n superset -f values.yaml

6. Initialize the Database

kubectl -n superset exec -it deploy/superset -- bash -c "superset db upgrade && superset fab create-admin"

On new clusters run migrations and create the first admin user.

Persistent Storage

Superset is largely stateless once it connects to Postgres and Redis. However, you may enable a shared uploads volume for CSV exports and custom images. Use a PersistentVolume (EFS, Filestore, etc.) mounted to both web and worker pods.

Scaling for Production

• Web server: Add replicas; set Gunicorn workers per pod based on CPU.
• Celery workers: Isolate chart schedules from web traffic; use separate Deployment with its own HorizontalPodAutoscaler.
• Database: Superset is read-heavy; enable connection pooling (PgBouncer) if concurrency spikes.

Security & Secrets Management

Follow these guidelines:

• Store SECRET_KEY, DB creds, and OAuth client secrets in Secrets or a cloud key-manager.
• Restrict ServiceAccount RBAC to namespace-scoped permissions.
• Enable TLS termination at the Ingress layer.
• Harden the base image by pinning tags and scanning for CVEs.

CI/CD & GitOps

Keep your values.yaml and overlays in Git. A GitOps engine like Argo CD or Flux will diff and sync changes automatically. To upgrade Superset, commit a version bump (image.tag: 3.0.1) and let the controller perform a rolling deploy.

Monitoring & Logging

• Expose Prometheus metrics from Gunicorn and Celery with the prometheus-flask-exporter plugin.
• Ship logs to Elasticsearch/Loki via a DaemonSet fluent-bit.
• Create alerts for 5xx error rates, worker queue depth, and dashboard latency.

Common Mistakes and How to Fix Them

1. Forgetting Persistent Storage for the Metadata DB

Why it’s wrong: Running Postgres inside a pod without a PVC risks data loss on reschedule.
Fix: Use a StatefulSet+PVC or an external managed database.

2. Not Setting SECRET_KEY

Why it’s wrong: Superset will generate a random key on each pod start, invalidating sessions.
Fix: Supply a stable SECRET_KEY via a Secret mounted as env var.

3. Oversubscribing Celery Workers

Why it’s wrong: Running too many concurrent tasks exhausts Redis and DB connections, causing dashboard timeouts.
Fix: Benchmark workload, tune worker_concurrency, and use an HPA driven by queue_latency.

Where Does Galaxy Fit In?

Superset’s built-in SQL Lab editor is convenient for analysts, but many engineers prefer a dedicated IDE-style experience. Galaxy offers a lightning-fast SQL editor with AI copilot and versioned query sharing. Teams often:

• Prototype and optimize SQL in Galaxy’s desktop app.
• Copy validated queries into Superset to build production dashboards.
• Leverage Galaxy’s query endorsement to ensure only vetted SQL reaches Superset.

This workflow separates ad-hoc exploration (Galaxy) from governed visualization (Superset).

Conclusion

Deploying Apache Superset on Kubernetes marries a powerful BI platform with a resilient orchestration layer. By containerizing web and worker services, externalizing state, and embracing Helm or GitOps, teams achieve repeatable, scalable analytics infrastructure. Avoid common pitfalls—persistent storage, secret management, and right-sizing workers—and you’ll unlock interactive dashboards that scale right alongside your data.