Reporting setup

The FlowX Reporting solution provides data analytics and visualization capabilities for your FlowX platform. This guide offers step-by-step instructions for setting up and configuring all components of the reporting system. The reporting solution consists of three main components:

Reporting Plugin: Extracts and processes data from the FlowX Engine
Spark Application: Handles data transformation and loading operations
Apache Superset: Provides the visualization interface and dashboard capabilities

Dependencies

The reporting plugin, available as a Docker image, requires the following dependencies:

PostgreSQL or Oracle: Dedicated instance for reporting data storage.
MongoDB: Read access to the app-runtime and integration-designer databases, which the plugin reads to resolve app versions and active builds.
Reporting-plugin Helm Chart:
- Utilizes a Spark Application to extract data from the FLOWX.AI Engine database and populate the Reporting plugin database.
- Utilizes Spark Operator. For more information, see the Spark Operator documentation.
Superset:
- Requires a dedicated PostgreSQL database for its operation.
- Utilizes Redis for efficient caching.
- Exposes its user interface via an ingress.

Prerequisites

Before starting the installation, ensure you have:

Kubernetes cluster with Helm installed
Access to the source and destination databases, each on PostgreSQL or Oracle:
- FlowX Engine database (source)
- Reporting database (destination)
A PostgreSQL database for Superset metadata
Access to the MongoDB databases app-runtime and integration-designer, each with its own user (typically app-runtime and integration-designer)
Read access is enough on the engine database and on both MongoDB databases. Only the reporting database is written to.
Access to the container registry that serves your reporting plugin image and Helm chart
Redis instance for Superset caching
Ingress controller for exposing Superset UI

The engine and reporting databases each run on PostgreSQL or Oracle. Set ENGINE_DATABASE_TYPE and REPORTING_DATABASE_TYPE to postgres or oracle to match. The runtime and integration databases are always MongoDB.On Oracle, the plugin reads the UUID columns as RAW(16), which is what the FlowX Liquibase scripts create.

Create a separate MongoDB user for each of the two databases, inside the database it reads. The plugin’s connection string carries no authSource, so each connection authenticates against the database it targets. A single user created in a shared authentication database fails to authenticate.

Reporting plugin helm chart configuration

Configuring the reporting plugin involves several steps:

Installation of Spark Operator

Install the Spark Operator using Helm:

helm install local-spark-release spark-operator/spark-operator \
--namespace spark-operator --create-namespace \
--set webhook.enable=true \
--set logLevel=6

Apply RBAC configurations:

kubectl apply -f spark-rbac.yaml

Pull the reporting plugin image from the container registry provisioned for your deployment. Building the image locally is not required.
Update the reporting-image URL in the spark-app.yml file.
Configure the correct database ENV variables in the spark-app.yml file (check them in the above examples with/without webhook).
Deploy the application:

kubectl apply -f operator/spark-app.yaml

Spark Operator deployment options

Without webhook

For deployments without a webhook, manage secrets and environmental variables for security:

sparkApplication: #Defines the Spark application configuration.
  enabled: "true" #Indicates that the Spark application is enabled for deployment.
  schedule: "@every 5m" #A cronJob that should run at every 5 minutes.
  driver: # This section configures the driver component of the Spark application.
    envVars: #Environment variables for driver setup.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
      ENGINE_DATABASE_PASSWORD: "password"
      REPORTING_DATABASE_PASSWORD: "password"
      RUNTIME_DATABASE_PASSWORD: "password"
      INTEGRATION_DATABASE_PASSWORD: "password"

  executor: #This section configures the executor component of the Spark application.
    envVars: #Environment variables for executor setup.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
      ENGINE_DATABASE_PASSWORD: "password"
      REPORTING_DATABASE_PASSWORD: "password"
      RUNTIME_DATABASE_PASSWORD: "password"
      INTEGRATION_DATABASE_PASSWORD: "password"
      

NOTE: Passwords are currently set as plain strings, which is not secure practice in a production environment.

With webhook

When using the webhook, employ environmental variables with secrets for a balanced security approach:

sparkApplication:
  enabled: "true"
  schedule: "@every 5m"
  driver:
    env: #Environment variables for driver setup with secrets.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle     
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys: 
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
  executor:
    env: #Environment variables for executor setup with secrets.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys:
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password

In Kubernetes-based Spark deployments managed by the Spark Operator, you can define the sparkApplication configuration to customize the behavior, resources, and environment for both the driver and executor components of Spark jobs. The driver section allows fine-tuning of parameters specifically pertinent to the driver part of the Spark application.

Below are the configurable values within the chart values.yml file (with webhook):

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: ScheduledSparkApplication
metadata:
  name: reporting-plugin-spark-app
  namespace: dev
  labels:
    app.kubernetes.io/component: reporting
    app.kubernetes.io/instance: reporting-plugin
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: reporting-plugin
    app.kubernetes.io/release: 0.0.1-FLOWXRELEASE
    app.kubernetes.io/version: 0.0.1-FLOWXVERSION
    helm.sh/chart: reporting-plugin-0.2.4
spec:
  schedule: '@every 5m'
  concurrencyPolicy: Forbid
  template:
    type: Python
    pythonVersion: "3"
    mode: cluster
    image: <your-registry>/reporting-plugin:<version>
    imagePullPolicy: IfNotPresent
    mainApplicationFile: local:///opt/spark/work-dir/main.py
    sparkVersion: "3.5.8"
    restartPolicy:
      type: Never
      onFailureRetries: 0
      onFailureRetryInterval: 10
      onSubmissionFailureRetries: 5
      onSubmissionFailureRetryInterval: 20
    driver:
      cores: 1
      coreLimit: 1200m
      memory: 1024m
      labels:
        version: 3.5.8
      serviceAccount: spark
      env:
        ENGINE_DATABASE_USER: flowx
        ENGINE_DATABASE_URL: postgresql:5432
        ENGINE_DATABASE_NAME: process_engine
        ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
        REPORTING_DATABASE_USER: flowx
        REPORTING_DATABASE_URL: postgresql:5432
        REPORTING_DATABASE_NAME: reporting
        REPORTING_DATABASE_TYPE: postgres
        RUNTIME_DATABASE_USER: app-runtime
        RUNTIME_DATABASE_URL: mongodb:27017
        RUNTIME_DATABASE_NAME: app-runtime
        RUNTIME_DATABASE_TYPE: mongo
        INTEGRATION_DATABASE_USER: integration-designer
        INTEGRATION_DATABASE_URL: mongodb:27017
        INTEGRATION_DATABASE_NAME: integration-designer
        INTEGRATION_DATABASE_TYPE: mongo
        ENGINE_DATABASE_PASSWORD: "password"
        REPORTING_DATABASE_PASSWORD: "password"
        RUNTIME_DATABASE_PASSWORD: "password"
        INTEGRATION_DATABASE_PASSWORD: "password"
    extraEnvVarsMultipleSecretsCustomKeys: 
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
    executor:
      cores:  1
      instances: 1
      memory: 512m
      labels:
        version: 3.5.8
      env: #Environment variables for executor setup with secrets.
        ENGINE_DATABASE_USER: flowx
        ENGINE_DATABASE_URL: postgresql:5432
        ENGINE_DATABASE_NAME: process_engine
        ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
        REPORTING_DATABASE_USER: flowx
        REPORTING_DATABASE_URL: postgresql:5432
        REPORTING_DATABASE_NAME: reporting
        REPORTING_DATABASE_TYPE: postgres
        RUNTIME_DATABASE_USER: app-runtime
        RUNTIME_DATABASE_URL: mongodb:27017
        RUNTIME_DATABASE_NAME: app-runtime
        RUNTIME_DATABASE_TYPE: mongo
        INTEGRATION_DATABASE_USER: integration-designer
        INTEGRATION_DATABASE_URL: mongodb:27017
        INTEGRATION_DATABASE_NAME: integration-designer
        INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys:
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password

sparkApplication.sparkVersion has no default in the chart, so it must be set explicitly. It has to match the Spark version in the plugin image.

The values below are the chart defaults.

Parameter	Default	Description
sparkApplication.driver.cores	`1`	Driver CPU request
sparkApplication.driver.coreLimit	`1200m`	Driver CPU limit
sparkApplication.driver.memory	`1024m`	Driver memory
sparkApplication.executor.cores	`1`	See the note on executor sizing below
sparkApplication.executor.instances	`1`	See the note on executor sizing below
sparkApplication.executor.memory	`512m`	See the note on executor sizing below
sparkApplication.schedule	`@every 5m`	Extraction interval
sparkApplication.concurrencyPolicy	`Forbid`	Prevents overlapping runs

Driver cores and memory are the values to raise if the extraction job does not finish inside the refresh interval.

Executor sizing is not driven by the chart values. The plugin builds its Spark session from the spark.conf file in the image, which ships spark.executor.cores = 2, spark.executor.memory = 1024m and spark.executor.inst.max = 5, the ceiling on the number of executors. It then recalculates the executor count at every run from the volume of data to be processed: one executor for small deltas, up to that ceiling for large ones.

Limiting the reporting scope

By default the plugin reports on every workspace. To restrict it to a single workspace, set WORKSPACE_ID on the driver and executor to that workspace UUID.

Superset configuration

Detailed Superset Configuration Guide:

Superset configuration

Superset docker image

Refer to Superset Documentation for in-depth information:

Superset documentation

Post-installation steps

After installation, perform the following essential configurations:

Datasource configuration

For document-related data storage, configure these environment variables:

SPRING_DATASOURCE_URL
SPRING_DATASOURCE_USERNAME
SPRING_DATASOURCE_PASSWORD

Ensure accurate details to prevent startup errors. The Liquibase script manages schema and migrations.

Redis configuration

The following values should be set with the corresponding Redis-related values:

SPRING_REDIS_HOST
SPRING_REDIS_PORT

Superset single sign-on

Superset authenticates users against your identity provider over OpenID Connect, using Flask-AppBuilder’s OAuth support, and maps a claim from the token to a Superset role at every login.

Configure AUTH_OAUTH, not AUTH_OID. The older flask-oidc approach is not the configuration FlowX runs and is not what the Superset image is built for.

Image requirements

The Superset image needs authlib for OAuth and psycopg2-binary for its PostgreSQL metadata database. Build them into the image rather than installing them at pod start, so the cluster needs no access to a public package index.

Superset configuration

Add the following to configOverrides in your Superset values, replacing the provider details with your own:

import os
from flask_appbuilder.security.manager import AUTH_OAUTH
from superset.security import SupersetSecurityManager

SECRET_KEY = os.environ["SUPERSET_SECRET_KEY"]

ENABLE_PROXY_FIX = True

AUTH_TYPE = AUTH_OAUTH
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Public"
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_ROLES_MAPPING = {
    "superset-admins": ["Admin"],
    "superset-analysts": ["Alpha"],
}

OAUTH_PROVIDERS = [
    {
        "name": "keycloak",
        "icon": "fa-key",
        "token_key": "access_token",
        "remote_app": {
            "client_id": "superset",
            "client_secret": os.environ["OAUTH_CLIENT_SECRET"],
            "server_metadata_url": f"{os.environ['KEYCLOAK_ISSUER']}/.well-known/openid-configuration",
            "client_kwargs": {"scope": "openid email profile"},
        },
    }
]

class KeycloakSecurityManager(SupersetSecurityManager):
    def get_oauth_user_info(self, provider, response=None):
        if provider == "keycloak":
            remote = self.appbuilder.sm.oauth_remotes[provider]
            me = (response or {}).get("userinfo") or remote.userinfo()
            name = (me.get("name") or "").strip()
            first, _, last = name.partition(" ")
            return {
                "username": me.get("email"),
                "email": me.get("email"),
                "first_name": me.get("given_name") or first,
                "last_name": me.get("family_name") or last,
                "role_keys": me.get("groups", []),
            }
        return super().get_oauth_user_info(provider, response)

CUSTOM_SECURITY_MANAGER = KeycloakSecurityManager

role_keys is what AUTH_ROLES_MAPPING matches on, so the group names returned by your identity provider have to match the keys in that map. A user who is in none of the mapped groups gets the role set in AUTH_USER_REGISTRATION_ROLE.

SECRET_KEY must be a strong random value, supplied as a Kubernetes secret, and it has to stay stable for the lifetime of the installation. Rotating it invalidates active sessions and any encrypted metadata Superset holds.

Identity provider setup

Set the redirect URI

The callback path is /oauth-authorized/{provider-name}, where the provider name is the name field from OAUTH_PROVIDERS. With the example above, the redirect URI is https://{your-superset-host}/oauth-authorized/keycloak.

Add the scopes and a group claim

Request openid email profile, and add a mapper that puts the user’s groups into the user-info response. Superset reads that claim at every login to assign a role, so without it no user gets access beyond the default role.On Keycloak, attach the mapper to the client itself, as described below. Some other providers expose groups only through a dedicated scope, in which case add that scope to client_kwargs.

Create the access groups

Create one group per Superset role you want to grant and add the members. Map them in AUTH_ROLES_MAPPING.

Superset runs behind a proxy-aware configuration. Your ingress must forward the original protocol and host through X-Forwarded-* headers, otherwise the sign-on redirect is built with the wrong scheme and login fails.

Group claim on Keycloak

The groups claim comes from a Group Membership mapper on the Superset client, the same mapper type FlowX uses on the platform client. For the step-by-step, see Group membership mapper, and set it as follows:

Setting	Value	Why
Mapper Type	Group Membership	Puts the user’s group memberships into the token
Token Claim Name	`groups`	Must match the claim the security manager reads
Full group path	Toggle OFF for plain names, ON for paths	Determines whether the claim carries `superset-admins` or `/superset-admins`
Add to userinfo	Toggle ON	The security manager above reads the claim from the user-info response, so this is required
Add to access token	Toggle ON

The keys in AUTH_ROLES_MAPPING must match the claim exactly. With Full group path ON, the group appears as /superset-admins and a key of superset-admins never matches, leaving every user on the default role.

If your users are federated from another identity provider, a Group Membership mapper emits nothing for them. Federated users are not members of any Keycloak group unless something puts them in one: FlowX’s own IDP federation carries group names in a user attribute instead. You have two options:

Map the incoming claim to real Keycloak groups with an Advanced Claim to Group mapper on the federated identity provider, then use the Group Membership mapper above.
Or emit the user attribute directly, by replacing the Group Membership mapper with a User Attribute mapper whose token claim name is groups.

Either way, confirm which form actually reaches Superset before writing the AUTH_ROLES_MAPPING keys. Group identifiers from an external provider are often opaque IDs rather than names.

For the full list of provider options, see the Flask-AppBuilder OAuth documentation.

Microservices

AI Platform

Plugins

Observability

Access management

Reporting setup

Dependencies

Prerequisites

Reporting plugin helm chart configuration

Installation of Spark Operator

Spark Operator deployment options

Without webhook

With webhook

Limiting the reporting scope

Superset configuration

Superset configuration

Superset docker image

Superset documentation

Post-installation steps

Datasource configuration

Redis configuration

Superset single sign-on

Image requirements

Superset configuration

Identity provider setup

Group claim on Keycloak

​Dependencies

​Prerequisites

​Reporting plugin helm chart configuration

​Installation of Spark Operator

​Spark Operator deployment options

​Without webhook

​With webhook

​Limiting the reporting scope

​Superset configuration

Superset configuration

Superset docker image

Superset documentation

​Post-installation steps

​Datasource configuration

​Redis configuration

​Superset single sign-on

​Image requirements

​Superset configuration

​Identity provider setup

​Group claim on Keycloak

Dependencies

Prerequisites

Reporting plugin helm chart configuration

Installation of Spark Operator

Spark Operator deployment options

Without webhook

With webhook

Limiting the reporting scope

Superset configuration

Post-installation steps

Datasource configuration

Redis configuration

Superset single sign-on

Image requirements

Superset configuration

Identity provider setup

Group claim on Keycloak