> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Reporting setup

> The Reporting setup guide assists in configuring the reporting plugin, relying on specific dependencies and configurations.

The FlowX Reporting solution provides data analytics and visualization capabilities for your FlowX platform. This guide offers step-by-step instructions for setting up and configuring all components of the reporting system.

The reporting solution consists of three main components:

* **Reporting Plugin**: Extracts and processes data from the FlowX Engine
* **Spark Application**: Handles data transformation and loading operations
* **Apache Superset**: Provides the visualization interface and dashboard capabilities

## Dependencies

The reporting plugin, available as a Docker image, requires the following dependencies:

* **PostgreSQL or Oracle**: Dedicated instance for reporting data storage.
* **MongoDB**: Read access to the `app-runtime` and `integration-designer` databases, which the plugin reads to resolve app versions and active builds.
* **Reporting-plugin Helm Chart**:
  * Utilizes a Spark Application to extract data from the FLOWX.AI Engine database and populate the Reporting plugin database.
  * Utilizes Spark Operator. For more information, see the [Spark Operator documentation](https://www.kubeflow.org/docs/components/spark-operator).
* **Superset**:
  * Requires a dedicated PostgreSQL database for its operation.
  * Utilizes Redis for efficient caching.
  * Exposes its user interface via an ingress.

## Prerequisites

Before starting the installation, ensure you have:

* Kubernetes cluster with Helm installed
* Access to the source and destination databases, each on PostgreSQL or Oracle:
  * FlowX Engine database (source)
  * Reporting database (destination)
* A PostgreSQL database for Superset metadata
* Access to the MongoDB databases `app-runtime` and `integration-designer`, each with its own user (typically `app-runtime` and `integration-designer`)
* Read access is enough on the engine database and on both MongoDB databases. Only the reporting database is written to.
* Access to the container registry that serves your reporting plugin image and Helm chart
* Redis instance for Superset caching
* Ingress controller for exposing Superset UI

> [!NOTE]
>
> The engine and reporting databases each run on PostgreSQL or Oracle. Set `ENGINE_DATABASE_TYPE` and `REPORTING_DATABASE_TYPE` to `postgres` or `oracle` to match. The runtime and integration databases are always MongoDB.
>
> On Oracle, the plugin reads the UUID columns as `RAW(16)`, which is what the FlowX Liquibase scripts create.

> [!WARNING]
>
> Create a separate MongoDB user for each of the two databases, inside the database it reads. The plugin's connection string carries no `authSource`, so each connection authenticates against the database it targets. A single user created in a shared authentication database fails to authenticate.

## Reporting plugin helm chart configuration

Configuring the reporting plugin involves several steps:

### Installation of Spark Operator

1. Install the Spark Operator using Helm:

```bash theme={"dark"}
helm install local-spark-release spark-operator/spark-operator \
--namespace spark-operator --create-namespace \
--set webhook.enable=true \
--set logLevel=6
```

2. Apply RBAC configurations:

```bash theme={"dark"}
kubectl apply -f spark-rbac.yaml
```

3. Pull the reporting plugin image from the container registry provisioned for your deployment. Building the image locally is not required.

4. Update the `reporting-image` URL in the `spark-app.yml` file.

5. Configure the correct database ENV variables in the `spark-app.yml` file (check them in the above examples with/without webhook).

6. Deploy the application:

```bash theme={"dark"}
kubectl apply -f operator/spark-app.yaml
```

## Spark Operator deployment options

### Without webhook

For deployments without a webhook, manage secrets and environmental variables for security:

```yaml theme={"dark"}
sparkApplication: #Defines the Spark application configuration.
  enabled: "true" #Indicates that the Spark application is enabled for deployment.
  schedule: "@every 5m" #A cronJob that should run at every 5 minutes.
  driver: # This section configures the driver component of the Spark application.
    envVars: #Environment variables for driver setup.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
      ENGINE_DATABASE_PASSWORD: "password"
      REPORTING_DATABASE_PASSWORD: "password"
      RUNTIME_DATABASE_PASSWORD: "password"
      INTEGRATION_DATABASE_PASSWORD: "password"

  executor: #This section configures the executor component of the Spark application.
    envVars: #Environment variables for executor setup.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
      ENGINE_DATABASE_PASSWORD: "password"
      REPORTING_DATABASE_PASSWORD: "password"
      RUNTIME_DATABASE_PASSWORD: "password"
      INTEGRATION_DATABASE_PASSWORD: "password"
      
```

> [!NOTE]
>
> NOTE: Passwords are currently set as plain strings, which is not secure practice in a production environment.

### With webhook

When using the webhook, employ environmental variables with secrets for a balanced security approach:

```yaml theme={"dark"}
sparkApplication:
  enabled: "true"
  schedule: "@every 5m"
  driver:
    env: #Environment variables for driver setup with secrets.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle     
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys: 
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
  executor:
    env: #Environment variables for executor setup with secrets.
      ENGINE_DATABASE_USER: flowx
      ENGINE_DATABASE_URL: postgresql:5432
      ENGINE_DATABASE_NAME: process_engine
      ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
      REPORTING_DATABASE_USER: flowx
      REPORTING_DATABASE_URL: postgresql:5432
      REPORTING_DATABASE_NAME: reporting
      REPORTING_DATABASE_TYPE: postgres
      RUNTIME_DATABASE_USER: app-runtime
      RUNTIME_DATABASE_URL: mongodb:27017
      RUNTIME_DATABASE_NAME: app-runtime
      RUNTIME_DATABASE_TYPE: mongo
      INTEGRATION_DATABASE_USER: integration-designer
      INTEGRATION_DATABASE_URL: mongodb:27017
      INTEGRATION_DATABASE_NAME: integration-designer
      INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys:
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
```

> [!NOTE]
>
> In Kubernetes-based Spark deployments managed by the Spark Operator, you can define the sparkApplication configuration to customize the behavior, resources, and environment for both the driver and executor components of Spark jobs. The driver section allows fine-tuning of parameters specifically pertinent to the driver part of the Spark application.

Below are the configurable values within the chart values.yml file (with webhook):

```yml theme={"dark"}
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: ScheduledSparkApplication
metadata:
  name: reporting-plugin-spark-app
  namespace: dev
  labels:
    app.kubernetes.io/component: reporting
    app.kubernetes.io/instance: reporting-plugin
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: reporting-plugin
    app.kubernetes.io/release: 0.0.1-FLOWXRELEASE
    app.kubernetes.io/version: 0.0.1-FLOWXVERSION
    helm.sh/chart: reporting-plugin-0.2.4
spec:
  schedule: '@every 5m'
  concurrencyPolicy: Forbid
  template:
    type: Python
    pythonVersion: "3"
    mode: cluster
    image: <your-registry>/reporting-plugin:<version>
    imagePullPolicy: IfNotPresent
    mainApplicationFile: local:///opt/spark/work-dir/main.py
    sparkVersion: "3.5.8"
    restartPolicy:
      type: Never
      onFailureRetries: 0
      onFailureRetryInterval: 10
      onSubmissionFailureRetries: 5
      onSubmissionFailureRetryInterval: 20
    driver:
      cores: 1
      coreLimit: 1200m
      memory: 1024m
      labels:
        version: 3.5.8
      serviceAccount: spark
      env:
        ENGINE_DATABASE_USER: flowx
        ENGINE_DATABASE_URL: postgresql:5432
        ENGINE_DATABASE_NAME: process_engine
        ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
        REPORTING_DATABASE_USER: flowx
        REPORTING_DATABASE_URL: postgresql:5432
        REPORTING_DATABASE_NAME: reporting
        REPORTING_DATABASE_TYPE: postgres
        RUNTIME_DATABASE_USER: app-runtime
        RUNTIME_DATABASE_URL: mongodb:27017
        RUNTIME_DATABASE_NAME: app-runtime
        RUNTIME_DATABASE_TYPE: mongo
        INTEGRATION_DATABASE_USER: integration-designer
        INTEGRATION_DATABASE_URL: mongodb:27017
        INTEGRATION_DATABASE_NAME: integration-designer
        INTEGRATION_DATABASE_TYPE: mongo
        ENGINE_DATABASE_PASSWORD: "password"
        REPORTING_DATABASE_PASSWORD: "password"
        RUNTIME_DATABASE_PASSWORD: "password"
        INTEGRATION_DATABASE_PASSWORD: "password"
    extraEnvVarsMultipleSecretsCustomKeys: 
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
    executor:
      cores:  1
      instances: 1
      memory: 512m
      labels:
        version: 3.5.8
      env: #Environment variables for executor setup with secrets.
        ENGINE_DATABASE_USER: flowx
        ENGINE_DATABASE_URL: postgresql:5432
        ENGINE_DATABASE_NAME: process_engine
        ENGINE_DATABASE_TYPE: postgres # To set the type of engine database, can be also changed to oracle
        REPORTING_DATABASE_USER: flowx
        REPORTING_DATABASE_URL: postgresql:5432
        REPORTING_DATABASE_NAME: reporting
        REPORTING_DATABASE_TYPE: postgres
        RUNTIME_DATABASE_USER: app-runtime
        RUNTIME_DATABASE_URL: mongodb:27017
        RUNTIME_DATABASE_NAME: app-runtime
        RUNTIME_DATABASE_TYPE: mongo
        INTEGRATION_DATABASE_USER: integration-designer
        INTEGRATION_DATABASE_URL: mongodb:27017
        INTEGRATION_DATABASE_NAME: integration-designer
        INTEGRATION_DATABASE_TYPE: mongo
    extraEnvVarsMultipleSecretsCustomKeys:
      - name: postgresql-generic
        secrets: #Secrets retrieved from a generic source.
          ENGINE_DATABASE_PASSWORD: postgresql-password
          REPORTING_DATABASE_PASSWORD: postgresql-password
      - name: mongodb-generic
        secrets:
          RUNTIME_DATABASE_PASSWORD: mongodb-password
          INTEGRATION_DATABASE_PASSWORD: mongodb-password
```

> [!NOTE]
>
> `sparkApplication.sparkVersion` has no default in the chart, so it must be set explicitly. It has to match the Spark version in the plugin image.

The values below are the chart defaults.

| Parameter | Default | Description |
| - | - | - |
| **sparkApplication.driver.cores** | `1` | Driver CPU request |
| **sparkApplication.driver.coreLimit** | `1200m` | Driver CPU limit |
| **sparkApplication.driver.memory** | `1024m` | Driver memory |
| **sparkApplication.executor.cores** | `1` | See the note on executor sizing below |
| **sparkApplication.executor.instances** | `1` | See the note on executor sizing below |
| **sparkApplication.executor.memory** | `512m` | See the note on executor sizing below |
| **sparkApplication.schedule** | `@every 5m` | Extraction interval |
| **sparkApplication.concurrencyPolicy** | `Forbid` | Prevents overlapping runs |

Driver cores and memory are the values to raise if the extraction job does not finish inside the refresh interval.

> [!NOTE]
>
> Executor sizing is not driven by the chart values. The plugin builds its Spark session from the `spark.conf` file in the image, which ships `spark.executor.cores = 2`, `spark.executor.memory = 1024m` and `spark.executor.inst.max = 5`, the ceiling on the number of executors. It then recalculates the executor count at every run from the volume of data to be processed: one executor for small deltas, up to that ceiling for large ones.

### Limiting the reporting scope

By default the plugin reports on every workspace. To restrict it to a single workspace, set `WORKSPACE_ID` on the driver and executor to that workspace UUID.

***

### Superset configuration

Detailed Superset Configuration Guide:

> [card] [Superset configuration](https://github.com/apache/superset/blob/master/helm/superset/README.md)

> [card] [Superset docker image](https://hub.docker.com/r/apache/superset)

Refer to Superset Documentation for in-depth information:

> [card] [Superset documentation](https://superset.apache.org/docs/intro/)

## Post-installation steps

After installation, perform the following essential configurations:

### Datasource configuration

For document-related data storage, configure these environment variables:

* `SPRING_DATASOURCE_URL`
* `SPRING_DATASOURCE_USERNAME`
* `SPRING_DATASOURCE_PASSWORD`

Ensure accurate details to prevent startup errors. The Liquibase script manages schema and migrations.

### Redis configuration

The following values should be set with the corresponding Redis-related values:

* `SPRING_REDIS_HOST`
* `SPRING_REDIS_PORT`

## Superset single sign-on

Superset authenticates users against your identity provider over OpenID Connect, using Flask-AppBuilder's OAuth support, and maps a claim from the token to a Superset role at every login.

> [!WARNING]
>
> Configure `AUTH_OAUTH`, not `AUTH_OID`. The older `flask-oidc` approach is not the configuration FlowX runs and is not what the Superset image is built for.

### Image requirements

The Superset image needs `authlib` for OAuth and `psycopg2-binary` for its PostgreSQL metadata database. Build them into the image rather than installing them at pod start, so the cluster needs no access to a public package index.

### Superset configuration

Add the following to `configOverrides` in your Superset values, replacing the provider details with your own:

```python theme={"dark"}
import os
from flask_appbuilder.security.manager import AUTH_OAUTH
from superset.security import SupersetSecurityManager

SECRET_KEY = os.environ["SUPERSET_SECRET_KEY"]

ENABLE_PROXY_FIX = True

AUTH_TYPE = AUTH_OAUTH
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Public"
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_ROLES_MAPPING = {
    "superset-admins": ["Admin"],
    "superset-analysts": ["Alpha"],
}

OAUTH_PROVIDERS = [
    {
        "name": "keycloak",
        "icon": "fa-key",
        "token_key": "access_token",
        "remote_app": {
            "client_id": "superset",
            "client_secret": os.environ["OAUTH_CLIENT_SECRET"],
            "server_metadata_url": f"{os.environ['KEYCLOAK_ISSUER']}/.well-known/openid-configuration",
            "client_kwargs": {"scope": "openid email profile"},
        },
    }
]

class KeycloakSecurityManager(SupersetSecurityManager):
    def get_oauth_user_info(self, provider, response=None):
        if provider == "keycloak":
            remote = self.appbuilder.sm.oauth_remotes[provider]
            me = (response or {}).get("userinfo") or remote.userinfo()
            name = (me.get("name") or "").strip()
            first, _, last = name.partition(" ")
            return {
                "username": me.get("email"),
                "email": me.get("email"),
                "first_name": me.get("given_name") or first,
                "last_name": me.get("family_name") or last,
                "role_keys": me.get("groups", []),
            }
        return super().get_oauth_user_info(provider, response)

CUSTOM_SECURITY_MANAGER = KeycloakSecurityManager
```

`role_keys` is what `AUTH_ROLES_MAPPING` matches on, so the group names returned by your identity provider have to match the keys in that map. A user who is in none of the mapped groups gets the role set in `AUTH_USER_REGISTRATION_ROLE`.

> [!NOTE]
>
> `SECRET_KEY` must be a strong random value, supplied as a Kubernetes secret, and it has to stay stable for the lifetime of the installation. Rotating it invalidates active sessions and any encrypted metadata Superset holds.

### Identity provider setup

Register a confidential client for Superset using the authorization code flow, then:

**1. Set the redirect URI**

The callback path is `/oauth-authorized/{provider-name}`, where the provider name is the `name` field from `OAUTH_PROVIDERS`. With the example above, the redirect URI is `https://{your-superset-host}/oauth-authorized/keycloak`.

**2. Add the scopes and a group claim**

Request `openid email profile`, and add a mapper that puts the user's groups into the user-info response. Superset reads that claim at every login to assign a role, so without it no user gets access beyond the default role.

On Keycloak, attach the mapper to the client itself, as described below. Some other providers expose groups only through a dedicated scope, in which case add that scope to `client_kwargs`.

**3. Create the access groups**

Create one group per Superset role you want to grant and add the members. Map them in `AUTH_ROLES_MAPPING`.

> [!WARNING]
>
> Superset runs behind a proxy-aware configuration. Your ingress must forward the original protocol and host through `X-Forwarded-*` headers, otherwise the sign-on redirect is built with the wrong scheme and login fails.

### Group claim on Keycloak

The groups claim comes from a **Group Membership** mapper on the Superset client, the same mapper type FlowX uses on the platform client. For the step-by-step, see [Group membership mapper](https://docs.flowx.ai/6.x/setup-guides/access-management/configuring-an-iam-solution.md#group-membership-mapper), and set it as follows:

| Setting | Value | Why |
| - | - | - |
| **Mapper Type** | Group Membership | Puts the user's group memberships into the token |
| **Token Claim Name** | `groups` | Must match the claim the security manager reads |
| **Full group path** | Toggle **OFF** for plain names, **ON** for paths | Determines whether the claim carries `superset-admins` or `/superset-admins` |
| **Add to userinfo** | Toggle **ON** | The security manager above reads the claim from the user-info response, so this is required |
| **Add to access token** | Toggle **ON** | |

> [!WARNING]
>
> The keys in `AUTH_ROLES_MAPPING` must match the claim exactly. With **Full group path** ON, the group appears as `/superset-admins` and a key of `superset-admins` never matches, leaving every user on the default role.

> [!WARNING]
>
> **If your users are federated from another identity provider**, a Group Membership mapper emits nothing for them. Federated users are not members of any Keycloak group unless something puts them in one: FlowX's own [IDP federation](https://docs.flowx.ai/6.x/setup-guides/access-management/runtime-authorization.md#per-organization-idp-federation) carries group names in a user attribute instead. You have two options:
>
> * Map the incoming claim to real Keycloak groups with an **Advanced Claim to Group** mapper on the federated identity provider, then use the Group Membership mapper above.
> * Or emit the user attribute directly, by replacing the Group Membership mapper with a **User Attribute** mapper whose token claim name is `groups`.
>
> Either way, confirm which form actually reaches Superset before writing the `AUTH_ROLES_MAPPING` keys. Group identifiers from an external provider are often opaque IDs rather than names.

For the full list of provider options, see the [Flask-AppBuilder OAuth documentation](https://flask-appbuilder.readthedocs.io/en/latest/security.html).


## Related topics

- [Reporting plugin](https://docs.flowx.ai/6.x/docs/platform-deep-dive/plugins/custom-plugins/reporting/reporting-overview.md)
- [Microservices setup guides](https://docs.flowx.ai/6.x/setup-guides/setup-guides-overview.md)
- [Event reporting](https://docs.flowx.ai/6.x/observatory/sdk/event-reporting.md)
- [License service setup](https://docs.flowx.ai/6.x/setup-guides/license-engine-setup.md)
- [Plugins setup](https://docs.flowx.ai/6.x/setup-guides/plugins-setup-guide/plugins-setup-guide-overview.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.