Reporting plugin

​

Quick walkthrough video

Watch this quick walkthrough video to get started:

​

Data exploration and visualization

The plugin uses Superset as a free data exploration and visualization tool. Use the suggested query structure and logic to ensure accurate reporting and avoid duplicates, even during database updates. Do not just use select on single reporting tables. Apache Superset is an open-source software application for data exploration and data visualization able to handle data at a large scale. It enables users to connect their company’s databases and perform data analysis, and build charts and assemble dashboards. Superset is also an SQL IDE, so you can write SQL, join data, create datasets, and more.

​

Plugin architecture

Here’s an overview of the reporting plugin architecture:

​

Setting up reporting for a process definition

​

Reporting data model

1. Open the branch view.

2. If the current project version is committed, start a new version.

3. Make sure the new work-in-progress branch is selected, then go back to the process definition page.
4. Click on the Data Model icon and navigate in the object model to the target parameters.
5. Set up the business data structure to be reported by using the “Use in reporting” flag.

• Click “Show details”.
• Click “Use in reporting”.

You can do this also for each parameter individually.

​

Primitive parameters reporting

In the following example, there are 3 simple parameters that will be extracted: loan_approvalFee, loan_currency and loan_downPayment. They will be reported in the table params_{process_alias}, one row per instance.

​

Arrays of primitives reporting

In the following example, the applicant can submit more than one email address, so each email will be extracted to a separate row in a dedicated table in the reporting database.

​

Array of objects reporting

In this example, the applicant can have several real estate assets, so a subset of data items (currentValue, mortgageBalance, propertyType) will be extracted for each one of them.

​

Enabling process reporting

Enable reporting by checking the “Use process in reporting” button on the settings/general page of each process.

• Modifying the data model of a process version has no impact until the moment the version or build of the project is set in the “Active policy” tab and is also set to “use_in_reporting”.

The reporting refresh schedule is set for a fixed interval (currently 5 minutes):

• No processing overlaps are allowed. If processing takes more than 5 minutes, the next processing is automatically postponed until the current one is finished.
• The reporting plugin chooses the number of Spark executors at each run, based on the volume of data to be processed: one executor for small deltas, up to the configured maximum for large ones. It does not use Spark dynamic allocation, and the instances value in the chart is only the driver’s starting point.
• Rebuilding the whole history for a process (if the master data model, process name or the Published version change) typically takes more time. It is better to make these changes after the working hours.

​

Reporting database

​

Main tables

Common fields for joins:

• inst_id: Unique identifier for each process instance
• query_time: Timestamp for when the plugin extracts data
• organization_id, workspace_id, build_app_version_id: The organization, workspace and app version the instance belongs to

Useful fields from instances table:

• date_started/finished timestamps
• process_alias (process name with ”_” instead of spaces)

• State of the process (started, finished, etc.)
• context_name: Useful if the process is started by another process

Useful fields from current_nodes table:

• node_started/finished timestamps for the nodes
• node_name, node_type
• swimlane of the current node
• prev_node_end: For calculating when a token is waiting for another process branch

Useful fields from token_history table: Similar to current_nodes but includes all nodes in the instance history.

​

Parameters and object tables

Common fields, on which the joins are built:

• inst_id, unique identifier for each process instance
• query_time, recorded at the moment the plugin extracts data from the database
• organization_id, workspace_id, build_app_version_id, carried on every parameter and array table

For the params_ table, there is a single row per process instance. For arrays and arrays of objects tables, there are multiple rows per process instance.

​

Reporting for UI Flows and workflows

Besides process instances, the reporting plugin extracts UI Flow sessions and integration workflow runs into the same reporting database.

​

Turning it on

For a UI Flow, open its General settings and toggle Use UI flow in reporting ON. For an integration workflow, open its Settings and toggle Use workflow in reporting ON. As with processes, only resources that belong to the active build are extracted.

​

How the columns are decided

Processes are reported attribute by attribute: you mark each data model attribute for reporting, and the plugin builds the columns from those definitions. UI Flows and workflows have no per-attribute marking. The plugin infers the structure from the data itself on every run, so each variable present in a session or a token becomes a column, and every array is exploded into its own child table.

​

UI Flow tables

uiflow_sessions also carries app_uiflow_id, resource_id, status, uf_started and uf_updated.

​

Workflow tables

wf_instances also carries app_workflow_id, workflow_resource_id, wf_status, wf_started and wf_finished. wf_tokens adds token_status and wf_updated, and links to its instance through wf_inst_uuid. The same naming difference applies here: the token identifier is token_uuid in wf_tokens and wf_token_id in the parameter and array tables. Every table listed above also carries organization_id, workspace_id and build_app_version_id.

---

​

Using Superset for data exploration

​

Data sources

The Data tab represents the sources of all information:

• Databases
• CSV files
• Tables
• Excel files

Reporting plugin can be used with Superset by connecting it with a PostgreSQL DB.

​

Charts

Charts represent the output of the information. There are multiple visualization charts/ plugins available.

​

Dashboards

With the use of dashboards, you can share persuading flows, show how metrics change in various scenarios and match your company efforts with logical, evidence‐based visual indicators.

​

Datasets

Contains all the information for extracting and processing data from the DB, including SQL queries, calculated metrics information, cache settings, etc. Datasets can also be exported / imported.

​

Connecting to a database

Before using Superset, ensure you have a PostgreSQL database installed and configured. Follow these guides for setup: FlowX Engine DB configuration Reporting DB configuration To connect Superset to a database, follow the next steps:

1. From the Toolbar, hover your cursor over the ”+” icon, then select Data, and then select Connect Database.

2. The Connect a database window appears. Select the appropriate database card (in this case - PostgreSQL).

3. After you selected the DB, click Connect this database with a SQLAlchemy URI string instead?.
4. Fill in the SQLALCHEMY URI and then click Connect.

​

Creating and configuring charts

There are multiple ways in which you can create/configure a chart.

​

Creating charts using Datasets tab

To create a Chart using the first method, you must follow the next steps:

1. From the top toolbar, select Data and then Datasets.

2. Select the desired dataset.
3. On the explore page, choose the visualization type and click Create chart.

4. When you select a dataset, by default table visualization type is selected.

​

Creating charts using Chart gallery

Using the Chart gallery is a useful method when you are not quite sure about which chart will be best for you. To create a Chart using the second method, you must follow the next steps:

1. Select the ”+” icon from the top toolbar and choose Chart.
2. Choose the dataset and chart type.
3. Review the description and example graphics of the selected chart, then click Create new chart.

​

Configuring a Chart

Configure the Query fields by dragging and dropping relevant columns to the matching fields. Note that time and query form attributes vary by chart type.

​

Exporting/importing a Chart

You can export and import charts to help you analyze your data and manipulate dashboards. To export/import a chart, follow the next steps:

1. Open Superset and navigate to Charts from the top navigation bar.
2. Select the desired chart and click the breadcrumbs menu in the top-right corner.
3. Choose an export option: .CSV, .JSON, or Download as image.

Table example:

• Time - time related form attributes
• Query - query attributes
  • Dimensions - one or many columns to group by
  • Metrics - metrics to display
  • Percentage metrics - metrics for which percentage of total are to be displayed, calculated from only data within the row limit
  • Filters - metric used for filtering
  • Sort by - metric used to define how the top series are sorted if a series or row limit is present
  • Row limit - limits the number of rows that get displayed

​

Creating a dashboard

To create a dashboard follow the next steps:

1. Create a new chart and save it to a new dashboard.
2. To publish, click Save and go to Dashboard.

For details on how to configure the FlowX.AI reporting plugin, check the following section: