Documents plugin setup

The plugin is available as a docker image.

Dependencies

Before setting up the plugin, ensure that you have the following dependencies installed and configured:

PostgreSQL Database: You will need a PostgreSQL database to store data related to document templates and documents.
MongoDB Database: MongoDB is required for the HTML templates feature of the plugin.
Kafka: Establish a connection to the Kafka instance used by the FLOWX.AI engine.
Redis: Set up a Redis instance for caching purposes.
S3-Compatible File Storage Solution: Deploy an S3-compatible file storage solution to store document files.

Configuration

The plugin comes with pre-filled configuration properties, but you need to set up a few custom environment variables to tailor it to your specific setup. Here are the key configuration steps:

Redis server

The plugin can utilize the Redis component already deployed for the FLOWX.AI engine. Make sure it is configured properly.

Authorization configuration

To connect to the identity management platform, set the following environment variables:

Environment Variable	Description	Default Value
`SECURITY_OAUTH2_BASESERVERURL`	Base URL of the OAuth2/OIDC server	`https://keycloak.example.com/auth`
`SECURITY_OAUTH2_CLIENT_CLIENTID`	OAuth2 client ID for the Documents Plugin	`document-plugin`
`SECURITY_OAUTH2_CLIENT_CLIENTSECRET`	OAuth2 client secret	`secret`
`SECURITY_OAUTH2_REALM`	OAuth2 realm name	`flowx`

Document processing configuration

Environment Variable	Description	Default Value
`FLOWX_DEFAULTCLIENTTYPE`	Default client type	`PF`
`FLOWX_HTML_TEMPLATES_ENABLED`	Enable HTML templates feature	`false`
`FLOWX_HTML_TEMPLATES_PDFFONTPATHS`	Paths to fonts for HTML templates	`/statics/fonts/Calibri.ttf, /statics/fonts/DejaVuSans.ttf`
`FLOWX_CONVERT_DPI`	DPI setting for PDF to image conversion	`150`

Set the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config to select the font used for generating documents based on PDF templates.

If you want to use specific fonts in your PDF templates, override the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config. By default, Calibri and DejaVuSans are available fonts.

FLOWX_CONVERT_DPI: Sets the DPI (dots per inch) for PDF to JPEG conversion. Higher values result in higher resolution images. (Default value: 150).

After making these configurations, the fonts will be available for use within PDF templates.

Database configuration

SQL database (PostgreSQL)

The Documents Plugin uses a PostgreSQL database for relational data storage.

Primary MongoDB configuration

Environment Variable	Description	Example Value
`SPRING_DATA_MONGODB_URI`	MongoDB connection URI	`mongodb://flowx:password@jx-document-mongodb:27017/document`
`MONGODB_USERNAME`	MongoDB username	`flowx`
`MONGODB_PASSWORD`	MongoDB password	`password`
`SPRING_DATA_MONGODB_UUID_REPRESENTATION`	UUID representation format	`standard`
`SPRING_DATA_MONGODB_STORAGE`	MongoDB storage type	`mongodb`

Runtime MongoDB configuration

Environment Variable	Description	Example Value
`SPRING_DATA_MONGODB_RUNTIME_ENABLED`	Enable runtime MongoDB connection	`true`
`SPRING_DATA_MONGODB_RUNTIME_URI`	Runtime MongoDB connection URI	`mongodb://${MONGODB_USERNAME:flowx}:${MONGODB_PASSWORD:password}@jx-document-mongodb:27017/document`
`MONGODB_USERNAME`	Runtime MongoDB username	`paperflow`
`MONGODB_PASSWORD`	Runtime MongoDB password	`password`
`SPRING_DATA_MONGODB_RUNTIME_AUTOINDEXCREATION`	Enable automatic index creation	`false`
`SPRING_DATA_MONGODB_RUNTIME_UUIDREPRESENTATION`	UUID representation format for runtime connection	`standard`

Redis configuration

Set the following values with the corresponding Redis-related values:

Environment Variable	Description	Default Value
`SPRING_DATA_REDIS_HOST`	Hostname of the Redis server	`localhost`
`SPRING_DATA_REDIS_PORT`	Port number of the Redis server	`6379`
`SPRING_DATA_REDIS_PASSWORD`	Password for Redis authentication	`defaultpassword`
`REDIS_TTL`	Time-to-live (TTL) for Redis cache	`5000000` (miliseconds)

Multipart upload configuration

Environment Variable	Description	Default Value
`SPRING_SERVLET_CONTEXTPATH`	Servlet context path	`/`
`SPRING_SERVLET_MULTIPART_MAXFILESIZE`	Maximum file size for uploads	`50MB`
`SPRING_SERVLET_MULTIPART_MAXREQUESTSIZE`	Maximum request size for multipart uploads	`50MB`

Basic Kafka configuration

Environment Variable	Description	Default Value
`SPRING_KAFKA_BOOTSTRAPSERVERS`	Address of Kafka server(s)	`localhost:9092`
`SPRING_KAFKA_SECURITY_PROTOCOL`	Security protocol for Kafka	`PLAINTEXT`
`SPRING_KAFKA_CONSUMER_GROUPID`	Consumer group ID for the service	`kafka-svc-document-consumer-local-test2`
`KAFKA_MESSAGE_MAX_BYTES`	Maximum message size in bytes	`52428800` (50MB)
`KAFKA_AUTHEXCEPTIONRETRYINTERVAL`	Retry interval for auth exceptions	`10` (seconds)

Topic naming configuration

Environment Variable	Description	Default Value
`KAFKA_TOPIC_NAMING_SEPARATOR`	Primary separator for topic names	`.`
`KAFKA_TOPIC_NAMING_SEPARATOR2`	Secondary separator for topic names	`-`
`KAFKA_TOPIC_NAMING_PACKAGE`	Package prefix for topic names	`ai.flowx.`
`KAFKA_TOPIC_NAMING_ENVIRONMENT`	Environment segment for topic names	`dev.`
`KAFKA_TOPIC_NAMING_VERSION`	Version suffix for topic names	`.v1`

Thread configuration

Environment Variable	Description	Default Value
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGENERATEHTMLIN`	Threads for HTML document generation	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTPERSISTIN`	Threads for document persistence	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTSPLITTIN`	Threads for document splitting	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGETURLSIN`	Threads for document URL retrieval	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_OCRIN`	Threads for OCR processing	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEDELETEIN`	Threads for file deletion	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEUPDATEIN`	Threads for file updates	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECONVERTIN`	Threads for file conversion	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEPERSISTIN`	Threads for file persistence	`5`
`KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECOMBINEIN`	Threads for file combining	`5`

Document topics

Generate HTML documents

Environment Variable	Description	Default Value
`KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_IN`	Topic for incoming HTML generation requests	`ai.flowx.dev.plugin.document.trigger.generate.html.v1`
`KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_OUT`	Topic for HTML generation results	`ai.flowx.dev.engine.receive.plugin.document.generate.html.results.v1`

Document persistence

Environment Variable	Description	Default Value
`KAFKA_TOPIC_DOCUMENT_PERSIST_IN`	Topic for incoming document persistence requests	`ai.flowx.dev.plugin.document.trigger.persist.document.v1`
`KAFKA_TOPIC_DOCUMENT_PERSIST_OUT`	Topic for document persistence results	`ai.flowx.dev.engine.receive.plugin.document.persist.document.results.v1`

Document splitting

Environment Variable	Description	Default Value
`KAFKA_TOPIC_DOCUMENT_SPLIT_IN`	Topic for incoming document splitting requests	`ai.flowx.dev.plugin.document.trigger.split.document.v1`
`KAFKA_TOPIC_DOCUMENT_SPLIT_OUT`	Topic for document splitting results	`ai.flowx.dev.engine.receive.plugin.document.split.document.results.v1`

Document URL retrieval

Environment Variable	Description	Default Value
`KAFKA_TOPIC_DOCUMENT_GET_URLS_IN`	Topic for incoming URL retrieval requests	`ai.flowx.dev.plugin.document.retrieve.urls.v1`
`KAFKA_TOPIC_DOCUMENT_GET_URLS_OUT`	Topic for URL retrieval results	`ai.flowx.dev.engine.receive.plugin.document.urls.results.v1`

OCR processing

Environment Variable	Description	Default Value
`KAFKA_TOPIC_OCR_IN`	Topic for incoming OCR requests	`ai.flowx.dev.plugin.document.store.ocr.v1`
`KAFKA_TOPIC_OCR_OUT`	Topic for OCR processing results	`ai.flowx.dev.engine.receive.plugin.document.ocr.results.v1`

File operation topics

File deletion

Environment Variable	Description	Default Value
`KAFKA_TOPIC_FILE_DELETE_IN`	Topic for incoming file deletion requests	`ai.flowx.dev.plugin.document.trigger.delete.file.v1`
`KAFKA_TOPIC_FILE_DELETE_OUT`	Topic for file deletion results	`ai.flowx.dev.engine.receive.plugin.document.delete.file.results.v1`

File update

Environment Variable	Description	Default Value
`KAFKA_TOPIC_FILE_UPDATE_IN`	Topic for incoming file update requests	`ai.flowx.dev.plugin.document.trigger.update.file.v1`
`KAFKA_TOPIC_FILE_UPDATE_OUT`	Topic for file update results	`ai.flowx.dev.engine.receive.plugin.document.update.file.results.v1`

File conversion

Environment Variable	Description	Default Value
`KAFKA_TOPIC_FILE_CONVERT_OUT`	Topic for file conversion results	`ai.flowx.dev.engine.receive.plugin.document.convert.file.results.v1`

File persistence

Environment Variable	Description	Default Value
`KAFKA_TOPIC_FILE_PERSIST_IN`	Topic for incoming file persistence requests	`ai.flowx.dev.plugin.document.trigger.persist.file.v1`
`KAFKA_TOPIC_FILE_PERSIST_IN`	Topic for incoming file persistence requests	`ai.flowx.dev.plugin.document.trigger.persist.file.v1`
`KAFKA_TOPIC_FILE_PERSIST_OUT`	Topic for file persistence results	`ai.flowx.dev.engine.receive.plugin.document.persist.file.results.v1`

File combination

Environment Variable	Description	Default Value
`KAFKA_TOPIC_FILE_COMBINE_IN`	Topic for incoming file combination requests	`ai.flowx.dev.plugin.document.trigger.combine.file.v1`
`KAFKA_TOPIC_FILE_COMBINE_OUT`	Topic for file combination results	`ai.flowx.dev.engine.receive.plugin.document.combine.file.results.v1`

Audit

Environment Variable	Description	Default Value
`KAFKA_TOPIC_AUDIT_OUT`	Topic for sending audit logs	`ai.flowx.dev.core.trigger.save.audit.v1`

General storage configuration

Environment Variable	Description	Default Value
`APPLICATION_DEFAULTLOCALE`	Default locale for the application	`en`
`APPLICATION_SUPPORTEDLOCALES`	Comma-separated list of supported locales	`en, ro`
`APPLICATION_FILESTORAGE_TYPE`	Storage type to use (`s3` or `fileSystem`)	`s3`
`APPLICATION_FILESTORAGE_DISKDIRECTORY`	Directory for file storage when using filesystem	`MS_SVC_DOCUMENT`
`APPLICATION_FILESTORAGE_PARTITIONSTRATEGY`	Set the partition strategy for file storage. Use `NONE` to save documents in `minio/amazon-s3` as before, with a bucket for each process instance. Use `PROCESS_DATE` to save documents in a single bucket with a subfolder structure, for example: `bucket/2022/2022-07-04/process-id-xxxx/customer-id/file.pdf`	`NONE`
`APPLICATION_FILESTORAGE_DELETIONSTRATEGY`	Strategy for deleting files (`delete`, `disabled`, or `deleteBypassingGovernanceRetention`)	`delete`

APPLICATION_FILESTORAGE_DELETIONSTRATEGY:

disabled: This will disable entirely the deletion of temporary files from the temporary bucket, and the responsibility to delete and clean up the bucket will move in the ownership of the admins of the implementing project.
deleteBypassingGovernanceRetention: This will still delete the temporary files and further more will add in the delete request the header: x-amz-bypass-governance-retention:true , to enable deletion of governed files, in case the s3 configured user for document-plugin, will have the s3:BypassGovernanceRetention permission.

S3-Compatible storage configuration

Environment Variable	Description	Default Value
`APPLICATION_FILESTORAGE_S3_ENABLED`	Enable S3-compatible storage	`true`
`APPLICATION_FILESTORAGE_S3_SERVERURL`	URL of MinIO or S3-compatible server	`http://minio-service:9000`
`APPLICATION_FILESTORAGE_S3_ACCESSKEY`	Access key for MinIO/S3	`minio`
`APPLICATION_FILESTORAGE_S3_SECRETKEY`	Secret key for MinIO/S3	`minio123`
`APPLICATION_FILESTORAGE_S3_BUCKETPREFIX`	Prefix for bucket names	`qdevlocal-preview-paperflow`
`APPLICATION_FILESTORAGE_S3_TEMPBUCKET`	Name of temporary bucket for initial file uploads	`temp-bucket`
`APPLICATION_FILESTORAGE_S3_ENCRYPTIONENABLED`	Enable server-side encryption	`false`

Make sure to follow the recommended bucket naming rules when choosing the bucket prefix name.

Logging configuration

Environment Variable	Description	Default Value
`LOGGING_LEVEL_APP`	Log level for application logs	`DEBUG`
`LOGGING_LEVEL_MONGO_DRIVER`	Log level for MongoDB driver	`INFO`
`LOGGING_LEVEL_LIQUIBASE`	Log level for Liquibase database migrations	`INFO`
`LOGGING_LEVEL_REDIS`	Log level for Redis/Lettuce client	`OFF`
`LOGGING_LEVEL_ROOT`	Root logging level for the application	-

Microservices

Plugins

Observability

Microservices access rights

Plugins access rights

Documents plugin setup

Dependencies

Configuration

Redis server

Authorization configuration

Document processing configuration

Database configuration

SQL database (PostgreSQL)

Primary MongoDB configuration

Runtime MongoDB configuration

Redis configuration

Multipart upload configuration

Basic Kafka configuration

Topic naming configuration

Thread configuration

Document topics

Generate HTML documents

Document persistence

Document splitting

Document URL retrieval

OCR processing

File operation topics

File deletion

File update

File conversion

File persistence

File combination

Audit

General storage configuration

S3-Compatible storage configuration

Logging configuration

​Dependencies

​Configuration

​Redis server

​Authorization configuration

​Document processing configuration

​Database configuration

​SQL database (PostgreSQL)

​Primary MongoDB configuration

​Runtime MongoDB configuration

​Redis configuration

​Multipart upload configuration

​Basic Kafka configuration

​Topic naming configuration

​Thread configuration

​Document topics

​Generate HTML documents

​Document persistence

​Document splitting

​Document URL retrieval

​OCR processing

​File operation topics

​File deletion

​File update

​File conversion

​File persistence

​File combination

​Audit

​General storage configuration

​S3-Compatible storage configuration

​Logging configuration

Dependencies

Configuration

Redis server

Authorization configuration

Document processing configuration

Database configuration

SQL database (PostgreSQL)

Primary MongoDB configuration

Runtime MongoDB configuration

Redis configuration

Multipart upload configuration

Basic Kafka configuration

Topic naming configuration

Thread configuration

Document topics

Generate HTML documents

Document persistence

Document splitting

Document URL retrieval

OCR processing

File operation topics

File deletion

File update

File conversion

File persistence

File combination

Audit

General storage configuration

S3-Compatible storage configuration

Logging configuration