Documents plugin setup
The Documents plugin provides functionality for generating, persisting, combining, and manipulating documents within the FlowX.AI system.
The plugin is available as a docker image.
Dependencies
Before setting up the plugin, ensure that you have the following dependencies installed and configured:
- PostgreSQL Database: You will need a PostgreSQL database to store data related to document templates and documents.
- MongoDB Database: MongoDB is required for the HTML templates feature of the plugin.
- Kafka: Establish a connection to the Kafka instance used by the FLOWX.AI engine.
- Redis: Set up a Redis instance for caching purposes.
- S3-Compatible File Storage Solution: Deploy an S3-compatible file storage solution, such as Min.io, to store document files.
Configuration
The plugin comes with pre-filled configuration properties, but you need to set up a few custom environment variables to tailor it to your specific setup. Here are the key configuration steps:
Redis server
The plugin can utilize the Redis component already deployed for the FLOWX.AI engine. Make sure it is configured properly.
Authorization configuration
To connect to the identity management platform, set the following environment variables:
| Environment Variable | Description | Default Value |
|---|---|---|
SECURITY_OAUTH2_BASESERVERURL | Base URL of the OAuth2/OIDC server | https://keycloak.example.com/auth |
SECURITY_OAUTH2_CLIENT_CLIENTID | OAuth2 client ID for the Documents Plugin | document-plugin |
SECURITY_OAUTH2_CLIENT_CLIENTSECRET | OAuth2 client secret | secret |
SECURITY_OAUTH2_REALM | OAuth2 realm name | flowx |
Document processing configuration
| Environment Variable | Description | Default Value |
|---|---|---|
FLOWX_DEFAULTCLIENTTYPE | Default client type | PF |
FLOWX_HTML_TEMPLATES_ENABLED | Enable HTML templates feature | false |
FLOWX_HTML_TEMPLATES_PDFFONTPATHS | Paths to fonts for HTML templates | /statics/fonts/Calibri.ttf, /statics/fonts/DejaVuSans.ttf |
FLOWX_CONVERT_DPI | DPI setting for PDF to image conversion | 150 |
Set the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config to select the font used for generating documents based on PDF templates.
If you want to use specific fonts in your PDF templates, override the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config. By default, Calibri and DejaVuSans are available fonts.
FLOWX_CONVERT_DPI: Sets the DPI (dots per inch) for PDF to JPEG conversion. Higher values result in higher resolution images. (Default value: 150).
After making these configurations, the fonts will be available for use within PDF templates.
Database configuration
SQL database (PostgreSQL)
The Documents Plugin uses a PostgreSQL database for relational data storage.
Primary MongoDB configuration
| Environment Variable | Description | Example Value |
|---|---|---|
SPRING_DATA_MONGODB_URI | MongoDB connection URI | mongodb://flowx:password@jx-document-mongodb:27017/document |
MONGODB_USERNAME | MongoDB username | flowx |
MONGODB_PASSWORD | MongoDB password | password |
SPRING_DATA_MONGODB_UUID_REPRESENTATION | UUID representation format | standard |
SPRING_DATA_MONGODB_STORAGE | MongoDB storage type | mongodb |
Runtime MongoDB configuration
| Environment Variable | Description | Example Value |
|---|---|---|
SPRING_DATA_MONGODB_RUNTIME_ENABLED | Enable runtime MongoDB connection | true |
SPRING_DATA_MONGODB_RUNTIME_URI | Runtime MongoDB connection URI | mongodb://${MONGODB_USERNAME:flowx}:${MONGODB_PASSWORD:password}@jx-document-mongodb:27017/document |
MONGODB_USERNAME | Runtime MongoDB username | paperflow |
MONGODB_PASSWORD | Runtime MongoDB password | password |
SPRING_DATA_MONGODB_RUNTIME_AUTOINDEXCREATION | Enable automatic index creation | false |
SPRING_DATA_MONGODB_RUNTIME_UUIDREPRESENTATION | UUID representation format for runtime connection | standard |
Redis configuration
Set the following values with the corresponding Redis-related values:
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_DATA_REDIS_HOST | Hostname of the Redis server | localhost |
SPRING_DATA_REDIS_PORT | Port number of the Redis server | 6379 |
SPRING_DATA_REDIS_PASSWORD | Password for Redis authentication | defaultpassword |
REDIS_TTL | Time-to-live (TTL) for Redis cache | 5000000 (miliseconds) |
Multipart upload configuration
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_SERVLET_CONTEXTPATH | Servlet context path | / |
SPRING_SERVLET_MULTIPART_MAXFILESIZE | Maximum file size for uploads | 50MB |
SPRING_SERVLET_MULTIPART_MAXREQUESTSIZE | Maximum request size for multipart uploads | 50MB |
Basic Kafka configuration
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_KAFKA_BOOTSTRAPSERVERS | Address of Kafka server(s) | localhost:9092 |
SPRING_KAFKA_SECURITY_PROTOCOL | Security protocol for Kafka | PLAINTEXT |
SPRING_KAFKA_CONSUMER_GROUPID | Consumer group ID for the service | kafka-svc-document-consumer-local-test2 |
KAFKA_MESSAGE_MAX_BYTES | Maximum message size in bytes | 52428800 (50MB) |
KAFKA_AUTHEXCEPTIONRETRYINTERVAL | Retry interval for auth exceptions | 10 (seconds) |
Topic naming configuration
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_NAMING_SEPARATOR | Primary separator for topic names | . |
KAFKA_TOPIC_NAMING_SEPARATOR2 | Secondary separator for topic names | - |
KAFKA_TOPIC_NAMING_PACKAGE | Package prefix for topic names | ai.flowx. |
KAFKA_TOPIC_NAMING_ENVIRONMENT | Environment segment for topic names | dev. |
KAFKA_TOPIC_NAMING_VERSION | Version suffix for topic names | .v1 |
Thread configuration
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGENERATEHTMLIN | Threads for HTML document generation | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTPERSISTIN | Threads for document persistence | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTSPLITTIN | Threads for document splitting | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGETURLSIN | Threads for document URL retrieval | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_OCRIN | Threads for OCR processing | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEDELETEIN | Threads for file deletion | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEUPDATEIN | Threads for file updates | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECONVERTIN | Threads for file conversion | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEPERSISTIN | Threads for file persistence | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECOMBINEIN | Threads for file combining | 5 |
Document topics
Generate HTML documents
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_IN | Topic for incoming HTML generation requests | ai.flowx.dev.plugin.document.trigger.generate.html.v1 |
KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_OUT | Topic for HTML generation results | ai.flowx.dev.engine.receive.plugin.document.generate.html.results.v1 |
Document persistence
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_PERSIST_IN | Topic for incoming document persistence requests | ai.flowx.dev.plugin.document.trigger.persist.document.v1 |
KAFKA_TOPIC_DOCUMENT_PERSIST_OUT | Topic for document persistence results | ai.flowx.dev.engine.receive.plugin.document.persist.document.results.v1 |
Document splitting
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_SPLIT_IN | Topic for incoming document splitting requests | ai.flowx.dev.plugin.document.trigger.split.document.v1 |
KAFKA_TOPIC_DOCUMENT_SPLIT_OUT | Topic for document splitting results | ai.flowx.dev.engine.receive.plugin.document.split.document.results.v1 |
Document URL retrieval
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_GET_URLS_IN | Topic for incoming URL retrieval requests | ai.flowx.dev.plugin.document.retrieve.urls.v1 |
KAFKA_TOPIC_DOCUMENT_GET_URLS_OUT | Topic for URL retrieval results | ai.flowx.dev.engine.receive.plugin.document.urls.results.v1 |
OCR processing
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_OCR_IN | Topic for incoming OCR requests | ai.flowx.dev.plugin.document.store.ocr.v1 |
KAFKA_TOPIC_OCR_OUT | Topic for OCR processing results | ai.flowx.dev.engine.receive.plugin.document.ocr.results.v1 |
File operation topics
File deletion
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_DELETE_IN | Topic for incoming file deletion requests | ai.flowx.dev.plugin.document.trigger.delete.file.v1 |
KAFKA_TOPIC_FILE_DELETE_OUT | Topic for file deletion results | ai.flowx.dev.engine.receive.plugin.document.delete.file.results.v1 |
File update
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_UPDATE_IN | Topic for incoming file update requests | ai.flowx.dev.plugin.document.trigger.update.file.v1 |
KAFKA_TOPIC_FILE_UPDATE_OUT | Topic for file update results | ai.flowx.dev.engine.receive.plugin.document.update.file.results.v1 |
File conversion
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_CONVERT_OUT | Topic for file conversion results | ai.flowx.dev.engine.receive.plugin.document.convert.file.results.v1 |
File persistence
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_PERSIST_IN | Topic for incoming file persistence requests | ai.flowx.dev.plugin.document.trigger.persist.file.v1 |
KAFKA_TOPIC_FILE_PERSIST_IN | Topic for incoming file persistence requests | ai.flowx.dev.plugin.document.trigger.persist.file.v1 |
KAFKA_TOPIC_FILE_PERSIST_OUT | Topic for file persistence results | ai.flowx.dev.engine.receive.plugin.document.persist.file.results.v1 |
File combination
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_COMBINE_IN | Topic for incoming file combination requests | ai.flowx.dev.plugin.document.trigger.combine.file.v1 |
KAFKA_TOPIC_FILE_COMBINE_OUT | Topic for file combination results | ai.flowx.dev.engine.receive.plugin.document.combine.file.results.v1 |
Audit
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_AUDIT_OUT | Topic for sending audit logs | ai.flowx.dev.core.trigger.save.audit.v1 |
General storage configuration
| Environment Variable | Description | Default Value |
|---|---|---|
APPLICATION_DEFAULTLOCALE | Default locale for the application | en |
APPLICATION_SUPPORTEDLOCALES | Comma-separated list of supported locales | en, ro |
APPLICATION_FILESTORAGE_TYPE | Storage type to use (s3 or fileSystem) | s3 |
APPLICATION_FILESTORAGE_DISKDIRECTORY | Directory for file storage when using filesystem | MS_SVC_DOCUMENT |
APPLICATION_FILESTORAGE_PARTITIONSTRATEGY | Set the partition strategy for file storage. Use NONE to save documents in minio/amazon-s3 as before, with a bucket for each process instance. Use PROCESS_DATE to save documents in a single bucket with a subfolder structure, for example: bucket/2022/2022-07-04/process-id-xxxx/customer-id/file.pdf | NONE |
APPLICATION_FILESTORAGE_DELETIONSTRATEGY | Strategy for deleting files (delete, disabled, or deleteBypassingGovernanceRetention) | delete |
APPLICATION_FILESTORAGE_DELETIONSTRATEGY:
disabled: This will disable entirely the deletion of temporary files from the temporary bucket, and the responsibility to delete and clean up the bucket will move in the ownership of the admins of the implementing project.deleteBypassingGovernanceRetention: This will still delete the temporary files and further more will add in the delete request the header:x-amz-bypass-governance-retention:true, to enable deletion of governed files, in case the s3 configured user for document-plugin, will have thes3:BypassGovernanceRetentionpermission.
S3-Compatible storage configuration
| Environment Variable | Description | Default Value |
|---|---|---|
APPLICATION_FILESTORAGE_S3_ENABLED | Enable S3-compatible storage | true |
APPLICATION_FILESTORAGE_S3_SERVERURL | URL of MinIO or S3-compatible server | http://minio-service:9000 |
APPLICATION_FILESTORAGE_S3_ACCESSKEY | Access key for MinIO/S3 | minio |
APPLICATION_FILESTORAGE_S3_SECRETKEY | Secret key for MinIO/S3 | minio123 |
APPLICATION_FILESTORAGE_S3_BUCKETPREFIX | Prefix for bucket names | qdevlocal-preview-paperflow |
APPLICATION_FILESTORAGE_S3_TEMPBUCKET | Name of temporary bucket for initial file uploads | temp-bucket |
APPLICATION_FILESTORAGE_S3_ENCRYPTIONENABLED | Enable server-side encryption | false |
Make sure to follow the recommended bucket naming rules when choosing the bucket prefix name.
Logging configuration
| Environment Variable | Description | Default Value |
|---|---|---|
LOGGING_LEVEL_APP | Log level for application logs | DEBUG |
LOGGING_LEVEL_MONGO_DRIVER | Log level for MongoDB driver | INFO |
LOGGING_LEVEL_LIQUIBASE | Log level for Liquibase database migrations | INFO |
LOGGING_LEVEL_REDIS | Log level for Redis/Lettuce client | OFF |
LOGGING_LEVEL_ROOT | Root logging level for the application | - |
Documents plugin setup
The Documents plugin provides functionality for generating, persisting, combining, and manipulating documents within the FlowX.AI system.
The plugin is available as a docker image.
Dependencies
Before setting up the plugin, ensure that you have the following dependencies installed and configured:
- PostgreSQL Database: You will need a PostgreSQL database to store data related to document templates and documents.
- MongoDB Database: MongoDB is required for the HTML templates feature of the plugin.
- Kafka: Establish a connection to the Kafka instance used by the FLOWX.AI engine.
- Redis: Set up a Redis instance for caching purposes.
- S3-Compatible File Storage Solution: Deploy an S3-compatible file storage solution, such as Min.io, to store document files.
Configuration
The plugin comes with pre-filled configuration properties, but you need to set up a few custom environment variables to tailor it to your specific setup. Here are the key configuration steps:
Redis server
The plugin can utilize the Redis component already deployed for the FLOWX.AI engine. Make sure it is configured properly.
Authorization configuration
To connect to the identity management platform, set the following environment variables:
| Environment Variable | Description | Default Value |
|---|---|---|
SECURITY_OAUTH2_BASESERVERURL | Base URL of the OAuth2/OIDC server | https://keycloak.example.com/auth |
SECURITY_OAUTH2_CLIENT_CLIENTID | OAuth2 client ID for the Documents Plugin | document-plugin |
SECURITY_OAUTH2_CLIENT_CLIENTSECRET | OAuth2 client secret | secret |
SECURITY_OAUTH2_REALM | OAuth2 realm name | flowx |
Document processing configuration
| Environment Variable | Description | Default Value |
|---|---|---|
FLOWX_DEFAULTCLIENTTYPE | Default client type | PF |
FLOWX_HTML_TEMPLATES_ENABLED | Enable HTML templates feature | false |
FLOWX_HTML_TEMPLATES_PDFFONTPATHS | Paths to fonts for HTML templates | /statics/fonts/Calibri.ttf, /statics/fonts/DejaVuSans.ttf |
FLOWX_CONVERT_DPI | DPI setting for PDF to image conversion | 150 |
Set the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config to select the font used for generating documents based on PDF templates.
If you want to use specific fonts in your PDF templates, override the FLOWX_HTML_TEMPLATES_PDFFONTPATHS config. By default, Calibri and DejaVuSans are available fonts.
FLOWX_CONVERT_DPI: Sets the DPI (dots per inch) for PDF to JPEG conversion. Higher values result in higher resolution images. (Default value: 150).
After making these configurations, the fonts will be available for use within PDF templates.
Database configuration
SQL database (PostgreSQL)
The Documents Plugin uses a PostgreSQL database for relational data storage.
Primary MongoDB configuration
| Environment Variable | Description | Example Value |
|---|---|---|
SPRING_DATA_MONGODB_URI | MongoDB connection URI | mongodb://flowx:password@jx-document-mongodb:27017/document |
MONGODB_USERNAME | MongoDB username | flowx |
MONGODB_PASSWORD | MongoDB password | password |
SPRING_DATA_MONGODB_UUID_REPRESENTATION | UUID representation format | standard |
SPRING_DATA_MONGODB_STORAGE | MongoDB storage type | mongodb |
Runtime MongoDB configuration
| Environment Variable | Description | Example Value |
|---|---|---|
SPRING_DATA_MONGODB_RUNTIME_ENABLED | Enable runtime MongoDB connection | true |
SPRING_DATA_MONGODB_RUNTIME_URI | Runtime MongoDB connection URI | mongodb://${MONGODB_USERNAME:flowx}:${MONGODB_PASSWORD:password}@jx-document-mongodb:27017/document |
MONGODB_USERNAME | Runtime MongoDB username | paperflow |
MONGODB_PASSWORD | Runtime MongoDB password | password |
SPRING_DATA_MONGODB_RUNTIME_AUTOINDEXCREATION | Enable automatic index creation | false |
SPRING_DATA_MONGODB_RUNTIME_UUIDREPRESENTATION | UUID representation format for runtime connection | standard |
Redis configuration
Set the following values with the corresponding Redis-related values:
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_DATA_REDIS_HOST | Hostname of the Redis server | localhost |
SPRING_DATA_REDIS_PORT | Port number of the Redis server | 6379 |
SPRING_DATA_REDIS_PASSWORD | Password for Redis authentication | defaultpassword |
REDIS_TTL | Time-to-live (TTL) for Redis cache | 5000000 (miliseconds) |
Multipart upload configuration
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_SERVLET_CONTEXTPATH | Servlet context path | / |
SPRING_SERVLET_MULTIPART_MAXFILESIZE | Maximum file size for uploads | 50MB |
SPRING_SERVLET_MULTIPART_MAXREQUESTSIZE | Maximum request size for multipart uploads | 50MB |
Basic Kafka configuration
| Environment Variable | Description | Default Value |
|---|---|---|
SPRING_KAFKA_BOOTSTRAPSERVERS | Address of Kafka server(s) | localhost:9092 |
SPRING_KAFKA_SECURITY_PROTOCOL | Security protocol for Kafka | PLAINTEXT |
SPRING_KAFKA_CONSUMER_GROUPID | Consumer group ID for the service | kafka-svc-document-consumer-local-test2 |
KAFKA_MESSAGE_MAX_BYTES | Maximum message size in bytes | 52428800 (50MB) |
KAFKA_AUTHEXCEPTIONRETRYINTERVAL | Retry interval for auth exceptions | 10 (seconds) |
Topic naming configuration
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_NAMING_SEPARATOR | Primary separator for topic names | . |
KAFKA_TOPIC_NAMING_SEPARATOR2 | Secondary separator for topic names | - |
KAFKA_TOPIC_NAMING_PACKAGE | Package prefix for topic names | ai.flowx. |
KAFKA_TOPIC_NAMING_ENVIRONMENT | Environment segment for topic names | dev. |
KAFKA_TOPIC_NAMING_VERSION | Version suffix for topic names | .v1 |
Thread configuration
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGENERATEHTMLIN | Threads for HTML document generation | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTPERSISTIN | Threads for document persistence | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTSPLITTIN | Threads for document splitting | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_DOCUMENTGETURLSIN | Threads for document URL retrieval | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_OCRIN | Threads for OCR processing | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEDELETEIN | Threads for file deletion | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEUPDATEIN | Threads for file updates | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECONVERTIN | Threads for file conversion | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILEPERSISTIN | Threads for file persistence | 5 |
KAFKA_CONSUMER_THREADPOOLS_THREADPOOLGENERIC_THREADCOUNTPERCONTAINER_FILECOMBINEIN | Threads for file combining | 5 |
Document topics
Generate HTML documents
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_IN | Topic for incoming HTML generation requests | ai.flowx.dev.plugin.document.trigger.generate.html.v1 |
KAFKA_TOPIC_DOCUMENT_GENERATE_HTML_OUT | Topic for HTML generation results | ai.flowx.dev.engine.receive.plugin.document.generate.html.results.v1 |
Document persistence
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_PERSIST_IN | Topic for incoming document persistence requests | ai.flowx.dev.plugin.document.trigger.persist.document.v1 |
KAFKA_TOPIC_DOCUMENT_PERSIST_OUT | Topic for document persistence results | ai.flowx.dev.engine.receive.plugin.document.persist.document.results.v1 |
Document splitting
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_SPLIT_IN | Topic for incoming document splitting requests | ai.flowx.dev.plugin.document.trigger.split.document.v1 |
KAFKA_TOPIC_DOCUMENT_SPLIT_OUT | Topic for document splitting results | ai.flowx.dev.engine.receive.plugin.document.split.document.results.v1 |
Document URL retrieval
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_DOCUMENT_GET_URLS_IN | Topic for incoming URL retrieval requests | ai.flowx.dev.plugin.document.retrieve.urls.v1 |
KAFKA_TOPIC_DOCUMENT_GET_URLS_OUT | Topic for URL retrieval results | ai.flowx.dev.engine.receive.plugin.document.urls.results.v1 |
OCR processing
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_OCR_IN | Topic for incoming OCR requests | ai.flowx.dev.plugin.document.store.ocr.v1 |
KAFKA_TOPIC_OCR_OUT | Topic for OCR processing results | ai.flowx.dev.engine.receive.plugin.document.ocr.results.v1 |
File operation topics
File deletion
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_DELETE_IN | Topic for incoming file deletion requests | ai.flowx.dev.plugin.document.trigger.delete.file.v1 |
KAFKA_TOPIC_FILE_DELETE_OUT | Topic for file deletion results | ai.flowx.dev.engine.receive.plugin.document.delete.file.results.v1 |
File update
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_UPDATE_IN | Topic for incoming file update requests | ai.flowx.dev.plugin.document.trigger.update.file.v1 |
KAFKA_TOPIC_FILE_UPDATE_OUT | Topic for file update results | ai.flowx.dev.engine.receive.plugin.document.update.file.results.v1 |
File conversion
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_CONVERT_OUT | Topic for file conversion results | ai.flowx.dev.engine.receive.plugin.document.convert.file.results.v1 |
File persistence
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_PERSIST_IN | Topic for incoming file persistence requests | ai.flowx.dev.plugin.document.trigger.persist.file.v1 |
KAFKA_TOPIC_FILE_PERSIST_IN | Topic for incoming file persistence requests | ai.flowx.dev.plugin.document.trigger.persist.file.v1 |
KAFKA_TOPIC_FILE_PERSIST_OUT | Topic for file persistence results | ai.flowx.dev.engine.receive.plugin.document.persist.file.results.v1 |
File combination
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_FILE_COMBINE_IN | Topic for incoming file combination requests | ai.flowx.dev.plugin.document.trigger.combine.file.v1 |
KAFKA_TOPIC_FILE_COMBINE_OUT | Topic for file combination results | ai.flowx.dev.engine.receive.plugin.document.combine.file.results.v1 |
Audit
| Environment Variable | Description | Default Value |
|---|---|---|
KAFKA_TOPIC_AUDIT_OUT | Topic for sending audit logs | ai.flowx.dev.core.trigger.save.audit.v1 |
General storage configuration
| Environment Variable | Description | Default Value |
|---|---|---|
APPLICATION_DEFAULTLOCALE | Default locale for the application | en |
APPLICATION_SUPPORTEDLOCALES | Comma-separated list of supported locales | en, ro |
APPLICATION_FILESTORAGE_TYPE | Storage type to use (s3 or fileSystem) | s3 |
APPLICATION_FILESTORAGE_DISKDIRECTORY | Directory for file storage when using filesystem | MS_SVC_DOCUMENT |
APPLICATION_FILESTORAGE_PARTITIONSTRATEGY | Set the partition strategy for file storage. Use NONE to save documents in minio/amazon-s3 as before, with a bucket for each process instance. Use PROCESS_DATE to save documents in a single bucket with a subfolder structure, for example: bucket/2022/2022-07-04/process-id-xxxx/customer-id/file.pdf | NONE |
APPLICATION_FILESTORAGE_DELETIONSTRATEGY | Strategy for deleting files (delete, disabled, or deleteBypassingGovernanceRetention) | delete |
APPLICATION_FILESTORAGE_DELETIONSTRATEGY:
disabled: This will disable entirely the deletion of temporary files from the temporary bucket, and the responsibility to delete and clean up the bucket will move in the ownership of the admins of the implementing project.deleteBypassingGovernanceRetention: This will still delete the temporary files and further more will add in the delete request the header:x-amz-bypass-governance-retention:true, to enable deletion of governed files, in case the s3 configured user for document-plugin, will have thes3:BypassGovernanceRetentionpermission.
S3-Compatible storage configuration
| Environment Variable | Description | Default Value |
|---|---|---|
APPLICATION_FILESTORAGE_S3_ENABLED | Enable S3-compatible storage | true |
APPLICATION_FILESTORAGE_S3_SERVERURL | URL of MinIO or S3-compatible server | http://minio-service:9000 |
APPLICATION_FILESTORAGE_S3_ACCESSKEY | Access key for MinIO/S3 | minio |
APPLICATION_FILESTORAGE_S3_SECRETKEY | Secret key for MinIO/S3 | minio123 |
APPLICATION_FILESTORAGE_S3_BUCKETPREFIX | Prefix for bucket names | qdevlocal-preview-paperflow |
APPLICATION_FILESTORAGE_S3_TEMPBUCKET | Name of temporary bucket for initial file uploads | temp-bucket |
APPLICATION_FILESTORAGE_S3_ENCRYPTIONENABLED | Enable server-side encryption | false |
Make sure to follow the recommended bucket naming rules when choosing the bucket prefix name.
Logging configuration
| Environment Variable | Description | Default Value |
|---|---|---|
LOGGING_LEVEL_APP | Log level for application logs | DEBUG |
LOGGING_LEVEL_MONGO_DRIVER | Log level for MongoDB driver | INFO |
LOGGING_LEVEL_LIQUIBASE | Log level for Liquibase database migrations | INFO |
LOGGING_LEVEL_REDIS | Log level for Redis/Lettuce client | OFF |
LOGGING_LEVEL_ROOT | Root logging level for the application | - |