Below is a table containing all of the configurable options within the Structured File Lens. To see how to set config variables, see the Quick Start Guide or the full User Guide. With version 2.0 and beyond, fewer configuration is required to have your Lens operational, just setting the Lens Directory is enough. In addition, no configuration is required at startup as config can be updated on a running Lens.
...
Environment Variable | Entry | Default Value | Description | |
|---|---|---|---|---|
KAFKAPROPERTY_GRAPH_BROKERSMODE | kafkaBrokerspropertyGraphMode | localhost:9092 | The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure | If you are using a Property Graph as your target graph type, set this configuration to true, otherwise leave as false. When set to true, the Lens will output Nodes and Edges CSV files instead of RDF files. |
PG_GRAPH | pgGraph | default | Set this property to the Property Graph provider you wish to use: |
Kafka Configuration
Environment Variable | Entry | Default Value | Description |
|---|---|---|---|
KAFKA_BROKERS | kafkaBrokers | localhost:9092 | The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure |
KAFKA_TOPIC_NAME_SOURCE | topicNameSource | source_urls | The topic used for the Consumer to read messages from containing input file URLs in order to ingest data. |
KAFKA_TOPIC_NAME_DLQ | topicNameDLQ | dead_letter_queue | The topic used to push messages containing reasons for failure within the Lens. These messages are represented as a JSON. |
KAFKA_TOPIC_NAME_SUCCESS | topicNameSuccess | success_queue | The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files. |
KAFKA_GROUP_ID_CONFIG | groupIdConfig | consumerGroup1 | The identifier of the group this consumer belongs to. |
KAFKA_AUTO_OFFSET_RESET_CONFIG | autoOffsetResetConfig | earliest | What to do when there is no initial offset in Kafka or if an offset is out of range.
|
KAFKA_MAX_POLL_RECORDS | maxPollRecords | 100 | The maximum number of records returned in a single call to poll. |
KAFKA_TIMEOUT | timeout | 1000000 | Kafka consumer polling time out. |
...
Environment Variable | Default Value | Description |
|---|---|---|
LOGGING_LEVEL | WARN | Global log level |
LOGGING_APPENDERS_CONSOLE_TIMEZONE | UTC | Timezone for console logging |
LOGGING_APPENDERS_TXT_FILE_THRESHOLD | ALL | Threashold for text logging |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages |
Current Log Filename (not overridable) | /var/log/datalens/text/current/application_${applicationName}_${timeStamp}.txt.log | Pattern for log file name |
LOGGING_APPENDERS_TXT_FILE_ARCHIVE | true | Archive log text files |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/text/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.txt.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly |
LOGGING_APPENDERS_TXT_FILE_ARCHIVED_TXT_FILE_COUNT | 7 | Max number of archived text files |
LOGGING_APPENDERS_TXT_FILE_TIMEZONE | UTC | Timezone for text file logging |
LOGGING_APPENDERS_JSON_FILE_THRESHOLD | ALL | Threashold for text logging |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages |
Current Log Filename (not overridable) | /var/log/datalens/json/current/application_${applicationName}_${timeStamp}.json.log | Pattern for log file name |
LOGGING_APPENDERS_JSON_FILE_ARCHIVE | true | Archive log text files |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/json/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.json.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly |
LOGGING_APPENDERS_JSON_FILE_ARCHIVED_FILE_COUNT | 7 | Max number of archived text files |
LOGGING_APPENDERS_JSON_FILE_TIMEZONE | UTC | Timezone for text file logging |
LOGGING_APPENDERS_JSON_FILE_LAYOUT_TYPE | json | The layout type for the json logger |
...