Below is a table containing all of the configurable options within the Lens Writer. To see how to set config variables, see the Quick Start Guide or the full User Guide. Mandatory variables are highlighted in red.
Table of Contents |
---|
Lens Writer Configuration
Environment Variable | Default Value | Description | Version |
---|---|---|---|
FRIENDLY_NAME | Lens-Writer | The name you wish to set your lens Writer up with. | v1.3+ |
LICENSE |
| The License key provided required for running the lensWriter. | v1.3+ |
TRIPLESTORE_ENDPOINT |
| The endpoint for your Triple Store you wish to upload your RDF to. | v1.3+ |
TRIPLESTORE_TYPE | sparql | The Triple Store type, some graphs will support the default | v1.3+ |
TRIPLESTORE_REASONING | false | Whether you want reasoning enabled or disabled. | v1.3+ |
TRIPLESTORE_USERNAME | The username of your Triple Store. Leave blank if your Triple Store does not require any authentication. | v1.3+ | |
TRIPLESTORE_PASSWORD | The password of your Triple Store. Leave blank if your Triple Store does not require any authentication. | v1.3+ | |
S3_REGION | us-east-1 | The region in AWS where your files and services reside. Note: all services must be in the same region. | v1.3+ |
AWS_ACCESS_KEY |
| Your access key for AWS. | v1.3+ |
AWS_SECRET_KEY |
| Your secret key for AWS. | v1.3+ |
LENS_RUN_STANDALONE | false | The Lens Writer is designed to run as part of a larger end to end system with the Lens providing the Writer with RDF files to write to a Triple Store. As part of this process, Kafka is used to communicate between services. This is enabled by default, however if you want to run the Lens Writer as standalone without communicating to other services, set this property to true. | v1.3+ |
...
Environment Variable | Default Value | Description | Version | ||||
---|---|---|---|---|---|---|---|
KAFKA_BROKERS | localhost:9092 | The The Kafka Broker is what tells the Lens Writer where to look for your Kafka Cluster. Set with the following structure | KAFKA_MESSAGE_COUNT | 1000 |
| v1.3+ | |
KAFKA_TOPIC_NAME_SOURCE | source_urls | The topic used for the Consumer to read messages from containing input file URLs in order to ingest data. | v1.3+ | ||||
KAFKA_TOPIC_NAME_DLQ | dead_letter_queue | The topic used for to push messages containing reasons for failure within the LensWriter. These messages are represented as a JSON. | v1.3+ | ||||
KAFKA_TOPIC_NAME_SUCCESS | success_queue | The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files. | v1.3+ | ||||
KAFKA_GROUP_ID_CONFIG | consumerGroup1 | The The identifier of the group this consumer belongs to. | v1.3+ | ||||
KAFKA_MAX_NO_MESSAGE_FOUND_COUNT | 100 |
| v1.3+ | KAFKA_AUTO_OFFSET_RESET_CONFIG | earliest | What to do when there is no initial offset in Kafka or if an offset is out of range.
| v1.3+ |
KAFKA_MAX_POLL_RECORDS | 100 | The The maximum number of records returned in a single call to poll. | v1.3+ | ||||
KAFKA_TIMEOUT | 1000000 | Kafka consumer polling time out. | v1.3+ |
...
Environment Variable | Default Value | Description | Version |
---|---|---|---|
RECORD_PROVO | false | Currently, the Lens Writer does not generate its own provenance meta-data and so this option is set to false | v1.3+ |
Logging Configuration
Environment Variable | Default Value | Description | Version |
---|---|---|---|
LOGGING_LEVEL | WARN | Global log level | v1.3+ |
LOGGING_LOGGERS_DATALENS | DEBUG | Log level for Data Lens loggers | v1.3+ |
LOGGING_LOGGERS_DROPWIZARD | INFO | Log level for Dropwizard loggers | v1.3+ |
LOGGING_APPENDERS_CONSOLE_TIMEZONE | UTC | Timezone for console logging | v1.3+ |
LOGGING_APPENDERS_TXT_FILE_THRESHOLD | ALL | Threashold for text logging | v1.3+ |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages | v1.3+ |
Current Log Filename (not overridable) | /var/log/datalens/text/current/application_${applicationName}_${timeStamp}.txt.log | Pattern for log file name | v1.3+ |
LOGGING_APPENDERS_TXT_FILE_ARCHIVE | true | Archive log text files | v1.3+ |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/text/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.txt.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly | v1.3+ |
LOGGING_APPENDERS_TXT_FILE_ARCHIVED_TXT_FILE_COUNT | 7 | Max number of archived text files | v1.3+ |
LOGGING_APPENDERS_TXT_FILE_TIMEZONE | UTC | Timezone for text file logging | v1.3+ |
LOGGING_APPENDERS_JSON_FILE_THRESHOLD | ALL | Threashold for text logging | v1.3+ |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages | v1.3+ |
Current Log Filename (not overridable) | /var/log/datalens/json/current/application_${applicationName}_${timeStamp}.json.log | Pattern for log file name | v1.3+ |
LOGGING_APPENDERS_JSON_FILE_ARCHIVE | true | Archive log text files | v1.3+ |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/json/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.json.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly | v1.3+ |
LOGGING_APPENDERS_JSON_FILE_ARCHIVED_FILE_COUNT | 7 | Max number of archived text files | v1.3+ |
LOGGING_APPENDERS_JSON_FILE_TIMEZONE | UTC | Timezone for text file logging | v1.3+ |
LOGGING_APPENDERS_JSON_FILE_LAYOUT_TYPE | json | The layout type for the json logger | v1.3+ |
...