Configurable Options - SQL Lens v1.3

Below is a table containing all of the configurable options within the SQL Lens. To see how to set config variables, see the Quick Start Guide or the full User Guide. Mandatory variables are highlighted in red.

1 Lens Configuration
2 Kafka Configuration
3 Provenance Configuration
4 Logging Configuration

Lens Configuration

Environment Variable	Default Value	Description	Version

Environment Variable	Default Value	Description	Version
FRIENDLY_NAME	SQL-Lens	The name you wish to set your Lens up with.	v1.3+
LICENSE		The License key provided required for running the Lens.	v1.3+
CRON_EXPRESSION	/50 * ? * * *	The Quartz Cron Expression. Used by the Lens to set up a time-based job scheduler which will schedule the Lens to ingest your specified data from your database(s) periodically at fixed times, dates, or intervals. This default Cron expression value used translates to triggering the Lens every 50 seconds starting at :00 seconds after the minute.	v1.3+
JDBC_CONNECTOR	com.mysql.cj.jdbc.Driver	The JDBC driver is a software component enabling a Java application to interact with a database. This default is the driver for a MySQL database, however, you can override this with the JDBC of the database you wish to use.	v1.3+
SQL_LIMIT	0	The SQL Limit provides the maximum limit to the number of records that can be processed in any one query. This means that if your database contains more records that this set variable, the Lens will batch process the records from the query and output multiple RDF files. This value must be an integer greater than zero. It defaults to zero, meaning that iterative queries are switched off.	v1.3+
SQL_OFFSET	0	The SQL Offset provides the ability to offset the start index of the iterative processing.	v1.3+
MAPPINGS_DIR_URL		The URL of the directory containing the mapping file(s). Can be local or remote, see here for more details.	v1.3+
MASTER_MAPPING_FILE	mapping.ttl	The filename of the master mapping file.	v1.3+
OUTPUT_DIR_URL		The URL of the directory you wish the generated RDF to be output to. Can be local or remote, see here for more details.	v1.3+
S3_REGION	us-east-1	The region in AWS where your files and services reside. Note: all services must be in the same region.	v1.3+
AWS_ACCESS_KEY		Your access key for AWS.	v1.3+
AWS_SECRET_KEY		Your secret key for AWS.	v1.3+
LENS_RUN_STANDALONE	false	Each of the Lenses are designed to run as part of a larger end to end system with the end result being the data is uploaded to a Knowledge or Property Graph. As part of this process, Kafka is used to communicate between services. This is enabled by default, however if you want to run the Lens as standalone without communicating to other services, set this property to true.	v1.3+

Kafka Configuration

Environment Variable	Default Value	Description	Version

Environment Variable	Default Value	Description	Version
KAFKA_BROKERS	localhost:9092	The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure `<kafka-ip>:<kafka-port>`. The recommended port is `9092`.	v1.3+
KAFKA_TOPIC_NAME_DLQ	dead_letter_queue	The topic used to push messages containing reasons for failure within the Lens. These messages are represented as a JSON.	v1.3+
KAFKA_TOPIC_NAME_SUCCESS	success_queue	The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files.	v1.3+
KAFKA_GROUP_ID_CONFIG	consumerGroup1	The identifier of the group this consumer belongs to.	v1.3+
KAFKA_AUTO_OFFSET_RESET_CONFIG	earliest	What to do when there is no initial offset in Kafka or if an offset is out of range. `earliest`: automatically reset the offset to the earliest offset `latest`: automatically reset the offset to the latest offset	v1.3+
KAFKA_MAX_POLL_RECORDS	100	The maximum number of records returned in a single call to poll.	v1.3+
KAFKA_TIMEOUT	1000000	Kafka consumer polling time out.	v1.3+

Provenance Configuration

Environment Variable	Default Value	Description	Version

Environment Variable	Default Value	Description	Version
RECORD_PROVO	true	Parameter indicating if the provenance meta-data should be generated.	v1.3+
PROV_OUTPUT_DIR_URL		The URL of the directory for the provenance meta-data.	v1.3+
PROV_S3_REGION	us-east-1	The region in AWS where you wish to upload the generated provenance files.	v1.3+
PROV_AWS_ACCESS_KEY		Your access key for AWS.	v1.3+
PROV_AWS_SECRET_KEY		Your secret key for AWS.	v1.3+
PROV_KAFKA_BROKERS	localhost:9092	This is the location of your Kafka Cluster for provenance. This can be the same or different as your broker for the Lens.	v1.3+
PROV_KAFKA_TOPIC_NAME_DLQ	prov_dead_letter_queue	The topic used for your dead letter queue provenance messages. This can be the same or different as your DLQ topic for the Lens	v1.3+
PROV_KAFKA_TOPIC_NAME_SUCCESS	prov_success_queue	The topic used for the messages sent containing the file URLs of the successfully generated provenance files. This can be the same or different as your success queue topic for the Lens	v1.3+
SWITCHED_OFF_ACTIVITIES		A comma-separated list of the provenance processes you which to turn off. The Lens contains the following processes: `main-execution`, `queryProcessing`, `tableProcessing`, and `singleQueryExecution`	v1.3+

Logging Configuration

Environment Variable	Default Value	Description	Version

Environment Variable	Default Value	Description	Version
LOGGING_LEVEL	WARN	Global log level	v1.3+
LOGGING_LOGGERS_DATALENS	DEBUG	Log level for Data Lens loggers	v1.3+
LOGGING_LOGGERS_DROPWIZARD	INFO	Log level for Dropwizard loggers	v1.3+
LOGGING_APPENDERS_CONSOLE_TIMEZONE	UTC	Timezone for console logging	v1.3+
LOGGING_APPENDERS_TXT_FILE_THRESHOLD	ALL	Threashold for text logging	v1.3+
Log Format (not overridable)	%-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n	Pattern for logging messages	v1.3+
Current Log Filename (not overridable)	/var/log/datalens/text/current/application_${applicationName}_${timeStamp}.txt.log	Pattern for log file name	v1.3+
LOGGING_APPENDERS_TXT_FILE_ARCHIVE	true	Archive log text files	v1.3+
Archived Log Filename Pattern (not overridable)	/var/log/datalens/text/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.txt.log	Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly	v1.3+
LOGGING_APPENDERS_TXT_FILE_ARCHIVED_TXT_FILE_COUNT	7	Max number of archived text files	v1.3+
LOGGING_APPENDERS_TXT_FILE_TIMEZONE	UTC	Timezone for text file logging	v1.3+
LOGGING_APPENDERS_JSON_FILE_THRESHOLD	ALL	Threashold for text logging	v1.3+
Log Format (not overridable)	%-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n	Pattern for logging messages	v1.3+
Current Log Filename (not overridable)	/var/log/datalens/json/current/application_${applicationName}_${timeStamp}.json.log	Pattern for log file name	v1.3+
LOGGING_APPENDERS_JSON_FILE_ARCHIVE	true	Archive log text files	v1.3+
Archived Log Filename Pattern (not overridable)	/var/log/datalens/json/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.json.log	Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly	v1.3+
LOGGING_APPENDERS_JSON_FILE_ARCHIVED_FILE_COUNT	7	Max number of archived text files	v1.3+
LOGGING_APPENDERS_JSON_FILE_TIMEZONE	UTC	Timezone for text file logging	v1.3+