Configurable Options - RESTful Lens v1.1

Below is a table containing all of the configurable options within the Structured File Lens. To see how to set config variables, see the Quick Start Guide or the full User Guide. Mandatory variables are highlighted in red.



Lens Configuration

Environment Variable

Default Value

Description

Version

Environment Variable

Default Value

Description

Version

FRIENDLY_NAME

RESTful-Lens

The name you wish to set your Lens up with.

v1.1+

LICENSE

 

The License key provided required for running the Lens.

v1.1+

CRON_EXPRESSION

* */30 * ? * * *

The Quartz Cron Expression. Used by the Lens to set up a time-based job scheduler which will schedule the Lens to ingest your specified data from your database(s) periodically at fixed times, dates, or intervals. This default Cron expression value used translates to triggering the Lens every 50 seconds starting at :00 seconds after the minute.

v1.1+

MAPPINGS_DIR_URL

 

The URL of the directory containing the mapping file(s). Can be local or remote, see here for more details. 

v1.1+

MASTER_MAPPING_FILE

mapping.ttl

The filename of the master mapping file 

v1.1+

OUTPUT_DIR_URL

 

The URL of the directory you wish the generated RDF to be output to. Can be local or remote, see here for more details.

v1.1+

OUTPUT_FILE_FORMAT

nquads

The format that the RDF is transformed into, must be either nquads or json-ld.

v1.1+

JSON_REST_CONFIG_URL

 

The URL where the Lens can obtain the JSON REST config file. Either this value or the JSON_API_CONFIG_URL value must be set, dependant on the ENDPOINT_MODE value.

v1.1+

JSON_API_CONFIG_URL

 

The URL where the Lens can obtain the JSON:API config file. Either this value or the JSON_REST_CONFIG_URL value must be set, dependant on the ENDPOINT_MODE value.

v1.1+

S3_REGION

us-east-1

The region in AWS where your files and services reside. Note: all services must be in the same region.

v1.1+

AWS_ACCESS_KEY

 

Your access key for AWS

v1.1+

AWS_SECRET_KEY

 

Your secret key for AWS

v1.1+

ENDPOINT_MODE

json-api

There are two types of mode in which the RESTful Lens can operate in, this depends on whether the Lens is being used to work with a standard RESTful Endpoint, or with one that conforms to the JSON:API specification. The endpoint mode is set by specifying either json-api or rest-api. Specifying a JSON config file is a mandatory requirement, and is done dependant on the mode by providing a value for the JSON_API_CONFIG_URL or JSON_REST_CONFIG_URL configurable variables respectively.

v1.1+

LENS_RUN_STANDALONE

false

Each of the Lenses are designed to run as part of a larger end to end system with the end result being the data is uploaded to a Knowledge or Property Graph. As part of this process, Kafka is used to communicate between services. This is enabled by default, however if you want to run the Lens as standalone without communicating to other services, set this property to true.

v1.1+

 

 

Kafka Configuration

Environment Variable

Default Value

Description

Version

Environment Variable

Default Value

Description

Version

KAFKA_BROKERS

localhost:9092

The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure <kafka-ip>:<kafka-port>. The recommended port is 9092.

v1.1+

KAFKA_TOPIC_NAME_SOURCE

source_urls

The topic used for the Consumer to read messages from containing input file URLs in order to ingest data.

v1.1+

KAFKA_TOPIC_NAME_DLQ

dead_letter_queue

The topic used to push messages containing reasons for failure within the Lens. These messages are represented as a JSON.

v1.1+

KAFKA_TOPIC_NAME_SUCCESS

success_queue

The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files.

v1.1+

KAFKA_GROUP_ID_CONFIG

consumerGroup1

 The identifier of the group this consumer belongs to.

v1.1+

KAFKA_AUTO_OFFSET_RESET_CONFIG

earliest

What to do when there is no initial offset in Kafka or if an offset is out of range.

earliest: automatically reset the offset to the earliest offset

latest: automatically reset the offset to the latest offset

v1.1+

KAFKA_MAX_POLL_RECORDS

100

 The maximum number of records returned in a single call to poll.

v1.1+

KAFKA_TIMEOUT

1000000

Kafka consumer polling time out.

v1.1+

 

Provenance Configuration

Environment Variable

Default Value

Description

Version

Environment Variable

Default Value

Description

Version

RECORD_PROVO

true

Parameter indicating whether any provenance meta-data should be generated

v1.1+

PROV_OUTPUT_DIR_URL

 

The URL of the directory for the provenance meta-data

v1.1+

PROV_S3_REGION

us-east-1

The region in AWS where you wish to upload the generated provenance files

v1.1+

PROV_AWS_ACCESS_KEY

 

Your access key for AWS

v1.1+

PROV_AWS_SECRET_KEY

 

Your secret key for AWS

v1.1+

PROV_KAFKA_BROKERS

localhost:9092

This is the location of your Kafka Cluster for provenance. This can be the same or different as your broker for the Lens

v1.1+

PROV_KAFKA_TOPIC_NAME_DLQ

prov_dead_letter_queue

The topic used for your dead letter queue provenance messages. This can be the same or different as your DLQ topic for the Lens

v1.1+

PROV_KAFKA_TOPIC_NAME_SUCCESS

prov_success_queue

The topic used for the messages sent containing the file URLs of the successfully generated provenance files. This can be the same or different as your success queue topic for the Lens

v1.1+

SWITCHED_OFF_ACTIVITIES

 

A comma-separated list of the provenance processes you which to turn off.

The Lens contains the following processes: main-execution and lens-iteration

v1.1+

 

Logging Configuration

Environment Variable

Default Value

Description

Version

Environment Variable

Default Value

Description

Version

LOGGING_LEVEL

WARN

Global log level

v1.1+

LOGGING_LOGGERS_DATALENS

DEBUG

Log level for Data Lens loggers

v1.1+

LOGGING_LOGGERS_DROPWIZARD

INFO

Log level for Dropwizard loggers

v1.1+

LOGGING_APPENDERS_CONSOLE_TIMEZONE

UTC

Timezone for console logging

v1.1+

LOGGING_APPENDERS_TXT_FILE_THRESHOLD

ALL

Threashold for text logging

v1.1+

Log Format (not overridable)

%-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n

Pattern for logging messages

v1.1+

Current Log Filename (not overridable)

/var/log/datalens/text/current/application_${applicationName}_${timeStamp}.txt.log

Pattern for log file name

v1.1+

LOGGING_APPENDERS_TXT_FILE_ARCHIVE

true

Archive log text files

v1.1+

Archived Log Filename Pattern (not overridable)

/var/log/datalens/text/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.txt.log

Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly

v1.1+

LOGGING_APPENDERS_TXT_FILE_ARCHIVED_TXT_FILE_COUNT

7

Max number of archived text files

v1.1+

LOGGING_APPENDERS_TXT_FILE_TIMEZONE

UTC

Timezone for text file logging

v1.1+

LOGGING_APPENDERS_JSON_FILE_THRESHOLD

ALL

Threashold for text logging

v1.1+

Log Format (not overridable)

%-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n

Pattern for logging messages

v1.1+

Current Log Filename (not overridable)

/var/log/datalens/json/current/application_${applicationName}_${timeStamp}.json.log

Pattern for log file name

v1.1+

LOGGING_APPENDERS_JSON_FILE_ARCHIVE

true

Archive log text files

v1.1+

Archived Log Filename Pattern (not overridable)

/var/log/datalens/json/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.json.log

Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly

v1.1+

LOGGING_APPENDERS_JSON_FILE_ARCHIVED_FILE_COUNT

7

Max number of archived text files

v1.1+

LOGGING_APPENDERS_JSON_FILE_TIMEZONE

UTC

Timezone for text file logging

v1.1+

LOGGING_APPENDERS_JSON_FILE_LAYOUT_TYPE

json

The layout type for the json logger

v1.1+