Configurable Options - RESTful Lens v1.2
- Russ Waterson (Unlicensed)
Below is a table containing all of the configurable options within the Structured File Lens. To see how to set config variables, see the Quick Start Guide or the full User Guide. Mandatory variables are highlighted in red.
Lens Configuration
Environment Variable | Default Value | Description | Version |
|---|---|---|---|
FRIENDLY_NAME | RESTful-Lens | The name you wish to set your Lens up with. | v1.1+ |
LICENSE |
| The License key provided required for running the Lens. | v1.1+ |
CRON_EXPRESSION |
| The Quartz Cron Expression. Used by the Lens to set up a time-based job scheduler which will schedule the Lens to ingest your specified data from your database(s) periodically at fixed times, dates, or intervals. This default Cron expression value used translates to triggering the Lens every 30 minutes starting at :00 or :30 minutes after the hour. | v1.1+ |
MAPPINGS_DIR_URL |
| The URL of the directory containing the mapping file(s). Can be local or remote, see here for more details. | v1.1+ |
MASTER_MAPPING_FILE | mapping.ttl | The filename of the master mapping file | v1.1+ |
OUTPUT_DIR_URL |
| The URL of the directory you wish the generated RDF to be output to. Can be local or remote, see here for more details. | v1.1+ |
OUTPUT_FILE_FORMAT | nquads | The format that the RDF is transformed into, must be either nquads or json-ld. | v1.1+ |
JSON_REST_CONFIG_URL |
| The URL where the Lens can obtain the JSON REST config file. Either this value or the | v1.1+ |
JSON_API_CONFIG_URL |
| The URL where the Lens can obtain the JSON:API config file. Either this value or the | v1.1+ |
S3_REGION | us-east-1 | The region in AWS where your files and services reside. Note: all services must be in the same region. | v1.1+ |
AWS_ACCESS_KEY |
| Your access key for AWS | v1.1+ |
AWS_SECRET_KEY |
| Your secret key for AWS | v1.1+ |
ENDPOINT_MODE | json-api | There are two types of mode in which the RESTful Lens can operate in, this depends on whether the Lens is being used to work with a standard RESTful Endpoint, or with one that conforms to the JSON:API specification. The endpoint mode is set by specifying either | v1.1+ |
LENS_RUN_STANDALONE | false | Each of the Lenses are designed to run as part of a larger end to end system with the end result being the data is uploaded to a Knowledge or Property Graph. As part of this process, Kafka is used to communicate between services. This is enabled by default, however if you want to run the Lens as standalone without communicating to other services, set this property to true. | v1.1+ |
UPLOAD_CUSTOM_FUNCTIONS | false | If you require a function to be executed that doesn’t perform the required operation using the built-in functions, it is possible to create and use your own. To do this, set this variable to true, and follow the instructions laid out in this guide. | v1.2+ |
Kafka Configuration
Environment Variable | Default Value | Description | Version |
|---|---|---|---|
KAFKA_BROKERS | localhost:9092 | The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure | v1.1+ |
KAFKA_TOPIC_NAME_SOURCE | source_urls | The topic used for the Consumer to read messages from containing input file URLs in order to ingest data. | v1.1+ |
KAFKA_TOPIC_NAME_DLQ | dead_letter_queue | The topic used to push messages containing reasons for failure within the Lens. These messages are represented as a JSON. | v1.1+ |
KAFKA_TOPIC_NAME_SUCCESS | success_queue | The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files. | v1.1+ |
KAFKA_GROUP_ID_CONFIG | consumerGroup1 | The identifier of the group this consumer belongs to. | v1.1+ |
KAFKA_AUTO_OFFSET_RESET_CONFIG | earliest | What to do when there is no initial offset in Kafka or if an offset is out of range.
| v1.1+ |
KAFKA_MAX_POLL_RECORDS | 100 | The maximum number of records returned in a single call to poll. | v1.1+ |
KAFKA_TIMEOUT | 1000000 | Kafka consumer polling time out. | v1.1+ |
Provenance Configuration
Environment Variable | Default Value | Description | Version |
|---|---|---|---|
RECORD_PROVO | true | Parameter indicating whether any provenance meta-data should be generated | v1.1+ |
PROV_OUTPUT_DIR_URL |
| The URL of the directory for the provenance meta-data | v1.1+ |
PROV_S3_REGION | us-east-1 | The region in AWS where you wish to upload the generated provenance files | v1.1+ |
PROV_AWS_ACCESS_KEY |
| Your access key for AWS | v1.1+ |
PROV_AWS_SECRET_KEY |
| Your secret key for AWS | v1.1+ |
PROV_KAFKA_BROKERS | localhost:9092 | This is the location of your Kafka Cluster for provenance. This can be the same or different as your broker for the Lens | v1.1+ |
PROV_KAFKA_TOPIC_NAME_DLQ | prov_dead_letter_queue | The topic used for your dead letter queue provenance messages. This can be the same or different as your DLQ topic for the Lens | v1.1+ |
PROV_KAFKA_TOPIC_NAME_SUCCESS | prov_success_queue | The topic used for the messages sent containing the file URLs of the successfully generated provenance files. This can be the same or different as your success queue topic for the Lens | v1.1+ |
SWITCHED_OFF_ACTIVITIES |
| A comma-separated list of the provenance processes you which to turn off. The Lens contains the following processes: | v1.1+ |
Logging Configuration
Environment Variable | Default Value | Description | Version |
|---|---|---|---|
LOGGING_LEVEL | WARN | Global log level | v1.1+ |
LOGGING_LOGGERS_DATALENS | DEBUG | Log level for Data Lens loggers | v1.1+ |
LOGGING_LOGGERS_DROPWIZARD | INFO | Log level for Dropwizard loggers | v1.1+ |
LOGGING_APPENDERS_CONSOLE_TIMEZONE | UTC | Timezone for console logging | v1.1+ |
LOGGING_APPENDERS_TXT_FILE_THRESHOLD | ALL | Threashold for text logging | v1.1+ |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages | v1.1+ |
Current Log Filename (not overridable) | /var/log/datalens/text/current/application_${applicationName}_${timeStamp}.txt.log | Pattern for log file name | v1.1+ |
LOGGING_APPENDERS_TXT_FILE_ARCHIVE | true | Archive log text files | v1.1+ |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/text/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.txt.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly | v1.1+ |
LOGGING_APPENDERS_TXT_FILE_ARCHIVED_TXT_FILE_COUNT | 7 | Max number of archived text files | v1.1+ |
LOGGING_APPENDERS_TXT_FILE_TIMEZONE | UTC | Timezone for text file logging | v1.1+ |
LOGGING_APPENDERS_JSON_FILE_THRESHOLD | ALL | Threashold for text logging | v1.1+ |
Log Format (not overridable) | %-6level [%d{HH:mm:ss.SSS}] [%t] %logger{5} - %X{code} %msg %n | Pattern for logging messages | v1.1+ |
Current Log Filename (not overridable) | /var/log/datalens/json/current/application_${applicationName}_${timeStamp}.json.log | Pattern for log file name | v1.1+ |
LOGGING_APPENDERS_JSON_FILE_ARCHIVE | true | Archive log text files | v1.1+ |
Archived Log Filename Pattern (not overridable) | /var/log/datalens/json/archive/application_${applicationName}_${timeStamp}_to_%d{yyyy-MM-dd}.json.log | Log file rollover frequency depends on pattern in following property. For example %d{yyyy-MM-ww} declares rollover weekly | v1.1+ |
LOGGING_APPENDERS_JSON_FILE_ARCHIVED_FILE_COUNT | 7 | Max number of archived text files | v1.1+ |
LOGGING_APPENDERS_JSON_FILE_TIMEZONE | UTC | Timezone for text file logging | v1.1+ |
LOGGING_APPENDERS_JSON_FILE_LAYOUT_TYPE | json | The layout type for the json logger | v1.1+ |