FRIENDLY_NAME

friendlyName

Structured-File-Lens

The name you wish to call your Lens.

LICENSE

license

The License key provided required for running the Lens. Only required when running a non AWS Marketplace version of the Lens.

LENS_DIRECTORY

lensDirectory

file:///var/local/

This is the directory where all Lens files are stored (assuming individual file dir config haven’t been edited). On Lens startup, if this has been declared, it will create folders at the specified location for mapping, output, yaml-mapping, prov output, and config backup.

MAPPINGS_DIR_URL

mappingsDirUrl

 file:///var/local/mapping/

The URL of the directory containing the mapping file(s). Can be local or remote, see here for more details. 

MASTER_MAPPING_FILE

masterMappingFile

mapping.ttl

The filename of the master mapping file 

YAML_MAPPINGS_DIR_URL

yamlMappingsDirUrl

file:///var/local/yaml-mapping/

The URL of the directory containing the yaml mapping file(s) if used.

OUTPUT_DIR_URL

outputDirUrl

 file:///var/local/output/

The URL of the directory you wish the generated RDF to be output to. Can be local or remote, see here for more details.

OUTPUT_FILE_FORMAT

outputFileFormat

nquads

The file type that will be constructed when the RDF is created. The options are: nquads, ntriples, jsonld, turtle, trig, and trix.

CONFIG_BACKUP

configBackup

file:///var/local/config-backup/

The URL directory where the config will be backed up to when calling the upload config endpoint

MAX_CSV_ROWS

maxCsvRows

100000

The maximum number of rows a CSV file can be before it is split into smaller files (of specified length) and processed individually.

VALIDATE_CSV

validateCsv

true

By default, the Lens will check a CSV file is valid and remove any invalid or multiline rows. To turn this off, set this property to false.

S3_REGION

s3Region

us-east-1

The region in AWS where your files and services reside. Note: all services must be in the same region.

AWS_ACCESS_KEY

awsAccessKey

Your access key for AWS

AWS_SECRET_KEY

awsSecretKey

Your secret key for AWS

LENS_RUN_STANDALONE

runStandalone

true

Each of the Lenses are designed to run as part of a larger end to end system with the end result being the data is uploaded to a Knowledge or Property Graph. As part of this process, Kafka is used to communicate between services. This is enabled by default, however if you want to run the Lens as standalone without communicating to other services, set this property to true.

UPLOAD_CUSTOM_FUNCTIONS

uploadCustomFunctions

false

If you require a function to be executed that doesn’t perform the required operation using the built-in functions, it is possible to create and use your own. To do this, set this variable to true, and follow the instructions laid out in this guide.

KAFKA_BROKERS

kafkaBrokers

localhost:9092

The Kafka Broker is what tells the Lens where to look for your Kafka Cluster. Set with the following structure <kafka-ip>:<kafka-port>. The recommended port is 9092.

KAFKA_TOPIC_NAME_SOURCE

topicNameSource

source_urls

The topic used for the Consumer to read messages from containing input file URLs in order to ingest data.

KAFKA_TOPIC_NAME_DLQ

topicNameDLQ

dead_letter_queue

The topic used to push messages containing reasons for failure within the Lens. These messages are represented as a JSON.

KAFKA_TOPIC_NAME_SUCCESS

topicNameSuccess

success_queue

The topic used for the messages sent containing the file URLs of the successfully transformed RDF data files.

KAFKA_GROUP_ID_CONFIG

groupIdConfig

consumerGroup1

 The identifier of the group this consumer belongs to.

KAFKA_AUTO_OFFSET_RESET_CONFIG

autoOffsetResetConfig

earliest

What to do when there is no initial offset in Kafka or if an offset is out of range.

earliest: automatically reset the offset to the earliest offset

latest: automatically reset the offset to the latest offset

KAFKA_MAX_POLL_RECORDS

maxPollRecords

100

 The maximum number of records returned in a single call to poll.

KAFKA_TIMEOUT

timeout

1000000

Kafka consumer polling time out.