...

• License - LICENSE

  • This is the license key required to operate the Lens, request your new unique license key here.

• Mapping Directory URL - MAPPINGS_DIR_URL

  • This is the directory where your mapping file(s) is located. As with all directories, this can be either local or on a remote S3 bucket. Mapping files for the Stream Lens can be created using our Mapping Config Web App and can be pushed directly to a running Lens.

  • For this quick start, we will use a local directory.

• Output Directory URL - OUTPUT_DIR_URL

  • This is the directory where all generated RDF files are saved to. This also supports local and remote URLs.

  • For this quick start, we will use a local directory.

• Run Standalone Mode - LENS_RUN_STANDALONE

  • Each of the Lenses are designed to be run as part of a larger end-to-end system, with the end result of data being uploaded into Semantic Knowledge Graphs or Property Graphs. As part of this process, Apache Kafka message queues are used for communicating between services.

  • For this quick start, we are going to enable standalone mode by setting this value to true, so that the Lens won't attempt to connect to external services.

• Record Provenance / Provenance Output Directory URL - RECORD_PROVO or PROV_OUTPUT_DIR_URL

  • In the Lenses, time-series data is supported as standard, so every time a Lens ingests some data, we add provenance information. This means that you have a full record of data over time. Allowing you to see what the state if the data was at any moment. 

  • For this quick start, we can take two approaches. One would be to turn provenance off by setting RECORD_PROVO to false, or the other is to leave it on (true by default) and set the PROV_OUTPUT_DIR_URL to a directory for your provenance RDF files to be saved to. The latter is the approach we will take in the following examples.

3. Running the Lens

Ask about using local files…

All of our Lenses are designed and built to be versatile, allowing them to be set up and ran on a number of environments, including in cloud or on-premise. This is achieved through the use of Docker Containers. For this quick start guide, we are going to use the simplest method of deployment, and this is to run the Lens' Docker image locally. To do this, we will utilise a docker-compose file to build our stack. First, please ensure you have Docker installed.

1. Once you have pulled the latest version of the Stream Lens, tag your local image as latest using: docker tag datalensltd/lens-stream:{VERSION} datalensltd/lens-stream:latest.

2. Next, download the docker-compose file and the lens-configuration file. We will go into more detail about how the docker-compose file works in the full user guide.

3. Configure your Lens by opening and adding your variables to the lens-configuration.env file.

   1. In this example, we have set our mapping and output directories to a local directory simply by assigning the variable a string value. As for the License, in this format, the value will be taken from the machines environment variable named ‘License’.

   2. The /tmp directory on the local machine is a mounted volume to the /mnt/efs directory on the running docker images. Therefore if we wish to use locally stored input, mapping, and output files, using the provided docker-compose file, we should store our files in the /tmp directory on your local machine.

      Code Block
      # Please configure your lens, example: KEY=VALUE. Leaving a field blank will take the value from your environment variables. LICENSE MAPPINGS_DIR_URL=file:/var//mnt/localefs/mapping-files/ OUTPUT_DIR_URL=file:///varmnt/localefs/output/ PROV_OUTPUT_DIR_URL=file://var/mnt/localefs/prov-output/

4. Please ensure the /tmp/data/ directory exists on the host machine using: mkdir -p /tmp/data/.

5. Now from within the directory you downloaded the compose and config files, run the Lens by using docker-compose up.

...

{
    "input": "file:///varmnt/localefs/input/input-data.csv",
    "failedIterations": 0,
    "successfulIterations": 1,
    "outputFileLocations": [
        "file:/var//mnt/localefs/output/Stream-Lens-44682bd6-3fbc-429b-988d-40dda8892328.nq"
    ]
}

...