Quick Start Guide - Structured File Lens v1.3

Owned by Russ Waterson (Unlicensed)
Last updated: May 11, 2020
4 min read

This is a quick start guide to get the Structured File Lens up and running in the quickest and simplest possible way so you can start ingesting and transforming data straight away. For a more in-depth set of instructions go to the User Guide.

In this guide we will be setting up and running the Lens as a docker image deployed to your local machine, however we support a number of cloud deployments technologies, including full support of AWS. Once deployed, you can utilise any of our ready-made sample input, mapping, and expected output files to test your Lens.

 

1. Creating the Mapping File

The first step in configuring the Structured File Lens is to create a mapping file. The mapping file is what creates the links between your source data and your target model (ontology). This can be created using our online Data Lens Mapping Tool utilising an intuitive web-based UI. Log in here to get started, and select the option for Structured File Lens. The Structured File Lens is capable of ingesting XML, CSV, and JSON files, creation of mapping files differ slightly between file types so ensure to select the correct options for your use case. Alternatively, if you wish you create your RML mapping files manually, there is a detailed step by step guide on creating one from scratch, along with a number of examples.

 

2. Configuring the Lens

All Lenses supplied by Data Lens are configurable through the use of Environment Variables. The following config options are required for running the Lens:

  • License - LICENSE

    • This is the license key required to operate the Lens, request your new unique license key here.

  • Mapping Directory URL - MAPPINGS_DIR_URL

    • This is the directory where your mapping file(s) is located. As with all directories, this can be either local or on a remote S3 bucket. Mapping files for the Structured File Lens can be created using our Mapping Config Web App and can be pushed directly to a running Lens.

    • For this quick start, we will use a local directory.

  • Output Directory URL - OUTPUT_DIR_URL

    • This is the directory where all generated RDF files are saved to. This also supports local and remote URLs.

    • For this quick start, we will use a local directory.

  • Run Standalone Mode - LENS_RUN_STANDALONE

    • Each of the Lenses are designed to be run as part of a larger end-to-end system, with the end result of data being uploaded into Semantic Knowledge Graphs or Property Graphs. As part of this process, Apache Kafka message queues are used for communicating between services.

    • For this quick start, we are going to enable standalone mode by setting this value to true, so that the Lens won't attempt to connect to external services.

  • Record Provenance - RECORD_PROVO or PROV_OUTPUT_DIR_URL

    • In the Lenses, time-series data is supported as standard, so every time a Lens ingests some data, we add provenance information. This means that you have a full record of data over time. Allowing you to see what the state if the data was at any moment. 

    • For this quick start, we can take two approaches. One would be to turn provenance off by setting RECORD_PROVO to false, or the other is to leave it on (true by default) and set the PROV_OUTPUT_DIR_URL to a directory for your provenance RDF files to be saved to. The latter is the approach we will take in the following examples.

 

3. Running the Lens

All of our Lenses are designed and built to be versatile, allowing them to be set up and ran on a number of environments, including in cloud or on-premise. This is achieved through the use of Docker Containers. For this quick start guide, we are going to use the simplest method of deployment, and this is to run the Lens' Docker image locally. To do this, please first ensure you have Docker installed. Once installed, simply by running a command with the following structure, Docker will start the container and run the Lens from your downloaded image.

For UNIX based machines (macOS and Linux):

docker run \ --env LICENSE=$LICENSE \ --env MAPPINGS_DIR_URL=/var/local/mapping-files/ \ --env OUTPUT_DIR_URL=/var/local/output/ \ --env LENS_RUN_STANDALONE=true \ --env PROV_OUTPUT_DIR_URL=/var/local/prov-output/ \ -p 8080:8080 \ -v /var/local/:/var/local/ \ lens-static:latest

For Windows

docker run ^ --env LICENSE=%LICENSE% ^ --env MAPPINGS_DIR_URL="/data/mapping-files/" ^ --env OUTPUT_DIR_URL="/data/output/" ^ --env LENS_RUN_STANDALONE=true ^ --env PROV_OUTPUT_DIR_URL="/data/prov-output/" ^ -p 8080:8080 ^ -v /data/:/data/ ^ lens-static:latest

The above examples demonstrate how to override configuration options using environment variables in your Lens. Line 2 shows the use of passing in an environment variable saved to the machine, whereas lines 3-6 show simply a string value being passed it. Given the Lens is ran on port 8080, line 7 exposes and binds that port of the host machine so that the APIs can be triggered. The -v flag seen on line 8 mounts the working directory into the container; when the host directory of a bind-mounted volume doesn’t exist, Docker will automatically create this directory on the host for you. And finally, line 9 is the name and version of the Docker image you wish to run.

For more information of running Docker Images, see the official Docs.

 

4. Ingesting Data / Triggering the Lens

The easiest way to ingest a file into the Structured File Lens is to use the built-in endpoint. Using the process endpoint, you can specify the URL of a file to ingest, and in return, you will be provided with the URL of the generated RDF data file.

For example, using a GET request:

<lens-ip>:<lens-port>/process?inputFileURL=<input-file-url> → http://127.0.0.1:8080/process?inputFileURL=file:///var/local/input-data.csv

Once an input file has successfully been processed, the response returned from the Lens is in the form of a JSON, and within the JSON response is the outputFileLocations element. This element contains a list of all the URLs of generated RDF files. (Multiple files are only generated when ingesting large CSV files).

Sample output:

{     "input": "file:///var/local/input/input-data.csv",     "failedIterations": 0,     "successfulIterations": 1,     "outputFileLocations": [         "/var/local/output/Structured-File-Lens-44682bd6-3fbc-429b-988d-40dda8892328.nq"     ] }