This is a quick start guide to get the RESTful Lens up and running in the quickest and simplest possible way so you can start ingesting and transforming data straight away. For a more in-depth set of instructions go to the User Guide.
In this guide we will be setting up and running the Lens as a docker image deployed to your local machine, however we support a number of cloud deployments technologies, including full support of AWS. The RESTful Lens supports the standard endpoints that supply JSON or XML and also has an additional mode that works with endpoints that conform to the JsonApi specification (Ref. JSON:API Specification). A separate JSON config file is required depending on which mode is used. All files presented in this Quick Guide are available in our example files repository.
The first step in configuring the RESTful Lens is to create a mapping file. The mapping file is what creates the links between your source database and your target model (ontology). This can be created using our online Data Lens Mapping Tool utilising an intuitive web-based UI. Log in here to get started, and select the option for RESTful Lens. Alternatively, if you wish you create your RML mapping files manually, there is a detailed step by step guide on creating one from scratch, along with a number of examples.
There are two types of JSON configuration files that can be used with the RESTful Lens, this depends on whether it is being used to work with a standard RESTful Endpoint, or with one that conforms to the JSON:API specification. The endpoint mode is set by specifying either json-api or rest-api under the ENDPOINT_MODE config option. By default the Lens is set to JSON:API mode, and is this approach we will take in this Quick Start Guide. Specifying a JSON config file is a mandatory requirement, and is done by providing a value for the JSON_API_CONFIG_URL or JSON_REST_CONFIG_URL configurable variables respectively.
More information about the RESTful modes can be found in the full User Guide. In addition, if you would prefer to try the Lens with ready-made example files, we have created a repository for them which you are able to clone to your local file system. In the example provided below, we have used these files pulled to the local directory /opt/Projects/DataLens/.
{
"startURL" : "http://example.com/articles",
"includeFields": [
"type",
"title",
"author.type",
"author.id",
"author.firstName",
"author.lastName",
"author.twitter",
"comments.type",
"comments.id",
"comments.body"
]
} |
The JSON:API config file needs to include the start URL of the JSON:API service and the list of fields that you want to take from the service to convert into RDF. The Lens will go through the JSON:API service to get the relevant fields, it will start its search on the first page referenced and if it cannot locate the required field on the current page it will use the self-reference for the relevant lower-level object to try to find the required fields. For example, if we take the author value, the author.type and author.id fields may be included on the initial http://example.com/articles page, however the author.firstName, author.lastName and author.twitter fields may not be.
For a specific article, if the author.type is “Person” and author.id is “9”, the Lens will look for an object with those values in type and id fields and follow the self-link to the relevant page for the author http://example.com/people/9. From here it will then look for the missing additional fields author.firstName, author.lastName and author.twitter. If the author object itself contained objects such as author.book, with fields author.book.type, author.book.id, author.book.price, author.book.title, and these additional objects were specified in the json config, then the JSON:API will follow the same process to use the id and type of the book object to go to the relevant self-link for the book and search for the remaining fields there. The Lens is able to follow this process for any depth of object. This can be seen in our example 1 within the examples repository, where the config file is called multipage-config.json.
All Lenses supplied by RESTful Lens are configurable through the use of Environment Variables. The following config options are required for running the Lens:
License - LICENSE
This is the license key required to operate the Lens, request your new unique license key here.
Mapping Directory URL - MAPPINGS_DIR_URL
This is the directory where your mapping file(s) is located. As with all directories, this can be either local or on a remote S3 bucket. Mapping files for the RESTful Lens can be created using our Mapping Config Web App and can be pushed directly to a running Lens.
For this quick start, we will use a local directory.
Output Directory URL - OUTPUT_DIR_URL
This is the directory where all generated RDF files are saved to. This also supports local and remote URLs.
For this quick start, we will use a local directory.
Configuration JSON file URL - JSON_API_CONFIG_URL or JSON_REST_CONFIG_URL
Depending on the mode in which the Lens is set to, either one of these two variables is required to be set.
For the purposes of this quick start guide, we will be using the default mode and therefore set the JSON_API_CONFIG_URL variable to a value of a URL referring to the JSON configuration file.
Run Standalone Mode - LENS_RUN_STANDALONE
Each of the Lenses are designed to be run as part of a larger end-to-end system, with the end result of data being uploaded into Semantic Knowledge Graphs or Property Graphs. As part of this process, Apache Kafka message queues are used for communicating between services.
For this quick start guide, we are going to enable standalone mode by setting this value to true, so that the Lens won't attempt to connect to external services.
Record Provenance - RECORD_PROVO and PROV_OUTPUT_DIR_URL
In the Lenses, time-series data is supported as standard, so every time a Lens ingests some data, we add provenance information. This means that you have a full record of data over time. Allowing you to see what the state if the data was at any moment.
For this quick start, we can take two approaches. One would be to turn provenance off by setting RECORD_PROVO to false, or the other is to leave it on (true by default) and set the PROV_OUTPUT_DIR_URL to a directory for your provenance RDF files to be saved to. The latter is the approach we will take in the following examples.
All of our Lenses are designed and built to be versatile, allowing them to be set up and ran on a number of environments, including in cloud or on-premise. This is achieved through the use of Docker Containers. For this quick start guide, we are going to use the simplest method of deployment, and this is to run the Lens' Docker image locally. To do this, please first ensure you have Docker installed. Once installed, simply by running a command with the following structure, Docker will start the container and run the Lens from your downloaded image.
For UNIX based machines (macOS and Linux):
docker run \ -e MAPPINGS_DIR_URL=file:///data/example1/mapping/ \ -e JSON_API_CONFIG_URL=file:///data/example1/input/multipage-config.json \ -e OUTPUT_DIR_URL=file:///data/example1/output \ -e RECORD_PROVO=false \ -e LENS_RUN_STANDALONE=true \ -e LICENSE \ -v /opt/Projects/DataLens/datalens-examples/restful-lens/example1:/data/example1 \ -p 8080:8080 \ lens-restful:latest |
For Windows
docker run ^ -e MAPPINGS_DIR_URL=file:///data/example1/mapping/ ^ -e JSON_API_CONFIG_URL=file:///data/example1/input/multipage-config.json ^ -e OUTPUT_DIR_URL=file:///data/example1/output ^ -e RECORD_PROVO=false ^ -e LENS_RUN_STANDALONE=true ^ -e LICENSE ^ -v /opt/Projects/DataLens/datalens-examples/restful-lens/example1:/data/example1 ^ -p 8080:8080 ^ lens-restful:latest |
The above examples demonstrate how to override configuration options using environment variables in your Lens. Line 7 shows the use of passing in an environment variable saved to the machine, whereas lines 2-6 show simply a string value being passed it. Given the Lens is ran on port 8080, line 9 exposes and binds that port of the host machine so that the APIs can be triggered. The -v flag seen on line 8 mounts the working directory into the container; when the host directory of a bind-mounted volume doesn’t exist, Docker will automatically create this directory on the host for you. And finally, line 10 is the name and version of the Docker image you wish to run.
For more information of running Docker Images, see the official Docs.
Using /process endpoint we will trigger the Lens process, for example, by following GET request: <lens-ip>:<lens-port>/process → http://127.0.0.1:8080/process.
Once the ingestion and transformation has successfully been processed, the response returned from the Lens is in the form of a JSON. This JSON response contains the input file location, the mapping files location and a list of all the URLs referring to the generated RDF files.
Sample output:
{
"lensProcessingReport": {
"inputFileLocation": "https://bitbucket.org/data-lens/datalens-examples/raw/master/restful-lens/example1/input/restful/json-api-no-data-array-page1.json",
"rmlFiles": [
"mapping.ttl"
],
"successfulIterations": 2,
"outputFileLocations": [
"file:///data/example1/output/RESTful-Lens-3142a3d1-956f-41c2-a456-a4b94402826f.nq",
"file:///data/example1/output/RESTful-Lens-069168f4-ae8e-420f-89f4-5b6d14f799bf.nq"
]
}
} |
The process to trigger the ingestion of your RESTful data can be done in two ways; in addition to the RESTful service, there is also a built-in Quartz Time Scheduler. This uses a user-configurable Cron Expression to set up a time-based job scheduler which will schedule the Lens to ingest your specified data from your JSON:API or REST:API remote service periodically at fixed times, dates, or intervals. More information on this can be found in the full User Guide.