Table of Contents
Introduction
A query language is the mechanism used to locate (search for and select) JSON values within a JSON document.
The JM2MP format offers its own query language, just named
native path syntax (or native for short). Both formats, JM2MP and
native, have been designed to be algebraically complete, in the sense
that they provide, with mathematical rigor, at least the minimal set of
operations necessary to achieve a complete transformation of JSON
documents from the source (or input) to the resultant (or
output), using a third document called the projection (which
contains the desired transformations).
There are several query languages designed for this purpose, ranging
from the simplest (which simply allow you to locate any element in the
document, like this native or the standard
JSON Pointer) to the
most sophisticated (like
JMESPath or
JSONata, for instance,
which offer pattern-based searches, as well as additional filtering and
sorting operations, among others).
Each query language offers its own syntax (or set of syntaxes) for writing a (simple) path or a (more complex) workflow, comprising of a sequence of steps necessary to locate (and perhaps process) one (or more) JSON input values from the source document, so that the query language engine (typically, the library that interprets such query language) can return them as the result (output) of that navigation, location or process.
The native query language features two syntactically different but
semantically equivalent notations: one based on text strings and another
based on an structured JSON syntax.
In addition, the JM2MP.JS library allows you to use several external
query languages by default (see
external references
section from
JM2MP Syntax tutorial for more details):
The following section presents the basic concepts and capabilities of
the native query language.
Basic Capabilities
The native path syntax is a simple query language designed to
unambiguously select a single JSON value within a document, and it
offers two easy-to-read syntaxes: one based on JSON arrays and the
other based on text string.
The native query language is designed for the null absorption
propagation of values. This means that, when it does not find the
specified value for a step, the resultant JSON value will always be
null, even if the path still has remaining steps (nodes not yet
traversed).
The concept of null absorption propagation is important to the
JM2MP format, as it ensures that any undefined result will be
returned as null, so: no errors will be raised due to an inapplicable
path and every template command will always return a valid JSON
value (remember that undefined is not a valid JSON value) or at least
a null one.
But errors will still occur when path is not in the correct or valid format!
Actually, the
QueryAdapter interface
described in the section about
Other Query Languages
of the JM2MP Syntax tutorial is the
way to standardize the behavior of any external query languages that
can be incorporated into the JM2MP.JS library, so that they can then
be used from JM2MP projection documents.
In the following sections, we'll show several examples of paths using
the native query language. For all of these examples, we will always
use the JSON document shown next as our (simple but representative)
source document:
{
"SubRootObject": {
"NullProperty" : null,
"FalseProperty" : false,
"TrueProperty" : true,
"TextProperty" : "Text value.",
"IntegerProperty" : 12,
"RealProperty" : Math.PI,
"EmptyArrayProperty" : [],
"ArrayProperty" : [ 1, "Two", { "Three" : 3 } ],
"EmptyObjectProperty": {},
"ObjectProperty": {
"Alpha" : 1,
"Bravo" : "B",
"Charlie" : 3.33
}
}
}
Next, we will present each notation of the native query language:
JSON Variant
The JSON-based syntactic variant of the native query language is
represented as an array whose items' values (called accessors) can
be of only two types:
-
Natural numbers: when referring to the zero-based index of an item within an array.
-
Text strings: when referring to the name of an object's property.
An empty array is considered as the empty path, which is a valid path that returns the same source value (input) as the resultant value (output).
Therefore, this JSON-based variant will only navigate (or locate) from the current context where it is being used.
Below are several valid examples of how to locate values in the source document presented above:
// Query:
[ "SubRootObject", "IntegerProperty" ]
// Resultant value:
12
// Query:
[ "SubRootObject", "ArrayProperty" ]
// Resultant value:
[ 1, "Two", { "Three" : 3 } ]
// Query:
[ "SubRootObject", "ArrayProperty", 2, "Three" ]
// Resultant value:
3
// Query:
[ "SubRootObject", "ObjectProperty" ]
// Resultant value:
{
"Alpha" : 1,
"Bravo" : "B",
"Charlie" : 3.33
}
// Query:
[ "SubRootObject", "ObjectProperty", "Bravo" ]
// Resultant value:
"B"
And, due to the null absorption propagation, inaccessible paths
will always return null:
// Query (index 5 is out of range):
[ "SubRootObject", "ArrayProperty", 5, "Single" ]
// Resultant value:
null
// Query (neither NonExistentObject nor Omega exists):
[ "SubRootObject", "NonExistentObject", "Omega" ]
// Resultant value:
null
But it is possible to declare an invalid native path that will raise
a ParseError exception:
// Invalid query:
[ false ]
// ParseError exception will be raised!
Textual String Variant
The textual syntactic variant of the native query language is
represented as a text string whose content is a complete path with
all its steps, always beginning with a selector from the
execution environment
considered for input:
-
$: it always references the root value of the source document. -
@: refers to the current context of the source document in which the template command is being execute; foldArr and foldObj template commands each define their own current step context due to their transformative nature. -
%AliasName: which must reference one of the previously defined (bound) alias.
An empty string is considered invalid (a syntax error), same as reference an undefined alias.
Again, due to the null absorption propagation, inaccessible paths
will always return null.
Below are the same examples from the previous section, but this time written using textual string variant:
// Query:
"$.SubRootObject.IntegerProperty"
// Resultant value:
12
// Query:
"$.SubRootObject.ArrayProperty"
// Resultant value:
[ 1, "Two", { "Three" : 3 } ]
// Query:
"$.SubRootObject.ArrayProperty.2.Three"
// Resultant value:
3
// Query:
"$.SubRootObject.ObjectProperty"
// Resultant value:
{
"Alpha" : 1,
"Bravo" : "B",
"Charlie" : 3.33
}
// Query:
"$.SubRootObject.ObjectProperty.Bravo"
// Resultant value:
"B"
Combining Operations
To fully understand the potential and versatility of JM2MP, it is
helpful to show some examples of how, by combining it with the native
query language (but also with any other equally capable language), it
is possible to perform and link common data processing operations, such
as compose (pipe), filter (map) and aggregate (reduce) data.
For all the examples presented below, we will always consider the following source document:
{
"Name": "Map-Reduce first example.",
"Records": [
{ "Id":1, "Title":"One", "Group":"Alpha", "Value": 3.00 },
{ "Id":2, "Title":"Two", "Group":"Alpha", "Value": 5.00 },
{ "Id":3, "Title":"Three", "Group":"Bravo", "Value": 7.00 },
{ "Id":4, "Title":"Four", "Group":"Bravo", "Value": 9.00 },
{ "Id":5, "Title":"Five", "Group":"Charlie", "Value":11.00 },
]
}
To embed comments within JSON values, all JM2MP projection
documents that are displayed will use the
JSONC format;
simply remove those comments to obtain fully compliant JSON documents.
Filtering
A typical filter operation to get only records with a value less than or equal to 6 should be like this:
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter
const expected_document = {
Name: source_document.Name,
Records: source_document.Records
.filter( (i) => (i.Value <= 6) )
};
where its equivalent JM2MP projection document would be (using the
appropriate template command foldArr):
{
// The root template, always mandatory!
"$": {
// Literal property name, which copies its property name.
"Name": { "$op":"get", "$path":"@.Name" },
// Literal property name, which iterates each item (FoldArr)
// filtering using a condition.
"Records":
{ "$op" : "foldArr",
// Array to iterate over; remember that 'foldArr'
// is a from-right-to-left operation.
"$over": { "$op":"get", "$path":"@.Records" },
// The initial value defines the data type of the result;
// in this case it will be an array.
// An the initial value is the empty array because it
// serves as the neutral element of array concatenation.
"$init": [],
// Step function, which actually is the filter.
"$step": {
// Condition to filter: value <= 6.
"$op": "if",
"$cond":
// LTE (less-than-or-equal) between current item's Value and
// constant 6 (literal number).
{ "$op" : "lte",
"$left" : { "$op":"get", "$path":"@.item.Value"},
"$right" : 6 },
// If passes the filter, then the current item is inserted at
// the beginning of the current aggregation.
"$then":
{ "$op" : "cons",
"$head" : { "$op":"get", "$path":"@.item" },
"$tail" : { "$op":"get", "$path":"@.acc" } },
// Otherwise, just pass the current aggregation to the next step.
"$else":
{ "$op":"get", "$path":"@.acc" }
}
}
}
}
Aggregation
Then, we can include the aggregation operation (sometimes called reduce or fold) using the appropriate template commands if and add:
{
// The root template, always mandatory!
"$": {
// Literal property name, which copies its property name.
"Name": { "$op":"get", "$path":"@.Name" },
// Literal property name, which iterates each item (foldArr)
// filtering using a condition... and then aggregates them all
// (reducing them to just a number).
"SumOfRecordValues":
{ "$op" : "foldArr",
// Array to iterate over; remember that 'foldArr'
// is a from-right-to-left operation.
"$over": { "$op":"get", "$path":"@.Records" },
// Because the final result will be a number, we need the
// neutral element for the addition of numbers, that is, zero.
"$init": 0 ,
"$step": {
// In this case, we are composing two operations: 'if'
// and 'add'; this way, we are making just one pass.
"$op": "if",
// First, we filter: value <= 6.
"$cond":
{ "$op" : "gt",
"$left" : { "$op":"get", "$path":"@.item.Value"},
"$right" : 6 },
// Second, but at the same time, we aggregate
// the value from the filtered items.
"$then":
{ "$op" : "add",
"$left" : { "$op":"get", "$path":"@.item.Value" },
"$right" : { "$op":"get", "$path":"@.acc" } },
"$else":
{ "$op":"get", "$path":"@.acc" }
}
}
}
}
The JavaScript code equivalent to filtering all records whose value is strictly greater than 6 and then adding all their values to return a single aggregate result would be as follows:
// The algorithm actually defined using JM2MP that
// requires only a single pass.
const expected_document = {
Name: source_document.Name,
SumOfRecordValues: source_document
.Records
.reduceRight( (acc, i)=>( (i.Value > 6)
? (acc + i.Value)
: acc ),
0 )
};
// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduceRight
Composition
However, another way to compose (combine) filtering and aggregation operations (although potentially less efficient, since each operation requires its own pass over the results) could be as follows:
// Another similar algorithm, but it requires two passes:
// filtering and reduction.
const expected_document_two_passes = {
Name: source_document.Name,
SumOfRecordValues: source_document
.Records
.filter( (i)=>(i.Value > 6) )
.reduceRight( (acc, i)=>(acc + i.Value),
0 )
};
which JM2MP equivalence involves the use of the
pipe template command:
{
// The root template, always mandatory!
"$": {
// Literal property name, which copies its property name.
"Name": { "$op":"get", "$path":"@.Name" },
// Literal property name for the final combined result (pipe).
"SumOfRecordValues": {
"$op" : "pipe",
"$stages" : [
// First stage: filtering (i-th.Value > 6).
{
"$op" : "foldArr",
"$over": { "$op":"get", "$path":"@.Records" },
// The resultant JSON value from
// this stage will be an array.
"$init": [],
"$step": {
// Actual filter.
"$op": "if",
"$cond": {
"$op" : "gt",
"$left" : { "$op":"get", "$path":"@.item.Value"},
"$right" : 6
},
// Preliminary results.
"$then": {
"$op" : "cons",
"$head" : { "$op":"get", "$path":"@.item" },
"$tail" : { "$op":"get", "$path":"@.acc" }
},
"$else": {
"$op":"get", "$path":"@.acc"
}
}
},
// Second stage: aggregation (0 + acc + i-th.Value).
{
"$op" : "foldArr",
// Its input will be the output from the previous stage;
// that is, the current context.
"$over": { "$op":"get", "$path":"@" },
"$init": 0,
"$step": {
"$op": "if",
"$cond": {
"$op" : "gt",
"$left" : { "$op":"get", "$path":"@.item.Value"},
"$right" : 6
},
"$then": {
"$op" : "add",
"$left" : { "$op":"get", "$path":"@.item.Value" },
"$right" : { "$op":"get", "$path":"@.acc" }
},
"$else":
{ "$op":"get", "$path":"@.acc" }
}
}
]
}
}
}