Many current tools and web services use JSON format for both input and output data. The jq tool proves very useful for extracting and processing JSON data for single-use situations or small scripts, especially shell scripts. Recently for a client I needed to create a small script to process API responses from Apache Marathon container orchestration system to identify any deployments exceeding a certain time limit. I took the opportunity to use the development of the script to present a brief tutorial on the use and capabilities of jq.
jq operates on the pipeline principle similar to many other UNIX utilities. However instead of single lines of text, the unit for jq is a JSON object. PIpeline operations can be chained together to extract data, construct summaries, or build new JSON objects and structures.
The Marathon API call "/v2/deployments" returns a JSON array of hashes, each representing a deployment in progress. Below is a sample Marathon JSON output. You may wish to adjust the dates in the "version" field so one or more of the deployments is not past the 30 minute "overdue" threshold.
[
{
"id": "bc12d628-f668-41af-8a3c-050aba1c8cb8",
"version": "2018-01-27T00:22:10.905Z",
"affectedApps": [
"/dev/serviceapp1"
],
"affectedPods": [],
"steps": [
{
"actions": [
{
"action": "ScaleApplication",
"app": "/dev/serviceapp1"
}
]
}
],
"currentActions": [
{
"action": "ScaleApplication",
"app": "/dev/serviceapp1",
"readinessCheckResults": []
}
],
"currentStep": 1,
"totalSteps": 1
},
{
"id": "48bca295-f668-41af-933acf938231c8cb8",
"version": "2018-01-27T05:00:10.905Z",
"affectedApps": [
"/qa/testapp"
],
"affectedPods": [],
"steps": [
{
"actions": [
{
"action": "ScaleApplication",
"app": "/qa/testapp"
}
]
}
],
"currentActions": [
{
"action": "ScaleApplication",
"app": "/qa/testapp",
"readinessCheckResults": []
}
],
"currentStep": 1,
"totalSteps": 1
},
{
"id": "bc12d628-f668-41af-8a3c-4aab392cc34d",
"version": "2018-01-27T00:55:10.905Z",
"affectedApps": [
"/prod/region1/app1a",
"/prod/region1/app1b"
],
"affectedPods": [],
"steps": [
{
"actions": [
{
"action": "ScaleApplication",
"app": "/prod/region1/app1b"
}
]
}
],
"currentActions": [
{
"action": "ScaleApplication",
"app": "/prod/region1/app1b",
"readinessCheckResults": []
}
],
"currentStep": 1,
"totalSteps": 1
}
]
Below is the Bash shell script, including intermediate stages to show the development and use of some jq features:
#!/bin/bash
# Evolution of a jq script to parse Marathon API output to identify
# "stuck" deployments i.e. exceeding 30 minutes
USER_PW="username:password"
URL="http://marathon-hostname:8080/v2/deployments"
input=deployments.json
curl -s -u ${USER_PW} ${URL} > $input
# pretty print JSON data
jq '.' $input
echo "============="
# extract first array element
jq '.[0]' $input
echo "============="
# extract single field from first array element
jq '.[0].version' $input
echo "============="
# extract single field from each array element
jq '.[].version' $input
echo "============="
# To extract multiple fields from an array element, construct a new JSON hash object { }
jq '.[0]|{timestamp: .version, appList: .affectedApps}' $input
echo "============="
# if you want only to extract the fields, retaining the names, you can use this shorthand
jq '.[0]|{version, affectedApps}' $input
echo "============="
# create the same object for each element in the array
# Note the output is not a JSON array, just a list of objects
jq '.[]|{version, affectedApps}|.affectedApps |= join(",")|.version+","+.affectedApps' $input
echo "============="
# to produce a JSON array of hashes, wrap the entire expression in [ ]
jq '[.[]|{version, affectedApps}]' $input
echo "============="
# The |= operator updates the value of a JSON element
# Here we convert the version value to UNIX epoc format in 3 steps
jq '.[]|{version, affectedApps}|.version |= (.[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime)' $input
echo "============="
# Mark deployments that have exceeded 30 minutes by adding
# an "overdue" element to each object with the result
jq '.[]|{version, affectedApps}|.version |= (.[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime)|if .version < (now - 1800) then .overdue = true else .overdue = false end' $input
echo "============="
# Alternatively use the "empty" operator to skip those objects
# that are not overdue
jq '.[]|{version, affectedApps}|.version |= (.[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime)|if .version < (now - 1800) then . else empty end' $input
echo "============="
# We want to use the human readable time format in the final
# output, so move the UNIX epoch conversion into the if
# statement expression
jq '.[]|{version, affectedApps}|if (.version[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime) < (now - 1800) then . else empty end' $input
echo "============="
# Join apps list into one comma delimited string, add some
# explanatory text, add -r option to omit quotes around JSON values
jq -r '.[]|{version, affectedApps}|if (.version[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime) < (now - 1800) then "Deploying since " + .version + ", apps affected: " + (.affectedApps|join(", ")) else empty end' $input
echo "============="
r=$(jq -r '.[]|{version, affectedApps}|if (.version[:-5]|strptime("%Y-%m-%dT%H:%M:%S")|mktime) < (now - 1800) then "Deploying since " + .version + ", Apps affected: " + (.affectedApps|join(", ")) else empty end' $input)
if [[ -n "$r" ]]
then
echo "List of overdue deployments:"
echo "$r"
fi
A small script such as this can't show the entire array of jq features, but should be enough to illustrate some basic concepts and provide a basis for further development. I find this iterative method useful in building up and testing complex operations, it lends itself well to experimentation and debugging.
For more information, see the jq manual