jq

jq linux command cheatsheet by Thamizhiniyan C S

Introduction

jq is a tool for processing JSON inputs, applying the given filter to its JSON text inputs and producing the filter's results as JSON on standard output.

---

Syntax

jq [options...] filter [files...]

---

Installation

For Installing on a Debian based OS or Ubuntu use the following command:

sudo apt-get install jq

---

Important Flags

Flag	Description
-c	Compact instead of pretty-printed output.
-n	Use `null` as the single input value.
-e	Set the exit status code based on the output.
-s	Read (slurp) all inputs into an array; apply filter to it.
-r	Output raw strings, not JSON texts. (prints without quotes)
-R	Read raw strings, not JSON texts.
-C	Colorize JSON.
-M	Monochrome (don't colorize JSON).
-S	Sort keys of objects on output.
--tab	Use tabs for indentation.
--arg a v	Set variable $a to value .
--argjson a v	Set variable $a to JSON value .
--slurpfile a f	Set variable $a to an array of JSON texts read from .
--rawfile a f	Set variable $a to a string consisting of the contents of .
--args	Remaining arguments are string arguments, not files.
--jsonargs	Remaining arguments are JSON arguments, not files.
--	Terminates argument processing.

---

jq Filters

Source: https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4

Basic concepts

The syntax for jq is pretty coherent:

Syntax	Description
,	Filters separated by a comma will produce multiple independent outputs
?	Will ignores error if the type is unexpected
[]	Array construction
{}	Object construction
+	Concatenate or Add
-	Difference of sets or Substract
length	Size of selected element
|	Pipes are used to chain commands in a similar fashion than bash

Dealing with json objects

Description	Command
Display all keys	jq 'keys'
Adds + 1 to all items	jq 'map_values(.+1)'
Delete a key	jq 'del(.foo)'
Convert an object to array	to_entries | map([.key, .value])

Dealing with fields

Description	Command
Concatenate two fields	fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

Description	Command
All	jq .[]
First	jq '.[0]'
Range	jq '.[2:4]'
First 3	jq '.[:3]'
Last 2	jq '.[-2:]'
Before Last	jq '.[-2]'
Select array of int by value	jq 'map(select(. >= 2))'
Select array of objects by value	jq '.[] | select(.id == "second")'
Select by type	jq '.[] | numbers' with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

Description	Command
Add + 1 to all items	jq 'map(.+1)'
Delete 2 items	jq 'del(.[1, 2])'
Concatenate arrays	jq 'add'
Flatten an array	jq 'flatten'
Create a range of numbers	jq '[range(2;4)]'
Display the type of each item	jq 'map(type)'
Sort an array of basic type	jq 'sort'
Sort an array of objects	jq 'sort_by(.foo)'
Group by a key - opposite to flatten	jq 'group_by(.foo)'
Minimun value of an array	jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)
Remove duplicates	jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'
Reverse an array	jq 'reverse'

---

Examples

In the given examples, the following JSON data is used which is stored in a file "json.json".

[{"layout":"en-gb","textType":"random","timeStamp":"2023-05-11T10:15:20.000Z","length":145,"time":31025,"errors":5,"speed":193.4567820395721,"histogram":[{"codePoint":32,"hitCount":20,"missCount":1,"timeToType":257},{"codePoint":101,"hitCount":29,"missCount":2,"timeToType":198},{"codePoint":105,"hitCount":18,"missCount":0,"timeToType":215},{"codePoint":108,"hitCount":10,"missCount":1,"timeToType":180},{"codePoint":110,"hitCount":12,"missCount":0,"timeToType":172},{"codePoint":114,"hitCount":7,"missCount":3,"timeToType":327},{"codePoint":116,"hitCount":33,"missCount":1,"timeToType":305}]},{"layout":"en-us","textType":"auto","timeStamp":"2024-01-15T08:25:35.000Z","length":132,"time":28567,"errors":3,"speed":204.2876359102458,"histogram":[{"codePoint":32,"hitCount":19,"missCount":1,"timeToType":263},{"codePoint":101,"hitCount":31,"missCount":0,"timeToType":210},{"codePoint":105,"hitCount":22,"missCount":1,"timeToType":225},{"codePoint":108,"hitCount":14,"missCount":0,"timeToType":190},{"codePoint":110,"hitCount":15,"missCount":0,"timeToType":180},{"codePoint":114,"hitCount":9,"missCount":2,"timeToType":335},{"codePoint":116,"hitCount":34,"missCount":1,"timeToType":315}]},{"layout":"fr-fr","textType":"generated","timeStamp":"2023-07-22T14:45:50.000Z","length":150,"time":32015,"errors":6,"speed":180.9324721938471,"histogram":[{"codePoint":32,"hitCount":21,"missCount":2,"timeToType":269},{"codePoint":101,"hitCount":28,"missCount":1,"timeToType":205},{"codePoint":105,"hitCount":20,"missCount":0,"timeToType":230},{"codePoint":108,"hitCount":13,"missCount":1,"timeToType":185},{"codePoint":110,"hitCount":16,"missCount":0,"timeToType":178},{"codePoint":114,"hitCount":11,"missCount":2,"timeToType":345},{"codePoint":116,"hitCount":32,"missCount":2,"timeToType":325}]},{"layout":"de-de","textType":"random","timeStamp":"2023-03-11T16:35:40.000Z","length":120,"time":27890,"errors":4,"speed":198.4783292857362,"histogram":[{"codePoint":32,"hitCount":17,"missCount":1,"timeToType":250},{"codePoint":101,"hitCount":27,"missCount":0,"timeToType":207},{"codePoint":105,"hitCount":19,"missCount":1,"timeToType":218},{"codePoint":108,"hitCount":11,"missCount":0,"timeToType":182},{"codePoint":110,"hitCount":13,"missCount":0,"timeToType":176},{"codePoint":114,"hitCount":10,"missCount":3,"timeToType":338},{"codePoint":116,"hitCount":30,"missCount":1,"timeToType":310}]},{"layout":"es-es","textType":"generated","timeStamp":"2023-11-05T12:55:25.000Z","length":125,"time":29235,"errors":2,"speed":205.6798234710598,"histogram":[{"codePoint":32,"hitCount":22,"missCount":2,"timeToType":255},{"codePoint":101,"hitCount":26,"missCount":1,"timeToType":215},{"codePoint":105,"hitCount":21,"missCount":0,"timeToType":228},{"codePoint":108,"hitCount":15,"missCount":0,"timeToType":188},{"codePoint":110,"hitCount":18,"missCount":1,"timeToType":170},{"codePoint":114,"hitCount":12,"missCount":2,"timeToType":350},{"codePoint":116,"hitCount":29,"missCount":1,"timeToType":300}]}]

Pretty Printing JSON

Command	Description
cat json.json | jq "."	Using jq along with cat for Pretty printing the JSON
jq '.' json.json	Output a JSON file, in pretty-print format
jq "." < json.json	To pretty print the json

---

Extracting Keys

Command	Description
jq keys json.json	To extract all keys from json
jq '.[0]' json.json | jq keys	To extract all keys from json

---

Array Manipulation

Command	Description
jq '.[]' json.json	Output all elements from arrays (or all key-value pairs from objects) in a JSON file
jq -cM ".[]" json.json | jq --slurp	Read JSON objects from a file into an array, and output it [ Note: In our case the given the given JSON data is already an array of JSON objects. Thus, first I extracted the JSON objects from the array and piped it again to jq to use -slurp, for the sake of demonstration ]
jq '.[] | .layout' json.json	Extract as stream of values instead of a list (array)
jq '.[] | length' json.json	To count elements
jq '[range(2;4)]' json.json	Create a range of numbers

---

Array Indexing

Command	Description
jq '.[0]' json.json	To access first list item
jq '.[0]' json.json	Output the first element in a JSON file
jq '.[0].layout' json.json	Output the value of a given key of the first element in a JSON file

---

Array Slicing

Command	Description
jq '.[1:4] | map(.layout)' json.json	Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as an array
jq '.[1:4] | .[] | .layout' json.json	Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as stream
jq '.[2:4]' json.json	Slicing example 1
jq '.[:3]' json.json	Slicing example 2
jq '.[-2:]' json.json	Slicing example 3

---

Filtering and Manipulating Data

Command	Description
jq 'add' json.json	Concatenate arrays
jq 'del(.[1, 2])' json.json	Delete 2 items
jq '.[] | select(.layout == "en-gb")' json.json	Only print records where given field matches a value
jq 'group_by(.textType)' json.json	Group by a key - opposite to flatten
jq 'min' json.json	Minimum value of an array
jq 'reverse' json.json	Reverse an array

---

Mapping

Command	Description
jq 'map(type)' json.json	Display the type of each item
jq 'map(.layout)' json.json	Output the value of a given key of each element in a JSON file
jq 'map({ layout, textType })' json.json	Dictionary subset shorthand
jq 'map(select(.textType == "random"))' json.json	Filter a list of objects
jq 'map(.length + 1)' json.json	Add + 1 to all items

---

Sorting

Command	Description
jq 'sort' json.json	Sort an array of basic type
jq 'sort_by(.length)' json.json	Sort an array of objects
jq --slurp '. | sort | .[]' json.json	Sort lines of a file

---

Unique

Command	Description
jq 'unique' json.json	Remove duplicates
jq 'unique_by(.layout)' json.json	Remove duplicates by key
jq 'unique_by(length)' json.json	Remove duplicates by length

---

Converting / Parsing / Serializing

Command	Description
jq 'flatten' json.json	Flatten an array
jq 'with_entries(.value |= fromjson)' --sort-keys	Parsing json
jq 'with_entries(.value |= tojson)' --sort-keys	Serializing json

---

Converting to CSV

Command	Description
jq '.[] | [.layout, .length] | @csv' -r json.json	Converting to csv
jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed] | @csv' json.json	This command takes each object in the JSON array, extracts the specified fields, and formats them as CSV.
jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed, (.histogram | map([.codePoint, .hitCount, .missCount, .timeToType] | join(",")) | join(";"))] | @csv' json.json	Same as the previous command but also includes histogram (nested array) data

---

Formatting

Command	Description
jq '.[] | {layout, length}' json.json	Print only selected fields
jq '.[] | {layout, length} | join(" ")' json.json	Print selected fields as text instead of json

---

URL Encoding

Command	Description
jq -sRr @uri json.json	URL Encode the data

---

Exporting Data / Environment Variables

Command	Description
export layout="a shell variable or an env" && jq -n --arg layout "$layout" '{"layout":$layout}'	To create proper JSON from a shell script and properly escape variables
export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"')	To fill environment variables from JSON object key
export $(jq -r '.[0] | "LAYOUT=\(.layout) LENGTH=\(.length)"' json.json)	To fill environment variables from the JSON file

---

With cat Command

Command	Description
cat json.json | jq --from-file script.jq	Execute a specific script
cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'	Pass specific arguments
cat path/to/file.json | jq '.key1, .key2, ...'	Print specific keys
cat path/to/file.json | jq '.[index1], .[index2], ...'	Print specific array items
cat path/to/file.json | jq '.[]	Print all array items/object keys
cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'	Add/remove specific keys