Linux
HomeWriteupsResourcesCheatsheets

jq

jq linux command cheatsheet by Thamizhiniyan C S

Introduction

jq is a tool for processing JSON inputs, applying the given filter to its JSON text inputs and producing the filter's results as JSON on standard output.

Syntax

jq [options...] filter [files...]

Installation

For Installing on a Debian based OS or Ubuntu use the following command:

sudo apt-get install jq

Important Flags

FlagDescription

-c

Compact instead of pretty-printed output.

-n

Use `null` as the single input value.

-e

Set the exit status code based on the output.

-s

Read (slurp) all inputs into an array; apply filter to it.

-r

Output raw strings, not JSON texts. (prints without quotes)

-R

Read raw strings, not JSON texts.

-C

Colorize JSON.

-M

Monochrome (don't colorize JSON).

-S

Sort keys of objects on output.

--tab

Use tabs for indentation.

--arg a v

Set variable $a to value .

--argjson a v

Set variable $a to JSON value .

--slurpfile a f

Set variable $a to an array of JSON texts read from .

--rawfile a f

Set variable $a to a string consisting of the contents of .

--args

Remaining arguments are string arguments, not files.

--jsonargs

Remaining arguments are JSON arguments, not files.

--

Terminates argument processing.

jq Filters

Source: https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4

Basic concepts

The syntax for jq is pretty coherent:

SyntaxDescription

,

Filters separated by a comma will produce multiple independent outputs

?

Will ignores error if the type is unexpected

[]

Array construction

{}

Object construction

+

Concatenate or Add

-

Difference of sets or Substract

length

Size of selected element

|

Pipes are used to chain commands in a similar fashion than bash

Dealing with json objects

DescriptionCommand

Display all keys

jq 'keys'

Adds + 1 to all items

jq 'map_values(.+1)'

Delete a key

jq 'del(.foo)'

Convert an object to array

to_entries | map([.key, .value])

Dealing with fields

DescriptionCommand

Concatenate two fields

fieldNew=.field1+' '+.field2

Dealing with json arrays

Slicing and Filtering

DescriptionCommand

All

jq .[]

First

jq '.[0]'

Range

jq '.[2:4]'

First 3

jq '.[:3]'

Last 2

jq '.[-2:]'

Before Last

jq '.[-2]'

Select array of int by value

jq 'map(select(. >= 2))'

Select array of objects by value

jq '.[] | select(.id == "second")'

Select by type

jq '.[] | numbers' with type been arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars

Mapping and Transforming

DescriptionCommand

Add + 1 to all items

jq 'map(.+1)'

Delete 2 items

jq 'del(.[1, 2])'

Concatenate arrays

jq 'add'

Flatten an array

jq 'flatten'

Create a range of numbers

jq '[range(2;4)]'

Display the type of each item

jq 'map(type)'

Sort an array of basic type

jq 'sort'

Sort an array of objects

jq 'sort_by(.foo)'

Group by a key - opposite to flatten

jq 'group_by(.foo)'

Minimun value of an array

jq 'min' .See also min, max, min_by(path_exp), max_by(path_exp)

Remove duplicates

jq 'unique' or jq 'unique_by(.foo)' or jq 'unique_by(length)'

Reverse an array

jq 'reverse'

Examples

In the given examples, the following JSON data is used which is stored in a file "json.json".

[{"layout":"en-gb","textType":"random","timeStamp":"2023-05-11T10:15:20.000Z","length":145,"time":31025,"errors":5,"speed":193.4567820395721,"histogram":[{"codePoint":32,"hitCount":20,"missCount":1,"timeToType":257},{"codePoint":101,"hitCount":29,"missCount":2,"timeToType":198},{"codePoint":105,"hitCount":18,"missCount":0,"timeToType":215},{"codePoint":108,"hitCount":10,"missCount":1,"timeToType":180},{"codePoint":110,"hitCount":12,"missCount":0,"timeToType":172},{"codePoint":114,"hitCount":7,"missCount":3,"timeToType":327},{"codePoint":116,"hitCount":33,"missCount":1,"timeToType":305}]},{"layout":"en-us","textType":"auto","timeStamp":"2024-01-15T08:25:35.000Z","length":132,"time":28567,"errors":3,"speed":204.2876359102458,"histogram":[{"codePoint":32,"hitCount":19,"missCount":1,"timeToType":263},{"codePoint":101,"hitCount":31,"missCount":0,"timeToType":210},{"codePoint":105,"hitCount":22,"missCount":1,"timeToType":225},{"codePoint":108,"hitCount":14,"missCount":0,"timeToType":190},{"codePoint":110,"hitCount":15,"missCount":0,"timeToType":180},{"codePoint":114,"hitCount":9,"missCount":2,"timeToType":335},{"codePoint":116,"hitCount":34,"missCount":1,"timeToType":315}]},{"layout":"fr-fr","textType":"generated","timeStamp":"2023-07-22T14:45:50.000Z","length":150,"time":32015,"errors":6,"speed":180.9324721938471,"histogram":[{"codePoint":32,"hitCount":21,"missCount":2,"timeToType":269},{"codePoint":101,"hitCount":28,"missCount":1,"timeToType":205},{"codePoint":105,"hitCount":20,"missCount":0,"timeToType":230},{"codePoint":108,"hitCount":13,"missCount":1,"timeToType":185},{"codePoint":110,"hitCount":16,"missCount":0,"timeToType":178},{"codePoint":114,"hitCount":11,"missCount":2,"timeToType":345},{"codePoint":116,"hitCount":32,"missCount":2,"timeToType":325}]},{"layout":"de-de","textType":"random","timeStamp":"2023-03-11T16:35:40.000Z","length":120,"time":27890,"errors":4,"speed":198.4783292857362,"histogram":[{"codePoint":32,"hitCount":17,"missCount":1,"timeToType":250},{"codePoint":101,"hitCount":27,"missCount":0,"timeToType":207},{"codePoint":105,"hitCount":19,"missCount":1,"timeToType":218},{"codePoint":108,"hitCount":11,"missCount":0,"timeToType":182},{"codePoint":110,"hitCount":13,"missCount":0,"timeToType":176},{"codePoint":114,"hitCount":10,"missCount":3,"timeToType":338},{"codePoint":116,"hitCount":30,"missCount":1,"timeToType":310}]},{"layout":"es-es","textType":"generated","timeStamp":"2023-11-05T12:55:25.000Z","length":125,"time":29235,"errors":2,"speed":205.6798234710598,"histogram":[{"codePoint":32,"hitCount":22,"missCount":2,"timeToType":255},{"codePoint":101,"hitCount":26,"missCount":1,"timeToType":215},{"codePoint":105,"hitCount":21,"missCount":0,"timeToType":228},{"codePoint":108,"hitCount":15,"missCount":0,"timeToType":188},{"codePoint":110,"hitCount":18,"missCount":1,"timeToType":170},{"codePoint":114,"hitCount":12,"missCount":2,"timeToType":350},{"codePoint":116,"hitCount":29,"missCount":1,"timeToType":300}]}]

Pretty Printing JSON

CommandDescription
cat json.json | jq "."

Using jq along with cat for Pretty printing the JSON

jq '.' json.json

Output a JSON file, in pretty-print format

jq "." < json.json

To pretty print the json

Extracting Keys

CommandDescription
jq keys json.json

To extract all keys from json

jq '.[0]' json.json | jq keys

To extract all keys from json

Array Manipulation

CommandDescription
jq '.[]' json.json

Output all elements from arrays (or all key-value pairs from objects) in a JSON file

jq -cM ".[]" json.json | jq --slurp

Read JSON objects from a file into an array, and output it [ Note: In our case the given the given JSON data is already an array of JSON objects. Thus, first I extracted the JSON objects from the array and piped it again to jq to use -slurp, for the sake of demonstration ]

jq '.[] | .layout' json.json

Extract as stream of values instead of a list (array)

jq '.[] | length' json.json

To count elements

jq '[range(2;4)]' json.json

Create a range of numbers

Array Indexing

CommandDescription
jq '.[0]' json.json

To access first list item

jq '.[0]' json.json

Output the first element in a JSON file

jq '.[0].layout' json.json

Output the value of a given key of the first element in a JSON file

Array Slicing

CommandDescription
jq '.[1:4] | map(.layout)' json.json

Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as an array

jq '.[1:4] | .[] | .layout' json.json

Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as stream

jq '.[2:4]' json.json

Slicing example 1

jq '.[:3]' json.json

Slicing example 2

jq '.[-2:]' json.json

Slicing example 3

Filtering and Manipulating Data

CommandDescription
jq 'add' json.json

Concatenate arrays

jq 'del(.[1, 2])' json.json

Delete 2 items

jq '.[] | select(.layout == "en-gb")' json.json

Only print records where given field matches a value

jq 'group_by(.textType)' json.json

Group by a key - opposite to flatten

jq 'min' json.json

Minimum value of an array

jq 'reverse' json.json

Reverse an array

Mapping

CommandDescription
jq 'map(type)' json.json

Display the type of each item

jq 'map(.layout)' json.json

Output the value of a given key of each element in a JSON file

jq 'map({ layout, textType })' json.json

Dictionary subset shorthand

jq 'map(select(.textType == "random"))' json.json

Filter a list of objects

jq 'map(.length + 1)' json.json

Add + 1 to all items

Sorting

CommandDescription
jq 'sort' json.json

Sort an array of basic type

jq 'sort_by(.length)' json.json

Sort an array of objects

jq --slurp '. | sort | .[]' json.json

Sort lines of a file

Unique

CommandDescription
jq 'unique' json.json

Remove duplicates

jq 'unique_by(.layout)' json.json

Remove duplicates by key

jq 'unique_by(length)' json.json

Remove duplicates by length

Converting / Parsing / Serializing

CommandDescription
jq 'flatten' json.json

Flatten an array

jq 'with_entries(.value |= fromjson)' --sort-keys

Parsing json

jq 'with_entries(.value |= tojson)' --sort-keys

Serializing json

Converting to CSV

CommandDescription
jq '.[] | [.layout, .length] | @csv' -r json.json

Converting to csv

jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed] | @csv' json.json

This command takes each object in the JSON array, extracts the specified fields, and formats them as CSV.

jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed, (.histogram | map([.codePoint, .hitCount, .missCount, .timeToType] | join(",")) | join(";"))] | @csv' json.json

Same as the previous command but also includes histogram (nested array) data

Formatting

CommandDescription
jq '.[] | {layout, length}' json.json

Print only selected fields

jq '.[] | {layout, length} | join(" ")' json.json

Print selected fields as text instead of json

URL Encoding

CommandDescription
jq -sRr @uri json.json

URL Encode the data

Exporting Data / Environment Variables

CommandDescription
export layout="a shell variable or an env" && jq -n --arg layout "$layout" '{"layout":$layout}'

To create proper JSON from a shell script and properly escape variables

export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"')

To fill environment variables from JSON object key

export $(jq -r '.[0] | "LAYOUT=\(.layout) LENGTH=\(.length)"' json.json)

To fill environment variables from the JSON file

With cat Command

CommandDescription
cat json.json | jq --from-file script.jq

Execute a specific script

cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'

Pass specific arguments

cat path/to/file.json | jq '.key1, .key2, ...'

Print specific keys

cat path/to/file.json | jq '.[index1], .[index2], ...'

Print specific array items

cat path/to/file.json | jq '.[]

Print all array items/object keys

cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'

Add/remove specific keys

PreviousgrepNextjoin

Last updated