# jq

## Introduction

jq is a tool for processing JSON inputs, applying the given filter to its JSON text inputs and producing the filter's results as JSON on standard output.

***

## Syntax

`jq [options...] filter [files...]`

***

## Installation

For Installing on a Debian based OS or Ubuntu use the following command:

```bash
sudo apt-get install jq
```

***

## Important Flags

| Flag              | Description                                                 |
| ----------------- | ----------------------------------------------------------- |
| `-c`              | Compact instead of pretty-printed output.                   |
| `-n`              | Use \`null\` as the single input value.                     |
| `-e`              | Set the exit status code based on the output.               |
| `-s`              | Read (slurp) all inputs into an array; apply filter to it.  |
| `-r`              | Output raw strings, not JSON texts. (prints without quotes) |
| `-R`              | Read raw strings, not JSON texts.                           |
| `-C`              | Colorize JSON.                                              |
| `-M`              | Monochrome (don't colorize JSON).                           |
| `-S`              | Sort keys of objects on output.                             |
| `--tab`           | Use tabs for indentation.                                   |
| `--arg a v`       | Set variable $a to value .                                  |
| `--argjson a v`   | Set variable $a to JSON value .                             |
| `--slurpfile a f` | Set variable $a to an array of JSON texts read from .       |
| `--rawfile a f`   | Set variable $a to a string consisting of the contents of . |
| `--args`          | Remaining arguments are string arguments, not files.        |
| `--jsonargs`      | Remaining arguments are JSON arguments, not files.          |
| `--`              | Terminates argument processing.                             |

***

## jq Filters

Source: <https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4>

### Basic concepts

The syntax for jq is pretty coherent:

<table><thead><tr><th width="207">Syntax</th><th>Description</th></tr></thead><tbody><tr><td><code>,</code></td><td>Filters separated by a comma will produce multiple independent outputs</td></tr><tr><td><code>?</code></td><td>Will ignores error if the type is unexpected</td></tr><tr><td><code>[]</code></td><td>Array construction</td></tr><tr><td><code>{}</code></td><td>Object construction</td></tr><tr><td><code>+</code></td><td>Concatenate or Add</td></tr><tr><td><code>-</code></td><td>Difference of sets or Substract</td></tr><tr><td><code>length</code></td><td>Size of selected element</td></tr><tr><td><code>|</code></td><td>Pipes are used to chain commands in a similar fashion than bash</td></tr></tbody></table>

### Dealing with json objects

| Description                |                 Command                 |
| -------------------------- | :-------------------------------------: |
| Display all keys           |               `jq 'keys'`               |
| Adds + 1 to all items      |          `jq 'map_values(.+1)'`         |
| Delete a key               |             `jq 'del(.foo)'`            |
| Convert an object to array | `to_entries &#124; map([.key, .value])` |

### Dealing with fields

| Description            |             Command            |
| ---------------------- | :----------------------------: |
| Concatenate two fields | `fieldNew=.field1+' '+.field2` |

### Dealing with json arrays

#### Slicing and Filtering

<table><thead><tr><th width="306">Description</th><th>Command</th></tr></thead><tbody><tr><td>All</td><td><code>jq .[]</code></td></tr><tr><td>First</td><td><code>jq '.[0]'</code></td></tr><tr><td>Range</td><td><code>jq '.[2:4]'</code></td></tr><tr><td>First 3</td><td><code>jq '.[:3]'</code></td></tr><tr><td>Last 2</td><td><code>jq '.[-2:]'</code></td></tr><tr><td>Before Last</td><td><code>jq '.[-2]'</code></td></tr><tr><td>Select array of int by value</td><td><code>jq 'map(select(. >= 2))'</code></td></tr><tr><td>Select array of objects by value</td><td><code>jq '.[] | select(.id == "second")'</code></td></tr><tr><td>Select by type</td><td><code>jq '.[] | numbers'</code>  with type been <code>arrays</code>, <code>objects</code>, <code>iterables</code>, <code>booleans</code>, <code>numbers</code>, <code>normals</code>, <code>finites</code>, <code>strings</code>, <code>nulls</code>, <code>values</code>, <code>scalars</code></td></tr></tbody></table>

#### Mapping and Transforming

| Description                          | Command                                                               |
| ------------------------------------ | --------------------------------------------------------------------- |
| Add + 1 to all items                 | `jq 'map(.+1)'`                                                       |
| Delete 2 items                       | `jq 'del(.[1, 2])'`                                                   |
| Concatenate arrays                   | `jq 'add'`                                                            |
| Flatten an array                     | `jq 'flatten'`                                                        |
| Create a range of numbers            | `jq '[range(2;4)]'`                                                   |
| Display the type of each item        | `jq 'map(type)'`                                                      |
| Sort an array of basic type          | `jq 'sort'`                                                           |
| Sort an array of objects             | `jq 'sort_by(.foo)'`                                                  |
| Group by a key - opposite to flatten | `jq 'group_by(.foo)'`                                                 |
| Minimun value of an array            | `jq 'min'` .See also min, max, min\_by(path\_exp), max\_by(path\_exp) |
| Remove duplicates                    | `jq 'unique'` or `jq 'unique_by(.foo)'` or `jq 'unique_by(length)'`   |
| Reverse an array                     | `jq 'reverse'`                                                        |

***

## Examples

In the given examples, the following JSON data is used which is stored in a file "json.json".

```
[{"layout":"en-gb","textType":"random","timeStamp":"2023-05-11T10:15:20.000Z","length":145,"time":31025,"errors":5,"speed":193.4567820395721,"histogram":[{"codePoint":32,"hitCount":20,"missCount":1,"timeToType":257},{"codePoint":101,"hitCount":29,"missCount":2,"timeToType":198},{"codePoint":105,"hitCount":18,"missCount":0,"timeToType":215},{"codePoint":108,"hitCount":10,"missCount":1,"timeToType":180},{"codePoint":110,"hitCount":12,"missCount":0,"timeToType":172},{"codePoint":114,"hitCount":7,"missCount":3,"timeToType":327},{"codePoint":116,"hitCount":33,"missCount":1,"timeToType":305}]},{"layout":"en-us","textType":"auto","timeStamp":"2024-01-15T08:25:35.000Z","length":132,"time":28567,"errors":3,"speed":204.2876359102458,"histogram":[{"codePoint":32,"hitCount":19,"missCount":1,"timeToType":263},{"codePoint":101,"hitCount":31,"missCount":0,"timeToType":210},{"codePoint":105,"hitCount":22,"missCount":1,"timeToType":225},{"codePoint":108,"hitCount":14,"missCount":0,"timeToType":190},{"codePoint":110,"hitCount":15,"missCount":0,"timeToType":180},{"codePoint":114,"hitCount":9,"missCount":2,"timeToType":335},{"codePoint":116,"hitCount":34,"missCount":1,"timeToType":315}]},{"layout":"fr-fr","textType":"generated","timeStamp":"2023-07-22T14:45:50.000Z","length":150,"time":32015,"errors":6,"speed":180.9324721938471,"histogram":[{"codePoint":32,"hitCount":21,"missCount":2,"timeToType":269},{"codePoint":101,"hitCount":28,"missCount":1,"timeToType":205},{"codePoint":105,"hitCount":20,"missCount":0,"timeToType":230},{"codePoint":108,"hitCount":13,"missCount":1,"timeToType":185},{"codePoint":110,"hitCount":16,"missCount":0,"timeToType":178},{"codePoint":114,"hitCount":11,"missCount":2,"timeToType":345},{"codePoint":116,"hitCount":32,"missCount":2,"timeToType":325}]},{"layout":"de-de","textType":"random","timeStamp":"2023-03-11T16:35:40.000Z","length":120,"time":27890,"errors":4,"speed":198.4783292857362,"histogram":[{"codePoint":32,"hitCount":17,"missCount":1,"timeToType":250},{"codePoint":101,"hitCount":27,"missCount":0,"timeToType":207},{"codePoint":105,"hitCount":19,"missCount":1,"timeToType":218},{"codePoint":108,"hitCount":11,"missCount":0,"timeToType":182},{"codePoint":110,"hitCount":13,"missCount":0,"timeToType":176},{"codePoint":114,"hitCount":10,"missCount":3,"timeToType":338},{"codePoint":116,"hitCount":30,"missCount":1,"timeToType":310}]},{"layout":"es-es","textType":"generated","timeStamp":"2023-11-05T12:55:25.000Z","length":125,"time":29235,"errors":2,"speed":205.6798234710598,"histogram":[{"codePoint":32,"hitCount":22,"missCount":2,"timeToType":255},{"codePoint":101,"hitCount":26,"missCount":1,"timeToType":215},{"codePoint":105,"hitCount":21,"missCount":0,"timeToType":228},{"codePoint":108,"hitCount":15,"missCount":0,"timeToType":188},{"codePoint":110,"hitCount":18,"missCount":1,"timeToType":170},{"codePoint":114,"hitCount":12,"missCount":2,"timeToType":350},{"codePoint":116,"hitCount":29,"missCount":1,"timeToType":300}]}]
```

### Pretty Printing JSON

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat json.json | jq "."
</code></pre></td><td>Using jq along with cat for Pretty printing the JSON</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.' json.json
</code></pre></td><td>Output a JSON file, in pretty-print format</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq "." &#x3C; json.json
</code></pre></td><td>To pretty print the json</td></tr></tbody></table>

***

### Extracting Keys

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq keys json.json
</code></pre></td><td>To extract all keys from json</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[0]' json.json | jq keys
</code></pre></td><td>To extract all keys from json</td></tr></tbody></table>

***

### Array Manipulation

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[]' json.json
</code></pre></td><td>Output all elements from arrays (or all key-value pairs from objects) in a JSON file</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq -cM ".[]" json.json | jq --slurp
</code></pre></td><td>Read JSON objects from a file into an array, and output it [ Note: In our case the given the given JSON data is already an array of JSON objects. Thus, first I extracted the JSON objects from the array and piped it again to jq to use -slurp, for the sake of demonstration ]</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | .layout' json.json
</code></pre></td><td>Extract as stream of values instead of a list (array)</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | length' json.json
</code></pre></td><td>To count elements</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '[range(2;4)]' json.json
</code></pre></td><td>Create a range of numbers</td></tr></tbody></table>

***

### Array Indexing

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[0]' json.json
</code></pre></td><td>To access first list item</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[0]' json.json
</code></pre></td><td>Output the first element in a JSON file</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[0].layout' json.json
</code></pre></td><td>Output the value of a given key of the first element in a JSON file</td></tr></tbody></table>

***

### Array Slicing

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[1:4] | map(.layout)' json.json
</code></pre></td><td>Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as an array</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[1:4] | .[] | .layout' json.json
</code></pre></td><td>Slicing the elements from 1 to 4 ( i.e, 1,2,3 ) and printing the value of the layout key as stream</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[2:4]' json.json
</code></pre></td><td>Slicing example 1</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[:3]' json.json
</code></pre></td><td>Slicing example 2</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[-2:]' json.json
</code></pre></td><td>Slicing example 3</td></tr></tbody></table>

***

### Filtering and Manipulating Data

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'add' json.json
</code></pre></td><td>Concatenate arrays</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'del(.[1, 2])' json.json
</code></pre></td><td>Delete 2 items</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | select(.layout == "en-gb")' json.json
</code></pre></td><td>Only print records where given field matches a value</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'group_by(.textType)' json.json
</code></pre></td><td>Group by a key - opposite to flatten</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'min' json.json
</code></pre></td><td>Minimum value of an array</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'reverse' json.json
</code></pre></td><td>Reverse an array</td></tr></tbody></table>

***

### Mapping

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'map(type)' json.json
</code></pre></td><td>Display the type of each item</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'map(.layout)' json.json
</code></pre></td><td>Output the value of a given key of each element in a JSON file</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'map({ layout, textType })' json.json
</code></pre></td><td>Dictionary subset shorthand</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'map(select(.textType == "random"))' json.json
</code></pre></td><td>Filter a list of objects</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'map(.length + 1)' json.json
</code></pre></td><td>Add + 1 to all items</td></tr></tbody></table>

***

### Sorting

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'sort' json.json
</code></pre></td><td>Sort an array of basic type</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'sort_by(.length)' json.json
</code></pre></td><td>Sort an array of objects</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq --slurp '. | sort | .[]' json.json
</code></pre></td><td>Sort lines of a file</td></tr></tbody></table>

***

### Unique

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'unique' json.json
</code></pre></td><td>Remove duplicates</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'unique_by(.layout)' json.json
</code></pre></td><td>Remove duplicates by key</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'unique_by(length)' json.json
</code></pre></td><td>Remove duplicates by length</td></tr></tbody></table>

***

### Converting / Parsing / Serializing

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'flatten' json.json
</code></pre></td><td>Flatten an array</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'with_entries(.value |= fromjson)' --sort-keys
</code></pre></td><td>Parsing json</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq 'with_entries(.value |= tojson)' --sort-keys
</code></pre></td><td>Serializing json</td></tr></tbody></table>

***

### Converting to CSV

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | [.layout, .length] | @csv' -r json.json
</code></pre></td><td>Converting to csv</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed] | @csv' json.json
</code></pre></td><td>This command takes each object in the JSON array, extracts the specified fields, and formats them as CSV.</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq -r '.[] | [.layout, .textType, .timeStamp, .length, .time, .errors, .speed, (.histogram | map([.codePoint, .hitCount, .missCount, .timeToType] | join(",")) | join(";"))] | @csv' json.json
</code></pre></td><td>Same as the previous command but also includes histogram (nested array) data</td></tr></tbody></table>

***

### Formatting

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | {layout, length}' json.json
</code></pre></td><td>Print only selected fields</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq '.[] | {layout, length} | join(" ")' json.json
</code></pre></td><td>Print selected fields as text instead of json</td></tr></tbody></table>

***

### URL Encoding

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">jq -sRr @uri json.json
</code></pre></td><td>URL Encode the data</td></tr></tbody></table>

***

### Exporting Data / Environment Variables

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">export layout="a shell variable or an env" &#x26;&#x26; jq -n --arg layout "$layout" '{"layout":$layout}'
</code></pre></td><td>To create proper JSON from a shell script and properly escape variables</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">export $(jq -r '@sh "FOO=\(.foo) BAZ=\(.baz)"')
</code></pre></td><td>To fill environment variables from JSON object key</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">export $(jq -r '.[0] | "LAYOUT=\(.layout) LENGTH=\(.length)"' json.json)
</code></pre></td><td>To fill environment variables from the JSON file</td></tr></tbody></table>

***

### With cat Command

<table><thead><tr><th>Command</th><th>Description</th></tr></thead><tbody><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat json.json | jq --from-file script.jq
</code></pre></td><td>Execute a specific script</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'
</code></pre></td><td>Pass specific arguments</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat path/to/file.json | jq '.key1, .key2, ...'
</code></pre></td><td>Print specific keys</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat path/to/file.json | jq '.[index1], .[index2], ...'
</code></pre></td><td>Print specific array items</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat path/to/file.json | jq '.[]
</code></pre></td><td>Print all array items/object keys</td></tr><tr><td><pre class="language-bash" data-overflow="wrap"><code class="lang-bash">cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'
</code></pre></td><td>Add/remove specific keys</td></tr></tbody></table>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://thamizhiniyancs.gitbook.io/cheatsheets/linux/strops-text-manipulation/jq.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.