Nowadays, JSON is everywhere, in web apis, in configuration files, even in logs. JSON is the default format used to structure data. Because it is used so much, there will be times when we will need to process JSON from the command line. Could you imagine doing this with grep, sed, or other conventional tools? That would be quite a challenge.
Luckily for us, there is a simple Command-line tool called jq that we can use to query JSON files. It comes with its own language syntax, as we will see in just a few minutes.
First let's install jq with the following command:
sudo apt install jq
Now let's use an example file, a dummy access log in JSON format: access.log, which we can also find in the course GitHub repository.
Let's start with some simple queries:
jq . access.log
We will print the JSON objects back to the screen, in a pretty format:
If we want to grab the request method from each request, run the following:
jq '.requestMethod' access.log
This will print the request method from each json object. Notice the double quotes around each method:
If we want to use the output as input to other scripts we probably don't want the double quotes and that is where the -r (raw output) comes in handy:
jq '.requestMethod' -r access.log
The jq is often used for big data queries at a much smaller scale:
Say, for example, if we want to calculate a statistic of request methods on the log file, we could run the following:
jq '.requestMethod' -r access.log | sort | uniq -c
Now we can see a count of get, put, post, and delete requests. If we want the same type of calculation for another field, say apikey, we can run the following:
jq '.requestHeaders.apikey' -r access.log | sort | uniq -c
Since that the syntax for accessing nested fields is to just use the dot as a delimiter between them. Also notice that we are using single quotes instead of double quotes to mark our query as a string. As you probably know, the difference between single and double quotes in shell scripting is that double-quoted strings will try to expand variables, while single quoted strings will be treated as a fixed string.
To query for the request bodies, we will be using the following command:
jq '.requestBody' access.log
As we can see from the output, even empty request bodies are logged and will be printed by jq:
To skip printing empty bodies, we can use jq's query language to select all documents without an empty body:
jq 'select(.requestBody != {}) | .requestBody' access.log
If we want to refine our search even more and only print the first element in the dataIds object of the request body, use the following:
jq 'select(.requestBody.dataIds[0] != null) | .requestBody.dataIds[0]' access.log
We can even perform arithmetic operations with the returned value, such as incrementing it:
jq 'select(.requestBody.dataIds[0] != null) | .requestBody.dataIds[0] + 1' access.log
There are many more examples and use cases for jq: just go to the official jq page and visit the tutorial there:
https://stedolan.github.io/jq/tutorial/
There we can see an example of consuming a rest API that returns json and pipes it to jq. To print a json with the commit messages from a github repository, run the following:
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq -r '[.[] | {message: .commit.message}]'
As we said, there are many more examples in the documentation, and many more use cases. jq is a pretty powerful tool, and a must when interacting with json from the command line.
The kernel and command line in Linux are stable and powerful. Their reliability has been proven throughout the years, with modern legends about Linux servers running for multiple years in a row without restarting. However, graphical interfaces are not the same, and they sometimes fail or become unresponsive. This can become annoying and it's always good to have a quick way of killing unresponsive windows. Prepare to meet xkill.
First, let's replicate an unresponsive window. Go to the terminal and start gedit: and then hit Ctrl + z. This will send gedit to the background, while the window is still visible. Trying to click inside the window a couple of times will tell Ubuntu that there is no process handling this window anymore and Ubuntu will make it gray:
Hit Ctrl + z:
This will send gedit to the background, while the window is still visible. Trying to click inside the window a couple of times will tell Ubuntu that there is no process handling this window anymore and Ubuntu will make it grey:
To avoid the process of grepping for the pid of the window and then killing that process we use a little trick. Go to the terminal and run the following:
xkill
Now we see that the mouse pointer has changed to an x.
Be careful not to click on anything. Hit Alt + Tab to bring back the gedit window, and then click it. The xkill command will find and kill the process of the window we just clicked on.
This trick can be used on any type of window; it's like shooting your windows!
OK, but what happens if the whole system becomes unresponsive and you can't type anything in the command line? That might happen, especially on older systems. You can hit the on/off button on your laptop or server, but in some circumstances, this is not possible.
What we are going to show you now is an old trick kept secret by Linux gurus for a very long time; nobody really talks about it because it's so powerful that it can do damage in the hands of the wrong people. Please make sure you save all your work and close all programs before trying the fatal keyboard shortcut that will force a restart of your Linux system. Hold down Alt + PrtScrn and at the same time type the following:
reisub
If you've tried it, it means that your computer restarted and you had to come back to this course and continue where you left off.
Practice this command with great caution and please don't use it to restart your computer on a regular basis. Use it only when the graphical user interface (GUI) is not responding.
Another trick: if the GUI is not responding and you have unsaved work, you can recover some of it from the command line, by accessing one of Linux's virtual terminals. Ubuntu starts, by default, seven virtual terminals and the graphical user interface starts on terminal 7. To access any of the seven terminals use Ctrl + Alt + F1 to F7. A prompt will appear asking you to log in and, after logging in, you can run some commands to close processes and save work before exiting. To get back to the user interface, hit Ctrl + Alt + F1.