A notebook on some tricks in Linux. For some of them, I learned them from the Internet. For the rest, I got it by playing around it myself.
1. cURL
1.1. Continue the interrupted downloading process
Use “-C” option. Specially, if we want cURL to automatically determine the location to continue transfer, then use “-C -“.
curl -C - https://sample.tgz
This would also save the failed file transfer from wget.
1.2 Save file using the original name
When downloading the ERA-5 data at ECMWF website, the simple command below does not work well, as it prints all the content directly to the screen.
curl https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib
In this case, we need to redirect it to the desired output:
curl https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib > output.grib
But this introduces a new problem: As the file size is huge (~10GB), it is quite often that the file transfer is interrupted. By this method, there is no way to utilize the capability of cURL to resume interruption. But if we download the file in the following way, then it works:
curl -C - -O -J -L https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib
What makes it work here is the “-J” option. See the man page of cURL for more information.
2. Python
In python, when using PIP, I got the following warming:
“/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.”
The solution is
pip install requests[security]
