Linux miscellaneous

A notebook on some tricks in Linux. For some of them, I learned them from the Internet. For the rest, I got it by playing around it myself.

1.    cURL

1.1. Continue the interrupted downloading process

Use “-C” option. Specially, if we want cURL to automatically determine the location to continue transfer, then use “-C -“.

 curl -C - https://sample.tgz 

This would also save the failed file transfer from wget.

 

1.2  Save file using the original name

When downloading the ERA-5 data at ECMWF website, the simple command below does not work well, as it prints all the content directly to the screen.

 curl https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib 

In this case, we need to redirect it to the desired output:

 curl https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib > output.grib 

But this introduces a new problem: As the file size is huge (~10GB), it is quite often that the file transfer is interrupted. By this method, there is no way to utilize the capability of cURL to resume interruption. But if we download the file in the following way, then it works:

 curl -C - -O -J -L https://stream.ecmwf.int/xxxxxxx/scratch/_mars-atls03-xxxxxx.grib 

What makes it work here is the “-J” option. See the man page of cURL for more information.

 

2.   Python

In python, when using PIP, I got the following warming:

“/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.”

The solution is

pip install requests[security] 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s