Linux proxy setup

Linux proxy setup

April 6, 2022
development

Sometimes we need to set up a proxy from the command line due to network issues. On Ubuntu, the common commands I use that require networking are curl, git, apt, wget, docker, maven, gradle, npm, yarn etc. Some of these commands access proxy settings in environment variables, some of them requires specific configurations. In this post, we are targeting a Debian-based distribution, e.g, Debian and Ubuntu…

Conventions #

Normally, Linux environment variables are all uppercase, but the proxy options are a bit unique in that they are generally case-sensitive, some commands do not recognize certain proxy environment variables in uppercase, such as curl can only recognize lowercase http_proxy.

Here we list the relevant environment variables in lowercase.

Environment variables curl git apt wget
http_proxy Y Y Y Y
https_proxy Y Y N Y
ftp_proxy Y Y Y N
all_proxy Y Y N Y
no_proxy Y Y N Y

In addition to the no_proxy these environment variables are set in the format

[protocol://][user[:password]@]proxyhost[:port]

For no_proxy, it’s a comma-separated list of hostnames (domains).

Also, ssh, docker, maen, gradle, npm, yarn does not use these environment variables, and has its own way of configuring proxies.

git #

As you can see from the git-config man page, git can use the proxies defined in the environment variables just like curl, or it can use the http.proxy You can also set it up with https.proxy option. http.proxy is set in the format

[protocol://][user[:password]@]proxyhost[:port]

Use these commands:

git config --global http.proxy http://username:password@host:port
git config --global https.proxy http://username:password@host:port

Or you can edit directly your ~/.gitconfig file:

[http]
        proxy = http://username:password@host:port
[https]
        proxy = http://username:password@host:port

This setting will override the environment variable setting.

The above setting is useful for using https protocol, if it is access via ssh protocol, we need to configure ssh proxy. For example, we set up the proxy by via ~/.ssh/config config file (this file should have read/write permissions of 644)

Host github.com
    User git
    ProxyCommand nc -X connect -x 127.0.0.1:10809 %h %p
  • Here -X connect option indicates that the proxy is http proxy,
  • For socks5, remove the -X connect option.
  • For other protol, check nc’s man page

ssh #

ssh does not use the above environment variables, so if the ssh connection requires a proxy, it must be set separately:

$ cat ~/.ssh/config

ProxyCommand /usr/bin/nc -X connect -x 127.0.0.1:8080 %h %p

Host gitlab.com
 ProxyCommand nc -X 5 -x 127.0.0.1:9999 %h %p

Host github.com
 ProxyCommand nc -X 5 -x 127.0.0.1:9999 %h %p

$ man ssh_config
...
ProxyCommand
        Specifies the command to use to connect to the server.  The command string extends to the end of the line, and is executed using the user's shell ‘exec’ directive to avoid a lingering
        shell process.

        Arguments to ProxyCommand accept the tokens described in the TOKENS section.  The command can be basically anything, and should read from its standard input and write to its standard
        output.  It should eventually connect an sshd(8) server running on some machine, or execute sshd -i somewhere.  Host key management will be done using the Hostname of the host being con‐
        nected (defaulting to the name typed by the user).  Setting the command to none disables this option entirely.  Note that CheckHostIP is not available for connects with a proxy command.

        This directive is useful in conjunction with nc(1) and its proxy support.  For example, the following directive would connect via an HTTP proxy at 192.0.2.0:

        ProxyCommand /usr/bin/nc -X connect -x 192.0.2.0:8080 %h %p
...

docker #

Many Linux distributions use systemd to start the Docker daemon. This document shows a few examples of how to customize Docker’s settings.

  1. Create a systemd drop-in directory for the docker service:
sudo mkdir -p /etc/systemd/system/docker.service.d
  1. Create a file named /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:
[Service]
Environment="HTTP_PROXY=http://localhost:10809/"
Environment="HTTPS_PROXY=http://localhost:10809/"
#Environment="HTTP_PROXY=socks5://localhost:10800/"
#Environment="HTTPS_PROXY=socks5://localhost:10800/"
#Environment="HTTP_PROXY=socks5://127.0.0.1:9999/"
#Environment="HTTPS_PROXY=socks5://127.0.0.1:9999/"
Environment="NO_PROXY=10.1.0.0/16,10.152.183.0/24,127.0.0.1,localhost,127.0.0.1,127.0.0.0/8,::1,192.168.0.0/16,10.96.0.0/12,172.16.0.0/12,172.17.0.0/12,*.internal,*.svc,*.local"
  1. Restar the dockerd service
sudo system restart docker

For mor details, check https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

go get #

Go command understand environment variables http_proxy and no_proxy, but that’s not enough because go get uses source control managers for retrieving code. So you have to set HTTP proxy settings for your SCM too, refer to git and ssh.

maven #

You can configure a proxy to use for some or all of your HTTP requests with Maven. The username and password are only required if your proxy requires basic authentication (note that later releases may support storing your passwords in a secured keystore - in the mean time, please ensure your settings.xml file (usually ${user.home}/.m2/settings.xml) is secured with permissions appropriate for your operating system).

The nonProxyHosts setting accepts wild cards, and each host not to proxy is separated by the | character. This matches the JDK configuration equivalent.

Edit the proxies section in your ~/.m2/settings.xml file:

  <proxies>
   <proxy>
      <id>example-proxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy.example.com</host>
      <port>8080</port>
      <username>proxyuser</username>
      <password>somepassword</password>
      <nonProxyHosts>www.google.com|*.example.com</nonProxyHosts>
    </proxy>
  </proxies>

Or Use System Properties:

-Dhttp.proxyHost=host 
-Dhttp.proxyPort=port 
-Dhttps.proxyHost=host 
-Dhttps.proxyPort=port 
-Dhttp.proxyUser=username 
-Dhttp.proxyPassword=password

Or Use Maven wrapper, Create a new file .mvn/jvm.config inside the project folder and set the properties accordingly:

-Dhttp.proxyHost=host 
-Dhttp.proxyPort=port 
-Dhttps.proxyHost=host 
-Dhttps.proxyPort=port 
-Dhttp.proxyUser=username 
-Dhttp.proxyPassword=password

gradle #

Add the below in your gradle.properties file and in your gradle/wrapper/gradle-wrapper.properties file if you are downloading the wrapper over a proxy If you want to set these properties globally then add it in USER_HOME/.gradle/gradle.properties file

## Proxy setup
systemProp.proxySet="true"
systemProp.http.keepAlive="true"
systemProp.http.proxyHost=host
systemProp.http.proxyPort=port
systemProp.http.proxyUser=username
systemProp.http.proxyPassword=password
systemProp.http.nonProxyHosts=local.net|some.host.com

systemProp.https.keepAlive="true"
systemProp.https.proxyHost=host
systemProp.https.proxyPort=port
systemProp.https.proxyUser=username
systemProp.https.proxyPassword=password
systemProp.https.nonProxyHosts=local.net|some.host.com
## end of proxy setup

npm #

If the HTTPS_PROXY or https_proxy or HTTP_PROXY or http_proxy environment variables are set, proxy settings will be honored by the underlying make-fetch-happen library.

Or you can Use these commands:

npm config set proxy http://username:password@host:port
npm config set https-proxy http://username:password@host:port

Or you can edit directly your ~/.npmrc file:

proxy=http://username:password@host:port
https-proxy=http://username:password@host:port
https_proxy=http://username:password@host:port

yarn #

Yarn respects the environment variables HTTP_PROXY, HTTPS_PROXY and NO_PROXY.

Or you can Use these commands:

yarn config set proxy http://username:password@host:port
yarn config set https-proxy http://username:password@host:port

curl #

The following environment variables in addition to http_proxy are allowed in all cases, but lowercase is preferred. Use environment variables to set proxy The effect and options of -x , --proxy has the same effect.

  • http_proxy [protocol://][:port] Set the HTTP protocol of the proxy .
  • HTTPS_PROXY [protocol://][:port] Set HTTPS protocol of the proxy .
  • [url-protocol]_PROXY [protocol://][:port] for [url-protocal] Set the proxy , where the url-protocol is curl supported protocols, such as > FTP , FTPS , POP3 , IMAP , SMTP , LDAP etc..
  • ALL_PROXY [protocol://][:port] for not explicitly setting proxy Protocol settings for the proxy .
  • NO_PROXY The list of hosts that will not be proxied (comma-separated list of hosts). If only set to * , then all hosts > are matched.
  • Translated from Curl Manual, Version 7.58.0 .

wget #

wget support both http_proxy, https_proxy, ftp_proxy and no_proxy

apt #

apt support http_proxy, no_proxy

$ man apt-transport-http
...
Proxy Configuration
    The environment variable http_proxy is supported for system wide configuration. Proxies specific to APT can be configured via the option Acquire::http::Proxy. Proxies which should be used only
    for certain hosts can be specified via Acquire::http::Proxy::host. Even more fine-grained control can be achieved via proxy autodetection, detailed further below. All these options use the URI
    format scheme://[[user][:pass]@]host[:port]/. Supported URI schemes are socks5h (SOCKS5 with remote DNS resolution), http and https. Authentication details can be supplied via apt_auth.conf(5)
    instead of including it in the URI directly.

    The various APT configuration options support the special value DIRECT meaning that no proxy should be used. The environment variable no_proxy is also supported for the same purpose.

    Furthermore, there are three settings provided for cache control with HTTP/1.1 compliant proxy caches: Acquire::http::No-Cache tells the proxy not to use its cached response under any
    circumstances.  Acquire::http::Max-Age sets the allowed maximum age (in seconds) of an index file in the cache of the proxy.  Acquire::http::No-Store specifies that the proxy should not store
    the requested archive files in its cache, which can be used to prevent the proxy from polluting its cache with (big) .deb files.
...