Proxy Authentication with Selenium

I’ve been working a lot with selenium for C# lately and one of my company’s projects required me to use proxies for certain automated tasks.

For basic proxies, selenium nicely uses the inbuilt functions that firefox offers in order to set up proxies, it’s really quite simple:


 var Proxy = new Proxy();          //Create proxy object
Proxy.Kind = ProxyKind.Manual;     //set to manual mode
Proxy.HttpProxy = ip + ":" + port; //add ip and port to proxy protocol

However, Firefox really doesn’t offer the best proxy configuration options, and I really needed to use a proxy with username:password authentication, which just does not work with selenium.

Why selenium sucks for proxy authentication

When trying to browse using a proxy with basic authentication, firefox will display a popup asking the user for the credentials to the proxy:

435345

None of the workarounds selenium offers for popups work on these basic authentication popups since they’re not part of any website, but rather standalone windows. According to some sources you could control these popups with earlier versions of selenium and/or firefox, but for me, this wasn’t an option.
After some long hours trying to work around this in various ways, I finally found a method that works and can even be adapted to other websites using basic authentication, not just proxies. The following code is written in C#, but you should be able to easily adapt this to any other language you are using. The code also requires you to use the development version of Firefox, which lets us install uncertified extensions.

Creating a firefox extension to include the proxy auth in the headers

Proxy authentication like this works by the proxy server sending back the status code 407 Proxy Authentication Required , the browser then showing the login window to the user and, once the user has entered their credentials, sending back new headers with a key-value pair like so:


"Proxy-Authorization" : "Basic [Base64Credentials]"

with [Base64Credentials] being the base64 representation of the string “username:password”.

My idea was to create an extension that adds a listener to webRequest.onBeforeSendHeaders and adds this key-value pair to the outgoing headers.
This is pretty straight-forward:


browser.webRequest.onBeforeSendHeaders.addListener(handler, { urls: ['']}, ['requestHeaders', 'blocking']);
function handler(details)
{
    details.requestHeaders.push({ name: 'Proxy-Authorization', value: 'Basic [BASE64CREDENTIALS]' });
    return { requestHeaders: details.requestHeaders };
}

If you’re not familiar with browser-extensions, these basically require a javascript file and a JSON manifest file. These then get zipped up and the extension is renamed to .xpi.
I named the javascript file from above background.js, which means my manifest.json will look like this:

{
  "manifest_version": 2,
  "name": "proxyauth",
  "version": "1.0",
  "description": "Description",
  "applications": {
      "gecko": {
                "id": "b.kuenzel@xoaproductions.com"
                }
   },
   "background": {
        "scripts": ["background.js"]
    },
    "permissions": [
          "webRequest", "webRequestBlocking", "<all_urls>"
     ]
}

We zip manifest.json and background.js up, change the .zip extension to .xpi and can now load the extension in Firefox (developer version). If you have replaced [BASE64CREDENTIALS] in background.js with the correct base64 version of your credentials, you’ll now see that firefox no longer asks for the proxy’s credentials and it just works™.

However, this means we’d need to create an extension by hand for every different proxy we are using and manually load it into the firefox profile we are using with selenium, which isn’t really useful if you plan to automate this with many selenium instances and many different proxies. To work around this, I wrote a C# class that will just plug into your existing selenium workflow and automate all of this.

Wrapping the extension in C#

In C#, the language of my choice for using Selenium, I wanted to create a class that would automatically build a new browser extension with some credentials I supply to it, return back the path to the created .xpi file for easy use with selenium and dispose of the .xpi when it’s not needed anymore.

For simplicity (and to ease copy&paste) I’ll start this off with the whole class and then go over what it does:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.IO.Compression;

namespace XOAProductions.Automation
{
    /// <summary>
    /// Creates a temporary firefox extension which automatically logs into a proxy by intercepting all traffic and appending a Proxy-Authentication header with credentials
    /// </summary>
    class ProxyConfigurationExtension : IDisposable
    {
        /// <summary>
        /// the script file for firefox extension
        /// </summary>
        const string scriptFile = @"browser.webRequest.onBeforeSendHeaders.addListener(handler, { urls: ['']}, ['requestHeaders', 'blocking']);
                                    function handler(details)
                                    {
                                        details.requestHeaders.push({ name: 'Proxy-Authorization', value: 'Basic [BASE64CREDENTIALS]' });
                                        return { requestHeaders: details.requestHeaders };
                                    }";

        /// <summary>
        /// The manifest file for firefox extension
        /// </summary>
        const string manifest = @"{
                                   'manifest_version': 2,
                                   'name': 'proxyauth',
                                   'version': '1.0',
                                   'description': 'Description',
                                   'applications': {
                                     'gecko': {
                                         'id': 'b.kuenzel@xoaproductions.com'
                                     }
                                    },
                                   'background': {
                                     'scripts': ['background.js']
                                     },
                                   'permissions': [
                                     'webRequest', 'webRequestBlocking', '' 
                                   ]
                                 }";

        /// <summary>
        /// The path to the temp extension created
        /// </summary>
        public string disposableExtensionPath { get; private set; }

        /// <summary>
        /// Creates the extension file. File can then be found at disposableExtensionPath.
        /// </summary>
        /// the username of proxy
        /// the password of proxy
        public ProxyConfigurationExtension(string username, string password)
        {
            //create temp directory for extension files
            var dir = Directory.CreateDirectory(System.Windows.Forms.Application.StartupPath + "\\" + Guid.NewGuid().ToString());
            var destinationBasePath = System.Windows.Forms.Application.StartupPath + "\\" + Guid.NewGuid().ToString();
            var zipFile = destinationBasePath + ".zip";
            var finalXPIFile = destinationBasePath + ".xpi";

            //create credentials and replace them in the script string
            string credentials = username + ":" + password;
            string b64Credentials = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(credentials));
            string scriptFileWithCredentials = scriptFile.Replace("[BASE64CREDENTIALS]", b64Credentials);

            //write script and manifest to temp directory
            File.WriteAllText(dir.FullName + "\\manifest.json", manifest.Replace("'", "\""));
            File.WriteAllText(dir.FullName + "\\background.js", scriptFileWithCredentials);

            //create zip out of directory and rename to xpi
            ZipFile.CreateFromDirectory(dir.FullName, zipFile);
            File.Move(zipFile, finalXPIFile);

            //delete tempdir
            Directory.Delete(dir.FullName, true);

            disposableExtensionPath = finalXPIFile;
        }

        /// <summary>
        /// gets rid of the temporary firefox extension.
        /// ONLY CALL AFTER FIREFOX IS CLOSED!
        /// </summary>
        public void Dispose()
        {
            try
            {
                File.Delete(disposableExtensionPath);
            }
            catch(Exception e)
            {
                Console.WriteLine("Warning! Temporary firefox extension could not be deleted at path: " + disposableExtensionPath + "\nSee error output below for more information.");
                Console.WriteLine(e.Message);
            }
        }
    }
}

As a side note,  since my code preview plugin likes to break any formatting and also introduces weird chars at times, you can find the whole project on github, too.

Starting from the top of the code:


class ProxyConfigurationExtension : IDisposable


We create the class and use the IDisposable interface, just because I like to use this on files that I don’t save in temp. It’s optional to do this.


const string scriptFile = ...
const string manifest = ...

We then define the const string scriptFile and manifest, which hold the exact code I showed in the previous code examples. These will be modified with the proper credentials and then written to disk and zipped up by the code.


public string disposableExtensionPath { get; private set; }

Whe then define disposableExtensionPath, which will become the path to the .xpi file created by the code after the class is constructed.


public ProxyConfigurationExtension(string username, string password)
{...



In the constructor we create a temporary directory, we base64 encode the credentials and replace the [BASE64CREDENTIALS] in the scriptFile string with it:


//create temp directory for extension files
var dir = Directory.CreateDirectory(System.Windows.Forms.Application.StartupPath + "\\" + Guid.NewGuid().ToString());
var destinationBasePath = System.Windows.Forms.Application.StartupPath + "\\" + Guid.NewGuid().ToString();
var zipFile = destinationBasePath + ".zip";
var finalXPIFile = destinationBasePath + ".xpi";

//create credentials and replace them in the script string
string credentials = username + ":" + password;
string b64Credentials = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(credentials));
string scriptFileWithCredentials = scriptFile.Replace("[BASE64CREDENTIALS]", b64Credentials);

We then write both the manifest string and the altered script string to manifest.json and background.js into the temporary directory:


//write script and manifest to temp directory
File.WriteAllText(dir.FullName + "\\manifest.json", manifest.Replace("'", "\""));
File.WriteAllText(dir.FullName + "\\background.js", scriptFileWithCredentials);

The contents of the temporary directory are then zipped and the extension is renamed from .zip to .xpi:


ZipFile.CreateFromDirectory(dir.FullName, zipFile);
File.Move(zipFile, finalXPIFile);

The temp directory is deleted and the path to the .xpi is assigned to disposableExtensionPath:


Directory.Delete(dir.FullName, true);
disposableExtensionPath = finalXPIFile;

Last but not least, the Dispose() method is created. In it, we simply try to delete the temporary file and catch any errors associated with that. For example, you should only dispose of the file once your Selenium instance and the Firefox browser have shut down, otherwise, the file can’t be deleted because Firefox still has it loaded.

Usage

The usage is pretty straightforward:


public ProxyConfigurationExtension tempProxyLoginExtension { get; private set; }

private void AddProxyToOptions(ref FirefoxOptions options)
{
    var Proxy = new Proxy();
    Proxy.Kind = ProxyKind.Manual;
    Proxy.HttpProxy =  ip + ":" + port;
    Proxy.SslProxy  =  ip + ":" + port;


    options.Proxy = Proxy;
    options.SetPreference("xpinstall.signatures.required", false);

    tempProxyLoginExtension = new ProxyConfigurationExtension(username, password);

    options.Profile.AddExtension(tempProxyLoginExtension.disposableExtensionPath);

}

This method creates a proxy object and assigns the ip and port to it.
I then make sure to set xpinstall.signatures.required to false, however, it probably works without this preference.
Then I populate a variable tempProxyLoginExtension with a reference to a new ProxyConfigurationExtension and pass in the username and password of the proxy.
The reference is there so I can later dispose of the temporary extension after I have completed my automation.
I then simply call options.Profile.AddExtension and pass in the path to the temporary .xpi file.

You can find all the code to this project on my github!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s