I have BurpSuite configured as a proxy. I have generated and installed a certificate for Burp in my Mac's keychain. On a Kali box I have also configured Iceweasel to proxy through my Burp Proxy running on my Mac. I can can browse from Kali to both http and https sites and burp intercepts the requests. I forward them through and confirm the security exceptions and the browser eventually displays web pages.

From the Kali command line I exported the following two variables

export http_proxy=http://172.xx.yy.z:8081
export https_proxy=http://172.xx.yy.z:8081

I wrote one python script:

(env) ojblass@kali:~/effective_python$ cat ojblass_urllib_http.py
import urllib
url = urllib.urlopen("http://www.hotmail.com")
data = url.read()
print data

This correctly hits the BurpSuite proxy; however, when I attempt to run https traffic through the request is not intercepted:

(env) ojblass@kali:~/effective_python$ cat ojblass_urllib_https.py
import urllib
url = urllib.urlopen("https://mail.live.com/default.aspx")
data = url.read()
print data

Instead the following error is printed:

<html><head><title>Burp Suite Professional</title>
<style type="text/css">
body { background: #dedede; font-family: Arial, sans-serif; color: #404042; -webkit-font-smoothing: antialiased; }
#container { padding: 0 15px; margin: 10px auto; background-color: #ffffff; }
a { word-wrap: break-word; }
a:link, a:visited { color: #e06228; text-decoration: none; }
a:hover, a:active { color: #404042; text-decoration: underline; }
h1 { font-size: 1.6em; line-height: 1.2em; font-weight: normal; color: #404042; }
h2 { font-size: 1.3em; line-height: 1.2em; padding: 0; margin: 0.8em 0 0.3em 0; font-weight: normal; color: #404042;}
.title, .navbar { color: #ffffff; background: #e06228; padding: 10px 15px; margin: 0 -15px 10px -15px; overflow: hidden; }
.title h1 { color: #ffffff; padding: 0; margin: 0; font-size: 1.8em; }
div.navbar {position: absolute; top: 18px; right: 25px;}div.navbar ul {list-style-type: none; margin: 0; padding: 0;}
div.navbar li {display: inline; margi-left: 20px;}
div.navbar a {color: white; padding: 10px}
div.navbar a:hover, div.navbar a:active {text-decoration: none; background: #404042;}
<div id="container">
<div class="title"><h1>Burp Suite Professional</h1></div>
<div class="request">GET&nbsp;https://mail.live.com/default.aspx&nbsp;HTTP/1.0<br>

The main content of the error states 'Invalid client request received. First line of request did not contain an absolute URL. I have googled this error but everything I seem to come does not appear relevant. I am new to this and would appreciate any insight you might have.


I found the answer. Urllib does not support https proxy. I found this site which has a recipe for connecting properly.

Here is the code in case the link goes dead:

# urllib2 opener to connection through a proxy using the CONNECT method, (useful for SSL)
# tested with python 2.4

import urllib2
import urllib
import httplib
import socket

class ProxyHTTPConnection(httplib.HTTPConnection):

    _ports = {'http' : 80, 'https' : 443}

    def request(self, method, url, body=None, headers={}):
        #request is called before connect, so can interpret url and get
        #real host/port to be used to make CONNECT request to proxy
        proto, rest = urllib.splittype(url)
        if proto is None:
            raise ValueError, "unknown URL type: %s" % url
        #get host
        host, rest = urllib.splithost(rest)
        #try to get port
        host, port = urllib.splitport(host)
        #if port is not defined try to get from proto
        if port is None:
                port = self._ports[proto]
            except KeyError:
                raise ValueError, "unknown protocol for: %s" % url
        self._real_host = host
        self._real_port = port
        httplib.HTTPConnection.request(self, method, url, body, headers)

    def connect(self):
        #send proxy CONNECT request
        self.send("CONNECT %s:%d HTTP/1.0\r\n\r\n" % (self._real_host, self._real_port))
        #expect a HTTP/1.0 200 Connection established
        response = self.response_class(self.sock, strict=self.strict, method=self._method)
        (version, code, message) = response._read_status()
        #probably here we can handle auth requests...
        if code != 200:
            #proxy returned and error, abort connection, and raise exception
            raise socket.error, "Proxy connection failed: %d %s" % (code, message.strip())
        #eat up header block from proxy....
        while True:
            #should not use directly fp probablu
            line = response.fp.readline()
            if line == '\r\n': break

class ProxyHTTPSConnection(ProxyHTTPConnection):

    default_port = 443

    def __init__(self, host, port = None, key_file = None, cert_file = None, strict = None):
        ProxyHTTPConnection.__init__(self, host, port)
        self.key_file = key_file
        self.cert_file = cert_file

    def connect(self):
        #make the sock ssl-aware
        ssl = socket.ssl(self.sock, self.key_file, self.cert_file)
        self.sock = httplib.FakeSocket(self.sock, ssl)

class ConnectHTTPHandler(urllib2.HTTPHandler):

    def do_open(self, http_class, req):
        return urllib2.HTTPHandler.do_open(self, ProxyHTTPConnection, req)

class ConnectHTTPSHandler(urllib2.HTTPSHandler):

    def do_open(self, http_class, req):
        return urllib2.HTTPSHandler.do_open(self, ProxyHTTPSConnection, req)

if __name__ == '__main__':

    import sys

    opener = urllib2.build_opener(ConnectHTTPHandler, ConnectHTTPSHandler)
    req = urllib2.Request(url='')
    req.set_proxy('', 'https')
    f = urllib2.urlopen(req)
    print f.read()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.