Python has a built in library for making HTTP requests. Well, it has two of them actually: urllib and urllib2. Unfortunately, both of them are not ideal. The API is thoroughly broken, and certain things are extremely hard.
Enter requests a fantastic third party python library. You can install it by typing
pip install requests in your terminal if you have Pip installed.
In this tutorial, we’ll cover making simple HTTP requests and dealing with cookies, URL query string parameters and POST data.
Making a Simple GET Request
resp is a Response object, which has several useful attributes.
resp.content would be the actually content of the page, for instance. and
resp.headers is a Python dictionary of the headers from the response. You can even see your request object with
Why do I love requests? Because it’s simple. The above is extremely intuitive, and there are very Pythonic ways of doing many other things. Dealing with cookies is a good example.
Making Requests with Cookies
Requests introduces a Session object, which acts as a sort of storage bin for data from previous requests. When you use
requests.get() it actually creates a session for you. Using the
with statement lets you make multiple requests with the same session easily. Meaning you can keep cookies around. You can also feed
requests.get() a session object using the
session keyword argument:
Proxies and Proxy Authentication
Need to make a request through proxy? No problem. The proxies keyword argument of
requests.post) makes it easy.
The value of
proxies keyword arg should be a dictionary with the various protocols (eg. http and https) as keys and the proxy URI’s as values.
What about Proxy Authentication? If your proxy uses HTTP basic auth, you can use the
auth keyword argument.
What about using the Proxy-Authentication header? Just use a the username and password in the URL like normal:
Query Strings & POST data
Need to send a URL query string with your request? You could just stick them on the end of the URI, but that’s no fun (and you’d have to worry about encoding them). Just use the
params keyword arg.
Making POST requests is just as simple. Use the
data keyword argument to send whatever you need to the remote server.
And the result: