This website uses cookies to ensure you get the best possible experience. See our Cookies Policy.

PMG Digital Made for Humans

Making HTTP Requests with Python

3 MINUTE READ | February 1, 2012

Making HTTP Requests with Python

{u‘arg1’: u‘value1’, u‘arg2’: u‘value2’}Python has a built in library for making HTTP requests. Well, it has two of them actually: urllib and urllib2. Unfortunately, both of them are not ideal. The API is thoroughly broken, and certain things are extremely hard.

Enter requests a fantastic third party python library. You can install it by typing pip install requests in your terminal if you have Pip installed.

In this tutorial, we’ll cover making simple HTTP requests and dealing with cookies, URL query string parameters and POST data.

Python>>> import requests
>>> resp = requests.get(‘http://httpbin.org’)
>>> resp
<Response [200]>
That’s it! resp is a Response object, which has several useful attributes. resp.content would be the actually content of the page, for instance. and resp.headers is a Python dictionary of the headers from the response. You can even see your request object with resp.request.

Why do I love requests? Because it’s simple. The above is extremely intuitive, and there are very Pythonic ways of doing many other things. Dealing with cookies is a good example.

Python

import requests

with requests.session() as s:

# set a cookie

s.get(‘http://httpbin.org/cookies/set/pmgcookie/mmmmmmcookie’)

#is the cookie still there?

resp = s.get(‘http://httpbin.org/cookies’)

print resp.content # yup

The result

Python

{

“cookies”: {

“pmgcookie”: “mmmmmmcookie”

}

}

Requests introduces a Session object, which acts as a sort of storage bin for data from previous requests. When you use requests.get() it actually creates a session for you. Using the with statement lets you make multiple requests with the same session easily. Meaning you can keep cookies around. You can also feed requests.get() a session object using the session keyword argument: requests.get('http://example.com', session=some_session)

Need to make a request through proxy? No problem. The proxies keyword argument of requests.get (and requests.post) makes it easy.

Python

import requests

proxy = {‘http’: ‘http://some-proxy.com’}

resp = requests.get(‘http://httpbin.org’, proxies=proxy)

The value of proxies keyword arg should be a dictionary with the various protocols (eg. http and https) as keys and the proxy URI’s as values.

What about Proxy Authentication? If your proxy uses HTTP basic auth, you can use the auth keyword argument.

Python

import requests

proxy = {‘http’: ‘http://some-proxy.com’}

auth = (‘username’, ‘password’,)

resp = requests.get(‘http://httpbin.org’, proxies=proxy, auth=auth)

What about using the Proxy-Authentication header? Just use a the username and password in the URL like normal: http://user:password@some-proxy.com.

Python

import requests

proxy = {‘http’: ‘http://user:password@some-proxy.com’}

resp = requests.get(‘http://httpbin.org’, proxies=proxy)

Need to send a URL query string with your request? You could just stick them on the end of the URI, but that’s no fun (and you’d have to worry about encoding them). Just use the params keyword arg.

Python

import json

import requests

# fetch!

params = {‘arg1’: ‘value1’, ‘arg2’: ‘value2’}

resp = requests.get(‘http://httpbin.org/get’, params=params)

# let’s see what we got back

j = json.loads(resp.content)

print j.get(‘args’)

The result:

Python

http://httpbin.org/get?arg1=value1&arg2=value2

{u‘arg1’: u‘value1’, u‘arg2’: u‘value2’}

Making POST requests is just as simple. Use the data keyword argument to send whatever you need to the remote server.

Python

import json

import requests

# fetch!

params = {‘arg1’: ‘value1’, ‘arg2’: ‘value2’}

resp = requests.post(‘http://httpbin.org/post’, data=params)

# what did we get back?

j = json.loads(resp.content)

print j.get(‘form’)

And the result:

Python

{u‘arg1’: u‘value1’, u‘arg2’: u‘value2’}

Insights meet inbox

Sign up for weekly articles & resources.

Requests is, by far, one of the most well though out python HTTP libraries today. Use it, and you’ll love its simplicity and easy. Head over to the requests docs to learn more!


Posted by Christopher Davis

Related Content

thumbnail image

Get Informed

PMG Innovation Challenge Inspires New Alli Technology Solutions

4 MINUTES READ | November 2, 2021

Get Informed

Applying Function Options to Domain Entities in Go

11 MINUTES READ | October 21, 2019

thumbnail image

Get Informed

My Experience Teaching Through Jupyter Notebooks

4 MINUTES READ | September 21, 2019

Get Informed

Trading Symfony’s Form Component for Data Transfer Objects

8 MINUTES READ | September 3, 2019

Get Inspired

Working with an Automation Mindset

5 MINUTES READ | August 22, 2019

Get Informed

Parsing Redshift Logs to Understand Data Usage

7 MINUTES READ | May 6, 2019

Get Inspired

3 Tips for Showing Value in the Tech You Build

5 MINUTES READ | April 24, 2019

thumbnail image

Get Informed

Testing React

13 MINUTES READ | March 12, 2019

Get Inspired

Tips for Designing & Testing Software Without a UX Specialist

4 MINUTES READ | March 6, 2019

Get Informed

A Beginner’s Experience with Terraform

4 MINUTES READ | December 20, 2018

ALL POSTS