• About Us
    • New York
  • Work
  • Capabilities
  • Careers
  • Technology
  • Blog
  • Contact Us
  • About Us
    • New York
  • Work
  • Capabilities
  • Careers
  • Technology
  • Blog
  • Contact Us
February 01, 2012

Making HTTP Requests with Python

Posted by Christopher Davis

Python has a built in library for making HTTP requests. Well, it has two of them actually: urllib and urllib2. Unfortunately, both of them are not ideal. The API is thoroughly broken, and certain things are extremely hard.

Enter requests a fantastic third party python library. You can install it by typing pip install requests in your terminal if you have Pip installed.

In this tutorial, we’ll cover making simple HTTP requests and dealing with cookies, URL query string parameters and POST data.

Making a Simple GET Request

Python
1
2
3
4
>>> import requests
>>> resp = requests.get(‘http://httpbin.org’)
>>> resp
<Response [200]>

That’s it! resp is a Response object, which has several useful attributes. resp.content would be the actually content of the page, for instance. and resp.headers is a Python dictionary of the headers from the response. You can even see your request object with resp.request.

Why do I love requests? Because it’s simple. The above is extremely intuitive, and there are very Pythonic ways of doing many other things. Dealing with cookies is a good example.

Making Requests with Cookies

Python
1
2
3
4
5
6
7
8
9
import requests
with requests.session() as s:
    # set a cookie
    s.get(‘http://httpbin.org/cookies/set/pmgcookie/mmmmmmcookie’)
    
    #is the cookie still there?
    resp = s.get(‘http://httpbin.org/cookies’)
    print resp.content # yup

The result

Python
1
2
3
4
5
{
  “cookies”: {
    “pmgcookie”: “mmmmmmcookie”
  }
}

Requests introduces a Session object, which acts as a sort of storage bin for data from previous requests. When you use requests.get() it actually creates a session for you. Using the with statement lets you make multiple requests with the same session easily. Meaning you can keep cookies around. You can also feed requests.get() a session object using the session keyword argument: requests.get('http://example.com', session=some_session)

Proxies and Proxy Authentication

Need to make a request through proxy? No problem. The proxies keyword argument of requests.get (and requests.post) makes it easy.

Python
1
2
3
4
import requests
proxy = {‘http’: ‘http://some-proxy.com’}
resp = requests.get(‘http://httpbin.org’, proxies=proxy)

The value of proxies keyword arg should be a dictionary with the various protocols (eg. http and https) as keys and the proxy URI’s as values.

What about Proxy Authentication? If your proxy uses HTTP basic auth, you can use the auth keyword argument.

Python
1
2
3
4
5
import requests
proxy = {‘http’: ‘http://some-proxy.com’}
auth = (‘username’, ‘password’,)
resp = requests.get(‘http://httpbin.org’, proxies=proxy, auth=auth)

What about using the Proxy-Authentication header? Just use a the username and password in the URL like normal: http://user:password@some-proxy.com.

Python
1
2
3
4
import requests
proxy = {‘http’: ‘http://user:password@some-proxy.com’}
resp = requests.get(‘http://httpbin.org’, proxies=proxy)

Query Strings & POST data

Need to send a URL query string with your request? You could just stick them on the end of the URI, but that’s no fun (and you’d have to worry about encoding them). Just use the params keyword arg.

Python
1
2
3
4
5
6
7
8
9
10
import json
import requests
# fetch!
params = {‘arg1’: ‘value1’, ‘arg2’: ‘value2’}
resp = requests.get(‘http://httpbin.org/get’, params=params)
# let’s see what we got back
j = json.loads(resp.content)
print j.get(‘args’)

The result:

Python
1
2
http://httpbin.org/get?arg1=value1&arg2=value2
{u‘arg1’: u‘value1’, u‘arg2’: u‘value2’}

Making POST requests is just as simple. Use the data keyword argument to send whatever you need to the remote server.

Python
1
2
3
4
5
6
7
8
9
10
import json
import requests
# fetch!
params = {‘arg1’: ‘value1’, ‘arg2’: ‘value2’}
resp = requests.post(‘http://httpbin.org/post’, data=params)
# what did we get back?
j = json.loads(resp.content)
print j.get(‘form’)

And the result:

Python
1
{u‘arg1’: u‘value1’, u‘arg2’: u‘value2’}

Requests is, by far, one of the most well though out python HTTP libraries today. Use it, and you’ll love its simplicity and easy. Head over to the requests docs to learn more!

apiDevelopmenthttpjqueryPython
Previous
Next

Latest White Papers

  • Shifting Plans for 2020 & Beyond
  • Game On: How Brands Can Log Into A Diverse Multi-Billion Dollar Industry
  • What CCPA Means For Brands
  • How Google is Improving Consumer Data Privacy
  • Ways to Prepare for the Cookieless Future
  • See all White Papers

Featured Posts

  • Ad Age Names PMG #1 Best Place to Work in 2021
  • MediaPost Names PMG Independent Agency of the Year
  • PMG Client Portfolio Trends During Amazon Prime Day 2020
  • A Closer Look at the Congressional Big Tech Market Power Report
  • What to Know About Reddit

Categories

  • Consumer Insights
  • Content
  • Creative Design
  • Data Analytics
  • Development
  • Digital TV & Video
  • Ecommerce
  • Industry News
  • Local
  • Mobile
  • Paid Search
  • PMG Culture
  • Programmatic & Display
  • SEO
  • Social Media
  • Structured Data
Fort Worth

2845 West 7th Street
Fort Worth, TX 76107

Dallas

3102 Oak Lawn Avenue
Suite 650
Dallas, TX 75219

Austin

823 Congress Avenue
Suite 800
Austin, TX 78701

London

33 Broadwick Street
London
W1F 0DQ

New York

120 East 23rd Street
New York, NY 10010

Get in touch

(817) 420 9970
info@pmg.com

Subscribe to the PMG Newsletter
© 2021 PMG Worldwide, LLC, All Rights Reserved
  • Contact
  • Privacy Policy
 Tweet
 Share
 Tweet
 Share
 Tweet
 Share
 LinkedIn
We and our partners use cookies to personalize content, analyze traffic, and deliver ads. By using our website, you agree to the use of cookies as described in our Cookie Policy.