2019-02-26 13:40:54 -08:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Python Requests\n",
"(c) 2019, Joe James.\n",
"MIT License.\n",
"\n",
"Tutorial on using the [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library to access HTTP requests, GET, POST, PUT, DELETE, HEAD, OPTIONS. \n",
"This notebook also covers how to use the Python [JSON](https://docs.python.org/3/library/json.html) library to parse values out of a GET response. \n",
"If you don't have the requests library installed you can run 'pip install requests' or some equivalent command for your system in the console window. "
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [],
"source": [
2020-05-27 10:01:14 -07:00
"import requests\n",
"import json"
2019-02-26 13:40:54 -08:00
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [],
"source": [
"r = requests.get('https://api.github.com/events')\n",
"r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})\n",
"r = requests.put('https://httpbin.org/put', data = {'name':'Joe'})\n",
"r = requests.delete('https://httpbin.org/delete')\n",
"r = requests.head('https://httpbin.org/get')\n",
"r = requests.options('https://httpbin.org/get')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GET Requests - Passing Parameters in URLs\n",
"A URL that returns an HTTP response in JSON format is called an API endpoint. \n",
"Here's an example, https://httpbin.org/get \n",
"\n",
"With GET requests we can add parameters onto the URL to retrieve specific data. \n",
"We define the params as a dictionary, and add params=payload to the Request. \n",
"The Requests library builds the whole URL for us."
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://httpbin.org/get?key1=value1&key2=value2\n"
]
}
],
"source": [
"payload = {'key1': 'value1', 'key2': 'value2'}\n",
"r = requests.get('https://httpbin.org/get', params=payload)\n",
"print(r.url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Passing a List as a parameter** \n",
"Still use key:value pairs, with the list as the value. \n",
"You can see here all the different attributes included in an HTTP Request response."
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"URL: https://httpbin.org/get?key1=value1&key2=value2&key2=value3\n",
"ENCODING: None\n",
"STATUS_CODE: 200\n",
"HEADERS: {'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Content-Type': 'application/json', 'Date': 'Tue, 26 Feb 2019 18:13:35 GMT', 'Server': 'nginx', 'Content-Length': '229', 'Connection': 'keep-alive'}\n",
"TEXT: {\n",
" \"args\": {\n",
" \"key1\": \"value1\", \n",
" \"key2\": [\n",
" \"value2\", \n",
" \"value3\"\n",
" ]\n",
" }, \n",
" \"headers\": {\n",
" \"Accept\": \"*/*\", \n",
" \"Accept-Encoding\": \"gzip, deflate\", \n",
" \"Host\": \"httpbin.org\", \n",
" \"User-Agent\": \"python-requests/2.21.0\"\n",
" }, \n",
" \"origin\": \"99.99.39.153, 99.99.39.153\", \n",
" \"url\": \"https://httpbin.org/get?key1=value1&key2=value2&key2=value3\"\n",
"}\n",
"\n",
"CONTENT: b'{\\n \"args\": {\\n \"key1\": \"value1\", \\n \"key2\": [\\n \"value2\", \\n \"value3\"\\n ]\\n }, \\n \"headers\": {\\n \"Accept\": \"*/*\", \\n \"Accept-Encoding\": \"gzip, deflate\", \\n \"Host\": \"httpbin.org\", \\n \"User-Agent\": \"python-requests/2.21.0\"\\n }, \\n \"origin\": \"99.99.39.153, 99.99.39.153\", \\n \"url\": \"https://httpbin.org/get?key1=value1&key2=value2&key2=value3\"\\n}\\n'\n",
"JSON: <bound method Response.json of <Response [200]>>\n"
]
}
],
"source": [
"payload = {'key1': 'value1', 'key2': ['value2', 'value3']}\n",
"r = requests.get('https://httpbin.org/get', params=payload)\n",
"print('URL: ', r.url)\n",
"print('ENCODING: ', r.encoding)\n",
"print('STATUS_CODE: ', r.status_code)\n",
"print('HEADERS: ', r.headers)\n",
"print('TEXT: ', r.text)\n",
"print('CONTENT: ', r.content)\n",
"print('JSON: ', r.json)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### POST Requests\n",
"We can add parameters to a POST request in Dictionary format, but we use data=payload. \n",
"POST requests are used to upload new records to the server. \n",
"POST would typically be used to get data from a web form and submit it to the server. "
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"args\": {}, \n",
" \"data\": \"\", \n",
" \"files\": {}, \n",
" \"form\": {\n",
" \"key1\": \"value1\", \n",
" \"key2\": \"value2\"\n",
" }, \n",
" \"headers\": {\n",
" \"Accept\": \"*/*\", \n",
" \"Accept-Encoding\": \"gzip, deflate\", \n",
" \"Content-Length\": \"23\", \n",
" \"Content-Type\": \"application/x-www-form-urlencoded\", \n",
" \"Host\": \"httpbin.org\", \n",
" \"User-Agent\": \"python-requests/2.21.0\"\n",
" }, \n",
" \"json\": null, \n",
" \"origin\": \"99.99.39.153, 99.99.39.153\", \n",
" \"url\": \"https://httpbin.org/post\"\n",
"}\n",
"\n"
]
}
],
"source": [
"r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})\n",
"\n",
"payload = {'key1': 'value1', 'key2': 'value2'}\n",
"r = requests.post(\"https://httpbin.org/post\", data=payload)\n",
"print(r.text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using Requests to GET Currency Exchange Data\n",
"Here's a handy endpoint where we can GET foreign currency exchange rates in JSON format, https://api.exchangeratesapi.io/latest"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"rates\":{\"MXN\":21.7145,\"AUD\":1.5897,\"HKD\":8.9178,\"RON\":4.7626,\"HRK\":7.4275,\"CHF\":1.1371,\"IDR\":15917.9,\"CAD\":1.5024,\"USD\":1.1361,\"ZAR\":15.752,\"JPY\":125.93,\"BRL\":4.2574,\"HUF\":317.06,\"CZK\":25.663,\"NOK\":9.7725,\"INR\":80.853,\"PLN\":4.3282,\"ISK\":136.1,\"PHP\":59.144,\"SEK\":10.5858,\"ILS\":4.1148,\"GBP\":0.86055,\"SGD\":1.5332,\"CNY\":7.6077,\"TRY\":6.0254,\"MYR\":4.6157,\"RUB\":74.6158,\"NZD\":1.652,\"KRW\":1270.0,\"THB\":35.583,\"BGN\":1.9558,\"DKK\":7.4616},\"base\":\"EUR\",\"date\":\"2019-02-26\"}\n"
]
}
],
"source": [
"url = 'https://api.exchangeratesapi.io/latest'\n",
"r = requests.get(url)\n",
"print(r.text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**It looks like the default is base:EUR, but we want exchange rates for USD, so we can pass in a parameter for base. \n",
"We can also put in any date we want.**"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"rates\":{\"MXN\":18.8315549401,\"AUD\":1.2571475116,\"HKD\":7.823572534,\"RON\":3.7694876599,\"HRK\":6.0552252179,\"CHF\":0.9610654069,\"IDR\":13307.03754989,\"CAD\":1.2432190274,\"USD\":1.0,\"JPY\":110.621487334,\"BRL\":3.1959762157,\"PHP\":50.2997474953,\"CZK\":20.7957970188,\"NOK\":7.8771686894,\"INR\":63.5175531482,\"PLN\":3.3954549157,\"MYR\":3.9560153132,\"ZAR\":12.302191089,\"ILS\":3.399609025,\"GBP\":0.7252830496,\"SGD\":1.3214140262,\"HUF\":251.6086991936,\"EUR\":0.8145312373,\"CNY\":6.4380548994,\"TRY\":3.7828459721,\"SEK\":8.0096929217,\"RUB\":56.4333306182,\"NZD\":1.3706931661,\"KRW\":1063.5660177568,\"THB\":31.9247373137,\"BGN\":1.5930601939,\"DKK\":6.0679319052},\"base\":\"USD\",\"date\":\"2018-01-15\"}\n"
]
}
],
"source": [
"url = 'https://api.exchangeratesapi.io/2018-01-15'\n",
"r = requests.get(url, params={'base':'USD'})\n",
"print(r.text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Decoding JSON data\n",
"Now we have the rates in JSON format. We need to convert that to usable data. \n",
"The JSON library basically has two functions: \n",
"- json.loads( ) converts a text string into Python dict/list objects. \n",
"- json.dumps( ) converts dict/list objects into a string. \n",
"\n",
"We need to decode the JSON data into a dictionary, then get the rate for GBP, convert it to a float, and do a conversion."
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'MXN': 18.8315549401, 'AUD': 1.2571475116, 'HKD': 7.823572534, 'RON': 3.7694876599, 'HRK': 6.0552252179, 'CHF': 0.9610654069, 'IDR': 13307.03754989, 'CAD': 1.2432190274, 'USD': 1.0, 'JPY': 110.621487334, 'BRL': 3.1959762157, 'PHP': 50.2997474953, 'CZK': 20.7957970188, 'NOK': 7.8771686894, 'INR': 63.5175531482, 'PLN': 3.3954549157, 'MYR': 3.9560153132, 'ZAR': 12.302191089, 'ILS': 3.399609025, 'GBP': 0.7252830496, 'SGD': 1.3214140262, 'HUF': 251.6086991936, 'EUR': 0.8145312373, 'CNY': 6.4380548994, 'TRY': 3.7828459721, 'SEK': 8.0096929217, 'RUB': 56.4333306182, 'NZD': 1.3706931661, 'KRW': 1063.5660177568, 'THB': 31.9247373137, 'BGN': 1.5930601939, 'DKK': 6.0679319052}\n",
"0.7252830496\n",
"200USD = 145.05660992 GBP\n"
]
}
],
"source": [
"rates_json = json.loads(r.text)['rates']\n",
"print(rates_json)\n",
"print(rates_json['GBP'])\n",
"gbp = float(rates_json['GBP'])\n",
"print('200USD = ', str(gbp * 200), 'GBP')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using Requests to GET Song Data\n",
"Every API has documentation on how to use it. \n",
"You can find the docs for this Song Data API [here.](https://documenter.getpostman.com/view/3719697/RzfarXB4)"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[{\"id\":12,\"name\":\"Beatles\",\"year_started\":1960,\"year_quit\":1970,\"text\":\"Beatles\"},{\"id\":14,\"name\":\"Dario G\",\"year_started\":1997,\"year_quit\":null,\"text\":\"Dario G\"},{\"id\":16,\"name\":\"Fleetwood Mac\",\"year_started\":1967,\"year_quit\":null,\"text\":\"Fleetwood Mac\"},{\"id\":17,\"name\":\"Blink 182\",\"year_started\":1992,\"year_quit\":null,\"text\":\"Blink 182\"},{\"id\":18,\"name\":\"Bloc Party\",\"year_started\":2002,\"year_quit\":null,\"text\":\"Bloc Party\"},{\"id\":19,\"name\":\"The Temper Trap\",\"year_started\":2005,\"year_quit\":null,\"text\":\"The Temper Trap\"},{\"id\":20,\"name\":\"MGMT\",\"year_started\":2002,\"year_quit\":null,\"text\":\"MGMT\"},{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\"},{\"id\":22,\"name\":\"\n"
]
}
],
"source": [
"url = 'https://musicdemons.com/api/v1/artist'\n",
"r = requests.get(url)\n",
"print(r.text[:700])"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\"}\n"
]
}
],
"source": [
"url = 'https://musicdemons.com/api/v1/artist/21'\n",
"r = requests.get(url)\n",
"print(r.text)"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\",\"songs\":[{\"id\":1,\"title\":\"Something Just Like This\",\"released\":\"02\\/22\\/2017\",\"text\":\"Something Just Like This\",\"youtube_id\":\"FM7MFYoylVs\",\"pivot\":{\"artist_id\":21,\"song_id\":1},\"subject\":{\"id\":226,\"subjectable_id\":1,\"subjectable_type\":\"App\\\\Entities\\\\MusicDemons\\\\Song\"}},{\"id\":11,\"title\":\"Hymn For The Weekend\",\"released\":\"01\\/25\\/2016\",\"text\":\"Hymn For The Weekend\",\"youtube_id\":\"YykjpeuMNEk\",\"pivot\":{\"artist_id\":21,\"song_id\":11},\"subject\":{\"id\":233,\"subjectable_id\":11,\"subjectable_type\":\"App\\\\Entities\\\\MusicDemons\\\\Song\"}},{\"id\":78,\"title\":\"Sky Full Of Stars\",\"released\":\"05\\/02\\/2014\",\"text\":\"Sky Full Of Stars\",\n"
]
}
],
"source": [
"url = 'https://musicdemons.com/api/v1/artist/21'\n",
"headers = {'with': 'songs,members'}\n",
"r = requests.get(url, headers=headers)\n",
"print(r.text[:700])"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Coldplay\n",
"Something Just Like This\n",
"Hymn For The Weekend\n",
"Sky Full Of Stars\n",
"Fix You\n",
"Brothers & Sisters\n",
"Shiver\n",
"The Scientist\n",
"Yellow\n",
"Trouble\n",
"Every Teardrop Is a Waterfall\n",
"Life in Technicolor ii\n",
"Adventure Of A Lifetime\n",
"Magic\n",
"The Hardest Part\n",
"Viva la Vida\n",
"1.36\n",
"42\n",
"A Head Full of Dreams\n",
"A Hopeful Transmission\n",
"A Message\n",
"A Rush of Blood to the Head\n",
"Princess of China\n"
]
}
],
"source": [
"import json\n",
"text_json = json.loads(r.text)\n",
"print(text_json['name'])\n",
"for song in text_json['songs']:\n",
" print(song['title'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tips on breaking down JSON \n",
"To get data out of a JSON object, which is a combination of lists and dictionaries, \n",
"just remember for lists you need a numerical index, and for key-value pairs you need a text index. \n",
"So if the object looks like this, {\"cars\":[\"id\":1,\"model\":\"Camry\"... you can access the model of the first car with text['cars'][0]['model']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
