{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Requests\n",
    "(c) 2019, Joe James.\n",
    "MIT License.\n",
    "\n",
    "Tutorial on using the [Requests](http://docs.python-requests.org/en/master/user/quickstart/) library to access HTTP requests, GET, POST, PUT, DELETE, HEAD, OPTIONS.  \n",
    "This notebook also covers how to use the Python [JSON](https://docs.python.org/3/library/json.html) library to parse values out of a GET response.  \n",
    "If you don't have the requests library installed you can run 'pip install requests' or some equivalent command for your system in the console window.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get('https://api.github.com/events')\n",
    "r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})\n",
    "r = requests.put('https://httpbin.org/put', data = {'name':'Joe'})\n",
    "r = requests.delete('https://httpbin.org/delete')\n",
    "r = requests.head('https://httpbin.org/get')\n",
    "r = requests.options('https://httpbin.org/get')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GET Requests - Passing Parameters in URLs\n",
    "A URL that returns an HTTP response in JSON format is called an API endpoint.  \n",
    "Here's an example, https://httpbin.org/get  \n",
    "\n",
    "With GET requests we can add parameters onto the URL to retrieve specific data.  \n",
    "We define the params as a dictionary, and add params=payload to the Request.  \n",
    "The Requests library builds the whole URL for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://httpbin.org/get?key1=value1&key2=value2\n"
     ]
    }
   ],
   "source": [
    "payload = {'key1': 'value1', 'key2': 'value2'}\n",
    "r = requests.get('https://httpbin.org/get', params=payload)\n",
    "print(r.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Passing a List as a parameter**  \n",
    "Still use key:value pairs, with the list as the value.  \n",
    "You can see here all the different attributes included in an HTTP Request response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL:  https://httpbin.org/get?key1=value1&key2=value2&key2=value3\n",
      "ENCODING:  None\n",
      "STATUS_CODE:  200\n",
      "HEADERS:  {'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Content-Type': 'application/json', 'Date': 'Tue, 26 Feb 2019 18:13:35 GMT', 'Server': 'nginx', 'Content-Length': '229', 'Connection': 'keep-alive'}\n",
      "TEXT:  {\n",
      "  \"args\": {\n",
      "    \"key1\": \"value1\", \n",
      "    \"key2\": [\n",
      "      \"value2\", \n",
      "      \"value3\"\n",
      "    ]\n",
      "  }, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.21.0\"\n",
      "  }, \n",
      "  \"origin\": \"99.99.39.153, 99.99.39.153\", \n",
      "  \"url\": \"https://httpbin.org/get?key1=value1&key2=value2&key2=value3\"\n",
      "}\n",
      "\n",
      "CONTENT:  b'{\\n  \"args\": {\\n    \"key1\": \"value1\", \\n    \"key2\": [\\n      \"value2\", \\n      \"value3\"\\n    ]\\n  }, \\n  \"headers\": {\\n    \"Accept\": \"*/*\", \\n    \"Accept-Encoding\": \"gzip, deflate\", \\n    \"Host\": \"httpbin.org\", \\n    \"User-Agent\": \"python-requests/2.21.0\"\\n  }, \\n  \"origin\": \"99.99.39.153, 99.99.39.153\", \\n  \"url\": \"https://httpbin.org/get?key1=value1&key2=value2&key2=value3\"\\n}\\n'\n",
      "JSON:  <bound method Response.json of <Response [200]>>\n"
     ]
    }
   ],
   "source": [
    "payload = {'key1': 'value1', 'key2': ['value2', 'value3']}\n",
    "r = requests.get('https://httpbin.org/get', params=payload)\n",
    "print('URL: ', r.url)\n",
    "print('ENCODING: ', r.encoding)\n",
    "print('STATUS_CODE: ', r.status_code)\n",
    "print('HEADERS: ', r.headers)\n",
    "print('TEXT: ', r.text)\n",
    "print('CONTENT: ', r.content)\n",
    "print('JSON: ', r.json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POST Requests\n",
    "We can add parameters to a POST request in Dictionary format, but we use data=payload.  \n",
    "POST requests are used to upload new records to the server.  \n",
    "POST would typically be used to get data from a web form and submit it to the server. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"data\": \"\", \n",
      "  \"files\": {}, \n",
      "  \"form\": {\n",
      "    \"key1\": \"value1\", \n",
      "    \"key2\": \"value2\"\n",
      "  }, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Content-Length\": \"23\", \n",
      "    \"Content-Type\": \"application/x-www-form-urlencoded\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.21.0\"\n",
      "  }, \n",
      "  \"json\": null, \n",
      "  \"origin\": \"99.99.39.153, 99.99.39.153\", \n",
      "  \"url\": \"https://httpbin.org/post\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r = requests.post('https://httpbin.org/post', data = {'name':'Joe'})\n",
    "\n",
    "payload = {'key1': 'value1', 'key2': 'value2'}\n",
    "r = requests.post(\"https://httpbin.org/post\", data=payload)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Requests to GET Currency Exchange Data\n",
    "Here's a handy endpoint where we can GET foreign currency exchange rates in JSON format, https://api.exchangeratesapi.io/latest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"rates\":{\"MXN\":21.7145,\"AUD\":1.5897,\"HKD\":8.9178,\"RON\":4.7626,\"HRK\":7.4275,\"CHF\":1.1371,\"IDR\":15917.9,\"CAD\":1.5024,\"USD\":1.1361,\"ZAR\":15.752,\"JPY\":125.93,\"BRL\":4.2574,\"HUF\":317.06,\"CZK\":25.663,\"NOK\":9.7725,\"INR\":80.853,\"PLN\":4.3282,\"ISK\":136.1,\"PHP\":59.144,\"SEK\":10.5858,\"ILS\":4.1148,\"GBP\":0.86055,\"SGD\":1.5332,\"CNY\":7.6077,\"TRY\":6.0254,\"MYR\":4.6157,\"RUB\":74.6158,\"NZD\":1.652,\"KRW\":1270.0,\"THB\":35.583,\"BGN\":1.9558,\"DKK\":7.4616},\"base\":\"EUR\",\"date\":\"2019-02-26\"}\n"
     ]
    }
   ],
   "source": [
    "url = 'https://api.exchangeratesapi.io/latest'\n",
    "r = requests.get(url)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**It looks like the default is base:EUR, but we want exchange rates for USD, so we can pass in a parameter for base.  \n",
    "We can also put in any date we want.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"rates\":{\"MXN\":18.8315549401,\"AUD\":1.2571475116,\"HKD\":7.823572534,\"RON\":3.7694876599,\"HRK\":6.0552252179,\"CHF\":0.9610654069,\"IDR\":13307.03754989,\"CAD\":1.2432190274,\"USD\":1.0,\"JPY\":110.621487334,\"BRL\":3.1959762157,\"PHP\":50.2997474953,\"CZK\":20.7957970188,\"NOK\":7.8771686894,\"INR\":63.5175531482,\"PLN\":3.3954549157,\"MYR\":3.9560153132,\"ZAR\":12.302191089,\"ILS\":3.399609025,\"GBP\":0.7252830496,\"SGD\":1.3214140262,\"HUF\":251.6086991936,\"EUR\":0.8145312373,\"CNY\":6.4380548994,\"TRY\":3.7828459721,\"SEK\":8.0096929217,\"RUB\":56.4333306182,\"NZD\":1.3706931661,\"KRW\":1063.5660177568,\"THB\":31.9247373137,\"BGN\":1.5930601939,\"DKK\":6.0679319052},\"base\":\"USD\",\"date\":\"2018-01-15\"}\n"
     ]
    }
   ],
   "source": [
    "url = 'https://api.exchangeratesapi.io/2018-01-15'\n",
    "r = requests.get(url, params={'base':'USD'})\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding JSON data\n",
    "Now we have the rates in JSON format. We need to convert that to usable data. \n",
    "The JSON library basically has two functions:  \n",
    "- json.loads( ) converts a text string into Python dict/list objects.  \n",
    "- json.dumps( ) converts dict/list objects into a string.  \n",
    "\n",
    "We need to decode the JSON data into a dictionary, then get the rate for GBP, convert it to a float, and do a conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'MXN': 18.8315549401, 'AUD': 1.2571475116, 'HKD': 7.823572534, 'RON': 3.7694876599, 'HRK': 6.0552252179, 'CHF': 0.9610654069, 'IDR': 13307.03754989, 'CAD': 1.2432190274, 'USD': 1.0, 'JPY': 110.621487334, 'BRL': 3.1959762157, 'PHP': 50.2997474953, 'CZK': 20.7957970188, 'NOK': 7.8771686894, 'INR': 63.5175531482, 'PLN': 3.3954549157, 'MYR': 3.9560153132, 'ZAR': 12.302191089, 'ILS': 3.399609025, 'GBP': 0.7252830496, 'SGD': 1.3214140262, 'HUF': 251.6086991936, 'EUR': 0.8145312373, 'CNY': 6.4380548994, 'TRY': 3.7828459721, 'SEK': 8.0096929217, 'RUB': 56.4333306182, 'NZD': 1.3706931661, 'KRW': 1063.5660177568, 'THB': 31.9247373137, 'BGN': 1.5930601939, 'DKK': 6.0679319052}\n",
      "0.7252830496\n",
      "200USD =  145.05660992 GBP\n"
     ]
    }
   ],
   "source": [
    "rates_json = json.loads(r.text)['rates']\n",
    "print(rates_json)\n",
    "print(rates_json['GBP'])\n",
    "gbp = float(rates_json['GBP'])\n",
    "print('200USD = ', str(gbp * 200), 'GBP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Requests to GET Song Data\n",
    "Every API has documentation on how to use it.  \n",
    "You can find the docs for this Song Data API [here.](https://documenter.getpostman.com/view/3719697/RzfarXB4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"id\":12,\"name\":\"Beatles\",\"year_started\":1960,\"year_quit\":1970,\"text\":\"Beatles\"},{\"id\":14,\"name\":\"Dario G\",\"year_started\":1997,\"year_quit\":null,\"text\":\"Dario G\"},{\"id\":16,\"name\":\"Fleetwood Mac\",\"year_started\":1967,\"year_quit\":null,\"text\":\"Fleetwood Mac\"},{\"id\":17,\"name\":\"Blink 182\",\"year_started\":1992,\"year_quit\":null,\"text\":\"Blink 182\"},{\"id\":18,\"name\":\"Bloc Party\",\"year_started\":2002,\"year_quit\":null,\"text\":\"Bloc Party\"},{\"id\":19,\"name\":\"The Temper Trap\",\"year_started\":2005,\"year_quit\":null,\"text\":\"The Temper Trap\"},{\"id\":20,\"name\":\"MGMT\",\"year_started\":2002,\"year_quit\":null,\"text\":\"MGMT\"},{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\"},{\"id\":22,\"name\":\"\n"
     ]
    }
   ],
   "source": [
    "url = 'https://musicdemons.com/api/v1/artist'\n",
    "r = requests.get(url)\n",
    "print(r.text[:700])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\"}\n"
     ]
    }
   ],
   "source": [
    "url = 'https://musicdemons.com/api/v1/artist/21'\n",
    "r = requests.get(url)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":21,\"name\":\"Coldplay\",\"year_started\":1996,\"year_quit\":null,\"text\":\"Coldplay\",\"songs\":[{\"id\":1,\"title\":\"Something Just Like This\",\"released\":\"02\\/22\\/2017\",\"text\":\"Something Just Like This\",\"youtube_id\":\"FM7MFYoylVs\",\"pivot\":{\"artist_id\":21,\"song_id\":1},\"subject\":{\"id\":226,\"subjectable_id\":1,\"subjectable_type\":\"App\\\\Entities\\\\MusicDemons\\\\Song\"}},{\"id\":11,\"title\":\"Hymn For The Weekend\",\"released\":\"01\\/25\\/2016\",\"text\":\"Hymn For The Weekend\",\"youtube_id\":\"YykjpeuMNEk\",\"pivot\":{\"artist_id\":21,\"song_id\":11},\"subject\":{\"id\":233,\"subjectable_id\":11,\"subjectable_type\":\"App\\\\Entities\\\\MusicDemons\\\\Song\"}},{\"id\":78,\"title\":\"Sky Full Of Stars\",\"released\":\"05\\/02\\/2014\",\"text\":\"Sky Full Of Stars\",\n"
     ]
    }
   ],
   "source": [
    "url = 'https://musicdemons.com/api/v1/artist/21'\n",
    "headers = {'with': 'songs,members'}\n",
    "r = requests.get(url, headers=headers)\n",
    "print(r.text[:700])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coldplay\n",
      "Something Just Like This\n",
      "Hymn For The Weekend\n",
      "Sky Full Of Stars\n",
      "Fix You\n",
      "Brothers & Sisters\n",
      "Shiver\n",
      "The Scientist\n",
      "Yellow\n",
      "Trouble\n",
      "Every Teardrop Is a Waterfall\n",
      "Life in Technicolor ii\n",
      "Adventure Of A Lifetime\n",
      "Magic\n",
      "The Hardest Part\n",
      "Viva la Vida\n",
      "1.36\n",
      "42\n",
      "A Head Full of Dreams\n",
      "A Hopeful Transmission\n",
      "A Message\n",
      "A Rush of Blood to the Head\n",
      "Princess of China\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "text_json = json.loads(r.text)\n",
    "print(text_json['name'])\n",
    "for song in text_json['songs']:\n",
    "    print(song['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tips on breaking down JSON  \n",
    "To get data out of a JSON object, which is a combination of lists and dictionaries,  \n",
    "just remember for lists you need a numerical index, and for key-value pairs you need a text index.  \n",
    "So if the object looks like this, {\"cars\":[\"id\":1,\"model\":\"Camry\"... you can access the model of the first car with text['cars'][0]['model']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}