Python - JSON

rgbeard · 01-22-2018, 01:38 PM

Python newbie, JSON newbie. Trying to do something simple. I hope someone here has patience to help lead me down the right path.

I want to print from a news feed API.

I want to be able to print titles and descriptions from the top news articles of the moment. JSON is loading the data into an array from what I can tell, but I don't understand how to print an individual piece of the data.

I WANT to create something like:

Here's today's top stories from the BBC:
Larry Nassar: USA Gymnastics' executive leadership resigns over abuse scandal
Advance cargo information is now required for flights to the US from five countries.

Here's what I have so far....

PYTHON CODE:

import requests, pprint
url = ('https://newsapi.org/v2/top-headlines?'
'sources=bbc-news&'
'apiKey=9e64da8475f74d9ea5e54137fd702a1c')
response = requests.get(url)
print response.json()

JSON RESPONSE:

{u'status': u'ok', u'articles': [{u'description': u"The executive leadership of USA Gymnastics' board of directors resigns in the wake of the Larry Nassar sexual abuse case.", u'title': u"Larry Nassar: USA Gymnastics' executive leadership resigns over abuse scandal", u'url': u'http://www.bbc.co.uk/sport/gymnastics/42782387', u'author': u'BBC Sport', u'publishedAt': u'2018-01-22T18:43:04Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/onesport/cps/624/cpsprodpb/11A80/production/_99702327_nassar_reuters.jpg'}, {u'description': u'Advance cargo information is now required for flights to the US from five countries.', u'title': u'New US cargo checks for Mid-East flights', u'url': u'http://www.bbc.co.uk/news/world-us-canada-42780223', u'author': u'BBC News', u'publishedAt': u'2018-01-22T17:22:51Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/15ED2/production/_99701898_gettyimages-106387173.jpg'}, {u'description': u'US government shutdown to end after Senate Republicans and Democrats reach temporary deal on federal budget', u'title': u"Senate deal 'to halt government shutdown'", u'url': u'http://www.bbc.co.uk/news/world-us-canada-42781880', u'author': u'BBC News', u'publishedAt': u'2018-01-22T17:22:15Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/7A23/production/_97176213_breaking_news_bigger.png'}, {u'description': u'Pope Francis apologises for "hurting" sex abuse victims whom he accused of slandering a Chilean bishop', u'title': u'Pope sorry for upsetting abuse victims', u'url': u'http://www.bbc.co.uk/news/world-latin-america-42780327', u'author': u'BBC News', u'publishedAt': u'2018-01-22T15:42:14Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef-1.bbci.co.uk/news/1024/branded_news/7A23/production/_97176213_breaking_news_bigger.png'}, {u'description': u'Vice-President Mike Pence makes the announcement in a speech to the Israeli parliament.', u'title': u'US embassy to move to Jerusalem by 2019', u'url': u'http://www.bbc.co.uk/news/world-middle-east-42773456', u'author': u'BBC News', u'publishedAt': u'2018-01-22T13:23:14Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/149F/production/_99697250_mediaitem99697248.jpg'}, {u'description': u'Ancient fragments written in code reveal new details about an ancient Jewish sect.', u'title': u'Mysterious Dead Sea Scroll deciphered', u'url': u'http://www.bbc.co.uk/news/world-middle-east-42773878', u'author': u'BBC News', u'publishedAt': u'2018-01-22T12:18:04Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/B4B3/production/_99695264_rsz_dead_sea_scroll.jpg'}, {u'description': u'The dark comedy scoops three prizes including best actress for Frances McDormand.', u'title': u'Three Billboards triumphs at SAG awards', u'url': u'http://www.bbc.co.uk/news/entertainment-arts-42772498', u'author': u'BBC News', u'publishedAt': u'2018-01-22T11:17:18Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/10F3F/production/_99693496_gettyimages-mcdormand.jpg'}, {u'description': u"The city's mayor promises justice for four Japanese tourists charged \u20ac1,100 for a meal.", u'title': u'Tourists gulp at huge Venice steak bill', u'url': u'http://www.bbc.co.uk/news/world-europe-42772609', u'author': u'BBC News', u'publishedAt': u'2018-01-22T11:16:59Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/87B3/production/_99693743_steakgetty15.jpg'}, {u'description': u'The ground offensive in the Afrin region is aimed at ousting Kurdish fighters, Turkey says.', u'title': u"Turkey 'takes villages' in north Syria", u'url': u'http://www.bbc.co.uk/news/world-middle-east-42771469', u'author': u'BBC News', u'publishedAt': u'2018-01-22T10:57:29Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/124A7/production/_99691947_hi044215452.jpg'}, {u'description': u'Amazon opens its first grocery store without cashiers to shoppers in Seattle on Monday.', u'title': u'The supermarket with no checkouts', u'url': u'http://www.bbc.co.uk/news/business-42769096', u'author': u'BBC News', u'publishedAt': u'2018-01-21T22:43:11Z', u'source': {u'id': u'bbc-news', u'name': u'BBC News'}, u'urlToImage': u'https://ichef.bbci.co.uk/news/1024/branded_news/1BBA/production/_99689070_hi044213884.jpg'}], u'totalResults': 10}

Sefyir · 01-22-2018, 02:00 PM

Use the reponse.json with the json module to make sense of it

Code:

import json
data = response.json()
print(data.keys())
print(data['articles'][0])
print(data['articles'][0]['description'])

Code:

dict_keys(['status', 'totalResults', 'articles'])
{'source': {'id': 'bbc-news', 'name': 'BBC News'}, 'author': 'BBC Sport', 'title': "Larry Nassar: USA Gymnastics' executive leadership resigns over abuse scandal", 'description': "The executive leadership of USA Gymnastics' board of directors resigns in the wake of the Larry Nassar sexual abuse case.", 'url': 'http://www.bbc.co.uk/sport/gymnastics/42782387', 'urlToImage': 'https://ichef.bbci.co.uk/onesport/cps/624/cpsprodpb/11A80/production/_99702327_nassar_reuters.jpg', 'publishedAt': '2018-01-22T18:43:04Z'}
The executive leadership of USA Gymnastics' board of directors resigns in the wake of the Larry Nassar sexual abuse case.

rgbeard · 01-22-2018, 02:10 PM

That's quite helpful! Thank you so very much for the response!

astrogeek · 01-22-2018, 02:17 PM

Please place your code and data snippets inside [CODE]...[/CODE] tags for better readability. You may type those yourself or click the "#" button in the edit controls.

As sefyir has pointed out, using the facilities of your language of choice to access the JSON response is the way to go. The idea is that JSON imposes a well defined structure on the information which may then be used to navigate through it. Programming languages understand that structure and provide methods for the navigation and data access, Python's json library in this case.

Sefyir · 01-22-2018, 04:23 PM

Quote:

Originally Posted by Sefyir

Use the reponse.json with the json module to make sense of it

Code:

import json
data = response.json()
print(data.keys())
print(data['articles'][0])
print(data['articles'][0]['description'])

Code:

dict_keys(['status', 'totalResults', 'articles'])
{'source': {'id': 'bbc-news', 'name': 'BBC News'}, 'author': 'BBC Sport', 'title': "Larry Nassar: USA Gymnastics' executive leadership resigns over abuse scandal", 'description': "The executive leadership of USA Gymnastics' board of directors resigns in the wake of the Larry Nassar sexual abuse case.", 'url': 'http://www.bbc.co.uk/sport/gymnastics/42782387', 'urlToImage': 'https://ichef.bbci.co.uk/onesport/cps/624/cpsprodpb/11A80/production/_99702327_nassar_reuters.jpg', 'publishedAt': '2018-01-22T18:43:04Z'}
The executive leadership of USA Gymnastics' board of directors resigns in the wake of the Larry Nassar sexual abuse case.

I wrote the above in about a minute to help explain the information you were looking at.
In doing so, I might of created some confusion about the role about what the json module is doing, what requests is doing.

The requests module, when using the requests.json method, is actually converting it into something python can make sense of.

When you call response.json(), the () part of it is running some internal code, likely something along the lines of

Code:

return json.loads(self.content)

.. Which you can then make use of.

Some code:

Code:

import json
import requests

response = requests.get(url)
data_raw = response.content
data = response.json()

print(data['articles'][0]['description'])
# print(data_raw['articles'][0]['description']) # Will error!
converted_content = json.loads(data_raw)
print(converted_content['articles'][0]['description'])

Code:

The government will reopen. Everyone involved is declaring victory. And everybody's right. And wrong.
The government will reopen. Everyone involved is declaring victory. And everybody's right. And wrong.

When taking the raw content and trying to access it like a dictionary will error because it's considered to just be a string.

When json.loads(data_raw) is run on the raw content (whether in the requests module or directly with json module), it turns it into a python dict (dictionary) object that you can perform actions on, like data.keys()