I’m currently trying to scrape this website using BeautifulSoup in Pycharm to sort all the articles from most upvotes to least upvotes: https://news.ycombinator.com/news
I have successfully parsed the HTML code to get all the articles but now I need to put them in an ordered list by score. I did this by using the following code:
This block of code converts all of the articles into dictionaries giving them titles, links, indexes, and scores from other lists. After the loop ends, it removes the last item of the list since it is just a ‘More’ button and not an actual article.
for a in article_texts:
ALL_ARTICLES.append({"title": a, "link": article_links[article_texts.index(a)], "score": article_scores[article_texts.index(a)], "index": ordered_scores.index(article_scores[article_texts.index(a)])})
ALL_ARTICLES.remove(ALL_ARTICLES[30])
This block of code removes the first occurrence of every article and puts it in the position that it’s dictionary says it has.
for A in ALL_ARTICLES:
ORDERED_ARTICLES.pop(A["index"])
ORDERED_ARTICLES.insert(A["index"], A)
This block of code prints all the articles in proper order and formatting to the user from the ORDERED_ARTICLES list.
for o in ORDERED_ARTICLES:
INDEX = int(o['index']) + 1
TITLE = o['title']
LINK = o['link']
SCORE = o['score']
print(f"{INDEX}. {TITLE} ({LINK})n{SCORE} Upvotes 🔼n")
I have tried this code several times and almost every time it prints differently. Sometimes it prints only the first 14 items, other times it will only print the first 25 items, and sometimes it only prints the first 7 items, and it’s the same with every number.
I know this code works because it eventually printed all 30 items in the list and to make sure it wasn’t a fluke, I did it 2-3 times.
Another problem I have is this block of code gives me TypeError: String indices must be integers every single time whether it be printed at the start or end of my code. This is in reference to the 2nd-5th lines where I try to access the values in o. This error confuses me because o is a dictionary and not a string and I didn’t spell the key name wrong so logically speaking, this is perfectly valid code. I have even tried putting print(type(o)) within the loop and it prints <class 'dict'>.
for o in ORDERED_ARTICLES:
INDEX = int(o['index']) + 1
TITLE = o['title']
LINK = o['link']
SCORE = o['score']
print(f"{INDEX}. {TITLE} ({LINK})n{SCORE} Upvotes 🔼n")
I don’t know if there’s something I’m missing or if it’s a software problem but someone help me 🙏
Update: This is how ALL_ARTICLES is created:
ALL_ARTICLES = []
ORDERED_ARTICLES = ["" for _ in range(31)]
ordered_scores = article_scores
ordered_scores.sort(reverse=True)
for a in article_texts:
ALL_ARTICLES.append({"title": a, "link": article_links[article_texts.index(a)], "score": article_scores[article_texts.index(a)], "index": ordered_scores.index(article_scores[article_texts.index(a)])})
ALL_ARTICLES.remove(ALL_ARTICLES[30])
Update #2: This is the entire project
from bs4 import BeautifulSoup
import requests
import pprint
# import lxml
URL = "https://news.ycombinator.com/news"
response = requests.get(URL)
yc_webpage = response.text
yc_soup = BeautifulSoup(yc_webpage, "html.parser")
articles = yc_soup.find_all(name="td", class_="title")
# articles = yc_soup.select_all(selector='span>a')
article_texts = []
article_links = []
article_scores = []
for s in yc_soup.find_all(name="td", class_="subtext"):
if s.find(name="span", class_="score") is None:
article_scores.append(0)
else:
article_scores.append(int(s.find(name="span", class_="score").text.replace(" points", "")))
# article_scores = [a.text for a in yc_soup.find_all(name="span", class_="score")]
#
# print(article_scores)
# article_text = [tag.text for tag in article_tags]
# # article_link = yc_soup.select_one(selector="span.sitebit.comhead").getText().strip()
# article_links = [t.get("href") for t in article_tags]
# article_points = [p.text for p in yc_soup.find_all(name="span", class_="score")]
for tag in articles:
anchor_tag = tag.find(name="a")
if anchor_tag is None:
pass
else:
article_text = anchor_tag.getText()
article_texts.append(article_text)
article_link = anchor_tag.get("href")
article_links.append(article_link)
# if score is None:
# score = "0 points"
# article_scores.append(score)
article_scores.append(-22)
ALL_ARTICLES = []
ORDERED_ARTICLES = ["" for _ in range(31)]
ordered_scores = article_scores
ordered_scores.sort(reverse=True)
for a in article_texts:
ALL_ARTICLES.append({"title": a, "link": article_links[article_texts.index(a)], "score": article_scores[article_texts.index(a)], "index": ordered_scores.index(article_scores[article_texts.index(a)])})
ALL_ARTICLES.remove(ALL_ARTICLES[30])
for A in ALL_ARTICLES:
ORDERED_ARTICLES.pop(A["index"])
ORDERED_ARTICLES.insert(A["index"], A)
for o in ORDERED_ARTICLES:
print(type(o))
INDEX = int(o['index']) + 1
TITLE = o['title']
LINK = o['link']
SCORE = o['score']
print(f"{INDEX}. {TITLE} ({LINK})n{SCORE} Upvotes 🔼n")
15
From your code, it’s unclear why you would be getting different results for the same content. However, given that your code retrieves an online source, a possible reason for changes would be that the online source has changed – we have no way to verify this for you.
But you could just retrieve the online data once, save it to file and then write and test your code working off of the saved data, to avoid being caught by a surprise change in data.
Your code has some other issues:
- you repeatedly loop through things which seem to be related and could be dealt with in a single go.
- you sort the scores of articles, but don’t change the order of the articles themselves, so you’re not matching articles with their correct scores.
- there’s an unexplained mismatch in the number of articles and the number of scores. (you pad the list of scores with
-22for some reason) - minor issue: you’re naming variables arbitrarily with uppercase and lowercase names, when you should really just be using lowercase (by convention all uppercase is generally reserved for global variables, which should be avoided unless absolutely needed anyway).
- minor issue: you have commented out code sprinkled throughout – it’s unclear if some of the problems you’re talking about has anything to do with that; you’d be better off using some form of versioning like Git.
To achieve what you need, starting with your code and fixing the above:
from pathlib import Path
from bs4 import BeautifulSoup, Tag
import requests
import urllib.parse
# leaving this as a global, you could pass it around if you wanted
URL = "https://news.ycombinator.com/news"
# create a function to load the page, keep your code organised
# load the page from a cache file, or from the web if new, or if you tell it to
def load_yc(url: str, reload: bool = False) -> str:
# read the page from file if you already retrieved it
filename_safe_url = urllib.parse.quote(url, safe='')
cache = Path(f'cache_{filename_safe_url}.txt')
if cache.exists() and not reload:
with open(cache, 'r') as f:
return f.read()
else:
response = requests.get(URL)
with open(cache, 'w') as f:
f.write(response.text)
return response.text
# keeping everything else in a main function, to avoid creating a global mess
def main():
yc_webpage = load_yc(URL)
yc_soup = BeautifulSoup(yc_webpage, "html.parser")
# instead of just looking for 'titles', it appears articles are 'athing' tr tags
# the scores are their next sibling tr
article_tags = yc_soup.find_all(name="tr", class_="athing")
articles = []
# instead of many loops, just loop once over the articles, and get all you need
for article_tag in article_tags:
titleline_anchor = article_tag.find(name='span', class_='titleline').find('a')
title = titleline_anchor .text
link = titleline_anchor ['href']
score_tag = article_tag.next_sibling
# nitpick: if the score is not found, the score will be None
# we know it's a Tag, but it's technically a PageElement, so we should check
if isinstance(score_tag, Tag):
score = score_tag.find(name='span', class_='score')
# a recent article won't have a score yet
if score is None:
score_value = 0
else:
# get the part before 'points' as an integer
score_value = int(score.text.split()[0])
else:
# in case there's no score element, just -1 - something's wrong
score_value = -1
# collect a single list of tuples
articles.append((title, link, score_value))
# sort the articles by score
articles.sort(key=lambda x: x[2], reverse=True)
# print as you did
for i, (title, link, score) in enumerate(articles):
# enumerates numbers them from 0.., so adding 1
if score == -1:
print(f'{i + 1}. {title} ({link})nNo score (error)n')
elif score == 0:
print(f'{i + 1}. {title} ({link})nNo upvotes yet 🔼n')
else:
print(f'{i+1}. {title} ({link})n{score} Upvotes 🔼n')
# if this contains useful functions, you can import them elsewhere
# the line below only runs the main function if this script is run directly
if __name__ == '__main__':
main()
3