Exploring Hacker News Posts with Python

For this project we're going to explore some Hacker News Data using mostly vanilla python. Hacker News is a reddit like news website that allows users to submit posts, and then vote and comment on them.

The two types of posts we'll explore begin with either Ask HN or Show HN.

Users submit Ask HN posts to ask the Hacker News community a specific question, such as "What is the best online course you've ever taken?" Likewise, users submit Show HN posts to show the Hacker News community a project, product, or just generally something interesting.

We'll specifically compare these two types of posts to determine the following:

  • Do Ask HN or Show HN receive more comments on average?
  • Do posts created at a certain time receive more comments on average?

It should be noted that the data set we're working with was reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions.

Documentation for the dataset can be found here.

Opening the File

In [2]:
opened_file = open('hacker_news.csv')
from csv import reader
read_file = reader(opened_file)
hn = list(read_file)
In [3]:
# extract headers
headers = hn[0]

# reassign list of rows without header row
hn = hn[1:]

print(headers)
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']

Column Information

id: The unique identifier from Hacker News for the post

title: The title of the post

url: The URL that the posts links to, if it the post has a URL

num_points: The number of points the post acquired, calculated as the total number of upvotes minus the total number of downvotes

num_comments: The number of comments that were made on the post

author: The username of the person who submitted the post

created_at: The date and time at which the post was submitted

Sample of First 5 Rows of Data

In [4]:
hn[:5]
Out[4]:
[['12224879',
  'Interactive Dynamic Video',
  'http://www.interactivedynamicvideo.com/',
  '386',
  '52',
  'ne0phyte',
  '8/4/2016 11:52'],
 ['10975351',
  'How to Use Open Source and Shut the Fuck Up at the Same Time',
  'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/',
  '39',
  '10',
  'josep2',
  '1/26/2016 19:30'],
 ['11964716',
  "Florida DJs May Face Felony for April Fools' Water Joke",
  'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/',
  '2',
  '1',
  'vezycash',
  '6/23/2016 22:20'],
 ['11919867',
  'Technology ventures: From Idea to Enterprise',
  'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429',
  '3',
  '1',
  'hswarna',
  '6/17/2016 0:01'],
 ['10301696',
  'Note by Note: The Making of Steinway L1037 (2007)',
  'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0',
  '8',
  '2',
  'walterbell',
  '9/30/2015 4:12']]

Extracting Ask HN and Show HN Posts

First, we'll identify posts that begin with either Ask HN or Show HN and separate the data for those two types of posts into different lists. Separating the data makes it easier to analyze in the following steps.

In [5]:
ask_posts = []
show_posts = []
other_posts = []

for row in hn:
    title = row[1]
    if title.lower().startswith("ask hn"):
        ask_posts.append(row)
    elif title.lower().startswith("show hn"):
        show_posts.append(row)
    else:
        other_posts.append(row)

print("# Ask Posts: ", len(ask_posts))
print("# Show Posts: ", len(show_posts))
print("# Other Posts:", len(other_posts))
    
# Ask Posts:  1744
# Show Posts:  1162
# Other Posts: 17194

Do Ask HN or Show HN posts recieve more comments on average?

Now that we separated ask posts and show posts into different lists, we'll calculate the average number of comments each type of post receives.

In [6]:
# calculate average number of Show HN Comments
total_ask_comments = 0

for row in ask_posts:
    comments = int(row[4])
    total_ask_comments += comments
    
avg_ask_comments = total_ask_comments / len(ask_posts)

total_show_comments = 0

# calculate average number of Show HN Comments
for row in show_posts:
    comments= int(row[4])
    total_show_comments += comments
    
avg_show_comments = total_show_comments / len(show_posts)

print("Avg Ask HN Comments:", avg_ask_comments)
print("Avg Show HN Comments:", avg_show_comments)
    
Avg Ask HN Comments: 14.038417431192661
Avg Show HN Comments: 10.31669535283993

On average, ask posts in our sample receive approximately 14 comments, whereas show posts receive approximately 10. Since ask posts are more likely to receive comments, we'll focus our remaining analysis just on these posts.

Do Ask HN Posts Recieve More Comments at Certain Times?

Next, we'll determine if we can maximize the amount of comments an ask post receives by creating it at a certain time. First, we'll find the amount of ask posts created during each hour of day, along with the number of comments those posts received. Then, we'll calculate the average amount of comments ask posts created at each hour of the day receive.

In [7]:
import datetime as dt

result_list = []

# extract 'created_at' and 'num_comments' columns from ask_posts into new list
for row in ask_posts:
    result_list.append([row[6], int(row[4])])
    
counts_by_hour = {}
comments_by_hour = {}
date_format = "%m/%d/%Y %H:%M"

# calculate how many comments / counts there are per hour, sort into frequency tables
for row in result_list:
    date = row[0]
    comments = row[1]
    hour = dt.datetime.strptime(date, date_format).strftime("%H")
    if hour not in counts_by_hour:
        counts_by_hour[hour] = 1
        comments_by_hour[hour] = comments
    else: 
        counts_by_hour[hour] += 1
        comments_by_hour[hour] += comments
        

 
comments_by_hour
    
    
Out[7]:
{'00': 447,
 '01': 683,
 '02': 1381,
 '03': 421,
 '04': 337,
 '05': 464,
 '06': 397,
 '07': 267,
 '08': 492,
 '09': 251,
 '10': 793,
 '11': 641,
 '12': 687,
 '13': 1253,
 '14': 1416,
 '15': 4477,
 '16': 1814,
 '17': 1146,
 '18': 1439,
 '19': 1188,
 '20': 1722,
 '21': 1745,
 '22': 479,
 '23': 543}

Calculate the Average Number of Comments Per Post By Hour

In [8]:
avg_by_hour = []

for hr in comments_by_hour:
     avg_by_hour.append([hr, comments_by_hour[hr] / counts_by_hour[hr]])

avg_by_hour
Out[8]:
[['14', 13.233644859813085],
 ['05', 10.08695652173913],
 ['13', 14.741176470588234],
 ['06', 9.022727272727273],
 ['16', 16.796296296296298],
 ['01', 11.383333333333333],
 ['18', 13.20183486238532],
 ['21', 16.009174311926607],
 ['15', 38.5948275862069],
 ['03', 7.796296296296297],
 ['19', 10.8],
 ['12', 9.41095890410959],
 ['04', 7.170212765957447],
 ['08', 10.25],
 ['11', 11.051724137931034],
 ['07', 7.852941176470588],
 ['17', 11.46],
 ['09', 5.5777777777777775],
 ['22', 6.746478873239437],
 ['20', 21.525],
 ['00', 8.127272727272727],
 ['02', 23.810344827586206],
 ['10', 13.440677966101696],
 ['23', 7.985294117647059]]

Sorting and Printing Avg Posts By Hour

In [10]:
swap_avg_by_hour = []

for row in avg_by_hour:
    one = row[0]
    two = row[1]
    
    swap_avg_by_hour.append([two, one])
    
sorted_swap = sorted(swap_avg_by_hour, reverse=True)

sorted_swap
Out[10]:
[[38.5948275862069, '15'],
 [23.810344827586206, '02'],
 [21.525, '20'],
 [16.796296296296298, '16'],
 [16.009174311926607, '21'],
 [14.741176470588234, '13'],
 [13.440677966101696, '10'],
 [13.233644859813085, '14'],
 [13.20183486238532, '18'],
 [11.46, '17'],
 [11.383333333333333, '01'],
 [11.051724137931034, '11'],
 [10.8, '19'],
 [10.25, '08'],
 [10.08695652173913, '05'],
 [9.41095890410959, '12'],
 [9.022727272727273, '06'],
 [8.127272727272727, '00'],
 [7.985294117647059, '23'],
 [7.852941176470588, '07'],
 [7.796296296296297, '03'],
 [7.170212765957447, '04'],
 [6.746478873239437, '22'],
 [5.5777777777777775, '09']]
In [12]:
print("Top 5 Hours for Ask Posts Comments")

for avg, hr in sorted_swap[:5]:
    print(
        "{}: {: .2f} average comments per post".format(dt.datetime.strptime(hr,"%H").strftime("%H:%M"), avg)
    )
Top 5 Hours for Ask Posts Comments
15:00:  38.59 average comments per post
02:00:  23.81 average comments per post
20:00:  21.52 average comments per post
16:00:  16.80 average comments per post
21:00:  16.01 average comments per post

The hour that has the most comments per post on average is 15:00 (3:00 pm EST) with an average of 38.59 comments per post. There's roughly a 60% increase from the 2nd highest ranking hour 2:00 (2:00am EST).

According to the documentation, the timezone is for certain EST

Conclusion

In this project, we analyzed ask posts and show posts to determine which type of post and time receive the most comments on average. Based on our analysis, to maximize the amount of comments a post receives, we'd recommend the post be categorized as ask post and created between 15:00 and 16:00 (3:00 pm est - 4:00 pm est).

However, it should be noted that the data set we analyzed excluded posts without any comments. Given that, it's more accurate to say that of the posts that received comments, ask posts received more comments on average and ask posts created between 15:00 and 16:00 (3:00 pm est - 4:00 pm est) received the most comments on average.

In [ ]: