This project is about doing data analysis using mostly vanilla python for a company that builds Android and IOS mobile apps. They want to better understand what type of apps are most likely to attract users.
Their business model is that they make money from in-app ads within free apps in the Google Play and IOS App stores. Revenue for any given app is mostly influenced by how many users use the apps and egage with ads.
We want to see if we can find useful information on new profitable app ideas.
We'll first analyze relevant freely available data to minimize cost and resources.
apple_file = open('AppleStore.csv')
gplay_file = open('googleplaystore.csv')
from csv import reader
apple_read = reader(apple_file)
gplay_read = reader(gplay_file)
apple_data = list(apple_read)
gplay_data = list(gplay_read)
# get the column headers
apple_header = apple_data[0]
gplay_header = gplay_data[0]
# re assign without headers
apple_data = apple_data[1:]
gplay_data = gplay_data[1:]
# this function helps repeatedly print rows in a readable way
# inputs 4 parameters(list, int, boolean) | output list
def explore_data(dataset, start, end, rows_and_columns=False):
dataset_slice = dataset[start:end]
for row in dataset_slice:
print(row)
print('\n') # adds a new (empty) line after each row
if rows_and_columns:
print('Number of rows:', len(dataset))
print('Number of columns:', len(dataset[0]))
"id" : App ID
"track_name": App Name
"size_bytes": Size (in Bytes)
"currency": Currency Type
"price": Price amount
"ratingcounttot": User Rating counts (for all version)
"ratingcountver": User Rating counts (for current version)
"user_rating" : Average User Rating value (for all version)
"userratingver": Average User Rating value (for current version)
"ver" : Latest version code
"cont_rating": Content Rating
"prime_genre": Primary Genre
"sup_devices.num": Number of supporting devices
"ipadSc_urls.num": Number of screenshots showed for display
"lang.num": Number of supported languages
"vpp_lic": Vpp Device Based Licensing Enabled
print(apple_header)
explore_data(apple_data, 1, 5, True)
From this we can see that there are 7197 IOS apps in the data set. Columns that would be of interest to us are 'track_name', 'currency', 'price', 'rating_count_tot', 'rating_count_ver', and 'prime_genre'.
Further documentation can be found here
print(gplay_header)
explore_data(gplay_data, 1, 5, True
)
From this we can see that there are 10840 Android apps. Columns that might be of interest to us are 'App', 'Category', 'Reviews', 'Installs', 'Type', 'Price', and 'Genres'.
Details about the columns can be found here
We only want to know about apps that are free to download and install, and that are directed toward an English-speaking audience. This means that we'll need to:
Additionally the Google Play data set has a dedicated discussion section, and we can see that one of the discussions outlines an error for row 10472. Let's print this row and compare it against the header and another row that is correct.
print(gplay_data[10472]) # incorrect row
print('\n')
print(gplay_header) # header
print('\n')
print(gplay_data[0]) # correct row
Looking at row 10472 we can see that some of the information is incorrect. The rating column is listed as '19' but the maximum a rating can be is 1 to 5. We'll go ahead and delete this row.
print(len(gplay_data))
del gplay_data[10472]
print(len(gplay_data))
As we inspect the dataset further, there are a lot of duplicate app entries in the google play dataset. In order to properly analyze the data we'll need to remove these. First, lets figure out how many there are. For this we build a for loop that iterates through all the entries and categorizes them on whether they are unique or duplicate.
duplicate_apps = []
unique_apps = []
for app in gplay_data:
name = app[0]
if name in unique_apps:
duplicate_apps.append(name)
else:
unique_apps.append(name)
print('Number of duplicate apps:', len(duplicate_apps))
print('\n')
print('Examples of duplicate apps'), duplicate_apps[:15]
The next question is which method we should use to get rid of the duplicates. We could remove duplicates randomly, but looking at the data more closely it seems like it might be best to find the most recent duplicate and keep that. We can tell that the entries were collected at different times because of the 4th column which is the number of reviews. The entry with the highest number of reviews is the one we should keep.
To remove the duplicates we'll:
create a dictionary where each key is a unique app name and the corresponding dictionary value is the highest number of reviews of that app
use the information stored in teh dictionary and create a new data set, which will have only one entry per app (with the highest number of reviews as the selection criteria)
reviews_max = {}
for row in gplay_data:
name = row[0]
n_reviews = float(row[3])
if name in reviews_max and reviews_max[name] < n_reviews:
reviews_max[name] = n_reviews
elif name not in reviews_max:
reviews_max[name] = n_reviews
print(len(reviews_max))
Now that we have a dictionary of unique apps with the highest amount of reviews, we can use it to create a new data set.
android_clean = []
already_added = []
for row in gplay_data:
name = row[0]
n_reviews = float(row[3])
if (n_reviews == reviews_max[name]) and (name not in already_added):
android_clean.append(row)
already_added.append(name)
explore_data(android_clean, 0, 5, True)
The number of rows matches what we expect.
Now we need to get rid of the non-English apps, as they aren't useful to the company. Here are some examples
print(apple_data[813][1])
print(apple_data[6731][1])
print(android_clean[4412][0])
print(android_clean[7940][0])
One way to approach this is to look for characters outside of English text via the 'ord()' function. English characters are numerically represented in a range from 0 - 127.
If an app's name has a character above 127, it is probably non english.
The problem with this approach is that some of the characters above 127 are emoji's or trademarks. So to filter out as many as we can, we'll set a counter. If there's more than 3 characters above 127, then we'll flag it as non english.
It's not 100% perfect but should clean up the data to a reasonable degree.
# input: string | output: boolean
def is_english(string):
non_ascii = 0
for character in string:
if ord(character) > 127:
non_ascii +=1
if non_ascii > 3:
return False
return True
# test cases
print(is_english('Instagram'))
print(is_english('爱奇艺PPS -《欢乐颂2》电视剧热播'))
print(is_english('Docs To Go™ Free Office Suite'))
print(is_english('Instachat 😜'))
android_english = []
ios_english = []
for row in android_clean:
name = row[0]
if is_english(name):
android_english.append(row)
for row in apple_data:
name = row[1]
if is_english(name):
ios_english.append(row)
explore_data(android_english, 0, 3, True)
print('\n')
explore_data(ios_english, 0, 3, True)
Now we take those datasets and find all the free apps, since the company only works with apps that are free.
android_free = []
ios_free = []
for row in android_english:
price = row[7]
if price == '0':
android_free.append(row)
for row in ios_english:
price = row[4]
if price == '0.0':
ios_free.append(row)
print(len(android_free))
print(len(ios_free))
As we mentioned in the introduction, our aim is to determine the kinds of apps that are likely to attract more users because our revenue is highly influenced by the number of people using our apps.
To minimize risks and overhead, our validation strategy for an app idea is comprised of three steps:
Because our end goal is to add the app on both Google Play and the App Store, we need to find app profiles that are successful on both markets.
# input: list, int | output: dictionary
# finds percentage for one column
def freq_table(dataset, index):
table = {}
total = 0
for row in dataset:
total += 1
value = row[index]
if value in table:
table[value] += 1
else:
table[value] = 1
table_percentages = {}
for key in table:
percentage = (table[key]/ total) * 100
table_percentages[key] = percentage
return table_percentages
# input: list, int | output: tuple
# display percentages in a descending order
def display_table(dataset, index):
table = freq_table(dataset, index)
table_display = []
for key in table:
key_val_as_tuple = (table[key], key)
table_display.append(key_val_as_tuple)
table_sorted = sorted(table_display, reverse = True)
for entry in table_sorted:
print(entry[1], ':', entry[0])
We start by examining the frequency table for the prime_genre column of the App Store data set.
display_table(ios_free, -5) # prime_genre column
From here we can see that on IOS apps that are games come in with the highest percentage at at 58%. The focus on IOS seems to be towards having fun.
Let's continue with the android / google play data.
display_table(android_free, 1) # Category Column
On Android things seem to be different. The family category (which turns out to be about childrens games) is the biggest category coming in at 18%. If we add the subsequent games catgory gaming in total is around 27%. This is roughly half of the importance of games on IOS. There is also a "Genre' column that might be useful.
display_table(android_free, -4) # Genres Column
The difference between the Genres and the Category columns is not crystal clear, but one thing we can notice is that the Genres column is much more granular (it has more categories). We're only looking for the bigger picture at the moment, so we'll only work with the Category column moving forward.
Up to this point, we found that the App Store is dominated by apps designed for fun, while Google Play shows a more balanced landscape of both practical and for-fun apps. Now we'd like to get an idea about the kind of apps that have most users.
One way to find out what genres are the most popular (have the most users) is to calculate the average number of installs for each app genre. For the Google Play data set, we can find this information in the Installs column, but for the App Store data set this information is missing. As a workaround, we'll take the total number of user ratings as a proxy, which we can find in the rating_count_tot app.
Below, we calculate the average number of user ratings per app genre on the App Store:
genres_ios = freq_table(ios_free, -5) #prime_genre
for genre in genres_ios:
total = 0
len_genre = 0
for app in ios_free:
genre_app = app[-5]
if genre_app == genre:
n_ratings = float(app[5])
total += n_ratings
len_genre += 1
avg_n_ratings = total / len_genre
print(genre, ':', avg_n_ratings)
On average, navigation apps have the highest number of user reviews, but this figure is heavily influenced by Waze and Google Maps, which have close to half a million user reviews together:
for app in ios_free:
if app[-5] == 'Navigation':
print(app[1], ':', app[5]) # print name and number of ratings
The same pattern applies to social networking apps, where the average number is heavily influenced by a few giants like Facebook, Pinterest, Skype, etc. Same applies to music apps, where a few big players like Pandora, Spotify, and Shazam heavily influence the average number.
Our aim is to find popular genres, but navigation, social networking or music apps might seem more popular than they really are. The average number of ratings seem to be skewed by very few apps which have hundreds of thousands of user ratings, while the other apps may struggle to get past the 10,000 threshold. We could get a better picture by removing these extremely popular apps for each genre and then rework the averages, but we'll leave this level of detail for later.
Reference apps have 74,942 user ratings on average, but it's actually the Bible and Dictionary.com which skew up the average rating:
for app in ios_free:
if app[-5] == 'Reference':
print(app[1], ':', app[5])
However, this niche seems to show some potential. One thing we could do is take another popular book and turn it into an app where we could add different features besides the raw version of the book. This might include daily quotes from the book, an audio version of the book, quizzes about the book, etc. On top of that, we could also embed a dictionary within the app, so users don't need to exit our app to look up words in an external app.
This idea seems to fit well with the fact that the App Store is dominated by for-fun apps. This suggests the market might be a bit saturated with for-fun apps, which means a practical app might have more of a chance to stand out among the huge number of apps on the App Store.
Other genres that seem popular include weather, book, food and drink, or finance. The book genre seem to overlap a bit with the app idea we described above, but the other genres don't seem too interesting to us:
Weather apps — people generally don't spend too much time in-app, and the chances of making profit from in-app adds are low. Also, getting reliable live weather data may require us to connect our apps to non-free APIs.
Food and drink — examples here include Starbucks, Dunkin' Donuts, McDonald's, etc. So making a popular food and drink app requires actual cooking and a delivery service, which is outside the scope of our company.
Finance apps — these apps involve banking, paying bills, money transfer, etc. Building a finance app requires domain knowledge, and we don't want to hire a finance expert just to build an app.
Now let's analyze the Google Play market a bit.
First we'll have a look at the installs column.
display_table(android_free, 5) # Installs Column
If we want to use the installs column to count users, we need to decide what to do with this imprecise data. For our purposes since we don't need exact precision, we can just use the given number and strip the "+" character.
We also need to convert the values from strings to floats so we can total them. Then we need to display the results sorted by value. We'll modify the function we made earlier to just sort any random dictionary from high to low.
categories_android = freq_table(android_free, 1)
# create empty dictionary for string corrections
cat_string_replace = {}
# clean up string issues in the installs column
for category in categories_android:
total = 0
len_category = 0
for app in android_free:
category_app = app[1]
if category_app == category:
n_installs = app[5]
n_installs = n_installs.replace(',', '')
n_installs = n_installs.replace('+', '')
total += float(n_installs)
len_category += 1
avg_n_installs = total / len_category
cat_string_replace[category] = avg_n_installs
# input: dictioary | output: sorted list
# sorts a dictionary by value high to low
def sort_dict(dict):
table = dict
table_display = []
for key in table:
key_val_as_tuple = (table[key], key)
table_display.append(key_val_as_tuple)
table_sorted = sorted(table_display, reverse = True)
for entry in table_sorted:
print(entry[1], ':', entry[0])
print("\n")
sort_dict(cat_string_replace)
On average, communication apps have the most installs: 38,456,119. This number is heavily skewed up by a few apps that have over one billion installs (WhatsApp, Facebook Messenger, Skype, Google Chrome, Gmail, and Hangouts), and a few others with over 100 and 500 million installs:
for app in android_free:
if app[1] == 'COMMUNICATION' and (app[5] == '1,000,000,000+'
or app[5] == '500,000,000+'
or app[5] == '100,000,000+'):
print(app[0], ':', app[5])
If we removed all the communication apps that have over 100 million installs, the average would be reduced roughly ten times:
under_100_m = []
for app in android_free:
n_installs = app[5]
n_installs = n_installs.replace(',', '')
n_installs = n_installs.replace('+', '')
if (app[1] == 'COMMUNICATION') and (float(n_installs) < 100000000):
under_100_m.append(float(n_installs))
sum(under_100_m) / len(under_100_m)
We see the same pattern for the video players category, which is the runner-up with 24,727,872 installs. The market is dominated by apps like Youtube, Google Play Movies & TV, or MX Player. The pattern is repeated for social apps (where we have giants like Facebook, Instagram, Google+, etc.), photography apps (Google Photos and other popular photo editors), or productivity apps (Microsoft Word, Dropbox, Google Calendar, Evernote, etc.).
Again, the main concern is that these app genres might seem more popular than they really are. Moreover, these niches seem to be dominated by a few giants who are hard to compete against.
The game genre seems pretty popular, but previously we found out this part of the market seems a bit saturated, so we'd like to come up with a different app recommendation if possible.
The books and reference genre looks fairly popular as well, with an average number of installs of 8,767,811. It's interesting to explore this in more depth, since we found this genre has some potential to work well on the App Store, and our aim is to recommend an app genre that shows potential for being profitable on both the App Store and Google Play.
Let's take a look at some of the apps from this genre and their number of installs:
for app in android_free:
if app[1] == 'BOOKS_AND_REFERENCE':
print(app[0], ':', app[5])
The book and reference genre includes a variety of apps: software for processing and reading ebooks, various collections of libraries, dictionaries, tutorials on programming or languages, etc. It seems there's still a small number of extremely popular apps that skew the average:
for app in android_free:
if app[1] == 'BOOKS_AND_REFERENCE' and (app[5] == '1,000,000,000+'
or app[5] == '500,000,000+'
or app[5] == '100,000,000+'):
print(app[0], ':', app[5])
However, it looks like there are only a few very popular apps, so this market still shows potential. Let's try to get some app ideas based on the kind of apps that are somewhere in the middle in terms of popularity (between 1,000,000 and 100,000,000 downloads):
This niche seems to be dominated by software for processing and reading ebooks, as well as various collections of libraries and dictionaries, so it's probably not a good idea to build similar apps since there'll be some significant competition.
We also notice there are quite a few apps built around the book Quran, which suggests that building an app around a popular book can be profitable. It seems that taking a popular book (perhaps a more recent book) and turning it into an app could be profitable for both the Google Play and the App Store markets.
However, it looks like the market is already full of libraries, so we need to add some special features besides the raw version of the book. This might include daily quotes from the book, an audio version of the book, quizzes on the book, a forum where people can discuss the book, etc.
In this project, we analyzed data about the App Store and Google Play mobile apps with the goal of recommending an app profile that can be profitable for both markets.
We concluded that taking a popular book (perhaps a more recent book) and turning it into an app could be profitable for both the Google Play and the App Store markets. The markets are already full of libraries, so we need to add some special features besides the raw version of the book. This might include daily quotes from the book, an audio version of the book, quizzes on the book, a forum where people can discuss the book, etc.