FutureTap - Smart iPhone Apps

The FutureBlog

April 30, 2009 by Ortwin Gentz

Scraping App Store Featured Entries

iTunes Ranking.pngMany iPhone developers are monitoring their app’s ranking and reviews. There are some tools to automatically retrieve the ranking (MajicRank) and the reviews (Erica Sadun’s perl script) for all country-specific App Stores using web scraping.

What is missing so far is a solution to retrieve information about feature entries on the App Store start page such as “New and noteworthy”, “What’s hot” and “Staff favorites”. Mentions on the start page have a considerable influence on sales. To avoid monitoring the different country-specific stores manually in iTunes, I wrote a Perl script that can be scheduled to run daily. The script is launched with the app’s iTunes ID (the numeric ID at the end of the iTunes URL you can copy in iTunes using right click on your app title) as a parameter and puts out the current date, country and the type of the mention:

./iTunesFeatured.pl 314785156 Navigation
09.08.2011  AE  App Store: Home page, Staff Favourites  Navigation: Home p…
09.08.2011  AR  App Store: Home page, Staff Favourites  Navigation: Home p…
09.08.2011  AT  App Store: TOP STAGE, Home page  Navigation: Home page

The script takes a while to complete because it needs to fetch 8 9 pages per country. The output is in tab-delimited format to facilitate post processing of the data. In our case, we’re importing the data into a FileMaker database that allows an easy graphic representation (colors to indicate trends) and the creation of all kinds of reports.

So, let’s get to the meat: The script is available on http://github.com/futuretap/iTunesFeaturedCheck.

Do you find this useful? I’d love to know your feedback in the comments or via Twitter!

Update 23.11.2009: Modified to work with iTunes 9 App Store
Update 30.12.2009: Modified Japanese match term for New and Noteworthy
Update 29.07.2010: Implemented check for category level featuring, posted on GitHub
Update 01.09.2010: Added iPad support (thanks to @the_appmaker), adopted script to a small iTunes server change
Update 02.12.2010: Adopted script to a small iTunes server change (thanks to @triplef)
Update 15.04.2011: Adopted script to an iTunes server change; now fetches compressed data to save bandwidth; added Games subcategories
Update 09.08.2011: Now also fetches arbitrary featuring categories on the App Store homepage, including App Of The Week
Update 09.07.2012: Revised version with multi threading

 

5 Comments

  1. @kkonotchick via Twitter

    wrote on April 30, 2009 at 2:11am

    RT @futuretap: How to scrape #appstore feature entries: http://tr.im/k3Tv #iphonedev

  2. @alexmarktl via Twitter

    wrote on May 6, 2009 at 3:41pm

    @futuretap thanks that worked! Really nice and helpful script for tracking your iPhone apps. http://tr.im/k3Tv

  3. Jeremy

    wrote on April 21, 2012 at 7:43am

    @futuretap Hi Ortwin! We’re collecting data as well .. putting it into some cool formats to analyze. We’d love to talk shop and hear what you think. @appmonsta and we’ll share a data sample.

  4. Nathan

    wrote on May 29, 2013 at 5:13pm

    Hi, I am looking for some solution to export the newly released app details from itune page to my website. Would it be possible do export the ‘App description’, ‘icon’ and ‘screen shots’ and publish it in my website?

    Example websites:
    http://www.appalerts.com
    http://www.iheartthispp.com

  5. Ortwin Gentz

    wrote on May 29, 2013 at 5:32pm

    @Nathan: Certainly that’s possible. Feel free to fork the code and add it!

Leave a comment