Using Phoenix & Elixir to turn HackerNews frontpage into a JSON API

I love turning web pages into structured API’s and Lately, I’ve been learning both Elixir & Phoenix. It was only natural to mix the two and put my freshly acquired skills to turn one of my favorite website (ie: Hacker News) into an API. I’m aware of the Official HackerNews API which is way better, but I just wanted to play around and satisfy my itch with Elixir & Phoenix, so I went ahead and spent a couple hours on this small side project.

Here’s the final Live result of how the API behaves:

Complete source code is available on the Github Repo here:

I started out by creating a new Phoenix project:

and connect a simple route to a controller action in Phoenix and just have it output “Hello World” in JSON format for now.

I decided to use “/api/top-stories” as my route that will eventually render the HN frontpage stories.

A StoriesController.top_stories/2 was necessary to respond to any requests on this endpoint:

Visiting, http://localhost:4000/api/top-stories in my development environment was now rendering the “hello world” I was expecting. The relevant git commit is here

Time to make things interesting now, Let’s fetch HN Frontpage, parse the response and turn it into a structured map that can be rendered as JSON by Phoenix. HTTPoison and Floki are two excellent Elixir libraries that we can use for the task. If you are interested, you can refer to this git commit on how and where to add Hex packages to your Phoenix project.

I started from the controller first, because I like to hash out the API of my code before I’ve even written any code. It’s clear to me that my controller needs to handle two simple cases of success and error for now. There may be more edge cases to handle, but we’ll keep things simple for now.

Having modified our controller with the above code, I needed a Hackernews.Frontpage.fetch/0 function which shall return the desired tuple of either {:ok, response} or {:error, reason}. Locations, such as lib/ in Phoenix would force me to restart the app everytime I make a change, So, I decided to add this module to my web/models folder.

HTTPoison offers a HTTPoison.Base module that can be used as a mixin within your own modules to make it easier for you to create HTTP Clients. So, I decided to use that.

Invoking Hackernews.Frontpage.get("/") fires off the request to HN’s frontpage and because we are using HTTPoison.Base in our module, we have to define Hackernews.Frontpage.process_url/1 and Hackernews.Frontpage.process_response_body/1 functions. Hackernews.Frontpage.process_response_body/1 is invoked when the HTTP request finishes and the argument contains the HTML part of the response.

From here onwards it’s upto us to parse the HTML and turn it into structured Map that our controller can easily encode into valid JSON. After looking at the HTML structure, it became clear that 3 table rows (<tr>) were being used to represent 1 single story on HN frontpage, and because we have 30 stories on the homepage, I just take the first 90 table rows (Enum.take(90)) and then group them in groups of 3 (Enum.chunk(3)).

This gives us a List of 30 elements, where each element contains the necessary markup to extract out everything there is about an HN story.

Each of this element is then passed onto the Hackernews.Frontpage.extract_story/1 function which takes care of turning the markup into a structured Map. This was really the most time consuming part of this whole project, as I had to refer to each attribute of a story and figure out how to extract it out.

Once extract, the returned Map is simply put back into a List which is then passed back into the controller and rendered as JSON. You can see the result here:

I already have ideas on how to further refactor my extract_story/1 function, but I decided to stop here as this was supposed to be a fun side project only. And I’m quite happy with how it turned out.

Provided that I’m still learning Elixir/Phoenix and yet I invested only approximately 2 hours on this project from start to finish, I was quite happy about my productivity during the process, also the fact that I was not exchanging my productivity for performance was the icing on the cake.

I’ve spent fair amount of time learning more about OTP/Ecto/Model Layer/Phoenix Channels etc as well and I’m already planning to adopt and push more for Elixir & Phoenix for our client projects here at Metaware.

Please feel free to write to me at jasdeep [at] metawarelabs [dot] com

Toronto Rails and Angular Development Agency

Today, I want to announce Metaware Labs to the world.  I founded Metaware Labs with my co-founder and brother Manpreet Narang in late 2012 and since then we are helping startups and SME’s to build and innovate on their ideas and solve business problems using technology.



We are a premium Rails Consulting and Development Agency based in Toronto. Our core strength or our core stack includes technologies such as Ruby, Rails, NodeJS, AngularJS and Mobile Apps Development.  Even though we are based out of Toronto, we are always happy to serve clients without any geographical restrictions, such as San Francisco, New York, Palo Alto, Chicago, Detroit, Texas etc. And as such we’ve been providing Rails Consulting services throughout Canada and USA to our clients. Our latest addition to our technology stack includes AngularJS and as such almost all our projects have seen an outburst in AngularJS usage. Apart from the standard web stack, our team has extensive experience with technologies and API’s such as Solr, Redis, Sidekiq, Ionic, Phonegap, Twilio etc.

If you have a project in mind, just need to drop a friendly hello or require expert opinion and Rails or Angular Consulting Services for your company, please feel free to get in touch with us here: [email protected]

Installing Ruby 1.9.3 on Mac OS X Yosemite using RVM

Apple announced their Mac OS X  Yosemite Beta Developer Program recently. Like a usual Apple fanboy, I went ahead and installed the Beta version of Mac OS X Yosemite and as expected was stuck with a non-working copy of Ruby 1.9.3 on my system. So, I just tried a reinstall of Ruby 1.9.3 on my computer:

Hope, that helps.

Installing Nokogiri Fails on OS X Mavericks


So, I upgraded to OS X Mavericks this weekend. The upgrade went smooth and everything had been running extremely fine for me until today. I wanted to install the Ruby OpenGraph gem for a project I was working on. This gem depends on the Nokogiri Gem. However, unfortunately I was facing this issue which was failing on building of Nokogiri Native Extensions as follows.

Continue reading “Installing Nokogiri Fails on OS X Mavericks” »

Rails Pub Night, Ruby Meetups & Learn to Code Bootcamps in Toronto

iltni2014 is right around the corner and you’ve decided to pick up some new chops. Perhaps you’ve decided to learn how to write code and software? If yes, there couldn’t have been a better time to make such a decision – Software is eating up the world and demand has increased insanely for developers, engineers and programmers. If you happen to be living in the beautiful city of Toronto, there are plenty of opportunities for one to network with like minded people and plenty of learning opportunities available as well.

Continue reading “Rails Pub Night, Ruby Meetups & Learn to Code Bootcamps in Toronto” »