Introducing the HNPWA API

Delivering Hacker News JSON data faster than before.

Published in

HNPWA

5 min readMar 1, 2018

Rocket by Wahyuntitle from the Noun Project. Edited by David East.

I’m useless without coffee. That’s why there’s a brewed pot waiting for me each morning. This keeps me from wasting time fumbling for the ingredients and painfully watching the coffee drip into the pot.

I need coffee the most when I wake up. I need my critical resources ready before I ask for them. Building and delivering a JSON API is no different.

The data bottleneck

Just as I’m useless without coffee, a HNPWA Is useless without its data. The performance bottleneck for most HNPWAs is fetching data over the network. The app waits the response of the API call whether the render occurs on the server or client.

This can take anywhere from hundreds of milliseconds to tens of seconds depending on the network. This bottleneck can put an end to hitting the 5 second page load benchmark for emerging markets. If we could speed up the delivery of this JSON data, the page load would get much faster. We worked on doing just that.

Today we’re excited to announce the alpha release of the HNPWA API!

The API

https://api.hnpwa.com/v0/news/1.json

The HNPWA API is an aggregated Hacker News JSON API with a few extra perks.

Make fewer requests. HNPWA API is designed for user interfaces to make as few requests as possible.
Load faster. Data is served over a CDN through Google Cloud Functions and Firebase Hosting.
Develop while offline. Using the hnpwa-api node module you can save the data locally and serve while developing without an internet connection.

Feed endpoints

Each feed returns an array of items in the feed. Each end point returns a list of size 30, and by incrementing the page number you can skip to another page.

Single item endpoints

Using the IDs from the feed lists, you can retrieve detailed information about the feed item. Each single story endpoint contains a nested list of comments for easily rendering a comment tree view.

Check out the documentation for a full list of properties.

Built for performance

In a few sample HNPWA apps we’ve seen around a 1 second improvement on slower networks.

What makes the HNPWA API special is how it’s delivered. Like a brewed pot of coffee this data is cached in a CDN waiting for you to use. The HNPWA API is served over a CDN through Google Cloud Functions and Firebase Hosting.

When a user requests an endpoint the request first stops at the local CDN edge. If the data exists in the edge, it will return the cached data back to the user. This results in extremely fast round-trips. Near CDN edge locations you can see response times of 20ms or less.

Prioritizing fresh content

The HNPWA API publishes new files to Firebase Hosting every 20 minutes. These files are from the top pages available on Hacker News. But what about older content?

If the content is not available as a static file Firebase Hosting forwards the request to Cloud Functions which dynamically generates the API data. The response comes back through the CDN edge and is cached for 20 minutes.

Offline Development

Software development doesn’t always happen around an internet connection. We wanted to combat this typically frustrating experience by provided an offline enabled API.

You can serve locally by installing the hnpwa-api node module.

npm i -g hnpwa-api # or use .bin to avoid a global install
hnpwa-api --save # saves the data offline (~10mb)
hnpwa-api --serve --offline

This will spin up an API server at localhost:3002 and you can access the endpoints via this local server. You’ll need a connection to do the initial save, but after that you’re good to go.

How is this different than the official Hacker News API?

The official Hacker News API returns a list of IDs for news feeds.

https://hacker-news.firebaseio.com/v0/topstories.json

[15124809,15125551,15124881,15125572,15124306,15124016,...]

This works well for realtime communications when using the Firebase SDK. However, it requires several requests to get one aggregated news feed. The HNPWA API consolidates these calls into one dataset.

Hasn’t this been done before?

Yes. A few times. Most noticeably cheeaun’s node-hnapi, which was the inspiration and guidance for this project. We mirrored the API of node-hnapi for the sake of consistency.

The difference here is in performance and an offline development experience.

Let’s build a network effect

The latency of HNPWA API calls can be extraordinarily fast when the data exists at the CDN edge.

A latency test tool like latency.apex.sh shows fast round trips across the world. In some places we can even break the 10ms barrier.

This is all provided that the data exists in the CDN edge. For the chances to increase we need real traffic to the API. The more apps that use the HNPWA API, the more everyone will benefit.

Try it today

The script below appends a list of Hacker News Items to the document in just 20 lines of code (minus comments!). Check the CodePen here.

(async function() {
  // get the first list of items as json
  let res = await fetch("https://api.hnpwa.com/v0/news/1.json");
  let newsItems = await res.json();
  
  // create a document fragment to hold all DOM creation
  // until the final add to the document
  let fragment = document.createDocumentFragment();
 
  // create a ul for the news items 
  let newsList = document.createElement("ul");
  newsList.classList.add('newsList');
  fragment.appendChild(newsList);  // for each news item create a li and an a on the list
  newsItems.forEach(item => {
    const li = document.createElement("li");
    const anchor = document.createElement("a");
    li.id = item.id;
    anchor.textContent = item.title;
    anchor.href = item.url;
    li.appendChild(anchor);
    newsList.appendChild(li);
  });
  
  // append the list to the document
  document.body.appendChild(fragment);
})();

We have a full set of documentation available for using the API. Pull requests and issues for docs and node module are always welcome as well! This is an alpha release, so we are ready to hear feedback and work towards a better v1 API .