Clever Caching a Stock Analysis app using Netlify’s fine-grained cache control

This is a submission for the Netlify Dynamic Site Challenge: Clever Caching

What I Built

I have build a Stock Analysis tool which shows price history, financial data and related News articles for any NYSE listed stock.
Github Link

Cache Mechanisms used in the Project

• Netlify-Vary headers
• Netlify-CDN-Cache-Control headers
• SWR (Stale while revalidate)

On-demand Cache Invalidation mechanism using

• purgeCache() helper function
• Netlify-Cache-Tag headers to purge cache by tags

//TODO Include Video

Demo

Link to app: https://market-lens-app.netlify.app/

Cache Status Screen shots

Platform Primitives

Tell us how you leveraged Netlify Cache Control.

To begin with I firstly configured cache as manual in the netlify.toml file for the paths I want to cache.

[[edge_functions]]
  function='news'
  path= '/api/news/:symbol'
  cache='manual'

[[edge_functions]]
  function='financial'
  path= '/api/financial'
  cache='manual'

In the above file you can see something called symbol, this stands for stock ticker symbol (Example AAPL is the symbol for Apple INC ). This symbol plays a key role for the caching mechanism in this project. Let see how.

For the sake of learning I have used two approaches when querying data
1: Ticker symbol in Path Params.

api/news/AAPL -> will fetch all news article related to Apple stock

2: Ticker symbol in Query Params.

api/financial?symbol=AAPL -> will fetch financial data for Apple stock

1. Caching Response when dynamic value in Path params

For fetching the news data the ticker symbol is passed in the URL’s path param. The response header looks like below.

const headers = {
  'Content-Type': 'application/json',
  'Cache-Control': 'public, max-age=0, must-revalidate',
  'Netlify-CDN-Cache-Control':'public, max-age=120, stale-while-revalidate=300',
  'Cache-Tag': `${symbol}, ${symbol}-news`,
};

I have used Netlify-CDN-Cache-Control with stale-while-revalidate
The stale-while-revalidate in the Netlify cache header allows cached content to be served even after it expires (max-age), while simultaneously refreshing the cache in the background. In this case, it allows serving stale content for up to 300 seconds while a new version is fetched and revalidated.

When the dynamic variable (the stock symbol) is used as a path param you dont have to do much as the symbol is part of the url path.

/api/news/AAPL -> Will have Apple news article Cached
/api/news/MSFT -> -> Will have Microsoft news Article Cached

The challenge is when symbol is passed in query params. lets see how its cached.

2. Caching with Netlify-Vary | Dynamic value in Query params

When the symbol is passed in query params then you have to add a custom header Netlify-Vary in-order to cache the response. The response header looks like below.

export default async (req: Request, context: Context) => {
  const parsedUrl = new URL(req.url);
  const queryParams = Object.fromEntries(parsedUrl.searchParams.entries());
  const symbol = queryParams.symbol
...
const headers = {
      'Content-Type': 'application/json',
      'Access-Control-Allow-Origin': '*',
      'Cache-Control': 'public, max-age=0, must-revalidate',
      'Netlify-CDN-Cache-Control':
        'public, max-age=604800 , stale-while-revalidate=2592000',
      'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
      'Netlify-Vary': 'query=symbol',
    };

This header object is similar to the one we saw above except that we have added an additional header field Netlify-Vary with value query=symbol. This makes sure to cache response taking the symbol query param into consideration.

/api/financial?symbol=MSFT -> Will cache Microsoft financial data
/api/financial?symbol=TSLA-> Will cache Tesla's financial data

More on Netlify-Vary by query params

On-demand Cache Purging

I have used Netlify’s purgeCache() helper to implement on-demand cache invalidation. I have made this function accessible under the path /manual-purge
Cache invalidation is one of the hardest things to do in Software development but I’m pretty impressed how Netlify has made it easy to implement with the helper function and cache tags.

Cache Tags

You may have noticed a field in my response headers labeled Netlify-Cache-Tag. This field is crucial for managing cache invalidation behavior effectively.

// news.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-news`,
...
}
// financial.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
...
}

The use of ticker symbol as tag names allows us to target specific company data for purging without affecting other company data.

The below tables might help you understand how responses are tagged

This granular approach ensures that updates or modifications related to particular stocks or financial information are accurately purged without affecting other data.

Example

1. /manual-purge?tag=MSFT
   Will purge all cached response related to Microsoft i.e News, Financial and Price data
2. /manual-purge?tag=MSFT-financial
   This will purge only the financial data related to Microsoft while other cached data related to Microsoft or any otehr company is not disturbed.

Personal Hurdles

I have been searching API providers for stock market data and
settled with Polygon io as the free plan allows 5 calls per minute.

To be honest, this limitation guided my cache strategy to prioritize minimizing external API calls. By serving cached data for previously accessed stocks, the app successfully avoids redundant calls to the same company.

Also as someone implementing edge caching for the first time I found Netlify’s docs quite comprehensive without the need to watch a lot of YouTube videos.

Thanks Dev.to and Netlify for the Hackathon. I learned something new.