How Social Media Crawlers Read Open Graph Metadata and Cache Link Previews

When a user shares a URL, the receiving service may request that page from its own crawler. The crawler downloads HTML, parses metadata and may fetch the image referenced by og:image. The platform then stores or transforms the data and renders a card in its app. The exact sequence and rules differ between services.

Stage 1: The crawler requests the page

The crawler needs to reach the URL over the public internet. DNS, TLS, redirects, firewall rules, access control and server response times all affect whether it can retrieve the page. A human browser succeeding does not always mean every automated crawler can access it.

Stage 2: Metadata is extracted

The service parses the HTML for properties such as og:title, og:description, og:image and og:url. Some services use fallback values, their own parsers or additional metadata. Server-rendered tags are generally more dependable than metadata that only appears after client-side scripts execute.

Stage 3: The image is fetched

The crawler may make a separate request for the image. That request can fail independently if the image is private, blocked, expired, too large or served with an unexpected response. This is why an existing og:image tag is not proof that the preview image is healthy.

Stage 4: The platform caches and renders the card

Platforms can store metadata or image data to avoid fetching the same URL repeatedly. After changing a page, the old preview may continue to appear until the cache expires or a platform-specific debugger requests a new scrape. Cache policies are not uniform.

How to debug by stage

1. Page request: check status code, redirect chain, TLS and crawler access.
2. Metadata parsing: inspect the actual HTML response and check duplicate/conflicting properties.
3. Image request: test the direct image URL, response headers, format and dimensions.
4. Cache/rendering: use the platform's official debugger where available and retest a fresh share.