Fix Fourthwall parsing breakage, add JSON-LD/sitemap sources, rework caching

Fourthwall moved the product title from <h2> to <h1>, which silently emptied
[fourthwall_single] and [fourthwall_random]: both gated rendering on a title
match that could no longer succeed. Verified against a live store.

Parsing
- Read schema.org JSON-LD first, falling back to scraped markup field by field,
  then to og: meta tags. Each field degrades independently, so a future markup
  change can only affect what it actually touched.
- Match CSS classes with a padded contains() predicate instead of @class="...",
  and select on data-testid where available. Exact class matching is what broke.
- Surface sku, availability and currency, which were previously discarded.

Product discovery
- Enumerate products from sitemap.xml rather than scraping the store page, which
  only sees the collection it features. [fourthwall_random count="10"] could not
  return more than the 3 items a featured homepage renders.
- New source="auto|sitemap|page" attribute. Off-host <loc> entries are rejected
  so a hostile sitemap cannot redirect fetches.

Caching
- Stale-while-revalidate: stale entries are served immediately and refreshed by
  WP-Cron, so cache expiry never costs a visitor a network round trip.
- Refresh scheduling dedupes via wp_next_scheduled and takes a lock, collapsing
  a stampede to a single job.
- Fetch uncached product pages concurrently with curl_multi instead of serially.
- Cache lifetime is now configurable (default 60 minutes).
- Entries written by earlier versions are read as stale and upgraded in place,
  so caches turn over without blanking a store.

Fixes
- Build absolute product links properly. "/collections/all" + "/products/x"
  produced 404s; a trailing slash produced "//products/x".
- Escape the price on output; it was interpolated raw.
- strpos() misuse meant an error string at offset 0 was treated as success.
- Guard a null description node that would fatal when show_description="true".
- Report non-200 responses instead of caching an empty body silently.
- Drop the shared /tmp/cookies.txt jar. It was cross-site mutable state and made
  concurrent fetching unsafe; all endpoints return 200 without it.
- Remove the ssl_verify setting. It was a development affordance and its
  checkbox never worked - unchecked meant "key absent", which read as true.
  Certificate verification is now pinned on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -18,18 +18,94 @@ This is a WordPress plugin that embeds Fourthwall store products into WordPress
 
 ### Key Components
 
-**HTTP Request Layer** (`fwembed_make_request` in shortcode.php):
+**HTTP Request Layer** (`fwembed_make_request` / `fwembed_make_requests` in shortcode.php):
 - Centralized cURL-based HTTP client with browser-like headers to avoid 403 blocks
-- 1-hour transient caching using WordPress transients (cache key: `fwembed_{md5(url)}`)
-- SSL verification configurable via admin settings
-- Cookie handling via `/tmp/cookies.txt` for session persistence
+- Stale-while-revalidate transient caching (cache key: `fwembed_{md5(url)}`), with
+  a configurable freshness window - see **Caching Strategy** below
+- `fwembed_make_requests()` fetches uncached URLs concurrently via `curl_multi`
+- TLS certificate verification is always on and not configurable
+- No cookie jar (see **Caching Strategy**)
 
 **HTML Parsing** (shortcode.php):
 - Uses DOMDocument/DOMXPath to extract product data from Fourthwall HTML
-- Looks for `div[data-testid="product"]` elements
+- Looks for `[data-testid="product"]` elements, falling back to `.product-tile`
 - Extracts product tiles, images, descriptions, titles, and prices via CSS class selectors
 - `loadHTML5()` wrapper ensures proper HTML5 parsing
 
+**Single-product extraction is layered** (`fwembed_parse_single_product`)
+
+Each field is resolved independently from the most stable source available, so a
+markup change can only degrade the field it actually touched:
+
+1. **JSON-LD** (`fwembed_product_from_jsonld`) - product pages embed a
+   `schema.org/Product` block. This is a published contract rather than styling,
+   so it survives theme changes. Source of truth for title, image, description
+   text, `sku`, `priceCurrency` and `availability`.
+2. **Scraped markup** (`fwembed_product_from_html`) - fills gaps, and *wins* for
+   two fields on purpose: the price (it carries the store's own formatting, e.g.
+   "from $20") and the description (JSON-LD flattens it to plain text).
+3. **OpenGraph `<meta>` tags** - last-resort title/image.
+
+`fwembed_find_jsonld_product` handles the shapes publishers emit: bare object,
+top-level list, and `@graph` / `itemListElement` / `mainEntity` wrappers.
+`fwembed_jsonld_offer` takes the cheapest offer, since variants each get their
+own `Offer` and `AggregateOffer` states it as `lowPrice`.
+
+**Product discovery** (`fwembed_collect_product_urls`)
+
+`sitemap.xml` is the full catalog; the store page only renders whichever
+collection it features. Sitemap first, page scrape as fallback - `[fourthwall_random
+source="auto|sitemap|page"]` overrides. `fwembed_read_sitemap` follows one level
+of sitemap index, caps at 10 nested sitemaps, and rejects any `<loc>` whose host
+differs from the configured store so a hostile sitemap cannot redirect fetches.
+
+`robots.txt` allows `/products/*` and `/collections/all` for `*`; `/cart.js`,
+`/checkout/*` and `/admin` are disallowed - do not fetch those.
+
+**Surviving Fourthwall markup changes**
+
+Fourthwall changes their theme markup without notice, and every past breakage has
+been an over-specific selector. Three helpers exist to keep that from recurring —
+use them for any new extraction:
+
+- `fwembed_class_predicate($class)` - builds a padded `contains()` predicate.
+  Never write `@class="foo"`; class order and extra classes change between releases.
+  (The `[fourthwall_single]` breakage was `//h2[@class="product-info__title"]` after
+  the title tag became `<h1>` — match the class or `data-testid`, not the tag name.)
+- `fwembed_query_first($xpath, [$q1, $q2, ...], $context)` - returns the first
+  matching query. Order selectors most-stable first: `data-testid` attributes,
+  then class names, then OpenGraph `<meta>` tags as a last resort.
+- `fwembed_resolve_url($href, $page_url)` - resolves relative hrefs. Product links
+  are root-relative (`/products/foo`), so never concatenate onto the store URL:
+  that breaks on a trailing slash and on store URLs with a path (`/collections/all`).
+
+Known-stable hooks as of the last verification: the JSON-LD `Product` block, the
+OpenGraph meta tags, and `data-testid` values `product`, `product.name`,
+`product.price`, `product.image`.
+
+### Markup emitted for single products
+
+Tiles carry `data-availability` (`InStock` / `OutOfStock`) and `data-sku` from
+JSON-LD, and gain a `product-tile--sold-out` class when out of stock. The class
+is deliberately unstyled - it is a hook for themes, not a built-in badge.
+
+### Verifying against a live store
+
+There is no test suite in-repo. To check parsing after a Fourthwall change, load
+`libs/shortcode.php` under PHP CLI with stubs for `get_option`, `get_transient`,
+`set_transient`, `delete_transient`, `add_shortcode`, `add_action`,
+`shortcode_atts`, `wp_next_scheduled`, `wp_schedule_single_event`, `esc_url`,
+`esc_attr`, `esc_html`, `wpautop`, plus the `MINUTE_IN_SECONDS` / `DAY_IN_SECONDS`
+constants. Then assert each shortcode returns a non-empty title, price, image and
+a well-formed absolute link. Worth covering:
+
+- Store URL with and without a trailing slash, and as `/collections/all`.
+- Degradation: strip the JSON-LD, then rename the CSS classes, then remove the
+  `og:` tags - the first two must still render, the last must render nothing.
+- Freshness: entries are aged by rewriting the stored `expires_at`, **not** by
+  faking the clock - the plugin calls real `time()`, so a fake clock silently
+  tests nothing.
+
 **Shortcodes**:
 - `[fourthwall]` - Displays all store products
 - `[fourthwall_single url="..." show_description="true"]` - Single product display
@@ -45,7 +121,16 @@ This is a WordPress plugin that embeds Fourthwall store products into WordPress
 
 WordPress options API key: `fourthwall_settings_name`
 - `fourth_url` - Default Fourthwall store URL
-- `ssl_verify` - Boolean for SSL certificate verification
+- `cache_ttl` - Freshness window in **minutes** (default 60); `fwembed_cache_ttl()`
+  converts to seconds and falls back to 60 minutes for values below 1
+
+Sanitised on save by `fourthwall_settings::sanitize_settings()`.
+
+The old `ssl_verify` option was removed - it was a development affordance, and
+its checkbox never worked anyway (unchecked meant "key absent", which the read
+treated as `true`). `fwembed_curl_handle()` now pins `CURLOPT_SSL_VERIFYPEER`
+and `CURLOPT_SSL_VERIFYHOST`. Do not make certificate verification configurable
+again; the plugin only ever talks to public HTTPS storefronts.
 
 ## Development Commands
 
@@ -101,16 +186,51 @@ fourthwall-store-embed.zip
 - Set `$dom->documentURI` for proper relative URL resolution
 
 ### Caching Strategy
-- All HTTP requests are cached for 1 hour (3600 seconds)
-- Cache is stored in WordPress transients, not direct database access
-- Error responses (non-200 status) are NOT cached
-- Cache keys use `md5($url)` to handle special characters
+
+Stale-while-revalidate. A cache entry carries its own `expires_at`, and the
+transient's own expiry is the much longer *retention* window:
+
+- **Fresh** (`now < expires_at`) - served directly.
+- **Stale** (past `expires_at`, still stored) - served **immediately**, and a
+  WP-Cron job is queued to refresh it. Expiry therefore never lands the cost of
+  a network round trip on a visitor.
+- **Absent** (past retention) - fetched synchronously. This is the only path
+  that blocks a page render.
+
+Consequences to keep in mind when changing this:
+
+- Never shorten transient expiry to the TTL. `fwembed_cache_retention()`
+  (>= 24h) is what keeps stale content available to serve; TTL only controls
+  when a refresh is triggered.
+- `fwembed_schedule_refresh()` dedupes via `wp_next_scheduled()`, so a thousand
+  visitors hitting one stale entry queue a single job. `fwembed_do_refresh()`
+  additionally takes a `_lock` transient so two overlapping cron runs cannot
+  both fetch.
+- Entries cached by older versions have no `expires_at`. They are read as stale,
+  served once, then rewritten in the new format - do not "clean up" that branch.
+- Error responses (non-200) are still never cached.
+- If `DISABLE_WP_CRON` is set with no real cron running, refreshes never fire and
+  content is served stale until retention lapses, then refetched synchronously.
+
+`fwembed_make_requests()` fetches everything uncached in one `curl_multi` batch
+(8 concurrent, chunked). `[fourthwall_random]` uses it so a cold cache costs one
+round trip instead of N sequential ones. Prefer it over looping
+`fwembed_make_request()` whenever the URL set is known up front.
+
+**No cookie jar.** The old code shared `/tmp/cookies.txt` across every request
+and every site on the host. It broke `curl_multi` (concurrent writes to one jar)
+and was unnecessary - the storefront, product pages and sitemap all return 200
+without cookies. Do not reintroduce a shared jar; give each handle its own file
+if session state ever becomes genuinely necessary.
+
+Cache keys use `md5($url)`; locks are `<key>_lock`, so the admin "Clear Cache"
+`LIKE '_transient_fwembed_%'` sweep clears both.
 
 ### Security Considerations
 - All user input sanitized via `esc_attr()`, `esc_html()`, `htmlspecialchars()`
 - Nonce verification for cache clearing: `wp_verify_nonce()`
 - Capability checks: `current_user_can('manage_options')`
-- SSL verification enabled by default (disable only for local dev)
+- TLS certificate verification is pinned on and cannot be disabled
 
 ### Version Management
 - Main plugin file contains placeholder: `Version: {auto_update_value_on_deploy}`