Fix Fourthwall parsing breakage, add JSON-LD/sitemap sources, rework caching

Fourthwall moved the product title from <h2> to <h1>, which silently emptied
[fourthwall_single] and [fourthwall_random]: both gated rendering on a title
match that could no longer succeed. Verified against a live store.

Parsing
- Read schema.org JSON-LD first, falling back to scraped markup field by field,
  then to og: meta tags. Each field degrades independently, so a future markup
  change can only affect what it actually touched.
- Match CSS classes with a padded contains() predicate instead of @class="...",
  and select on data-testid where available. Exact class matching is what broke.
- Surface sku, availability and currency, which were previously discarded.

Product discovery
- Enumerate products from sitemap.xml rather than scraping the store page, which
  only sees the collection it features. [fourthwall_random count="10"] could not
  return more than the 3 items a featured homepage renders.
- New source="auto|sitemap|page" attribute. Off-host <loc> entries are rejected
  so a hostile sitemap cannot redirect fetches.

Caching
- Stale-while-revalidate: stale entries are served immediately and refreshed by
  WP-Cron, so cache expiry never costs a visitor a network round trip.
- Refresh scheduling dedupes via wp_next_scheduled and takes a lock, collapsing
  a stampede to a single job.
- Fetch uncached product pages concurrently with curl_multi instead of serially.
- Cache lifetime is now configurable (default 60 minutes).
- Entries written by earlier versions are read as stale and upgraded in place,
  so caches turn over without blanking a store.

Fixes
- Build absolute product links properly. "/collections/all" + "/products/x"
  produced 404s; a trailing slash produced "//products/x".
- Escape the price on output; it was interpolated raw.
- strpos() misuse meant an error string at offset 0 was treated as success.
- Guard a null description node that would fatal when show_description="true".
- Report non-200 responses instead of caching an empty body silently.
- Drop the shared /tmp/cookies.txt jar. It was cross-site mutable state and made
  concurrent fetching unsafe; all endpoints return 200 without it.
- Remove the ssl_verify setting. It was a development affordance and its
  checkbox never worked - unchecked meant "key absent", which read as true.
  Certificate verification is now pinned on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -18,18 +18,94 @@ This is a WordPress plugin that embeds Fourthwall store products into WordPress
 
 ### Key Components
 
-**HTTP Request Layer** (`fwembed_make_request` in shortcode.php):
+**HTTP Request Layer** (`fwembed_make_request` / `fwembed_make_requests` in shortcode.php):
 - Centralized cURL-based HTTP client with browser-like headers to avoid 403 blocks
-- 1-hour transient caching using WordPress transients (cache key: `fwembed_{md5(url)}`)
-- SSL verification configurable via admin settings
-- Cookie handling via `/tmp/cookies.txt` for session persistence
+- Stale-while-revalidate transient caching (cache key: `fwembed_{md5(url)}`), with
+  a configurable freshness window - see **Caching Strategy** below
+- `fwembed_make_requests()` fetches uncached URLs concurrently via `curl_multi`
+- TLS certificate verification is always on and not configurable
+- No cookie jar (see **Caching Strategy**)
 
 **HTML Parsing** (shortcode.php):
 - Uses DOMDocument/DOMXPath to extract product data from Fourthwall HTML
-- Looks for `div[data-testid="product"]` elements
+- Looks for `[data-testid="product"]` elements, falling back to `.product-tile`
 - Extracts product tiles, images, descriptions, titles, and prices via CSS class selectors
 - `loadHTML5()` wrapper ensures proper HTML5 parsing
 
+**Single-product extraction is layered** (`fwembed_parse_single_product`)
+
+Each field is resolved independently from the most stable source available, so a
+markup change can only degrade the field it actually touched:
+
+1. **JSON-LD** (`fwembed_product_from_jsonld`) - product pages embed a
+   `schema.org/Product` block. This is a published contract rather than styling,
+   so it survives theme changes. Source of truth for title, image, description
+   text, `sku`, `priceCurrency` and `availability`.
+2. **Scraped markup** (`fwembed_product_from_html`) - fills gaps, and *wins* for
+   two fields on purpose: the price (it carries the store's own formatting, e.g.
+   "from $20") and the description (JSON-LD flattens it to plain text).
+3. **OpenGraph `<meta>` tags** - last-resort title/image.
+
+`fwembed_find_jsonld_product` handles the shapes publishers emit: bare object,
+top-level list, and `@graph` / `itemListElement` / `mainEntity` wrappers.
+`fwembed_jsonld_offer` takes the cheapest offer, since variants each get their
+own `Offer` and `AggregateOffer` states it as `lowPrice`.
+
+**Product discovery** (`fwembed_collect_product_urls`)
+
+`sitemap.xml` is the full catalog; the store page only renders whichever
+collection it features. Sitemap first, page scrape as fallback - `[fourthwall_random
+source="auto|sitemap|page"]` overrides. `fwembed_read_sitemap` follows one level
+of sitemap index, caps at 10 nested sitemaps, and rejects any `<loc>` whose host
+differs from the configured store so a hostile sitemap cannot redirect fetches.
+
+`robots.txt` allows `/products/*` and `/collections/all` for `*`; `/cart.js`,
+`/checkout/*` and `/admin` are disallowed - do not fetch those.
+
+**Surviving Fourthwall markup changes**
+
+Fourthwall changes their theme markup without notice, and every past breakage has
+been an over-specific selector. Three helpers exist to keep that from recurring —
+use them for any new extraction:
+
+- `fwembed_class_predicate($class)` - builds a padded `contains()` predicate.
+  Never write `@class="foo"`; class order and extra classes change between releases.
+  (The `[fourthwall_single]` breakage was `//h2[@class="product-info__title"]` after
+  the title tag became `<h1>` — match the class or `data-testid`, not the tag name.)
+- `fwembed_query_first($xpath, [$q1, $q2, ...], $context)` - returns the first
+  matching query. Order selectors most-stable first: `data-testid` attributes,
+  then class names, then OpenGraph `<meta>` tags as a last resort.
+- `fwembed_resolve_url($href, $page_url)` - resolves relative hrefs. Product links
+  are root-relative (`/products/foo`), so never concatenate onto the store URL:
+  that breaks on a trailing slash and on store URLs with a path (`/collections/all`).
+
+Known-stable hooks as of the last verification: the JSON-LD `Product` block, the
+OpenGraph meta tags, and `data-testid` values `product`, `product.name`,
+`product.price`, `product.image`.
+
+### Markup emitted for single products
+
+Tiles carry `data-availability` (`InStock` / `OutOfStock`) and `data-sku` from
+JSON-LD, and gain a `product-tile--sold-out` class when out of stock. The class
+is deliberately unstyled - it is a hook for themes, not a built-in badge.
+
+### Verifying against a live store
+
+There is no test suite in-repo. To check parsing after a Fourthwall change, load
+`libs/shortcode.php` under PHP CLI with stubs for `get_option`, `get_transient`,
+`set_transient`, `delete_transient`, `add_shortcode`, `add_action`,
+`shortcode_atts`, `wp_next_scheduled`, `wp_schedule_single_event`, `esc_url`,
+`esc_attr`, `esc_html`, `wpautop`, plus the `MINUTE_IN_SECONDS` / `DAY_IN_SECONDS`
+constants. Then assert each shortcode returns a non-empty title, price, image and
+a well-formed absolute link. Worth covering:
+
+- Store URL with and without a trailing slash, and as `/collections/all`.
+- Degradation: strip the JSON-LD, then rename the CSS classes, then remove the
+  `og:` tags - the first two must still render, the last must render nothing.
+- Freshness: entries are aged by rewriting the stored `expires_at`, **not** by
+  faking the clock - the plugin calls real `time()`, so a fake clock silently
+  tests nothing.
+
 **Shortcodes**:
 - `[fourthwall]` - Displays all store products
 - `[fourthwall_single url="..." show_description="true"]` - Single product display
@@ -45,7 +121,16 @@ This is a WordPress plugin that embeds Fourthwall store products into WordPress
 
 WordPress options API key: `fourthwall_settings_name`
 - `fourth_url` - Default Fourthwall store URL
-- `ssl_verify` - Boolean for SSL certificate verification
+- `cache_ttl` - Freshness window in **minutes** (default 60); `fwembed_cache_ttl()`
+  converts to seconds and falls back to 60 minutes for values below 1
+
+Sanitised on save by `fourthwall_settings::sanitize_settings()`.
+
+The old `ssl_verify` option was removed - it was a development affordance, and
+its checkbox never worked anyway (unchecked meant "key absent", which the read
+treated as `true`). `fwembed_curl_handle()` now pins `CURLOPT_SSL_VERIFYPEER`
+and `CURLOPT_SSL_VERIFYHOST`. Do not make certificate verification configurable
+again; the plugin only ever talks to public HTTPS storefronts.
 
 ## Development Commands
 
@@ -101,16 +186,51 @@ fourthwall-store-embed.zip
 - Set `$dom->documentURI` for proper relative URL resolution
 
 ### Caching Strategy
-- All HTTP requests are cached for 1 hour (3600 seconds)
-- Cache is stored in WordPress transients, not direct database access
-- Error responses (non-200 status) are NOT cached
-- Cache keys use `md5($url)` to handle special characters
+
+Stale-while-revalidate. A cache entry carries its own `expires_at`, and the
+transient's own expiry is the much longer *retention* window:
+
+- **Fresh** (`now < expires_at`) - served directly.
+- **Stale** (past `expires_at`, still stored) - served **immediately**, and a
+  WP-Cron job is queued to refresh it. Expiry therefore never lands the cost of
+  a network round trip on a visitor.
+- **Absent** (past retention) - fetched synchronously. This is the only path
+  that blocks a page render.
+
+Consequences to keep in mind when changing this:
+
+- Never shorten transient expiry to the TTL. `fwembed_cache_retention()`
+  (>= 24h) is what keeps stale content available to serve; TTL only controls
+  when a refresh is triggered.
+- `fwembed_schedule_refresh()` dedupes via `wp_next_scheduled()`, so a thousand
+  visitors hitting one stale entry queue a single job. `fwembed_do_refresh()`
+  additionally takes a `_lock` transient so two overlapping cron runs cannot
+  both fetch.
+- Entries cached by older versions have no `expires_at`. They are read as stale,
+  served once, then rewritten in the new format - do not "clean up" that branch.
+- Error responses (non-200) are still never cached.
+- If `DISABLE_WP_CRON` is set with no real cron running, refreshes never fire and
+  content is served stale until retention lapses, then refetched synchronously.
+
+`fwembed_make_requests()` fetches everything uncached in one `curl_multi` batch
+(8 concurrent, chunked). `[fourthwall_random]` uses it so a cold cache costs one
+round trip instead of N sequential ones. Prefer it over looping
+`fwembed_make_request()` whenever the URL set is known up front.
+
+**No cookie jar.** The old code shared `/tmp/cookies.txt` across every request
+and every site on the host. It broke `curl_multi` (concurrent writes to one jar)
+and was unnecessary - the storefront, product pages and sitemap all return 200
+without cookies. Do not reintroduce a shared jar; give each handle its own file
+if session state ever becomes genuinely necessary.
+
+Cache keys use `md5($url)`; locks are `<key>_lock`, so the admin "Clear Cache"
+`LIKE '_transient_fwembed_%'` sweep clears both.
 
 ### Security Considerations
 - All user input sanitized via `esc_attr()`, `esc_html()`, `htmlspecialchars()`
 - Nonce verification for cache clearing: `wp_verify_nonce()`
 - Capability checks: `current_user_can('manage_options')`
-- SSL verification enabled by default (disable only for local dev)
+- TLS certificate verification is pinned on and cannot be disabled
 
 ### Version Management
 - Main plugin file contains placeholder: `Version: {auto_update_value_on_deploy}`

README.md

@@ -30,6 +30,14 @@ Displays a random selection of products from your store.
 - `count` (optional): Number of products to display (default: 3)
 - `urls` (optional): Comma-separated list of specific product URLs to randomize from
 - `store_url` (optional): Custom store URL (uses default from settings if not provided)
+- `source` (optional): Where to discover products - `auto` (default), `sitemap`, or `page`
+
+**About `source`:** products are discovered from your store's `sitemap.xml`, which
+lists your entire catalog. The older behaviour scraped the store page, which only
+sees the collection that page happens to feature - so if your homepage shows 3
+featured items, `count="10"` could never return more than 3. `auto` uses the
+sitemap and falls back to scraping the page if the sitemap is unavailable; use
+`page` to force the old behaviour.
 
 #### Examples:
 
@@ -58,11 +66,16 @@ Displays a random selection of products from your store.
 
 In the WordPress admin, go to Settings > Fourthwall Store Embed to set your Fourthwall store URL.
 
+**Cache Lifetime** (default: 60 minutes) controls how often store content is
+refreshed. Content is cached and served instantly; once it passes the lifetime it
+is *still* served immediately while a background refresh runs, so raising or
+lowering this never slows a page down. Use **Clear Cache** to force an immediate
+refresh after changing products in Fourthwall.
+
 ## Features
 
 - Caches requests for better performance
 - Responsive design
-- SSL verification options
 - Error handling for failed requests
 - Random product selection
 - Support for multiple store URLs
@@ -77,12 +90,10 @@ In the WordPress admin, go to Settings > Fourthwall Store Embed to set your Four
 
 **Store URL**: Enter your Fourthwall store URL (e.g., `https://your-store.fourthwall.com`)
 
-**SSL Verification**: 
-- **Enabled (Recommended)**: Use for production sites to ensure secure connections
-- **Disabled**: Use only for local development when SSL certificates are not properly configured
+**Cache Lifetime**: How long store content stays fresh, in minutes (default: 60)
 
 **Cache Management**: 
-- Content is automatically cached for 1 hour to improve performance
+- Content is cached and served instantly, then refreshed in the background
 - Use the "Clear Cache" button if products are not updating
 
 #### Display your entire store
@@ -98,8 +109,8 @@ You can also display the product description by setting the `show_description` a
 
 ### Features
 
-- **Smart Caching**: Automatic caching system reduces server load and improves performance
-- **Configurable SSL**: Toggle SSL verification for development vs production environments
+- **Smart Caching**: Stale content is served instantly while it refreshes in the background, so cache expiry never slows a page down
+- **Parallel Fetching**: Uncached products are fetched concurrently rather than one at a time
 - **Error Handling**: Graceful fallbacks and clear error messages
 - **Admin Interface**: User-friendly settings page with clear instructions
 - **Cache Management**: Manual cache clearing for troubleshooting
@@ -113,21 +124,21 @@ You can also display the product description by setting the `show_description` a
 
 ### Performance Notes
 
-- Content is cached for 1 hour to reduce API calls to Fourthwall
-- Cache automatically refreshes when content changes
+- Content is cached to reduce requests to Fourthwall (default: 60 minutes)
+- Once the cache lifetime passes, content is still served instantly while a
+  background refresh runs, so visitors never wait on the network
 - Manual cache clearing available in admin settings
-- SSL verification can be disabled for local development
 
 ### Troubleshooting
 
 **Products not updating?**
 - Clear the cache using the "Clear Cache" button in admin settings
 - Check your store URL is correct
-- Verify SSL verification setting matches your environment
+- Lower the Cache Lifetime if you need changes to appear sooner
 
-**SSL errors in development?**
-- Disable SSL verification in admin settings (development only)
-- Ensure proper SSL certificates in production
+**Background refreshes not happening?**
+- Refreshes run via WP-Cron. If `DISABLE_WP_CRON` is set with no system cron
+  configured, content is served stale until it is refetched on demand.
 
 **403 Forbidden errors?**
 - Fourthwall may be blocking automated requests

libs/settings.php

@@ -27,7 +27,8 @@ class fourthwall_settings {
 
 		register_setting(
 			'fourthwall_settings_group',
-			'fourthwall_settings_name'
+			'fourthwall_settings_name',
+			array( 'sanitize_callback' => array( $this, 'sanitize_settings' ) )
 		);
 
 		add_settings_section(
@@ -46,15 +47,37 @@ class fourthwall_settings {
 		);
 
 		add_settings_field(
-			'ssl_verify',
-			__( 'SSL Verification', 'fourthwall_text_domain' ),
-			array( $this, 'render_ssl_verify_field' ),
+			'cache_ttl',
+			__( 'Cache Lifetime', 'fourthwall_text_domain' ),
+			array( $this, 'render_cache_ttl_field' ),
 			'fourthwall_settings_name',
 			'fourthwall_settings_name_section'
 		);
 
 	}
 
+	/**
+	 * Sanitize settings before they are stored.
+	 *
+	 * @param array $input Raw submitted values
+	 * @return array Cleaned values
+	 */
+	public function sanitize_settings( $input ) {
+
+		$input  = is_array( $input ) ? $input : array();
+		$output = array();
+
+		if ( isset( $input['fourth_url'] ) ) {
+			$output['fourth_url'] = esc_url_raw( trim( $input['fourth_url'] ) );
+		}
+
+		$ttl = isset( $input['cache_ttl'] ) ? intval( $input['cache_ttl'] ) : 60;
+		$output['cache_ttl'] = $ttl < 1 ? 60 : $ttl;
+
+		return $output;
+
+	}
+
 	public function fourthwall_page_layout() {
 
 		// Check required user capability
@@ -94,7 +117,9 @@ class fourthwall_settings {
 		echo '   <p><strong>' . __( 'Display random products:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="5"]</code></p>' . "\n";
 		echo '   <p><strong>' . __( 'Random from specific URLs:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="3" urls="https://store.com/product1,https://store.com/product2,https://store.com/product3"]</code></p>' . "\n";
 		echo '   <p><strong>' . __( 'Random from different store:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="2" store_url="https://different-store.fourthwall.com"]</code></p>' . "\n";
-		echo '   <p><em>' . __( 'Note: Disable SSL verification only for local development. Keep enabled for production sites.', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
+		echo '   <p><strong>' . __( 'Random from the store page instead of the sitemap:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="3" source="page"]</code></p>' . "\n";
+		echo '   <p><em>' . __( 'Random products are drawn from your store sitemap, which covers your whole catalog. Set source="page" to only use products shown on the store page itself.', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
+		echo '   <p><em>' . __( 'Tip: point the Store URL at your /collections/all page to list every product with [fourthwall].', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
 		echo ' </div>' . "\n";
 		echo '</div>' . "\n";
 
@@ -114,17 +139,20 @@ class fourthwall_settings {
 
 	}
 
-	function render_ssl_verify_field() {
+	function render_cache_ttl_field() {
 
 		// Retrieve data from the database.
 		$options = get_option( 'fourthwall_settings_name' );
 
 		// Set default value.
-		$value = isset( $options['ssl_verify'] ) ? $options['ssl_verify'] : '';
+		$value = isset( $options['cache_ttl'] ) ? intval( $options['cache_ttl'] ) : 60;
+		if ( $value < 1 ) {
+			$value = 60;
+		}
 
 		// Field output.
-		echo '<input type="checkbox" name="fourthwall_settings_name[ssl_verify]" value="1" ' . checked( $value, 1, false ) . '>';
-		echo '<p class="description">' . __( 'Enable SSL verification', 'fourthwall_text_domain' ) . '</p>';
+		echo '<input type="number" min="1" step="1" name="fourthwall_settings_name[cache_ttl]" class="small-text" value="' . esc_attr( $value ) . '"> ' . esc_html__( 'minutes', 'fourthwall_text_domain' );
+		echo '<p class="description">' . __( 'How long store content stays fresh before it is refreshed (default: 60). Expired content is still shown instantly while it refreshes in the background, so raising this does not slow your pages down.', 'fourthwall_text_domain' ) . '</p>';
 
 	}