Fix Fourthwall parsing breakage, add JSON-LD/sitemap sources, rework caching

Fourthwall moved the product title from <h2> to <h1>, which silently emptied
[fourthwall_single] and [fourthwall_random]: both gated rendering on a title
match that could no longer succeed. Verified against a live store.

Parsing
- Read schema.org JSON-LD first, falling back to scraped markup field by field,
  then to og: meta tags. Each field degrades independently, so a future markup
  change can only affect what it actually touched.
- Match CSS classes with a padded contains() predicate instead of @class="...",
  and select on data-testid where available. Exact class matching is what broke.
- Surface sku, availability and currency, which were previously discarded.

Product discovery
- Enumerate products from sitemap.xml rather than scraping the store page, which
  only sees the collection it features. [fourthwall_random count="10"] could not
  return more than the 3 items a featured homepage renders.
- New source="auto|sitemap|page" attribute. Off-host <loc> entries are rejected
  so a hostile sitemap cannot redirect fetches.

Caching
- Stale-while-revalidate: stale entries are served immediately and refreshed by
  WP-Cron, so cache expiry never costs a visitor a network round trip.
- Refresh scheduling dedupes via wp_next_scheduled and takes a lock, collapsing
  a stampede to a single job.
- Fetch uncached product pages concurrently with curl_multi instead of serially.
- Cache lifetime is now configurable (default 60 minutes).
- Entries written by earlier versions are read as stale and upgraded in place,
  so caches turn over without blanking a store.

Fixes
- Build absolute product links properly. "/collections/all" + "/products/x"
  produced 404s; a trailing slash produced "//products/x".
- Escape the price on output; it was interpolated raw.
- strpos() misuse meant an error string at offset 0 was treated as success.
- Guard a null description node that would fatal when show_description="true".
- Report non-200 responses instead of caching an empty body silently.
- Drop the shared /tmp/cookies.txt jar. It was cross-site mutable state and made
  concurrent fetching unsafe; all endpoints return 200 without it.
- Remove the ssl_verify setting. It was a development affordance and its
  checkbox never worked - unchecked meant "key absent", which read as true.
  Certificate verification is now pinned on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -0,0 +1,244 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+This is a WordPress plugin that embeds Fourthwall store products into WordPress sites via shortcodes. The plugin fetches product data from Fourthwall stores using cURL, parses HTML to extract product information, and displays them with custom CSS styling.
+
+## Core Architecture
+
+### File Structure
+
+- **fw-store-embed.php** - Main plugin entry point that loads libraries and registers CSS
+- **libs/settings.php** - Admin settings page, cache management, and WordPress options API integration
+- **libs/shortcode.php** - Core functionality for fetching, parsing, and displaying products
+- **libs/self-update.php** - Auto-update system that checks Gitea releases API
+- **css/fw-store-embed.css** - Styling for product tiles and admin interface
+
+### Key Components
+
+**HTTP Request Layer** (`fwembed_make_request` / `fwembed_make_requests` in shortcode.php):
+- Centralized cURL-based HTTP client with browser-like headers to avoid 403 blocks
+- Stale-while-revalidate transient caching (cache key: `fwembed_{md5(url)}`), with
+  a configurable freshness window - see **Caching Strategy** below
+- `fwembed_make_requests()` fetches uncached URLs concurrently via `curl_multi`
+- TLS certificate verification is always on and not configurable
+- No cookie jar (see **Caching Strategy**)
+
+**HTML Parsing** (shortcode.php):
+- Uses DOMDocument/DOMXPath to extract product data from Fourthwall HTML
+- Looks for `[data-testid="product"]` elements, falling back to `.product-tile`
+- Extracts product tiles, images, descriptions, titles, and prices via CSS class selectors
+- `loadHTML5()` wrapper ensures proper HTML5 parsing
+
+**Single-product extraction is layered** (`fwembed_parse_single_product`)
+
+Each field is resolved independently from the most stable source available, so a
+markup change can only degrade the field it actually touched:
+
+1. **JSON-LD** (`fwembed_product_from_jsonld`) - product pages embed a
+   `schema.org/Product` block. This is a published contract rather than styling,
+   so it survives theme changes. Source of truth for title, image, description
+   text, `sku`, `priceCurrency` and `availability`.
+2. **Scraped markup** (`fwembed_product_from_html`) - fills gaps, and *wins* for
+   two fields on purpose: the price (it carries the store's own formatting, e.g.
+   "from $20") and the description (JSON-LD flattens it to plain text).
+3. **OpenGraph `<meta>` tags** - last-resort title/image.
+
+`fwembed_find_jsonld_product` handles the shapes publishers emit: bare object,
+top-level list, and `@graph` / `itemListElement` / `mainEntity` wrappers.
+`fwembed_jsonld_offer` takes the cheapest offer, since variants each get their
+own `Offer` and `AggregateOffer` states it as `lowPrice`.
+
+**Product discovery** (`fwembed_collect_product_urls`)
+
+`sitemap.xml` is the full catalog; the store page only renders whichever
+collection it features. Sitemap first, page scrape as fallback - `[fourthwall_random
+source="auto|sitemap|page"]` overrides. `fwembed_read_sitemap` follows one level
+of sitemap index, caps at 10 nested sitemaps, and rejects any `<loc>` whose host
+differs from the configured store so a hostile sitemap cannot redirect fetches.
+
+`robots.txt` allows `/products/*` and `/collections/all` for `*`; `/cart.js`,
+`/checkout/*` and `/admin` are disallowed - do not fetch those.
+
+**Surviving Fourthwall markup changes**
+
+Fourthwall changes their theme markup without notice, and every past breakage has
+been an over-specific selector. Three helpers exist to keep that from recurring —
+use them for any new extraction:
+
+- `fwembed_class_predicate($class)` - builds a padded `contains()` predicate.
+  Never write `@class="foo"`; class order and extra classes change between releases.
+  (The `[fourthwall_single]` breakage was `//h2[@class="product-info__title"]` after
+  the title tag became `<h1>` — match the class or `data-testid`, not the tag name.)
+- `fwembed_query_first($xpath, [$q1, $q2, ...], $context)` - returns the first
+  matching query. Order selectors most-stable first: `data-testid` attributes,
+  then class names, then OpenGraph `<meta>` tags as a last resort.
+- `fwembed_resolve_url($href, $page_url)` - resolves relative hrefs. Product links
+  are root-relative (`/products/foo`), so never concatenate onto the store URL:
+  that breaks on a trailing slash and on store URLs with a path (`/collections/all`).
+
+Known-stable hooks as of the last verification: the JSON-LD `Product` block, the
+OpenGraph meta tags, and `data-testid` values `product`, `product.name`,
+`product.price`, `product.image`.
+
+### Markup emitted for single products
+
+Tiles carry `data-availability` (`InStock` / `OutOfStock`) and `data-sku` from
+JSON-LD, and gain a `product-tile--sold-out` class when out of stock. The class
+is deliberately unstyled - it is a hook for themes, not a built-in badge.
+
+### Verifying against a live store
+
+There is no test suite in-repo. To check parsing after a Fourthwall change, load
+`libs/shortcode.php` under PHP CLI with stubs for `get_option`, `get_transient`,
+`set_transient`, `delete_transient`, `add_shortcode`, `add_action`,
+`shortcode_atts`, `wp_next_scheduled`, `wp_schedule_single_event`, `esc_url`,
+`esc_attr`, `esc_html`, `wpautop`, plus the `MINUTE_IN_SECONDS` / `DAY_IN_SECONDS`
+constants. Then assert each shortcode returns a non-empty title, price, image and
+a well-formed absolute link. Worth covering:
+
+- Store URL with and without a trailing slash, and as `/collections/all`.
+- Degradation: strip the JSON-LD, then rename the CSS classes, then remove the
+  `og:` tags - the first two must still render, the last must render nothing.
+- Freshness: entries are aged by rewriting the stored `expires_at`, **not** by
+  faking the clock - the plugin calls real `time()`, so a fake clock silently
+  tests nothing.
+
+**Shortcodes**:
+- `[fourthwall]` - Displays all store products
+- `[fourthwall_single url="..." show_description="true"]` - Single product display
+- `[fourthwall_random count="5" urls="..." store_url="..."]` - Random product selection
+
+**Auto-Update System** (self-update.php):
+- Hooks into WordPress plugin update transients (`site_transient_update_plugins`)
+- Fetches latest release from `https://repo.anhonesthost.net/api/v1/repos/wp-plugins/fourth-wall-embed-wp/releases/latest`
+- Provides changelog via `plugins_api` filter
+- Version placeholder `{auto_update_value_on_deploy}` is replaced during CI/CD build
+
+### Settings Storage
+
+WordPress options API key: `fourthwall_settings_name`
+- `fourth_url` - Default Fourthwall store URL
+- `cache_ttl` - Freshness window in **minutes** (default 60); `fwembed_cache_ttl()`
+  converts to seconds and falls back to 60 minutes for values below 1
+
+Sanitised on save by `fourthwall_settings::sanitize_settings()`.
+
+The old `ssl_verify` option was removed - it was a development affordance, and
+its checkbox never worked anyway (unchecked meant "key absent", which the read
+treated as `true`). `fwembed_curl_handle()` now pins `CURLOPT_SSL_VERIFYPEER`
+and `CURLOPT_SSL_VERIFYHOST`. Do not make certificate verification configurable
+again; the plugin only ever talks to public HTTPS storefronts.
+
+## Development Commands
+
+### Testing the Plugin Locally
+
+1. Symlink or copy to WordPress plugins directory:
+   ```bash
+   ln -s $(pwd) /path/to/wordpress/wp-content/plugins/fourth-wall-embed-wp
+   ```
+
+2. Activate in WordPress admin at: **Plugins > Installed Plugins**
+
+3. Configure at: **Settings > Fourthwall Store Embed**
+
+### Cache Management
+
+Clear transient cache from admin UI or manually:
+```sql
+DELETE FROM wp_options WHERE option_name LIKE '_transient_fwembed_%';
+DELETE FROM wp_options WHERE option_name LIKE '_transient_timeout_fwembed_%';
+```
+
+## CI/CD Pipeline
+
+### Gitea Actions Workflows
+
+**`.gitea/workflows/release.yml`** - Runs on push to `main`:
+1. Generates version tag from date/time: `YYYY.MM.DD-HHMM`
+2. Creates release notes from commits since last tag
+3. Updates version placeholder in `fw-store-embed.php`
+4. Creates ZIP archive with plugin folder structure: `fourthwall-store-embed/`
+5. Creates GitHub-style release with ZIP attachment
+
+**`.gitea/workflows/update-version.yml`** - Version update automation
+
+### Release Process
+
+Releases are automatic on merge/push to `main`. The ZIP file structure must match WordPress conventions:
+```
+fourthwall-store-embed.zip
+└── fourthwall-store-embed/
+    ├── fw-store-embed.php
+    ├── libs/
+    ├── css/
+    └── README.md
+```
+
+## Important Implementation Notes
+
+### DOMDocument HTML Parsing
+- Always use `libxml_use_internal_errors(true)` to suppress HTML5 parsing warnings
+- Clear errors with `libxml_clear_errors()` after parsing
+- Set `$dom->documentURI` for proper relative URL resolution
+
+### Caching Strategy
+
+Stale-while-revalidate. A cache entry carries its own `expires_at`, and the
+transient's own expiry is the much longer *retention* window:
+
+- **Fresh** (`now < expires_at`) - served directly.
+- **Stale** (past `expires_at`, still stored) - served **immediately**, and a
+  WP-Cron job is queued to refresh it. Expiry therefore never lands the cost of
+  a network round trip on a visitor.
+- **Absent** (past retention) - fetched synchronously. This is the only path
+  that blocks a page render.
+
+Consequences to keep in mind when changing this:
+
+- Never shorten transient expiry to the TTL. `fwembed_cache_retention()`
+  (>= 24h) is what keeps stale content available to serve; TTL only controls
+  when a refresh is triggered.
+- `fwembed_schedule_refresh()` dedupes via `wp_next_scheduled()`, so a thousand
+  visitors hitting one stale entry queue a single job. `fwembed_do_refresh()`
+  additionally takes a `_lock` transient so two overlapping cron runs cannot
+  both fetch.
+- Entries cached by older versions have no `expires_at`. They are read as stale,
+  served once, then rewritten in the new format - do not "clean up" that branch.
+- Error responses (non-200) are still never cached.
+- If `DISABLE_WP_CRON` is set with no real cron running, refreshes never fire and
+  content is served stale until retention lapses, then refetched synchronously.
+
+`fwembed_make_requests()` fetches everything uncached in one `curl_multi` batch
+(8 concurrent, chunked). `[fourthwall_random]` uses it so a cold cache costs one
+round trip instead of N sequential ones. Prefer it over looping
+`fwembed_make_request()` whenever the URL set is known up front.
+
+**No cookie jar.** The old code shared `/tmp/cookies.txt` across every request
+and every site on the host. It broke `curl_multi` (concurrent writes to one jar)
+and was unnecessary - the storefront, product pages and sitemap all return 200
+without cookies. Do not reintroduce a shared jar; give each handle its own file
+if session state ever becomes genuinely necessary.
+
+Cache keys use `md5($url)`; locks are `<key>_lock`, so the admin "Clear Cache"
+`LIKE '_transient_fwembed_%'` sweep clears both.
+
+### Security Considerations
+- All user input sanitized via `esc_attr()`, `esc_html()`, `htmlspecialchars()`
+- Nonce verification for cache clearing: `wp_verify_nonce()`
+- Capability checks: `current_user_can('manage_options')`
+- TLS certificate verification is pinned on and cannot be disabled
+
+### Version Management
+- Main plugin file contains placeholder: `Version: {auto_update_value_on_deploy}`
+- CI/CD replaces this during build with actual version
+- Update checker compares version strings exactly (not semantic versioning)
+
+## WordPress Compatibility
+
+- **Requires**: WordPress 6.0+, PHP 7.4+
+- **Tested up to**: WordPress 6.8
+- **Required PHP extensions**: cURL, libxml, DOM

README.md

@@ -30,6 +30,14 @@ Displays a random selection of products from your store.
 - `count` (optional): Number of products to display (default: 3)
 - `urls` (optional): Comma-separated list of specific product URLs to randomize from
 - `store_url` (optional): Custom store URL (uses default from settings if not provided)
+- `source` (optional): Where to discover products - `auto` (default), `sitemap`, or `page`
+
+**About `source`:** products are discovered from your store's `sitemap.xml`, which
+lists your entire catalog. The older behaviour scraped the store page, which only
+sees the collection that page happens to feature - so if your homepage shows 3
+featured items, `count="10"` could never return more than 3. `auto` uses the
+sitemap and falls back to scraping the page if the sitemap is unavailable; use
+`page` to force the old behaviour.
 
 #### Examples:
 
@@ -58,11 +66,16 @@ Displays a random selection of products from your store.
 
 In the WordPress admin, go to Settings > Fourthwall Store Embed to set your Fourthwall store URL.
 
+**Cache Lifetime** (default: 60 minutes) controls how often store content is
+refreshed. Content is cached and served instantly; once it passes the lifetime it
+is *still* served immediately while a background refresh runs, so raising or
+lowering this never slows a page down. Use **Clear Cache** to force an immediate
+refresh after changing products in Fourthwall.
+
 ## Features
 
 - Caches requests for better performance
 - Responsive design
-- SSL verification options
 - Error handling for failed requests
 - Random product selection
 - Support for multiple store URLs
@@ -77,12 +90,10 @@ In the WordPress admin, go to Settings > Fourthwall Store Embed to set your Four
 
 **Store URL**: Enter your Fourthwall store URL (e.g., `https://your-store.fourthwall.com`)
 
-**SSL Verification**: 
+**Cache Lifetime**: How long store content stays fresh, in minutes (default: 60)
-- **Enabled (Recommended)**: Use for production sites to ensure secure connections
-- **Disabled**: Use only for local development when SSL certificates are not properly configured
 
 **Cache Management**: 
-- Content is automatically cached for 1 hour to improve performance
+- Content is cached and served instantly, then refreshed in the background
 - Use the "Clear Cache" button if products are not updating
 
 #### Display your entire store
@@ -98,8 +109,8 @@ You can also display the product description by setting the `show_description` a
 
 ### Features
 
-- **Smart Caching**: Automatic caching system reduces server load and improves performance
+- **Smart Caching**: Stale content is served instantly while it refreshes in the background, so cache expiry never slows a page down
-- **Configurable SSL**: Toggle SSL verification for development vs production environments
+- **Parallel Fetching**: Uncached products are fetched concurrently rather than one at a time
 - **Error Handling**: Graceful fallbacks and clear error messages
 - **Admin Interface**: User-friendly settings page with clear instructions
 - **Cache Management**: Manual cache clearing for troubleshooting
@@ -113,21 +124,21 @@ You can also display the product description by setting the `show_description` a
 
 ### Performance Notes
 
-- Content is cached for 1 hour to reduce API calls to Fourthwall
+- Content is cached to reduce requests to Fourthwall (default: 60 minutes)
-- Cache automatically refreshes when content changes
+- Once the cache lifetime passes, content is still served instantly while a
+  background refresh runs, so visitors never wait on the network
 - Manual cache clearing available in admin settings
-- SSL verification can be disabled for local development
 
 ### Troubleshooting
 
 **Products not updating?**
 - Clear the cache using the "Clear Cache" button in admin settings
 - Check your store URL is correct
-- Verify SSL verification setting matches your environment
+- Lower the Cache Lifetime if you need changes to appear sooner
 
-**SSL errors in development?**
+**Background refreshes not happening?**
-- Disable SSL verification in admin settings (development only)
+- Refreshes run via WP-Cron. If `DISABLE_WP_CRON` is set with no system cron
-- Ensure proper SSL certificates in production
+  configured, content is served stale until it is refetched on demand.
 
 **403 Forbidden errors?**
 - Fourthwall may be blocking automated requests

css/fw-store-embed.css

@@ -8,17 +8,29 @@
 }
 
 .product-tile {
-    align-content: center;
+    align-content: flex-start;
-    vertical-align: bottom;
+    vertical-align: top;
-    background: white;
+    display:inline-flex;
-    display:inline-block;
+    flex-direction: column;
     width:225px;
     margin:15px 15px 15px 15px;
+    background: inherit;
+}
 
+.product-tile .product-link {
+    display: flex;
+    flex-direction: column;
+    height: 100%;
+    text-decoration: none;
+    color: inherit;
 }
 
 .product-tile img {
-    max-height: 350px;
+    width: 100%;
+    height: 300px;
+    object-fit: cover;
+    object-position: center;
+    display: block;
 }
 
 .image__badges {
@@ -39,6 +51,28 @@
 .tile__heading {
     font-size: 0.88em;  /* Reduced font size for product titles */
     font-weight: bold;
+    margin: 0.5em 0;
+    color: inherit;
+    font-family: inherit;
+}
+
+.tile__description {
+    flex: 1;
+    display: flex;
+    flex-direction: column;
+    padding: 10px 5px;
+    color: inherit;
+    font-family: inherit;
+}
+
+.tile__prices {
+    margin-top: auto;
+    padding-top: 0.5em;
+}
+
+.tile__price {
+    color: inherit;
+    font-family: inherit;
 }
 
 /* Admin page styles */

libs/settings.php

@@ -27,7 +27,8 @@ class fourthwall_settings {
 
 		register_setting(
 			'fourthwall_settings_group',
-			'fourthwall_settings_name'
+			'fourthwall_settings_name',
+			array( 'sanitize_callback' => array( $this, 'sanitize_settings' ) )
 		);
 
 		add_settings_section(
@@ -46,15 +47,37 @@ class fourthwall_settings {
 		);
 
 		add_settings_field(
-			'ssl_verify',
+			'cache_ttl',
-			__( 'SSL Verification', 'fourthwall_text_domain' ),
+			__( 'Cache Lifetime', 'fourthwall_text_domain' ),
-			array( $this, 'render_ssl_verify_field' ),
+			array( $this, 'render_cache_ttl_field' ),
 			'fourthwall_settings_name',
 			'fourthwall_settings_name_section'
 		);
 
 	}
 
+	/**
+	 * Sanitize settings before they are stored.
+	 *
+	 * @param array $input Raw submitted values
+	 * @return array Cleaned values
+	 */
+	public function sanitize_settings( $input ) {
+
+		$input  = is_array( $input ) ? $input : array();
+		$output = array();
+
+		if ( isset( $input['fourth_url'] ) ) {
+			$output['fourth_url'] = esc_url_raw( trim( $input['fourth_url'] ) );
+		}
+
+		$ttl = isset( $input['cache_ttl'] ) ? intval( $input['cache_ttl'] ) : 60;
+		$output['cache_ttl'] = $ttl < 1 ? 60 : $ttl;
+
+		return $output;
+
+	}
+
 	public function fourthwall_page_layout() {
 
 		// Check required user capability
@@ -91,7 +114,12 @@ class fourthwall_settings {
 		echo '   <p><strong>' . __( 'Display entire store:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall]</code></p>' . "\n";
 		echo '   <p><strong>' . __( 'Display single product:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_single url="https://your-store.fourthwall.com/products/product-name"]</code></p>' . "\n";
 		echo '   <p><strong>' . __( 'With description:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_single url="https://your-store.fourthwall.com/products/product-name" show_description="true"]</code></p>' . "\n";
-		echo '   <p><em>' . __( 'Note: Disable SSL verification only for local development. Keep enabled for production sites.', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
+		echo '   <p><strong>' . __( 'Display random products:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="5"]</code></p>' . "\n";
+		echo '   <p><strong>' . __( 'Random from specific URLs:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="3" urls="https://store.com/product1,https://store.com/product2,https://store.com/product3"]</code></p>' . "\n";
+		echo '   <p><strong>' . __( 'Random from different store:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="2" store_url="https://different-store.fourthwall.com"]</code></p>' . "\n";
+		echo '   <p><strong>' . __( 'Random from the store page instead of the sitemap:', 'fourthwall_text_domain' ) . '</strong> <code>[fourthwall_random count="3" source="page"]</code></p>' . "\n";
+		echo '   <p><em>' . __( 'Random products are drawn from your store sitemap, which covers your whole catalog. Set source="page" to only use products shown on the store page itself.', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
+		echo '   <p><em>' . __( 'Tip: point the Store URL at your /collections/all page to list every product with [fourthwall].', 'fourthwall_text_domain' ) . '</em></p>' . "\n";
 		echo ' </div>' . "\n";
 		echo '</div>' . "\n";
 
@@ -111,17 +139,20 @@ class fourthwall_settings {
 
 	}
 
-	function render_ssl_verify_field() {
+	function render_cache_ttl_field() {
 
 		// Retrieve data from the database.
 		$options = get_option( 'fourthwall_settings_name' );
 
 		// Set default value.
-		$value = isset( $options['ssl_verify'] ) ? $options['ssl_verify'] : '';
+		$value = isset( $options['cache_ttl'] ) ? intval( $options['cache_ttl'] ) : 60;
+		if ( $value < 1 ) {
+			$value = 60;
+		}
 
 		// Field output.
-		echo '<input type="checkbox" name="fourthwall_settings_name[ssl_verify]" value="1" ' . checked( $value, 1, false ) . '>';
+		echo '<input type="number" min="1" step="1" name="fourthwall_settings_name[cache_ttl]" class="small-text" value="' . esc_attr( $value ) . '"> ' . esc_html__( 'minutes', 'fourthwall_text_domain' );
-		echo '<p class="description">' . __( 'Enable SSL verification', 'fourthwall_text_domain' ) . '</p>';
+		echo '<p class="description">' . __( 'How long store content stays fresh before it is refreshed (default: 60). Expired content is still shown instantly while it refreshes in the background, so raising this does not slow your pages down.', 'fourthwall_text_domain' ) . '</p>';
 
 	}