Apply per-path crawler rules from WebDecoy Cloud (#995)

admin/partials/settings-page.php

@@ -491,6 +491,22 @@
                                 <?php esc_html_e('Block AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.)', 'webdecoy'); ?>
                             </label>
                             <p class="description"><?php esc_html_e('AI crawlers are allowed by default. Enable this to block them.', 'webdecoy'); ?></p>
+                            <?php $cloud_policy = WebDecoy_Cloud_Policy::describe(); ?>
+                            <?php if ($cloud_policy['state'] === 'applied') : ?>
+                                <p class="description">
+                                    <?php
+                                    printf(
+                                        /* translators: 1: number of protected paths that refuse crawlers, 2: number that only watch, 3: how long ago the policy was read */
+                                        esc_html__('Per-path crawler rules from WebDecoy Cloud are applied here too: %1$d refusing, %2$d watching, read %3$s ago. They can only refuse; the custom allowlist does not override them. Change them in the WebDecoy dashboard under Enforcement.', 'webdecoy'),
+                                        (int) $cloud_policy['refusing'],
+                                        (int) $cloud_policy['watching'],
+                                        esc_html(human_time_diff($cloud_policy['fetched_at']))
+                                    );
+                                    ?>
+                                </p>
+                            <?php elseif ($cloud_policy['state'] === 'stale') : ?>
+                                <p class="description"><?php esc_html_e('The per-path crawler rules from WebDecoy Cloud have not been refreshed for over two days and are not being applied. They resume on the next successful refresh.', 'webdecoy'); ?></p>
+                            <?php endif; ?>
                         </td>
                     </tr>
                     <tr>

changelog.txt

@@ -1,6 +1,7 @@
 *** WebDecoy Bot Detection Changelog ***
 
 = Unreleased =
+* Added: per-path crawler rules set in WebDecoy Cloud now apply in WordPress too. If your site is connected and you have told WebDecoy to refuse, say, AI training crawlers on /premium/*, this plugin refuses them there as well, using the same rule the WebDecoy edge sensor uses. Watched paths and sites in Monitor count what would have been refused instead of refusing. The rules are read from WebDecoy twice daily and shown under the AI Crawlers setting; a copy older than two days is not applied. Precedence is simple: cloud rules can only refuse, never allow; Block AI crawlers still refuses site-wide; the custom allowlist exempts a bot from Block AI crawlers but not from a cloud path rule; and this plugin's monitor mode still gates every block.
 * Changed: known crawlers are now identified from the same crawler registry the WebDecoy dashboard and the other WebDecoy sensors use, instead of a list kept only in this plugin. A crawler is named the same thing here and in your reports. The registry knows 182 crawlers where the old list knew 54, so more search engines (Naver, Seznam, Ecosia, Mojeek and others), more AI crawlers (ByteSpider, Mistral, YouBot and others) and more link-preview, monitoring, SEO and feed services are recognised as what they are.
 * Changed: with Block AI crawlers on, AI agents and assistants that browse on a person's behalf (for example ChatGPT-User, Claude-User, Operator, Siri) are refused along with training crawlers, and so are the newly recognised AI crawlers above. ChatGPT-User and OAI-SearchBot are now classified as OpenAI documents them (an assistant and a search crawler, neither used for training), which matters once per-path crawler policy arrives; under this setting they are refused either way. The setting is the one instruction about AI traffic; if you want a specific one through, add it to the custom allowlist by name.
 * Changed: the W3C markup and Validator.nu checkers are no longer listed as known bots. They were never treated as good bots, so requests from them are scored exactly as before; they simply no longer carry a bot name.

includes/class-webdecoy-cloud-connect.php

@@ -204,8 +204,10 @@ public function maybe_handle_return(): void
         // Single-use: burn the nonce so the token can't be replayed.
         delete_transient(self::NONCE_TRANSIENT);
 
-        // Pull entitlements immediately, then keep them fresh twice daily.
+        // Pull entitlements and the site's cloud policy immediately, then keep
+        // both fresh twice daily.
         $this->sync_entitlements();
+        (new WebDecoy_Cloud_Policy())->sync();
         $this->schedule_sync();
 
         // What just happened is that credentials were stored. Whether this site
@@ -271,6 +273,7 @@ public function handle_disconnect(): void
 
         webdecoy()->clear_cloud_credentials();
         delete_option(self::ENTITLEMENTS_OPTION);
+        WebDecoy_Cloud_Policy::clear();
         delete_transient(self::NONCE_TRANSIENT);
         wp_clear_scheduled_hook(self::CRON_HOOK);
 

includes/class-webdecoy-cloud-policy.php

@@ -0,0 +1,182 @@
+<?php
+/**
+ * The site's enforcement policy from WebDecoy Cloud, fetched and cached.
+ *
+ * Until now every policy this plugin applied was a local option. A site
+ * protected by WebDecoy Cloud and this plugin had two policies that could
+ * disagree, and the per-path crawler refusals an owner sets in the dashboard
+ * never reached WordPress at all. This reads the same public, cacheable
+ * config the edge sensor reads, keyed by this site's organization and
+ * hostname, so the dashboard, the edge and this plugin resolve one policy.
+ *
+ * Precedence, stated once:
+ *  - The cloud policy governs per-path crawler refusals. It can only refuse;
+ *    it never grants a crawler access this plugin would otherwise deny.
+ *  - The local "Block AI crawlers" setting is unchanged and site-wide; it can
+ *    only add refusals.
+ *  - The custom allowlist exempts a bot from the local setting, not from a
+ *    cloud path refusal. Remove the refusal in the dashboard instead.
+ *  - This plugin's monitor mode still gates every block. A cloud refusal on a
+ *    site in monitor mode is counted, not applied.
+ *
+ * @package WebDecoy
+ */
+
+if (!defined('ABSPATH')) {
+    exit;
+}
+
+class WebDecoy_Cloud_Policy
+{
+    /** The public config endpoint the edge sensor reads (nothing in it is secret). */
+    private const CONFIG_ENDPOINT = 'https://in.webdecoy.com/api/v1/clearance/config';
+
+    /** Where the last good fetch is kept: the config body plus fetched_at. */
+    public const OPTION = 'webdecoy_cloud_policy';
+
+    /**
+     * How old a cached policy may be before it is not applied. The cron runs
+     * twice daily; two days covers a missed run and a host whose cron is
+     * sluggish, without letting a policy the owner changed last week keep
+     * refusing after they turned it off.
+     */
+    public const MAX_AGE = 2 * DAY_IN_SECONDS;
+
+    /**
+     * Wire the refresh onto the entitlements cron: same cadence, same
+     * lifecycle (scheduled on connect, cleared on disconnect).
+     */
+    public function register(): void
+    {
+        add_action(WebDecoy_Cloud_Connect::CRON_HOOK, [$this, 'sync']);
+    }
+
+    /**
+     * Fetch the policy for this site and cache it. On any failure the cached
+     * copy is left as it is; get_policy() decides whether it is still usable.
+     */
+    public function sync(): void
+    {
+        if (!function_exists('wp_remote_get')) {
+            return;
+        }
+        $organization_id = self::organization_id();
+        if ($organization_id === '') {
+            return;
+        }
+        $host = (string) wp_parse_url(home_url(), PHP_URL_HOST);
+        $url = add_query_arg(
+            ['aid' => $organization_id, 'host' => $host],
+            self::CONFIG_ENDPOINT
+        );
+        $response = wp_remote_get($url, [
+            'timeout' => 5,
+            'headers' => ['Accept' => 'application/json'],
+        ]);
+        if (is_wp_error($response)) {
+            return;
+        }
+        $code = (int) wp_remote_retrieve_response_code($response);
+        if ($code < 200 || $code >= 300) {
+            return;
+        }
+        $body = json_decode((string) wp_remote_retrieve_body($response), true);
+        if (!is_array($body) || !isset($body['mode'])) {
+            return;
+        }
+        update_option(self::OPTION, ['config' => self::relevant($body), 'fetched_at' => time()], false);
+    }
+
+    /**
+     * Keep only what this plugin reads. The config also carries public keys,
+     * credential ids and the deny-list, which the edge needs and this plugin
+     * does not; storing them here would be a second copy of things that
+     * change without us.
+     *
+     * @param array<string,mixed> $body
+     * @return array<string,mixed>
+     */
+    private static function relevant(array $body): array
+    {
+        $keep = ['scope', 'mode', 'routes', 'route_min_trust', 'monitor_routes', 'route_exceptions', 'route_refusals', 'generated_at'];
+        $out = [];
+        foreach ($keep as $k) {
+            if (array_key_exists($k, $body)) {
+                $out[$k] = $body[$k];
+            }
+        }
+        return $out;
+    }
+
+    /**
+     * The policy to apply on this request, or null when there is none to
+     * trust: not connected, never fetched, or fetched too long ago.
+     *
+     * @return array<string,mixed>|null
+     */
+    public static function get_policy(): ?array
+    {
+        $cached = get_option(self::OPTION, null);
+        if (!is_array($cached) || !is_array($cached['config'] ?? null)) {
+            return null;
+        }
+        $age = time() - (int) ($cached['fetched_at'] ?? 0);
+        if ($age > self::MAX_AGE) {
+            return null;
+        }
+        return $cached['config'];
+    }
+
+    /**
+     * What the settings page says about the cloud policy: whether one is
+     * applied, how old it is, and what it refuses.
+     *
+     * @return array{state:string,fetched_at:int,mode:string,refusing:int,watching:int}
+     */
+    public static function describe(): array
+    {
+        $cached = get_option(self::OPTION, null);
+        $config = is_array($cached) && is_array($cached['config'] ?? null) ? $cached['config'] : null;
+        $fetched_at = is_array($cached) ? (int) ($cached['fetched_at'] ?? 0) : 0;
+        if (self::organization_id() === '') {
+            $state = 'not_connected';
+        } elseif ($config === null) {
+            $state = 'not_fetched';
+        } elseif (time() - $fetched_at > self::MAX_AGE) {
+            $state = 'stale';
+        } else {
+            $state = 'applied';
+        }
+        $refusing = 0;
+        $watching = 0;
+        foreach ((array) ($config['route_refusals'] ?? []) as $r) {
+            if (!is_array($r) || empty($r['refuse_behaviors'])) {
+                continue;
+            }
+            if (($r['mode'] ?? '') === 'monitor') {
+                $watching++;
+            } else {
+                $refusing++;
+            }
+        }
+        return [
+            'state' => $state,
+            'fetched_at' => $fetched_at,
+            'mode' => (string) ($config['mode'] ?? ''),
+            'refusing' => $refusing,
+            'watching' => $watching,
+        ];
+    }
+
+    /** Drop the cached policy, on disconnect. */
+    public static function clear(): void
+    {
+        delete_option(self::OPTION);
+    }
+
+    private static function organization_id(): string
+    {
+        $options = get_option('webdecoy_options', []);
+        return is_array($options) ? (string) ($options['organization_id'] ?? '') : '';
+    }
+}

includes/class-webdecoy-detector.php

@@ -51,6 +51,9 @@ public function __construct(?array $options = null)
             'allow_social_bots' => $this->options['allow_social_bots'] ?? true,
             'block_ai_crawlers' => $this->options['block_ai_crawlers'] ?? false,
             'custom_allowlist' => $this->options['custom_allowlist'] ?? [],
+            // Per-path crawler refusals set in WebDecoy Cloud (#995); null
+            // when not connected or the cached copy is too old to trust.
+            'cloud_policy' => class_exists('WebDecoy_Cloud_Policy') ? WebDecoy_Cloud_Policy::get_policy() : null,
         ]);
     }
 

sdk/src/BotDetector.php

@@ -150,6 +150,10 @@ class BotDetector
      *   - block_ai_crawlers: bool (default: false)
      *   - custom_allowlist: array of bot names (default: [])
      *   - verify_bot_ips: bool (default: true) - Verify good bots via reverse DNS
+     *   - cloud_policy: array|null (default: null) - The site's WebDecoy Cloud
+     *     enforcement config (mode, protected paths, per-path crawler refusals),
+     *     as served by the clearance config endpoint. Null when the site is
+     *     not connected or the cached copy is too old to trust.
      */
     public function __construct(array $options = [])
     {
@@ -161,6 +165,7 @@ public function __construct(array $options = [])
             'custom_allowlist' => [],
             'verify_bot_ips' => true,
             'trusted_proxies' => [],
+            'cloud_policy' => null,
         ], $options);
 
         $this->goodBotList = new GoodBotList();
@@ -220,6 +225,31 @@ public function analyze(?array $signals = null, ?string $clientIP = null): Detec
                 return $result;
             }
 
+            // A per-path crawler refusal set in WebDecoy Cloud (#995): the same
+            // policy the dashboard shows and the edge sensor applies, resolved
+            // by the same rule. It acts on the User-Agent claim, like the
+            // setting above, and for the same reason. The custom allowlist does
+            // not override it: a local exemption that opened a path the owner
+            // protected in the cloud would be a policy granting access by
+            // accident, which is the one thing a policy must not do.
+            $refusal = $this->cloudRefusal($botInfo, (string) ($signals['request_path'] ?? ''));
+            if ($refusal !== null) {
+                $result->addMetadata('bot_info', $botInfo);
+                $result->addMetadata('cloud_policy', $refusal);
+                if ($refusal['mode'] === 'enforce') {
+                    $result->setIsGoodBot(false);
+                    $result->setScore(self::SCORE_POLICY_DENIED);
+                    $result->setScoreBreakdown(['crawler_refused' => self::SCORE_POLICY_DENIED]);
+                    $result->setConfidence(1.0);
+                    $result->addFlag('crawler_refused');
+                    $result->addMetadata('denied_by', 'cloud_policy');
+                    return $result;
+                }
+                // Monitoring: counted, not applied. The crawler continues as
+                // the good bot it is, carrying what would have happened.
+                $result->addFlag('crawler_would_be_refused');
+            }
+
             // Check if this bot should be allowed
             if ($this->shouldAllowBot($botInfo)) {
                 // Verify the bot's IP if verification is enabled
@@ -562,6 +592,48 @@ private function deniedByPolicy(array $botInfo): bool
             && !empty($this->options['block_ai_crawlers']);
     }
 
+    /**
+     * What the cloud policy says about this crawler on this path, or null when
+     * it says nothing: no policy, a path no protected pattern covers, a crawler
+     * of a kind the covering paths do not refuse, or a crawler the registry
+     * does not place in a kind at all.
+     *
+     * The returned mode is the deciding mode from route resolution: 'enforce'
+     * when the site enforces and a refusing path decides, 'monitor' when the
+     * site monitors or only watched paths cover the request.
+     *
+     * @param array $botInfo Bot information from identifyBot()
+     * @param string $requestPath The request URI (query string is ignored)
+     * @return array{mode:string,pattern:string,behavior:string,assurance:string}|null
+     */
+    public function cloudRefusal(array $botInfo, string $requestPath): ?array
+    {
+        $policy = $this->options['cloud_policy'];
+        if (!is_array($policy)) {
+            return null;
+        }
+        $behavior = (string) ($botInfo['behavior'] ?? '');
+        if ($behavior === '') {
+            return null;
+        }
+        $path = (string) (parse_url($requestPath, PHP_URL_PATH) ?: '/');
+        $siteMode = (string) ($policy['mode'] ?? 'monitor');
+        $resolution = RouteResolution::resolve($path, RouteResolution::rulesFromConfig($policy), $siteMode);
+        if ($resolution['deciding_mode'] === '' || !in_array($behavior, $resolution['refused_behaviors'], true)) {
+            return null;
+        }
+        return [
+            'mode' => $resolution['deciding_mode'],
+            'pattern' => $resolution['attributed'],
+            'behavior' => $behavior,
+            // What this plugin can say about the identity it acted on. It
+            // reads the User-Agent; it did not verify the source address
+            // before refusing, and does not need to: refusing a claim harms
+            // nobody who is not making it.
+            'assurance' => 'claimed',
+        ];
+    }
+
     /**
      * Check if a bot should be allowed based on policies
      *