Crawl activity

Overview

The Crawl Activity page shows the fetch status for every monitored policy page. Use it to check whether scans are running successfully, diagnose pages that are failing, and monitor response times. The page title is "Crawl activity" and the URL path is /policy-radar/activity.

Stat cards

Three stat cards across the top summarize overall crawl health:

• Monitored pages: The total number of policy page targets being tracked
• Success rate: The percentage of targets whose most recent fetch was successful. The card highlights when the rate drops below 80%.
• Avg response time: The average response time across the most recent fetch for every target, shown in milliseconds

Current status vs. full history

A tab bar lets you switch between two views:

• Current status: Shows only the most recent fetch result for each target — one row per target. This is the default view and gives you a quick snapshot of what is working and what is not.
• Full history: Shows every fetch attempt across all targets, up to the most recent 200 entries. Use this view to investigate intermittent failures or track response time trends.

Filtering by status

A status filter dropdown lets you narrow the table to show only successful fetches, only failed fetches, or all results.

Table columns

The activity table has sortable columns showing the details of each fetch:

• Provider: The vendor name with its logo. Sortable alphabetically.
• Target: The name of the policy page being fetched. Sortable alphabetically.
• Status: A green check for successful fetches or a red X for failures. Sortable.
• HTTP: The HTTP status code returned by the server (e.g., 200, 403, 500). Codes 400 and above are shown in red. Sortable.
• Response time: How long the fetch took in milliseconds. Sortable.
• Error: If the fetch failed, the error message is shown here. Hover over a truncated message to see the full text in a tooltip.
• Timestamp: When the fetch occurred. Sortable, defaults to most recent first.

Diagnosing failures

When a target fetch fails, common causes include:

• HTTP 403 Forbidden: The vendor is blocking automated access. The page may require authentication or have bot protection.
• HTTP 404 Not Found: The URL may have changed. Check the vendor's website for an updated policy page URL.
• Timeout or connection error: The server did not respond in time. This can be transient — check the full history to see if it is a recurring issue.