Web Crawler Documentation
Meet Jakka Bot
Your tireless guardian. A quiet force that crawls the web, finding issues before your users do. Polite, thorough, and always respectful of your server.
USER AGENT
jakka-bot
USER AGENT
jakka-bot
USER AGENT
jakka-bot
What Jakka Bot Does
What Jakka Bot Does
Our crawler visits your pages, runs 530+ tests, and reports back with actionable insights.
Our crawler visits your pages, runs 530+ tests, and reports back with actionable insights.
Map Every Client Page
Discover every page across all client sites automatically without missing a single URL.
Map Every Client Page
Discover every page across all client sites automatically without missing a single URL.
Catch Visual Errors Instantly
Capture screenshots of every page and spot layout issues before clients ever see them.
Catch Visual Errors Instantly
Capture screenshots of every page and spot layout issues before clients ever see them.
See What Real Users See
Test JavaScript-heavy sites with headless Chromium and catch issues basic crawlers miss.
See What Real Users See
Test JavaScript-heavy sites with headless Chromium and catch issues basic crawlers miss.
Never See a Broken Link Again
Track images, scripts, and all other resources across all sites and prevent broken links.
Never See a Broken Link Again
Track images, scripts, and all other resources across all sites and prevent broken links.
Run 530+ Checks Automatically
Audit SEO, accessibility, performance, security, and UX on every page you manage automatically.
Run 530+ Checks Automatically
Audit SEO, accessibility, performance, security, and UX on every page you manage automatically.
Prevent Security Issues
Scan for HTTPS issues, mixed content, and security gaps before hackers find them first.
Prevent Security Issues
Scan for HTTPS issues, mixed content, and security gaps before hackers find them first.
Map Every Client Page
Discover every page across all client sites automatically without missing a single URL.
Map Every Client Page
Discover every page across all client sites automatically without missing a single URL.
See What Real Users See
Test JavaScript-heavy sites with headless Chromium and catch issues basic crawlers miss.
See What Real Users See
Test JavaScript-heavy sites with headless Chromium and catch issues basic crawlers miss.
Run 530+ Checks Automatically
Audit SEO, accessibility, performance, security, and UX on every page you manage automatically.
Run 530+ Checks Automatically
Audit SEO, accessibility, performance, security, and UX on every page you manage automatically.
Catch Visual Errors Instantly
Capture screenshots of every page and spot layout issues before clients ever see them.
Catch Visual Errors Instantly
Capture screenshots of every page and spot layout issues before clients ever see them.
Never See a Broken Link Again
Track images, scripts, and all other resources across all sites and prevent broken links.
Never See a Broken Link Again
Track images, scripts, and all other resources across all sites and prevent broken links.
Prevent Security Issues
Scan for HTTPS issues, mixed content, and security gaps before hackers find them first.
Prevent Security Issues
Scan for HTTPS issues, mixed content, and security gaps before hackers find them first.
Polite by Design
Jakka Bot is built to be a good citizen of the web. Here's how we ensure minimal impact on your servers.
Respects robots.txt
By default, Jakka Bot obeys your robots.txt rules and avoids restricted areas. Only verified site owners can override this for their own properties.
Concurrent Request Limits
We limit parallel requests to avoid overwhelming your server. Static resources are cached to reduce repeat fetches.
Honors
Crawl-Delay
When you set a crawl-delay in robots.txt, we strictly follow it for HTML page requests. Your rate limits are respected.
Technical Specifications
Everything developers need to know about Jakka Bot
Crawler Identity
User-Agent String
jakka-bot
Full
User-Agent
Mozilla/5.0 (compatible; jakka-bot/1.0; +https://jakka.ai/jakka-bot)
Rendering Engine
Chromium (headless)
JavaScript
Enabled
Crawl Behavior
Default Delay
1 second between pages
Respects robots.txt
Yes (default)
Crawl-Delay Support
Yes
Error Backoff
Automatic on 4xx/5xx
robots.txt Configuration
Control how Jakka Bot crawls your site
⦿ Allow Jakka Bot (Default)
copy
copy
# Allow Jakka Bot to crawl your site User-agent: jakka-bot
Allow: /
# Optional: Set a crawl delay (in seconds) Crawl-delay: 2
# Point to your sitemap Sitemap: https://yoursite.com/sitemap.xml
⦿ Block Jakka Bot
copy
copy
# Block Jakka Bot from your entire site User-agent: jakka-bot
Disallow: /
# Or block specific sections
User-agent: jakka-bot
Disallow: /admin/
Disallow: /private/
Disallow: /staging/
Verify Your Site Ownership
Verified site owners get additional control over how Jakka Bot crawls their properties. Unlock the ability to crawl areas normally blocked by robots.txt and access advanced crawl settings.
Crawl restricted areas (with your permission)
Custom crawl speed settings
Test password-protected staging sites
Polite by Design
Polite by Design
Jakka Bot is built to be a good citizen of the web. Here's how we ensure minimal impact on your servers.
Respects robots.txt
By default, Jakka Bot obeys your robots.txt rules and avoids restricted areas. Only verified site owners can override this for their own properties.
Concurrent Request Limits
We limit parallel requests to avoid overwhelming your server. Static resources are cached to reduce repeat fetches.
Honors
Crawl-Delay
When you set a crawl-delay in robots.txt, we strictly follow it for HTML page requests. Your rate limits are respected.
Respects robots.txt
By default, Jakka Bot obeys your robots.txt rules and avoids restricted areas. Only verified site owners can override this for their own properties.
Concurrent Request Limits
We limit parallel requests to avoid overwhelming your server. Static resources are cached to reduce repeat fetches.
Honors
Crawl-Delay
When you set a crawl-delay in robots.txt, we strictly follow it for HTML page requests. Your rate limits are respected.
Technical Specifications
Technical Specifications
Everything developers need to know about Jakka Bot
Crawler Identity
User-Agent String
jakka-bot
Full
User-Agent
Mozilla/5.0 (compatible; jakka-bot/1.0; +https://jakka.ai/jakka-bot)
Rendering Engine
Chromium (headless)
JavaScript
Enabled
Crawl Behavior
Default Delay
1 second between pages
Respects robots.txt
Yes (default)
Crawl-Delay Support
Yes
Error Backoff
Automatic on 4xx/5xx
Crawler Identity
User-Agent String
jakka-bot
Full
User-Agent
Mozilla/5.0 (compatible; jakka-bot/1.0; +https://jakka.ai/jakka-bot)
Rendering Engine
Chromium (headless)
JavaScript
Enabled
Crawl Behavior
Default Delay
1 second between pages
Respects robots.txt
Yes (default)
Crawl-Delay Support
Yes
Error Backoff
Automatic on 4xx/5xx
robots.txt Configuration
Control how Jakka Bot crawls your site
⦿ Allow Jakka Bot (Default)
copy
copy
# Allow Jakka Bot to crawl your site User-agent: jakka-bot
Allow: /
# Optional: Set a crawl delay (in seconds) Crawl-delay: 2
# Point to your sitemap Sitemap: https://yoursite.com/sitemap.xml
⦿ Block Jakka Bot
copy
copy
# Block Jakka Bot from your entire site User-agent: jakka-bot
Disallow: /
# Or block specific sections
User-agent: jakka-bot
Disallow: /admin/
Disallow: /private/
Disallow: /staging/
Verify Your Site Ownership
Verified site owners get additional control over how Jakka Bot crawls their properties. Unlock the ability to crawl areas normally blocked by robots.txt and access advanced crawl settings.
Crawl restricted areas (with your permission)
Custom crawl speed settings
Test password-protected staging sites
robots.txt Configuration
Control how Jakka Bot crawls your site
⦿ Allow Jakka Bot (Default)
copy
# Allow Jakka Bot to crawl your site User-agent: jakka-bot
Allow: /
# Optional: Set a crawl delay (in seconds) Crawl-delay: 2
# Point to your sitemap Sitemap: https://yoursite.com/sitemap.xml
⦿ Block Jakka Bot
copy
# Block Jakka Bot from your entire site User-agent: jakka-bot
Disallow: /
# Or block specific sections
User-agent: jakka-bot
Disallow: /admin/
Disallow: /private/
Disallow: /staging/
Verify Your Site Ownership
Verified site owners get additional control over how Jakka Bot crawls their properties. Unlock the ability to crawl areas normally blocked by robots.txt and access advanced crawl settings.
Crawl restricted areas (with your permission)
Custom crawl speed settings
Test password-protected staging sites
Frequently Asked Questions
Common questions about Jakka Bot
Why is Jakka Bot crawling my site?
Will Jakka Bot slow down my server?
Why does Jakka Bot load JavaScript?
Can I request crawl data be deleted?
Does Jakka Bot respect noindex/nofollow?
How do I report abusive crawling?
Common questions about Jakka Bot
Frequently Asked Questions
Why is Jakka Bot crawling my site?
Will Jakka Bot slow down my server?
Why does Jakka Bot load JavaScript?
Can I request crawl data be deleted?
Does Jakka Bot respect noindex/nofollow?
How do I report abusive crawling?
Why is Jakka Bot crawling my site?
Will Jakka Bot slow down my server?
Why does Jakka Bot load JavaScript?
Can I request crawl data be deleted?
Does Jakka Bot respect noindex/nofollow?
How do I report abusive crawling?
Frequently Asked Questions
Common questions about Jakka Bot
Why is Jakka Bot crawling my site?
Will Jakka Bot slow down my server?
Why does Jakka Bot load JavaScript?
Can I request crawl data be deleted?
Does Jakka Bot respect noindex/nofollow?
How do I report abusive crawling?
Why is Jakka Bot crawling my site?
Will Jakka Bot slow down my server?
Why does Jakka Bot load JavaScript?
Can I request crawl data be deleted?
Does Jakka Bot respect noindex/nofollow?
How do I report abusive crawling?
