Following one link is easy. A real website crawl needs scope, redirects, failure handling, polite pacing, progress and repeatable tests. This is why I like spatie/crawler: it handles that plumbing while leaving the important decisions visible in ordinary PHP.
01
Crawling becomes complicated after the first page
A crawler can look like a small job. Fetch one page, find its links and repeat. That version fits comfortably in a few lines of PHP and gives you the pleasant feeling that the feature is nearly finished.
Then the real website turns up. Some links redirect. A few point outside the domain. One page takes ages to respond, another returns a PDF and part of the navigation only appears after JavaScript runs. The small loop starts collecting conditions until the crawler becomes the feature you are spending all week maintaining.
Imagine a release check for a business website. It should start from the home page, stay on the same site, discover internal links and report pages that fail. It would also be useful to record each page title so an empty or duplicated title can be spotted before the next campaign sends visitors there.
- Decide whether subdomains count as internal.
- Set how far the crawl can travel from the starting page.
- Handle redirects, duplicate URLs, timeouts and failed responses.
- Control how quickly requests reach the target server.
- Test the crawl without depending on a live website.
The first HTTP request is not the difficult part. The real work is deciding what belongs to the crawl and what should happen when the web refuses to behave neatly. spatie/crawler gives those decisions clear places instead of leaving the application to grow a home-made URL queue one edge case at a time.
02
The API starts with the job you are trying to do
Version 9 starts with Crawler::create(), the target URL and a callback such as onCrawled(). Calling start() begins the crawl and returns a FinishReason so the application knows whether it completed, hit a crawl limit, reached a time limit or was interrupted.
Our release check can make its policy readable in the same fluent chain. internalOnly() keeps discovery on the target site. depth() and limit() stop the crawl growing without control. concurrency() chooses how many URLs can be fetched at the same time.
| Concern | Quick custom loop | spatie/crawler |
|---|---|---|
| Scope | Conditions spread around link handling | Profiles and fluent scope methods |
| Responses | Raw body and status handled locally | A named CrawlResponse object |
| Progress | Counters and stop reasons built by the app | CrawlProgress and FinishReason |
| Request pressure | Sleep calls and concurrency logic | Concurrency, delays and throttles |
| Testing | Live requests or custom HTTP stubs | Crawler fake responses |
That readability is one of the reasons I like the package. Another developer can see the crawl's shape without first understanding a custom queue, URL normaliser and collection of callback arrays.
The package also supports observers when a dedicated class is a better home than closures. The application can begin with a small callback and move towards a reusable observer without changing the basic model of the crawl.
03
A response object should answer useful questions
Once a page arrives, the application needs more than a raw body string. Version 9's CrawlResponse provides a friendly layer over the PSR-7 response.
It exposes the status code, cached body, headers and a Symfony DOM crawler. It can tell you where the link was found, its anchor text, its depth and whether redirects occurred. Transfer statistics can also provide values such as total transfer time, DNS lookup time and time to first byte when the underlying request makes them available.
For the release check, dom() makes title and heading inspection straightforward. foundOnUrl() is useful when a broken URL must be traced back to the page that linked to it. redirectHistory() helps distinguish a deliberate move from a chain that has quietly grown over several redesigns.
The package still allows access to the underlying PSR-7 response when an unusual requirement needs it. Normal application code can use the clearer API, while the escape hatch remains available at the edge. The common path is pleasant, but the abstraction does not pretend the web has no odd corners.
04
Polite crawling is part of correctness
A crawler that gets the right answer while overwhelming somebody else's server is not correct enough.
spatie/crawler uses concurrent requests and currently defaults to a concurrency of 10. That may be fine for a site you control, but it is a setting to choose rather than forget. The package supports a normal delay, a fixed-delay throttle and an adaptive throttle that slows down when responses take longer.
The crawler respects robots.txt by default. It also lets you set a meaningful user agent, which is used when checking user-agent-specific robot rules. That does not replace permission, a site's terms or ordinary judgement, but it gives considerate behaviour a proper place in the code.
Limits matter for your own application too. A page count, depth limit and time limit stop one unexpected calendar archive or faceted navigation system turning a scheduled job into an accidental tour of the entire internet.
05
The crawler is testable without pretending the internet is stable
Network tests have a bad habit of failing for reasons unrelated to the behaviour under test. DNS has a wobble, the target site changes its navigation or a request is rate limited. The test has learned that the internet exists, but not necessarily that your crawler logic works.
Version 9's fake() method lets a test define the pages and links the crawler will receive without making live HTTP requests. That makes it practical to test scope, link discovery, failure handling and the database records created by the application.
A small live smoke test can still prove that the whole integration works against a site you control. The useful distinction is that the business rules do not need a real network request every time the test suite runs.
This matters once a crawl becomes part of a scheduled command or release process. A predictable test can cover the application's decisions, while a separate monitored check covers the external network boundary.
06
JavaScript rendering is optional for a good reason
Not every website puts its useful links in the first HTML response. A client-rendered application may need a browser before the navigation or content appears.
spatie/crawler can execute JavaScript, but it does not do so by default. Plain HTTP crawling is faster and simpler when the server already returns meaningful HTML. A headless browser adds installation, memory, time and more failure modes, so it should earn its place.
When rendering is needed, version 9 uses a driver-based approach. The crawler can work with Browsershot, Cloudflare Browser Rendering or an application-owned renderer. Browsershot is a suggested dependency rather than a cost every installation has to carry.
I like that the feature is available without becoming the baseline. Start with ordinary HTML. Add browser rendering only when a real page proves that the initial response is not enough.
07
It gives you a crawler, not permission to crawl everything
The package can support sitemaps, broken-link checks, content inventories, metadata audits and internal monitoring. It can also extract structured public information when crawling is the legitimate way to reach it.
It is not automatically the best answer for every collection job. If the provider offers an API, feed or export, that usually gives the application a more stable contract than parsing the provider's current page layout. A managed scraping service may be more sensible when the work needs large proxy networks, anti-bot handling or browser infrastructure the application should not own.
Security matters when users can supply the starting URL. An unrestricted server-side crawler can be abused to request private addresses, cloud metadata endpoints or other internal services. Validate and allowlist targets, block private network ranges, limit redirects and keep credentials out of arbitrary crawl requests.
Repeated crawls need a data policy as well. Use stable URL keys, make writes idempotent, record when each page was checked and separate a temporary request failure from confirmed content removal. Crawling gathers evidence. The application still has to interpret it carefully.
08
Why this package earns its place
I love spatie/crawler because it keeps the interesting code in the application.
The package takes care of discovering links, scheduling concurrent requests, exposing useful responses, tracking progress and providing test fakes. My code can concentrate on the real decision: which pages matter, what should be checked and what the result means.
It also leaves the important controls visible. Scope, limits, concurrency, throttling and JavaScript rendering are choices rather than surprises hidden inside a black box.
For the release checker we started with, that means the job can stay small. Crawl the internal pages, inspect each response, store a clear result and stop for a known reason. No home-made spider required, and far fewer odd corners waiting in next month's redesign.
If you are building a Laravel tool that needs reliable website checks, content discovery or monitoring, I can help shape the crawler, storage and operational limits so it remains useful after the first successful run.
Useful questions
Before adding a website crawl, check:
- Is an official API, feed or export a more stable source?
- Do you have permission to crawl the target and have you checked its terms?
- Is robots.txt respected with an honest user agent?
- Are the allowed host, subdomains, paths and redirect targets clearly restricted?
- Have page, depth and time limits been set?
- Will concurrency and throttling avoid unnecessary pressure on the server?
- Can normal crawl behaviour be covered with fake responses?
- Is JavaScript rendering only enabled where the first HTML response needs it?
- Are user-supplied URLs protected against requests to private infrastructure?
- Are repeated results stored idempotently with a clear checked-at time?
