Norconex Web and File System Crawlers collect content from websites, file systems, cloud storage, and content management systems, process it, and commit it to the repository of your choice — a search engine, a database, a vector store, or your own pipeline.
They run from the command line with file-based configuration on any OS, or embed into Java applications through documented APIs.
Website and documentation: crawler.norconex.com
| Status | Where | |
|---|---|---|
| Version 3 | Current stable release — recommended for production | 3.x-branch · docs |
| Version 4 | Beta — a ground-up rewrite, open for early adopters | main (this branch) · docs |
Version 4 is a beta: interfaces, configuration, and packaging may still change before the final release. Version 3 configurations are not compatible as-is and need conversion — the online configurator can import a V3 configuration and convert it for you.
Feedback on the beta is very welcome — please open an issue or start a discussion.
Images are published for each release. There is no latest tag yet — the
moving tags are reserved for the first stable 4.0.0.
| Image | Purpose |
|---|---|
norconex/crawler-web |
Web crawler (includes WebDriver support for JavaScript-rendered pages) |
norconex/crawler-fs |
File system crawler |
norconex/crawler-web-playwright |
Web crawler with Playwright and Chromium bundled, as an alternative to WebDriver |
docker run --rm \
-v "$PWD/configs:/opt/norconex/crawler/configs" \
-v "$PWD/logs:/opt/norconex/crawler/logs" \
norconex/crawler-web:4.0.0-beta.1Images are also mirrored to ghcr.io/norconex/.
Ready-to-run zips for the web and file system crawlers, plus each committer, are attached to every GitHub release.
All modules share the group id com.norconex.crawler and one version number.
<dependency>
<groupId>com.norconex.crawler</groupId>
<artifactId>nx-crawler-web</artifactId>
<version>4.0.0-beta.1</version>
</dependency>configurator.norconex.com builds and validates crawler configurations in your browser, and can convert an existing V3 configuration to V4.
This is a mono-repo: every Norconex crawler project that used to live in its own repository is here, and they are all released together under one version.
| Folder | Artifact Id | Quality gate |
|---|---|---|
| crawler/core/ | nx-crawler-core | |
| crawler/web/ | nx-crawler-web | |
| crawler/fs/ | nx-crawler-fs | |
| importer/ | nx-importer | |
| committer/core/ | nx-committer-core | |
| committer/apachekafka/ | nx-committer-apachekafka | |
| committer/azurecognitivesearch/ | nx-committer-azurecognitivesearch | |
| committer/elasticsearch/ | nx-committer-elasticsearch | |
| committer/googlecloudsearch/ | nx-committer-googlecloudsearch | |
| committer/idol/ | nx-committer-idol | |
| committer/neo4j/ | nx-committer-neo4j | |
| committer/solr/ | nx-committer-solr | |
| committer/sql/ | nx-committer-sql | |
| 🪦 committer/amazoncloudsearch/ | nx-committer-amazoncloudsearch | Deprecated |
Contributions are welcome — a bug report, a fix, a new feature, or better documentation. Please read CONTRIBUTING.md first: all contributions need a sign-off and signed commits.
Norconex Crawler is free and Apache-2.0 licensed, built and maintained by Norconex Inc. Sponsorship funds development beyond what we already invest, and puts your name in front of the people using it.
👉 Become a sponsor — tiers and what each includes are listed there. See SPONSORS.md for our sponsors, and for the many ways to support the project that cost nothing.
Sponsorship is recognition, not a support contract. If your team needs guaranteed response times or committed engineering, see Support.
Apache License 2.0 — see LICENSE.