When to use Crawlberg
Crawlberg is a purpose‑built web crawling engine that runs inside Flutter applications on mobile, web, and desktop platforms. By leveraging Dart isolates, it can perform thousands of HTTP requests in parallel without blocking the UI thread, delivering fast and reliable data extraction directly from the client side. The package abstracts the low‑level networking, queue management, and retry logic, allowing developers to focus on the parsing and business rules that matter most. Whether you need to scrape product listings, monitor price changes, or generate a site map for an internal tool, Crawlberg provides a clean, type‑safe API that integrates seamlessly with the rest of your Flutter codebase.
When to use Crawlberg? The engine shines in scenarios where real‑time or near‑real‑time web data is required inside a Flutter app. For example, a marketplace app can periodically crawl competitor sites to display price trends, a news aggregator can pull headlines from multiple sources without a dedicated backend, and a SEO dashboard can validate robots.txt compliance on the fly. Because the crawling runs on the client, you eliminate the need for an extra server layer, reduce latency, and keep the data pipeline under your direct control. However, for massive, enterprise‑scale crawling workloads that demand distributed processing, a dedicated backend service may still be more appropriate.
In a typical Flutter architecture, Crawlberg belongs to the data layer. You would wrap its API in a repository or service class that the domain layer consumes. This keeps the crawling logic isolated from UI code and makes it testable with mock implementations. The package works well with clean architecture, MVVM, or even simple Provider‑based setups. By exposing streams or Future‑based results, you can feed crawled data directly into state‑management solutions like Riverpod, Bloc, or GetX, allowing UI widgets to react instantly to new content. The isolation model also means you can run crawls in the background while the user continues to interact with the app, preserving a smooth experience.
Getting started is straightforward. Add the dependency with `flutter pub add crawlberg`, import the package, and configure a CrawlConfig that defines request headers, concurrency limits, and optional robots.txt handling. On Android and iOS you must declare internet permissions in the manifest files, and on the web you should be aware of CORS restrictions—Crawlberg can fall back to a proxy if needed. Once configured, you create a CrawlTask, supply a list of URLs, and provide a parser callback that transforms raw HTML into your domain models. The engine handles retries, exponential back‑off, and graceful shutdown of isolates, so you can focus on the parsing logic.
Production cautions are essential when dealing with web crawling. Respect the target site’s robots.txt and rate‑limit your requests to avoid being blocked or causing unintended load. Crawlberg includes built‑in throttling, but you should still configure sensible defaults based on the target domain. Network errors, malformed HTML, and dynamic content rendered by JavaScript are common pitfalls; consider pairing Crawlberg with a headless browser service for JavaScript‑heavy pages. Finally, be mindful of platform‑specific limitations—mobile devices have stricter background execution policies, and the web version may be constrained by the browser’s same‑origin policy. By following these best practices, Crawlberg can become a reliable component of any Flutter app that needs on‑device web data extraction.
Pros
- high concurrency with isolates
- single API for all Flutter platforms
- built‑in retry and throttling
- easy integration with state‑management solutions
Watch outs
- requires careful handling of robots.txt
- limited by device resources on mobile
- CORS restrictions on web
- not a replacement for large‑scale server crawlers
Setup notes
1. Run `flutter pub add crawlberg` to add the package. 2. Import the library: `import 'package:crawlberg/crawlberg.dart';` 3. Add internet permission to AndroidManifest.xml (`<uses-permission android:name="android.permission.INTERNET"/>`) and iOS Info.plist (`<key>NSAppTransportSecurity</key><dict><key>NSAllowsArbitraryLoads</key><true/></dict>`). 4. (Web only) Ensure the target server supports CORS or configure a proxy. 5. Initialize a CrawlConfig and start a CrawlTask as shown in the code example.
Crawlberg supports Flutter 3.0 and later on Android, iOS, macOS, Windows, Linux, and web. The package relies on Dart isolates, which are unavailable on the web; in that case it falls back to a single‑threaded implementation with the same API. Minimum SDK constraint is Dart 2.17.
import 'package:flutter/material.dart';
import 'package:crawlberg/crawlberg.dart';
void main() => runApp(const MyApp());
class MyApp extends StatelessWidget {
const MyApp({super.key});
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(title: const Text('Crawlberg Demo')),
body: const CrawlDemo(),
),
);
}
}
class CrawlDemo extends StatefulWidget {
const CrawlDemo({super.key});
@override
State<CrawlDemo> createState() => _CrawlDemoState();
}
class _CrawlDemoState extends State<CrawlDemo> {
final List<String> _titles = [];
bool _loading = false;
Future<void> _startCrawl() async {
setState(() => _loading = true);
final config = CrawlConfig(concurrency: 5, timeout: Duration(seconds: 10));
final task = CrawlTask(
urls: ['https://example.com', 'https://example.org'],
parser: (html) => RegExp(r'<title>(.*?)</title>', dotAll: true)
.firstMatch(html)?[1] ?? 'No title',
);
final results = await CrawlEngine(config).run(task);
setState(() {
_titles.clear();
_titles.addAll(results);
_loading = false;
});
}
@override
Widget build(BuildContext context) {
return Column(
children: [
ElevatedButton(onPressed: _loading ? null : _startCrawl, child: const Text('Start Crawl')),
if (_loading) const CircularProgressIndicator(),
Expanded(
child: ListView.builder(
itemCount: _titles.length,
itemBuilder: (_, i) => ListTile(title: Text(_titles[i])),
),
),
],
);
}
}