Crawl4AI Open-Source Tool Added to Amazon Bedrock Knowledge Bases

Building good AI answers requires clean data, and developers can now use the open-source Crawl4AI tool to prep web content for Amazon Bedrock Knowledge Bases. The built-in web crawler works for most public sites by handling scheduled syncs and JavaScript pages automatically. But some sources need extra work to strip out website clutter and attach custom metadata. That is where Crawl4AI steps in to turn web pages into structured Markdown files and metadata sidecars. An AWS Fargate task runs the crawler on a schedule, drops the files into an Amazon S3 bucket, and triggers a knowledge base ingestion job. The setup lets you target specific page regions with CSS selectors and filter out unwanted navigation menus. The catch is that this workflow only handles public pages and does not support sites requiring sign-in. Once the files reach Amazon S3, Bedrock takes over to parse, chunk, index, and retrieve the data for your AI apps.