Unlocking Business Intelligence with Document Processing
Many organizations face the challenge of extracting valuable business intelligence from vast collections of paper or electronic documents. With advancements in generative AI, particularly in large language models, it is now possible to leverage intelligent document processing to automate data extraction from these sources. This article explores how to dynamically extract data using on-demand and batch pipelines on Amazon Bedrock, optimizing for both time and cost.
Overview of Intelligent Document Processing Pipelines
The core focus of intelligent document processing is to efficiently handle high volumes of documents, such as scanned PDFs or text files, which often come in varying formats. Amazon Bedrock provides a flexible infrastructure to build two types of inference pipelines: on-demand (real-time) and batch (asynchronous), both of which can be dynamically invoked depending on specific business needs.
- On-demand pipeline: Ideal for time-sensitive requests, processing documents individually with rapid turnaround.
- Batch pipeline: Suited for cost-efficient processing of large data volumes, grouping multiple documents into a single job.
Both pipelines allow users to specify which large language model and prompt to use at the document level, enabling flexibility in extracting data from different document types using the same infrastructure.
On-Demand Inference Pipeline
The on-demand pipeline is triggered by an AWS SQS First-In, First-Out (FIFO) queue. Each message contains information such as the document ID, model ID, prompt ID/version, and system prompt ID/version. When a new message arrives, it triggers an AWS Lambda function that retrieves the document from Amazon S3, converts PDF pages to images if necessary, and fetches the relevant prompts from Amazon Bedrock Prompt Management.
This Lambda function then invokes the large language model via Amazon Bedrock, processes the response, and stores the extracted data in an Amazon DynamoDB table. The message is deleted from the queue once the operation is complete, ensuring reliable and ordered processing.
Batch Inference Pipeline
For organizations dealing with hundreds of millions of documents, the batch inference pipeline becomes invaluable. Using a standard AWS SQS queue for high throughput, this pipeline relies on scheduled triggers from Amazon EventBridge to start a batch Lambda function. The function collects messages, retrieves and processes documents, and prepares JSONL files required for Amazon Bedrock batch inference jobs.
To maximize efficiency, Python’s multiprocessing module is used, allowing the system to handle up to 1,000 documents in just 15 minutes. After the batch job completes, a post-processing Lambda function parses the output and saves the extracted data to DynamoDB.
Dynamic Prompt and Model Selection
A standout feature of this intelligent document processing solution is its ability to dynamically select prompts and models for each document. Since documents like land lease agreements may vary in structure—lists, tables, or drawings—specifying the appropriate prompt enhances extraction accuracy. Amazon Bedrock Prompt Management allows up to 50 prompts per region and 10 versions per prompt, supporting a wide range of document types and use cases.
Testing and Deploying Pipelines
To implement these pipelines, users need an AWS account with permissions to create and manage CloudFormation stacks. Deployment involves launching the stack via the AWS Management Console and uploading sample documents to the designated S3 bucket. By customizing queue messages with the correct prompt and system prompt IDs, users can trigger the pipeline and monitor results through DynamoDB and CloudWatch logs.
Testing the pipelines confirms their effectiveness in extracting detailed information such as land tract attributes from scanned documents. The batch pipeline, in particular, offers significant cost savings—up to 50% lower compared to on-demand processing in Amazon Bedrock.
Cost Optimization and Scalability
Choosing between on-demand and batch pipelines depends on your organization’s time sensitivity and data volume. For massive datasets, the batch pipeline is the most cost-effective option. Advanced users can further scale the solution by migrating Lambda code to AWS Batch, enabling processing of tens of thousands of documents in a single job.
Conclusion
Implementing intelligent document processing with Amazon Bedrock empowers organizations to unlock valuable insights from their document repositories. By leveraging on-demand and batch pipelines, businesses can dynamically extract data based on real-time needs or cost constraints. The flexibility to choose models and prompts per document type makes this approach adaptable to various industries and document formats. For those seeking to automate and scale data extraction, intelligent document processing on Amazon Bedrock represents a robust and future-proof solution.
This article is inspired by content from Original Source. It has been rephrased for originality. Images are credited to the original source.