Multimodal AI document triage
Document Robot
Multimodal AI that recognizes 26 document types, files each one in the right folder and sends the CRM a checklist of what is missing.
- document types recognized
- 26
- per document, on average
- 6.1 s
- type accuracy (89 of 95)
- 94%
- automated tests
- 860+
Figures measured from the repository.
- Extraction
- Classification
- Review
- SharePoint
- CRM
| File received | New name | Folder | Status |
|---|---|---|---|
| scan_0012.pdf | BALANCETE EMPRESA EXEMPLO 08.2026 | Contábil | ok |
| IMG_4471.jpg | CONTRATO SOCIAL EMPRESA EXEMPLO | Societário | ok |
| faturamento (2).xlsx | FATURAMENTO EMPRESA EXEMPLO 12.2025 A 08.2026 | Faturamento | ok |
| doc socio.pdf | RG SOCIO ANA EXEMPLO | Sócios | ok |
| extrato.pdf | ENDIVIDAMENTO BANCARIO EMPRESA EXEMPLO | Endividamento | Check: period not identified |
Check: period not identified (extrato.pdf)
Context and problem
To analyze credit, the sales team receives from each customer a package with dozens of files (trial balances, contracts, bank statements, partners’ IDs) as ZIP, RAR, photos and scanned PDFs. Someone opened file after file, named it, stored it in the right SharePoint folder and then filled the CRM with the folder link and what was still missing for the analysis.
The robot walks that path on its own: it receives the package from the CRM card, classifies each document with AI, renames it to the TYPE ENTITY MM.YYYY standard, files it in the right folder and returns the link and a checklist of pending items to the CRM.
Project constraints
- When in doubt, a person decides. The automation flags what needs checking and never rewrites type, period or name on its own.
- Nothing gets lost. Replace mode never deletes the previous folder, and an archive that fails to extract does not count as a delivered document.
- One run per customer. Two concurrent runs for the same customer are blocked at intake.
- Hostile files exist. Recursive extraction with limits against compression bombs, and symbolic links inside RAR files are refused.
- Cost and latency matter. Every document goes through a model; picking the setting is an engineering decision, not a matter of taste.
Decisions that shaped the product
Measure before choosing
Type and period correct (of 95)
higher is better
- No thinking (0)71
- 512 tokens74
- 1024 tokens73
- Dynamic (mean of 2 runs)72
Mean latency per document
lower is better
- No thinking (0)3.9 s
- 512 tokens6.1 s
- 1024 tokens7.3 s
- Dynamic (mean of 2 runs)8.6 s
Cost for 95 documents
lower is better
- No thinking (0)US$ 0.18
- 512 tokensUS$ 0.29
- 1024 tokensUS$ 0.36
- Dynamic (mean of 2 runs)US$ 0.44
Architecture
The analyst sends the package from the CRM card. The API identifies the department and the customer and guarantees a single run per customer. Files are extracted with limits against malicious archives, and each document is classified by a multimodal model with JSON output. After a fixed-rule review, the document is renamed to the standard, uploaded to the right SharePoint folder, and the CRM gets the link and a checklist of what is still missing.If the review finds something doubtful, the document is flagged for human check in the checklist, and the rule never rewrites type, period or name. If SharePoint throttles requests, the upload is retried with backoff and a breaker opens after 5 throttles. The checklist is saved in the database before reaching the CRM, so a failed write can be resent without repeating the upload.
Operating lessons
- Extensions lie. A video saved as
.pdfbroke classification. Now the real file type is detected before any other step. - Tests never touch production. A test run against the real database cancelled a real job and created 97 copies. A test and a config guard now prevent it.
- Success must be confirmed. Zero confirmed CRM writes become an error, never a success. The checklist is saved in the database, and a command resends only the CRM part without repeating the upload.
- Large files take their own path. Above 20 MB the document goes through the model’s file API, and SharePoint uploads are chunked above 100 MB.
- Throttling is expected. SharePoint calls retry with backoff behind a breaker that opens after 5 consecutive throttles.
My role
I conceived the robot and built its foundation over four months: the end-to-end pipeline, the CRM and SharePoint integrations, the naming and folder standard, the checklist, idempotency and operations. In October a collaborator ran a classifier-hardening pass: the thinking-budget experiment, parallel classification, the rule-based review, the large-file path and the guard against malicious RAR files.
What the product delivers
- Classification of 26 document types, including photos and scanned PDFs, with entity, accounting period and a flag for debt documents.
- Standardized names and folders, with separate folders for partners and group companies.
- A pending-items checklist in the CRM, computed over the whole folder and aligned with the credit-analysis team’s rules.
- Live progress for every run and a full history.
What I would do differently
- An anonymized reference set in CI. Today the labelled sample contains real documents and cannot live in the repository, so accuracy regression is still measured outside the pipeline.
- Ask the model for a confidence level. Without it, review relies on rules alone; a score would let human checks be prioritized.
- Plan the move off the single process earlier. A simple database and one worker serve today’s volume well, but they are the known scaling limit.