All work
Corporate caseTeam project

Multimodal AI document triage

Document Robot

Multimodal AI that recognizes 26 document types, files each one in the right folder and sends the CRM a checklist of what is missing.

Role
Lead author · with 1 collaborator
Period
Jun to Oct 2026
Stack
  • Python
  • FastAPI
  • Gemini
  • Claude
  • SharePoint
  • CRM
  • SQLite
  • SSE
  • Docker
document types recognized
26
per document, on average
6.1 s
type accuracy (89 of 95)
94%
automated tests
860+

Figures measured from the repository.

Run · Example Company Ltd12 files · 41 s · showing 5
  1. Extraction
  2. Classification
  3. Review
  4. SharePoint
  5. CRM
File receivedNew nameFolderStatus
scan_0012.pdfBALANCETE EMPRESA EXEMPLO 08.2026Contábilok
IMG_4471.jpgCONTRATO SOCIAL EMPRESA EXEMPLOSocietáriook
faturamento (2).xlsxFATURAMENTO EMPRESA EXEMPLO 12.2025 A 08.2026Faturamentook
doc socio.pdfRG SOCIO ANA EXEMPLOSóciosok
extrato.pdfENDIVIDAMENTO BANCARIO EMPRESA EXEMPLOEndividamentoCheck: period not identified

Check: period not identified (extrato.pdf)

Illustrative recreation with fictitious files and company. The real UI sits behind corporate login.

Context and problem

To analyze credit, the sales team receives from each customer a package with dozens of files (trial balances, contracts, bank statements, partners’ IDs) as ZIP, RAR, photos and scanned PDFs. Someone opened file after file, named it, stored it in the right SharePoint folder and then filled the CRM with the folder link and what was still missing for the analysis.

The robot walks that path on its own: it receives the package from the CRM card, classifies each document with AI, renames it to the TYPE ENTITY MM.YYYY standard, files it in the right folder and returns the link and a checklist of pending items to the CRM.

Project constraints

  • When in doubt, a person decides. The automation flags what needs checking and never rewrites type, period or name on its own.
  • Nothing gets lost. Replace mode never deletes the previous folder, and an archive that fails to extract does not count as a delivered document.
  • One run per customer. Two concurrent runs for the same customer are blocked at intake.
  • Hostile files exist. Recursive extraction with limits against compression bombs, and symbolic links inside RAR files are refused.
  • Cost and latency matter. Every document goes through a model; picking the setting is an engineering decision, not a matter of taste.

Decisions that shaped the product

Measure before choosing

Type and period correct (of 95)

higher is better

  • No thinking (0)71
  • 512 tokens74
  • 1024 tokens73
  • Dynamic (mean of 2 runs)72

Mean latency per document

lower is better

  • No thinking (0)3.9 s
  • 512 tokens6.1 s
  • 1024 tokens7.3 s
  • Dynamic (mean of 2 runs)8.6 s

Cost for 95 documents

lower is better

  • No thinking (0)US$ 0.18
  • 512 tokensUS$ 0.29
  • 1024 tokensUS$ 0.36
  • Dynamic (mean of 2 runs)US$ 0.44
Measured on 95 real documents against a human classification, no cache. Accuracy counts type and period together (type alone: 89 of 95). Two identical runs differed by about 2 documents, so accuracy differences are within noise. Choosing 512 rests on latency and cost.

Architecture

CRM cardCustomer packageIntake APIUpload + routingLock1 run per customerExtractionZIP, RAR, 7Z · limitsMultimodal AIType, entity, periodRule-based reviewNo AI · deterministicStandard nameTYPE ENTITY MM.YYYYSharePointRight folder · chunkedCRM updatedLink and checklistHuman checkFlagged in checklistBackoff retryBreaker after 5 errors

The analyst sends the package from the CRM card. The API identifies the department and the customer and guarantees a single run per customer. Files are extracted with limits against malicious archives, and each document is classified by a multimodal model with JSON output. After a fixed-rule review, the document is renamed to the standard, uploaded to the right SharePoint folder, and the CRM gets the link and a checklist of what is still missing.

Operating lessons

  • Extensions lie. A video saved as .pdf broke classification. Now the real file type is detected before any other step.
  • Tests never touch production. A test run against the real database cancelled a real job and created 97 copies. A test and a config guard now prevent it.
  • Success must be confirmed. Zero confirmed CRM writes become an error, never a success. The checklist is saved in the database, and a command resends only the CRM part without repeating the upload.
  • Large files take their own path. Above 20 MB the document goes through the model’s file API, and SharePoint uploads are chunked above 100 MB.
  • Throttling is expected. SharePoint calls retry with backoff behind a breaker that opens after 5 consecutive throttles.

My role

I conceived the robot and built its foundation over four months: the end-to-end pipeline, the CRM and SharePoint integrations, the naming and folder standard, the checklist, idempotency and operations. In October a collaborator ran a classifier-hardening pass: the thinking-budget experiment, parallel classification, the rule-based review, the large-file path and the guard against malicious RAR files.

What the product delivers

  • Classification of 26 document types, including photos and scanned PDFs, with entity, accounting period and a flag for debt documents.
  • Standardized names and folders, with separate folders for partners and group companies.
  • A pending-items checklist in the CRM, computed over the whole folder and aligned with the credit-analysis team’s rules.
  • Live progress for every run and a full history.

What I would do differently

  • An anonymized reference set in CI. Today the labelled sample contains real documents and cannot live in the repository, so accuracy regression is still measured outside the pipeline.
  • Ask the model for a confidence level. Without it, review relies on rules alone; a score would let human checks be prioritized.
  • Plan the move off the single process earlier. A simple database and one worker serve today’s volume well, but they are the known scaling limit.

Want something like this for your team?

Get in touch