What converted, and how well

A handbook over a folder has to survive a file it cannot read and flag a page it read badly. convert_all with one argument keeps going, and the confidence grade from lesson 18 says which results to trust.

Keep going on a bad file

Example

from pathlib import Path
from docling.datamodel.base_models import ConversionStatus
from docling.document_converter import DocumentConverter

files = ["shipping.md", "refunds.html", "orders.csv", "returns.pdf"]
for result in DocumentConverter().convert_all([Path(n) for n in files],
                                              raises_on_error=False):
    print(result.input.file.name, "|", result.status.name)

raises_on_error=False from lesson 19 is what a folder needs: one unreadable file becomes a FAILURE row instead of an exception that ends the run. result.status is a ConversionStatus, and SUCCESS is the one to index.

The confidence grade

Example

from pathlib import Path
from docling.document_converter import DocumentConverter

files = ["refunds.html", "returns.pdf"]
for result in DocumentConverter().convert_all([Path(n) for n in files]):
    print(result.input.file.name, "|", result.confidence.mean_grade.value)

The PDF is graded excellent; the HTML is unspecified, because grading is about how well a page was read and HTML was not read from a page, as lesson 18 showed. A report that mixes formats has to expect unspecified and not treat it as a failure.

A row per file

Example

from pathlib import Path
from docling.datamodel.base_models import ConversionStatus
from docling.document_converter import DocumentConverter

files = ["shipping.md", "refunds.html", "orders.csv", "returns.pdf"]
report = []
for result in DocumentConverter().convert_all([Path(n) for n in files],
                                              raises_on_error=False):
    name = result.input.file.name
    if result.status is ConversionStatus.SUCCESS:
        report.append((name, result.status.name, result.confidence.mean_grade.value))
    else:
        report.append((name, result.status.name, "-"))
for row in report:
    print(row)

One row per file: name, status, and grade where there is one. This is the report the handbook returns beside its index, so the caller can see what went in and, from the grade, decide whether to trust it or send it for review.

Converting the folder on every run reparses the PDF each time. To rebuild in seconds, save each parsed document to disk with the method from lesson 10 and load it back on the next run, reconverting only files that changed. The report stays the same; only where the document comes from changes.

Try it yourself

Add notes.xyz to the list and read its FAILURE row.
Print low_grade beside mean_grade for the PDF.
Filter the report to the rows worth indexing, graded good or excellent or unspecified.

PreviousSearching the chunks

This is what real progress feels like.