Four files in one call
Everything so far converted one file. A handbook is a folder, and a folder always contains one file that goes wrong.
convert_all takes a list of paths and yields one result per file, in the order given. It reuses the loaded models across the whole list, which is the difference lesson 13 warned about.
from pathlib import Path
from docling.document_converter import DocumentConverter
names = ["shipping.md", "refunds.html", "orders.csv", "returns.pdf"]
converter = DocumentConverter()
for result in converter.convert_all([Path(n) for n in names]):
print(result.input.file.name, result.status.name, len(result.document.texts))Four formats, one loop, and the status from lesson 3 beside each. It yields as it goes rather than building a list, so a folder of a thousand files does not sit in memory.
One bad file stops everything
Here is a fifth file with an extension Docling does not know.
Ring the courier about A17.from docling.exceptions import ConversionError
try:
for result in converter.convert_all([Path("notes.xyz"), Path("shipping.md")]):
print(result.input.file.name)
except ConversionError as stopped:
print(stopped)The good file after it never ran. For a folder that is the wrong behaviour: one unreadable file should not cost you the other nine hundred and ninety-nine.
Keeping going
for result in converter.convert_all(
[Path("notes.xyz"), Path("shipping.md")], raises_on_error=False):
print(result.input.file.name, "->", result.status.name)raises_on_error=False turns the exception back into a status. Now both files come back, one skipped and one successful, and it is your loop that decides what to do about it.
What went wrong, in detail
results = converter.convert_all(
[Path("notes.xyz")], raises_on_error=False)
for error in list(results)[0].errors:
print(error.component_type.value, "|", error.error_message)The errors list from lesson 3, filled in. Each entry says which part of the pipeline complained and what it said, and for a page level failure it carries the page number too.
raises_on_error=False for a folder, and always read the status. Without the first you lose the batch; without the second you quietly index an empty document.- Convert all five files with
raises_on_error=Falseand count the successes. - Add a file with a
.mdextension containing binary rubbish and see which status it gets. - Time
convert_allover the four files against four separateconvertcalls on one converter.
Slow is fine. Stopping is the only problem.