The configuration manager

The configuration manager is the one class that reads config.yaml, params.yaml and schema.yaml at start-up and hands each stage its own entity object, creating the folder that stage writes into as it goes.

Last updated: 19 Oct, 2026 · MLOps

Entity classes for stage config declared what a stage's settings look like, and built one by hand. Something has to build them from the files for real, and it should not be the stages: four stages each opening YAML is four places to get a path wrong. The manager is the single seam between the files and the typed objects, and it is the next file in the video's workflow order.

The clip's manager reads config.yaml and params.yaml; this one reads a third file, schema.yaml, because the student-performance pipeline validates its columns before it trains.

Reading the three files once

The imports name the four entity classes it builds and the two helpers it uses.

from pathlib import Path    # every path a stage receives is a Path, not text

from sp.entity.config_entity import (
    DataIngestionConfig,
    DataTransformationConfig,
    DataValidationConfig,
    ModelTrainerConfig,
)
from sp.utils.common import create_directories, read_yaml

The constructor takes the three file paths as defaults, so a test can point it at a different set of files, and reads each one into a ConfigBox.

class ConfigurationManager:
    """Reads the YAML files once, then hands each stage its own typed config."""

    def __init__(
        self,
        config_path=Path("config/config.yaml"),
        params_path=Path("params.yaml"),
        schema_path=Path("schema.yaml"),
    ):
        # read once at start-up: no stage opens a YAML file itself
        self.config = read_yaml(config_path)
        self.params = read_yaml(params_path)
        self.schema = read_yaml(schema_path)

Handing data ingestion its config

There is one getter per stage, and each one does three things: pick that stage's block out of config.yaml, make the folder, and return a filled entity. The return annotation says which entity, so a getter that returns the wrong shape is obvious.

    def get_data_ingestion_config(self) -> DataIngestionConfig:
        c = self.config.data_ingestion
        create_directories([c.root_dir])        # the stage can write the moment it starts

        return DataIngestionConfig(
            root_dir=Path(c.root_dir),
            source_url=c.source_url,
            local_data_file=Path(c.local_data_file),
            train_file=Path(c.train_file),
            test_file=Path(c.test_file),
            test_size=self.params.TEST_SIZE,    # wiring from config, numbers from params
            random_state=self.params.RANDOM_STATE,
        )

This is where the two files meet. The paths come from the stage's block in config.yaml, wrapped in Path on the way through; the split size and the seed come from params.yaml. The stage never learns that there were two files.

Giving validation the schema

The validation getter is the only one that touches the third file. It passes the columns as a plain dict, because that is what the entity's annotation asks for and @ensure_annotations is checking.

    def get_data_validation_config(self) -> DataValidationConfig:
        c = self.config.data_validation
        create_directories([c.root_dir])

        return DataValidationConfig(
            root_dir=Path(c.root_dir),
            status_file=Path(c.status_file),
            expected_columns=dict(self.schema.COLUMNS),   # a plain dict, as the entity asks
        )

The transformation getter is the shortest: a folder, the file the fitted preprocessor is saved to, and the name of the column being predicted.

    def get_data_transformation_config(self) -> DataTransformationConfig:
        c = self.config.data_transformation
        create_directories([c.root_dir])

        return DataTransformationConfig(
            root_dir=Path(c.root_dir),
            preprocessor_file=Path(c.preprocessor_file),
            target_column=self.params.TARGET_COLUMN,      # which column the model predicts
        )

Passing the hyperparameters to the trainer

The trainer's getter reaches furthest into params.yaml: the target column, both forest settings and the seed.

    def get_model_trainer_config(self) -> ModelTrainerConfig:
        c = self.config.model_trainer
        create_directories([c.root_dir])

        return ModelTrainerConfig(
            root_dir=Path(c.root_dir),
            model_file=Path(c.model_file),
            metrics_file=Path(c.metrics_file),
            target_column=self.params.TARGET_COLUMN,
            n_estimators=self.params.N_ESTIMATORS,        # tune these in params.yaml
            max_depth=self.params.MAX_DEPTH,
            random_state=self.params.RANDOM_STATE,        # the same seed on every run
        )

TARGET_COLUMN: math_score
TEST_SIZE: 0.2
RANDOM_STATE: 42
N_ESTIMATORS: 100
MAX_DEPTH: 12

COLUMNS:
  gender: object
  race_ethnicity: object
  parental_level_of_education: object
  lunch: object
  test_preparation_course: object
  math_score: int64
  reading_score: int64
  writing_score: int64

data_ingestion:
  root_dir: artifacts/data_ingestion
  source_url: https://raw.githubusercontent.com/krishnaik06/mlproject/main/notebook/data/stud.csv
  local_data_file: artifacts/data_ingestion/stud.csv
  train_file: artifacts/data_ingestion/train.csv
  test_file: artifacts/data_ingestion/test.csv

data_validation:
  root_dir: artifacts/data_validation
  status_file: artifacts/data_validation/status.json

data_transformation:
  root_dir: artifacts/data_transformation
  preprocessor_file: artifacts/data_transformation/preprocessor.joblib

model_trainer:
  root_dir: artifacts/model_trainer
  model_file: artifacts/model_trainer/model.joblib
  metrics_file: artifacts/model_trainer/metrics.json

import json, os
from pathlib import Path

import joblib
import yaml
from box import ConfigBox
from ensure import ensure_annotations


@ensure_annotations
def read_yaml(path: Path) -> ConfigBox:
    """Load a YAML file so its keys can be read with dots, not brackets."""
    with open(path) as f:
        return ConfigBox(yaml.safe_load(f))


@ensure_annotations
def create_directories(paths: list):
    for p in paths:
        os.makedirs(p, exist_ok=True)


@ensure_annotations
def save_json(path: Path, data: dict):
    with open(path, "w") as f:
        json.dump(data, f, indent=4)


@ensure_annotations
def save_object(path: Path, obj):
    joblib.dump(obj, path)


@ensure_annotations
def load_object(path: Path):
    return joblib.load(path)

from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class DataIngestionConfig:
    root_dir: Path
    source_url: str
    local_data_file: Path
    train_file: Path
    test_file: Path
    test_size: float
    random_state: int


@dataclass(frozen=True)
class DataValidationConfig:
    root_dir: Path
    status_file: Path
    expected_columns: dict


@dataclass(frozen=True)
class DataTransformationConfig:
    root_dir: Path
    preprocessor_file: Path
    target_column: str


@dataclass(frozen=True)
class ModelTrainerConfig:
    root_dir: Path
    model_file: Path
    metrics_file: Path
    target_column: str
    n_estimators: int
    max_depth: int
    random_state: int

Asking the manager for two stages' settings

One manager, built once, answers for every stage. Run this from the project root, where the three YAML files are.

import os

from sp.config.configuration import ConfigurationManager

manager = ConfigurationManager()      # the three files are read here, once

ingestion = manager.get_data_ingestion_config()
trainer = manager.get_model_trainer_config()

print("ingestion writes to:", ingestion.root_dir)
print("split and seed     :", ingestion.test_size, ingestion.random_state)
print("forest             :", trainer.n_estimators, trainer.max_depth)
print("target column      :", trainer.target_column)
print("folders now on disk:", sorted(os.listdir("artifacts")))

ingestion writes to: artifacts/data_ingestion
split and seed     : 0.2 42
forest             : 100 12
target column      : math_score
folders now on disk: ['data_ingestion', 'model_trainer']

What the manager produced

• Each stage got only its own block: the ingestion object knows about artifacts/data_ingestion and nothing about the trainer's files.
• The numbers arrived from params.yaml, the 0.2 split and the seed 42 on the ingestion object, 100 trees and a depth of 12 on the trainer's, with no number typed in Python.
• The target column is a setting too, math_score, so the trainer and the transformer agree on what is being predicted without either one hard-coding it.
• Two folders exist that did not before, because each getter called create_directories before returning; asking for all four settings creates all four folders.

YAML in each stage against one manager

Where one manager pays off

• A renamed key, where source_url becoming source is a one-line change in one getter rather than a search across the stages.
• Unit testing a stage, where you build its entity by hand and never put a YAML file in the test folder at all.
• A second environment, where pointing the manager at another config.yaml moves every output folder at once.

Related

• Previous: Entity classes for stage config
• Next: Stage components
• Reference: pathlib in the Python standard library