Core Concept: Loaders

What is a Loader?

A Loader is aiNXT's automatic code discovery system. It scans your Python modules to find and register classes and functions that can be used to create objects (datasets, models, etc.).

Real-world analogy: Imagine you're organizing a music festival. Instead of manually calling every musician, you send scouts (Loaders) to different cities (modules) to find performers (classes/functions) who match certain criteria (type, pattern, task).

Why Do We Need Loaders?

Without Loaders, you'd have to manually register every dataset and model:

# Without Loaders - tedious and error-prone
factory = Factory()
factory.register("classification", "resnet", ResNetModel)
factory.register("classification", "vgg", VGGModel)
factory.register("classification", "mobilenet", MobileNetModel)
# ... manually register hundreds of classes

With Loaders, this happens automatically:

# With Loaders - automatic discovery
loader = Loader(
    template="myproject.models.{task}",
    tasks=["classification"],
    kind=Model
)
factory = Factory(loader)  # All models automatically registered!

How Loaders Work

1. Module Template System

Loaders use templates with placeholders to know where to search:

from ainxt.factory import Loader
from ainxt.models import Model

# Template with {task} placeholder
loader = Loader(
    template="myproject.models.{task}",
    tasks=["classification", "detection", "segmentation"]
)

# This will search in:
# - myproject.models.classification
# - myproject.models.detection
# - myproject.models.segmentation

What happens under the hood:

For each task: "myproject.models.{task}" → "myproject.models.classification"
The module is imported
All classes and functions are scanned
Matching ones are registered with key (task, name)

2. Filtering What Gets Loaded

Loaders have multiple filtering mechanisms:

Filter by Type (`kind`)

from ainxt.data import Dataset

loader = Loader(
    template="myproject.data.{task}",
    tasks=["classification"],
    kind=Dataset  # Only load Dataset subclasses
)

Filter by Name Pattern (`pattern`)

loader = Loader(
    template="myproject.models.{task}",
    tasks=["classification"],
    pattern=r".*Model$"  # Only names ending with "Model"
)

Filter by Source (`strict`)

loader = Loader(
    template="myproject.data.{task}",
    strict=True  # Only classes defined IN this module, not imported ones
)

Filter by Custom Condition (`condition`)

def has_pretrained(constructor):
    """Only load models with pretrained weights available."""
    return hasattr(constructor, 'pretrained_urls')

loader = Loader(
    template="models.{task}",
    condition=has_pretrained
)

3. Name Inference and Formatting

Loaders automatically infer names from class/function names:

# Class name: ImageClassificationDataset
# Registered as: "imageclassificationdataset" (lowercase)

# You can clean this up:
loader = Loader(
    template="myproject.data.{task}",
    ignore_suffixes=["Dataset"],  # Strip these from names
)

# Now registered as: "imageclassification"

You can also format the final name:

loader = Loader(
    template="models.{task}",
    name_template="create_{name}",  # Add prefix
    ignore_suffixes=["Model"]
)

# Class "ResNetModel" becomes "create_resnet"

Practical Examples

Example 1: Loading Models

# File structure:
# myproject/
#   models/
#     classification/
#       __init__.py
#       resnet.py       # contains ResNetModel
#       vgg.py          # contains VGGModel
#       mobilenet.py    # contains MobileNetModel

# Loader setup:
from ainxt.factory import Loader
from ainxt.models import Model

model_loader = Loader(
    template="myproject.models.{task}",
    tasks=["classification", "detection"],
    kind=Model,
    ignore_suffixes=["Model"]  # Clean up names
)

# Automatically finds and registers:
# ("classification", "resnet") → ResNetModel
# ("classification", "vgg") → VGGModel
# ("classification", "mobilenet") → MobileNetModel
# ("detection", ...) → detection models

Example 2: Loading Datasets

# myproject/
#   data/
#     datasets/
#       classification/
#         imagenet.py   # contains ImageNetDataset
#         cifar.py      # contains CIFAR10Dataset, CIFAR100Dataset

from ainxt.data import Dataset

dataset_loader = Loader(
    template="myproject.data.datasets.{task}",
    tasks=["classification"],
    kind=Dataset,
    pattern=r"^(?!_).*",  # Exclude private classes (starting with _)
    ignore_suffixes=["Dataset"]
)

# Automatically finds:
# ("classification", "imagenet") → ImageNetDataset
# ("classification", "cifar10") → CIFAR10Dataset
# ("classification", "cifar100") → CIFAR100Dataset

Example 3: Loading Functions

Loaders work with functions too, not just classes:

# myproject/augmentation/image.py
def horizontal_flip(image):
    """Flips an image horizontally."""
    return image[:, ::-1]

def random_rotation(image, degrees=15):
    """Rotates an image."""
    pass

# Loader:
augmentation_loader = Loader(
    template="myproject.augmentation.{task}",
    tasks=["image", "video"]
)

# Registers:
# ("image", "horizontal_flip") → horizontal_flip function
# ("image", "random_rotation") → random_rotation function

Integration with Factories

Loaders are typically used with Factories:

from ainxt.factory import Factory, Loader

# Create loader
loader = Loader(template="myproject.models.{task}", tasks=["classification"])

# Use in factory
MODELS = Factory(loader)

# Now you can build models from config:
model = MODELS.build(task="classification", name="resnet", num_classes=10)

Helper Functions

aiNXT provides helper functions in ainxt/serving/singletons.py:

from ainxt.serving import create_dataset_factory, create_model_factory

# These functions create Loaders + Factories for you
DATASETS = create_dataset_factory(package="myapp", tasks=["classification"])
MODELS = create_model_factory(package="myapp", tasks=["classification"])

# Equivalent to:
# dataset_loader = Loader(template="myapp.data.datasets.{task}", ...)
# DATASETS = Factory(dataset_loader)

Real-World Usage: DigitalNXT.Vision

Let's see how DigitalNXT.Vision uses Loaders:

# File: DigitalNXT.Vision/vision/serving/singletons.py

from ainxt.serving import create_dataset_factory, create_model_factory
from vision.constants import PACKAGE
from vision.task import Task

# Create factories with automatic loading
DATASETS = create_dataset_factory(PACKAGE, Task)
MODELS = create_model_factory(PACKAGE, Task)

# This automatically scans:
# - vision/data/datasets/classification/*.py
# - vision/data/datasets/detection/*.py
# - vision/models/classification/*.py
# - vision/models/detection/*.py
# And registers all Dataset/Model classes found

Advanced Features

1. Multiple Module Paths

Loaders can scan multiple module patterns:

# Scan both base and task-specific modules
loader = Loader(
    template="myapp.{task}.models",
    tasks=["nlp", "vision"]
)

# Scans:
# - myapp.nlp.models
# - myapp.vision.models

2. Decorator Loading

Loaders can find decorators for datasets/models:

# Load dataset decorators
decorator_loader = Loader(
    template="myapp.data.datasets.{task}.decorators",
    tasks=["classification"]
)

factory = Factory(dataset_loader, decorator=decorator_loader)

3. Return Type Inference

Loaders can infer what type of objects they create:

from ainxt.data import Dataset

loader = Loader[Dataset](  # Type hint for what it loads
    template="myapp.data.{task}",
    kind=Dataset
)

# loader.return_type == Dataset

Troubleshooting

Issue 1: "Module not found"

Problem: ModuleNotFoundError: No module named 'myapp.models'

Solutions:

# 1. Check the template path is correct
import myapp.models.classification  # Test the import manually

# 2. Ensure package is installed
# pip install -e .

# 3. Check PYTHONPATH includes your project root
import sys
print(sys.path)

Issue 2: "No classes found"

Problem: Loader doesn't find any classes

Solutions:

# 1. Check the kind parameter matches your base class
loader = Loader(kind=Dataset)  # Ensure your classes inherit from Dataset

# 2. Check strict parameter
loader = Loader(strict=False)  # Include imported classes

# 3. Check pattern filter
loader = Loader(pattern=r".*")  # Match everything to debug

# 4. Manually verify the module content
import myapp.models.classification
import inspect
for name, obj in inspect.getmembers(myapp.models.classification):
    print(f"{name}: {obj}")

Issue 3: "Wrong classes loaded"

Problem: Loader finds classes you don't want

Solutions:

# 1. Use stricter kind filtering
loader = Loader(kind=Dataset, strict=True)

# 2. Use pattern to filter names
loader = Loader(pattern=r"^[A-Z].*Dataset$")  # Only public Dataset classes

# 3. Use custom condition
def is_valid_model(obj):
    return hasattr(obj, 'predict')

loader = Loader(condition=is_valid_model)

Best Practices

1. Consistent Module Structure

# Good - consistent structure
myproject/
  data/
    datasets/
      classification/  # Task-specific
      detection/
  models/
    classification/  # Task-specific
    detection/

2. Use Helper Functions

# Prefer helper functions over manual setup
from ainxt.serving import create_dataset_factory

# Good
DATASETS = create_dataset_factory("myapp", tasks)

# Verbose (but equivalent)
loader = Loader(template="myapp.data.datasets.{task}", tasks=tasks, kind=Dataset)
decorator = Loader(template="myapp.data.datasets.{task}.decorators", tasks=tasks)
DATASETS = Factory(loader, decorator=decorator)

3. Clear Naming Conventions

# Use clear, descriptive class names
class ImageClassificationDataset(Dataset):  # Good
class ICD(Dataset):  # Too abbreviated

# Use ignore_suffixes to clean up registered names
loader = Loader(
    template="...",
    ignore_suffixes=["Dataset", "Model"]  # "ImageClassificationDataset" → "imageclassification"
)

4. Type Hints

# Add type hints for clarity
from ainxt.data import Dataset

loader = Loader[Dataset](
    template="myapp.data.{task}",
    kind=Dataset
)

Summary

Loaders automatically discover and register classes/functions from modules
Templates with {task} placeholders define where to search
Filters (kind, pattern, strict, condition) control what gets loaded
Name inference converts class names to registry keys
Integration with Factories enables configuration-driven object creation
Helper functions in ainxt.serving simplify common patterns

Loaders eliminate manual registration, making your codebase more maintainable and automatically discovering new components as you add them.

Core Concept: Loaders

What is a Loader?

Why Do We Need Loaders?

How Loaders Work

1. Module Template System

2. Filtering What Gets Loaded

Filter by Type (kind)

Filter by Name Pattern (pattern)

Filter by Source (strict)

Filter by Custom Condition (condition)

3. Name Inference and Formatting

Practical Examples

Example 1: Loading Models

Example 2: Loading Datasets

Example 3: Loading Functions

Integration with Factories

Helper Functions

Real-World Usage: DigitalNXT.Vision

Advanced Features

1. Multiple Module Paths

2. Decorator Loading

3. Return Type Inference

Troubleshooting

Issue 1: "Module not found"

Issue 2: "No classes found"

Issue 3: "Wrong classes loaded"

Best Practices

1. Consistent Module Structure

2. Use Helper Functions

3. Clear Naming Conventions

4. Type Hints

Summary

See Also

Filter by Type (`kind`)

Filter by Name Pattern (`pattern`)

Filter by Source (`strict`)

Filter by Custom Condition (`condition`)