FedNLPDataset

class fl_sim.data_processing.FedNLPDataset(datadir: Path | str | None = None, seed: int = 0, **extra_config: Any)

Bases: FedDataset, ABC

Base class for all federated NLP datasets.

Methods that have to be implemented by subclasses:

• get_dataloader

• _preload

• evaluate

• get_word_dict

Properties that have to be implemented by subclasses:

• url

• candidate_models

• doi

Parameters:

• datadir (Union[str, pathlib.Path], optional) – The directory to store the dataset. If None, use default directory.

• seed (int, default 0) – The random seed.

• **extra_config (dict, optional) – Extra configurations.

abstract get_dataloader(train_bs: int, test_bs: int, client_idx: int | None = None) → Tuple[DataLoader, DataLoader]

Get dataloader for client client_idx or get global dataloader.

load_partition_data(batch_size: int | None = None) → tuple

Partition data into all local clients.

Parameters:

batch_size (int, optional) – Batch size for dataloader. If None, use default batch size.

Returns:

• train_clients_num: int

  Number of training clients.

• train_data_num: int

  Number of training data.

• test_data_num: int

  Number of testing data.

• train_data_global: torch.utils.data.DataLoader

  Global training dataloader.

• test_data_global: torch.utils.data.DataLoader

  Global testing dataloader.

• data_local_num_dict: dict

  Number of local training data for each client.

• train_data_local_dict: dict

  Local training dataloader for each client.

• test_data_local_dict: dict

  Local testing dataloader for each client.

• vocab_len: int

  Length of the vocabulary.

Return type:

tuple

load_partition_data_distributed(process_id: int, batch_size: int | None = None) → tuple

Get local dataloader at client process_id or get global dataloader.

Parameters:

• process_id (int) – Index of the client to get dataloader. If None, get the dataloader containing all data, usually used for centralized training.

• batch_size (int, optional) – Batch size for dataloader. If None, use default batch size.

Returns:

• train_clients_num: int

  Number of training clients.

• train_data_num: int

  Number of training data.

• train_data_global: torch.utils.data.DataLoader or None

  Global training dataloader.

• test_data_global: torch.utils.data.DataLoader or None

  Global testing dataloader.

• local_data_num: int

  Number of local training data.

• train_data_local: torch.utils.data.DataLoader or None

  Local training dataloader.

• test_data_local: torch.utils.data.DataLoader or None

  Local testing dataloader.

• vocab_len: int

  Length of the vocabulary.

Return type:

tuple