Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Deterministic TFT model prediction not reproducable #2385

Closed
xlsi opened this issue May 17, 2024 · 2 comments
Closed

[BUG] Deterministic TFT model prediction not reproducable #2385

xlsi opened this issue May 17, 2024 · 2 comments
Labels
question Further information is requested

Comments

@xlsi
Copy link

xlsi commented May 17, 2024

Describe the bug
Hi, I'm using a deterministic TFT model for forecast. But .predict always returns a different output.
Here's the model detail.

TFTModel(hidden_size=256, lstm_layers=4, num_attention_heads=32, full_attention=False, feed_forward=GatedResidualNetwork, dropout=0.23125297663928174, hidden_continuous_size=8, categorical_embedding_sizes=None, add_relative_index=False, loss_fn=None, likelihood=None, norm_type=LayerNorm, use_static_covariates=True, batch_size=32, n_epochs=20, nr_epochs_val_period=1, add_encoders={'cyclic': {'future': ['dayofweek', 'month', 'quarter', 'dayofyear', 'weekofyear']}, 'datetime_attribute': {'future': ['is_month_end', 'is_month_start', 'is_year_start', 'is_year_end']}, 'position': {'past': ['relative'], 'future': ['relative']}, 'transformer': Scaler}, pl_trainer_kwargs={'accelerator': 'gpu', 'devices': [0], 'callbacks': [<pytorch_lightning.callbacks.early_stopping.EarlyStopping object at 0x000001EC20E37100>]}, model_name=tft_model, force_reset=True, save_checkpoints=True, input_chunk_length=31, output_chunk_length=21, optimizer_kwargs={'lr': 0.0001884674313235514})

To Reproduce

model = TFTModel.load("./tft_model_deterministic.pt", map_location=torch.device('cpu'))
model.to_cpu()

preds = model.predict(
    series=[data.split_after(pd.Timestamp("2023-11-14"))[0] for data in target_scaled],
    n=model.output_chunk_length,
    future_covariates=[future_cov_scaled for _ in range(len(train_target_scaled))],
    past_covariates=[past_cov_scaled for _ in range(len(train_target_scaled))],
)

Expected behavior
The .predict should have returned the same result every time it's called, but it's not.

System (please complete the following information):

  • Python version: [e.g. 3.9.13]
  • darts version [e.g. 0.27.0]

Additional context
Add any other context about the problem here.

@xlsi xlsi added bug Something isn't working triage Issue waiting for triaging labels May 17, 2024
@dennisbader
Copy link
Collaborator

Hi @xlsi , by default TFTModel uses likelihood=QuantileRegression(), loss_fn=None which results in a probabilistic model. Since you create it with likelihood=None, loss_fn=None, it will use the default.

You have to specify a loss_fn (e.g. use MSELoss or another) to make it deterministic:

from torch.nn import MSELoss
from torch
TFTModel(
    ...
    likelihood=None,
    loss_fn=MSELoss(),
)

@dennisbader dennisbader removed bug Something isn't working triage Issue waiting for triaging labels May 17, 2024
@madtoinou madtoinou added the question Further information is requested label May 17, 2024
@tRosenflanz
Copy link
Contributor

Apologies. Initially wrongly mentioned this issue in the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants