Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to reproduce results for DAGMM #60

Open
jonomon opened this issue Jan 11, 2022 · 2 comments
Open

Unable to reproduce results for DAGMM #60

jonomon opened this issue Jan 11, 2022 · 2 comments

Comments

@jonomon
Copy link

jonomon commented Jan 11, 2022

Hello,

Thank you for the nice library!

I was just wondering if you managed to reproduce the results in Zong, Bo, et al. "Deep autoencoding gaussian mixture model for unsupervised anomaly detection." International conference on learning representations. 2018.

image

I used the following configuration:

DAGMMConfig(gmm_k=2, hidden_size=4, num_epochs=20000, lr=0.0001, batch_size=1024)

and only managed to get the following results on the Thyroid dataset (.mat obtained from http://odds.cs.stonybrook.edu):

Precision: 0.0238
Recall: 0.3571
F1: 0.0447
@aadyotb
Copy link
Contributor

aadyotb commented Jan 12, 2022

Hi @jonomon, there can be some subtle differences in the way precision/recall are computed, the way the detection threshold is chosen, and how the model handles point data (which the Thyroid dataset is) vs time series data. Before anything else, you should try to use PointwisePrecision, PointwiseRecall, and PointwiseF1 as your evaluation metrics, as the default ones are specialized for time series data. If this doesn't resolve the issue, @yangwenzhuo08 can you answer any further questions?

@jonomon
Copy link
Author

jonomon commented Jan 12, 2022

Hi @aadyotb

Thank you for the reply.

Using PointwisePrecision, PointwiseRecall, and PointwiseF1 had the following results:

Precision: 0.0238
Recall: 0.3409
F1: 0.0446

It seems like it help too much.

As a side note, both the Autoencoder and VAE achieved comparable results on the Thyroid dataset out of the box.

Model name: Autoencoder
Precision: 0.4444
Recall: 0.3636
F1: 0.4000

Model name: VAE
Precision: 0.4242
Recall: 0.3182
F1: 0.3636

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jonomon @aadyotb