Machine learning scripts for Fortranner.
https://qiita.com/advent-calendar/2022/fortlearner

cd test_script
make test_all

Forest Packing like implementation for Python
Re-Implement Automatic Differentiation
Benchmark and Optimize KDTree, Naive Bayse, Implement Linear/Kernel SVR, One-Class SVM, ica, HDBSCAN(not accelarated one)

2024/05/05: csr to dense.
2024/05/05: add optional argument for 'weighted_sampling', ignoring negative weightes.
2024/05/05: dense matrix to csr matrix
2024/05/04: mnist classifier example code
2024/04/21: add 'neural_network' (Re-implementation of automatic differentiation)
2024/04/20: remove 'multi_layer_perceptron'
2024/03/06: sparse matrix multiplication (csr-like sparse matrix and dense matrix)
2024/02/25: replace the weighted sampling method with a method that uses ↓
2024/02/18: branchless binary search and prefix sum with simd
2024/01/14: apply mydgemv to kdtree (replace openblas 'dgemv')
2024/01/09: my dgemv
2023/06/25: models can accept data_holder not only x
2023/06/10: fix and release random seed
2023/06/10: Forest Packing like implementation, 2x faster than naive implementation
2023/06/05: Fast Ingerence Engine for Decision Trees (https://cds.cern.ch/record/2688585/files/AA_main.pdf) 2023/04/08: Local Outlier Factor
2023/04/02: apply openmp for FOREST%predict(x, parallel=.true.)
2023/03/29: Thinnnig (isolation_forest)
2023/02/21: KNN classifier
2023/02/20: KNN regressor
2023/02/18: Implement and Optimize Kernel SVC (linear, poly, sigmoid, rbf) and add benchmark
2023/01/30: optimize 'linear' support vector machine classifier
2023/01/01: hash_map without delete
2022/12/25: svm (not so optimized)
2022/10/07: multi layer perceptron (wengert list, reverse mode)
2022/07/28: bug fix(kdtree, balltree)
2022/07/24: product_quantization
2022/06/11: adaboost
2022/05/03: nipals
2022/05/02: dbscan(modify 2022/05/19)
2022/04/27: oblivious_tree (not so optimized)
2022/04/22: minibatch_kmeans
2022/04/19: sliq (base estimator of xgboost, w/o openmp)
2022/02/15: threshold_tree
2022/01/22: breathing_kmeans
2022/01/09: locality_sensitive_hashing(random projection, p-stable random projection), balltree(n_neighbors, radius), exact_duplicate_search, one_at_a_time_hash, hash_table(no collision check)
2021/12/16: kdtree(n_neighbors, radius)
2021/11/23: (postponed) re-implement matrix-vector multiplication for accelerating kmeans
2021/10/12: add 'isolation_forest' benchmark
2021/10/10: Implemented elkan's method to speed up "kmeans", but it didn't make sense.
2021/09/28: speed up 'kmeans++'(see benchmark)
2021/09/24: isolation_tree and isolation_forest
2021/09/21: kmeans++
2021/09/20: simulated annealing initial temperature
2021/09/19: refactering simulated annealing
2021/09/17: implement sadt_regressor, simulated-annealing decision tree.
2021/09/14: Add Create Dataset Scripts.
2021/09/14: 'multi_mat_x_vec' with simd.
2021/09/04: Parallel implementation of 'extra_tree_regressor'.
2021/08/31: add new 'get_matrix_count_and_sum_up_parallel_r8' for extra_tree_regressor (useless).
2021/08/29: add new 'get_matrix_minmax_parallel' for extra_tree_regressor.
2021/08/24: add new 'get_minmax'.
2021/08/22: add benchmark for lawu_regressor.
2021/08/22: add benchmark for clouds_regressor.
2021/08/22: set '#include <stdint.h>' to inc_covariance_value_of_vectors.C
2021/08/22: add benchmark for decision_tree_regressor.
2021/08/22: add model dump & load to decision_tree_regressor, extra_tree_regressor, clouds_regressor, lawu_regressor
2021/08/20: add model dump & load to linear_regression, ridge_regression, lasso_regression

gfortran >= 7.4.0
gcc >= 7.4.0
openmp
Python >= 3.7.3 (to create sample datasets)
scikit-learn >= 0.23.2 (to create sample datasets)

x86_64
doxygen
make
graphviz

n_tree=1 and max_leaf_nodes=100, others are default.
SK = scikit-learn
FL = FortLearner

ET = extra_tree_regressor(FL), ExtraTreeRegressor(SK)
DT = decision_tree_regressor(FL), DecisionTreeRegressor(SK)
HG = clouds_regressor(FL), HistGradientBoostingRegressor(SK)
LW = lawu_regressor(FL) , There is no counterpart to scikit-learn.

KM = Kmeans()
KM_naive: Naive Implementation in Fortlearner
KM_fast: Fast Implementation in Fortlearner(Simplified Calculation of Euclid Distance + Fast Matrix-Vector Multiplication + Naive Centroid Update Skip)
FL: mean±std of 40 runs
SK: mean±std of 4 runs

IF = IsolationForest(n_estimators=100, max_samples=256, n_jobs=8)
Train:Validation:Test = 3:1:1
Datasets http://odds.cs.stonybrook.edu/
Thinnig = Remove trees with low average height.
Thinnig Detail https://www.scutum.jp/information/waf_tech_blog/2021/06/waf-blog-079.html .
100 iteration, auc mean±std

• Linear Regression:
  • linear_regression
  • lasso_regression
  • ridge_regression
• Logistic Regression:
  • logistic_regression
• Stochastic Gradient Descent:
  • sgd_regressor
• Decision Tree:
  • decision_tree_regressor
  • extra_tree_regressor
  • clouds_regressor
  • lawu_regressor
  • sliq_regressor
  • oblivious_tree_regressor
• Ensemble Trees:
  • random_forest_regressor
  • extra_trees_regressor
  • deep_forest_regressor(?)
• Boosting Trees:
  • gradient_boosting_tree_regressor
  • gradient_boosting_extra_tree_regressor
  • gradient_boosting_clouds_regressor
  • gradient_boosting_lawu_regressor
• Dimensionality Reduction:
  • pca
  • nipals
• Clustering
  • kmeans
  • minibatch_kmeans
  • breathing_kmeans
  • threshold_tree
  • dbscan
• Anomaly Detection
  • isolation_tree
  • isolation_forest
• Nearest Neighbours Search
  • kdtree
  • balltree
  • bruteforce
  • lsh
  • product_quantization
• Neural Network
  • simple mlp
• Support Vector Machine
  • support vector classifier

• Matrix Factorization
  • Matrix Factorization
  • Non-Negative Matrix Factorization
• Decision Tree
  • Axis-Parallel
    • residual likelihood tree
    • oblivious_tree
  • Oblique
    • simulated annealing decision tree
    • rotation tree
    • soft decision tree
    • CART-LC
    • OC1
    • householder cart
    • weighted oblique decision tree
    • residual likelihood forest
    • slow-growing tree
• Ensemble Trees
  • Extended Isolation Forest(anomaly detection)
  • PIDForest(anomaly detection)
  • Cross-Cluster Weighted Forests
  • WildWood
• Gradient Boosting Decision Tree
  • Axis-Parallel
    • Xgboost
    • LightGBM
    • CatBoost
    • SCORE: Selective Cascade of Residual ExtraTrees
    • NGBoost
• Multi-Layer Perceptron
• Neural Network
  • convolution
• Decomposition
  • Independent Component Analysis