The source files for all examples can be found in /examples.
Huber Fitting
This example sets up a $\ell_1$-regularized huber fitting problem using the MLSolver interface provided by GeNIOS. Huber fitting is a form of 'robust regression' that is less sensitive to outliers.
The huber loss function is defined as
\[f^\mathrm{hub}(x) = \begin{cases} \frac{1}{2}x^2 & \lvert x\rvert \leq 1 \\ |x| - \frac{1}{2} & \lvert x \rvert > 1 \end{cases}\]
We want to solve the problem
\[\begin{array}{ll} \text{minimize} & \sum_{i=1}^N f^\mathrm{hub}(a_i^T x - b_i) + \gamma \|x\|_1 \end{array}\]
using GeNIOS
using Random, LinearAlgebra, SparseArraysGenerating the problem data
Random.seed!(1)
N, n = 200, 400
A = randn(N, n)
A .-= sum(A, dims=1) ./ N
normalize!.(eachcol(A))
xstar = sprandn(n, 0.1)
b = A*xstar + 1e-3*randn(N)
# Add outliers
b += 10*collect(sprand(N, 0.05))
λ = 0.05*norm(A'*b, Inf)0.18184410798147943MLSolver interface
We just need to specify $f$ and the regularization parameters.
# Huber problem: min ∑ fʰᵘᵇ(aᵢᵀx - bᵢ) + λ||x||₁
f(x) = abs(x) <= 1 ? 0.5*x^2 : abs(x) - 0.5
λ1 = λ
λ2 = 0.0
solver = GeNIOS.MLSolver(f, λ1, λ2, A, b)
res = solve!(solver; options=GeNIOS.SolverOptions(use_dual_gap=false))
rmse = sqrt(1/N*norm(A*solver.zk - b, 2)^2)
println("Final RMSE: $(round(rmse, digits=8))")Starting setup...
Setup in 0.042s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Iteration Objective RMSE r_primal r_dual ρ Time
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0 5.505e+01 5.246e-01 Inf Inf 1.000e+00 0.000
1 5.505e+01 5.246e-01 0.000e+00 9.994e+00 1.000e+00 0.268
20 3.755e+01 3.297e-01 5.352e-02 2.085e-01 1.000e+00 0.272
40 3.735e+01 3.162e-01 1.053e-02 6.937e-02 1.000e+00 0.277
60 3.730e+01 3.104e-01 7.514e-03 4.175e-02 1.000e+00 0.282
80 3.728e+01 3.075e-01 3.500e-03 2.238e-02 1.000e+00 0.288
100 3.728e+01 3.058e-01 2.184e-03 1.384e-02 1.000e+00 0.295
120 3.728e+01 3.047e-01 1.377e-03 9.316e-03 1.000e+00 0.301
140 3.728e+01 3.041e-01 8.090e-04 6.417e-03 1.000e+00 0.307
160 3.728e+01 3.036e-01 6.316e-04 4.580e-03 1.000e+00 0.314
180 3.728e+01 3.033e-01 4.291e-04 3.225e-03 1.000e+00 0.321
199 3.728e+01 3.031e-01 4.722e-04 2.222e-03 1.000e+00 0.327
SOLVED in 0.327s, 199 iterations
Total time: 0.369s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Final RMSE: 0.62386089Duality gap
However, we can supply the conjugate function if we want to use the duality gap.
fconj(y) = y > 1 ? Inf : y^2/2.0
solver = GeNIOS.MLSolver(f, λ1, λ2, A, b; fconj=fconj)
res = solve!(solver; options=GeNIOS.SolverOptions(use_dual_gap=true))
rmse = sqrt(1/N*norm(A*solver.zk - b, 2)^2)
println("Final RMSE: $(round(rmse, digits=8))")Starting setup...
Setup in 0.039s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Iteration Objective RMSE Dual Gap r_primal r_dual ρ Time
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0 5.505e+01 5.246e-01 6.266e+00 Inf Inf 1.000e+00 0.000
1 5.505e+01 5.246e-01 6.266e+00 0.000e+00 9.994e+00 1.000e+00 0.000
20 3.755e+01 3.297e-01 1.529e-01 5.352e-02 2.085e-01 1.000e+00 0.005
40 3.735e+01 3.162e-01 6.772e-02 1.053e-02 6.937e-02 1.000e+00 0.009
60 3.730e+01 3.104e-01 3.585e-02 7.514e-03 4.175e-02 1.000e+00 0.015
80 3.728e+01 3.075e-01 1.983e-02 3.500e-03 2.238e-02 1.000e+00 0.021
100 3.728e+01 3.058e-01 1.283e-02 2.184e-03 1.384e-02 1.000e+00 0.027
120 3.728e+01 3.047e-01 9.128e-03 1.377e-03 9.316e-03 1.000e+00 0.034
140 3.728e+01 3.041e-01 6.302e-03 8.090e-04 6.417e-03 1.000e+00 0.040
160 3.728e+01 3.036e-01 4.532e-03 6.316e-04 4.580e-03 1.000e+00 0.047
180 3.728e+01 3.033e-01 3.218e-03 4.291e-04 3.225e-03 1.000e+00 0.054
200 3.728e+01 3.031e-01 2.172e-03 2.871e-04 2.185e-03 1.000e+00 0.061
220 3.728e+01 3.028e-01 1.272e-03 2.688e-04 1.277e-03 1.000e+00 0.068
240 3.728e+01 3.027e-01 7.490e-04 1.567e-04 7.492e-04 1.000e+00 0.074
260 3.728e+01 3.026e-01 4.418e-04 9.170e-05 4.402e-04 1.000e+00 0.081
280 3.728e+01 3.026e-01 2.607e-04 5.375e-05 2.590e-04 1.000e+00 0.087
300 3.728e+01 3.026e-01 1.538e-04 3.154e-05 1.525e-04 1.000e+00 0.093
317 3.728e+01 3.026e-01 9.821e-05 2.007e-05 9.730e-05 1.000e+00 0.098
SOLVED in 0.098s, 317 iterations
Total time: 0.137s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Final RMSE: 0.62163896This page was generated using Literate.jl.