The source files for all examples can be found in /examples.
Lasso
This example uses the LassoSolver to solve a lasso regression problem. We also show how to use the MLSolver.
The lasso regression problem is
\[\begin{array}{ll} \text{minimize} & (1/2)\|Ax - b\|_2^2 + \lambda \|x\|_1. \end{array}\]
using GeNIOS
using Random, LinearAlgebra, SparseArraysGenerating the problem data
Random.seed!(1)
m, n = 200, 400
A = randn(m, n)
A .-= sum(A, dims=1) ./ m
normalize!.(eachcol(A))
xstar = sprandn(n, 0.1)
b = A*xstar + 1e-3*randn(m)
λ = 0.05*norm(A'*b, Inf)0.14048198582260793LassoSolver interface
The easiest interface for this problem is the LassoSolver, where we just need to specify the regularization parameter (in addition to the problem data).
λ1 = λ
solver = GeNIOS.LassoSolver(λ1, A, b)
res = solve!(solver; options=GeNIOS.SolverOptions(use_dual_gap=true, dual_gap_tol=1e-4, verbose=true))
rmse = sqrt(1/m*norm(A*solver.zk - b, 2)^2)
println("Final RMSE: $(round(rmse, digits=8))")Starting setup...
Setup in 0.101s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Iteration Objective RMSE Dual Gap r_primal r_dual ρ Time
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0 1.995e+01 3.158e-01 9.256e+00 Inf Inf 1.000e+00 0.000
1 1.995e+01 3.158e-01 9.256e+00 0.000e+00 1.058e+01 1.000e+00 0.078
20 4.260e+00 5.491e-02 5.403e-02 2.000e-02 6.603e-02 1.000e+00 0.084
40 4.260e+00 5.409e-02 2.114e-04 1.546e-04 2.842e-04 1.000e+00 0.091
42 4.260e+00 5.409e-02 8.546e-05 8.677e-05 1.527e-04 1.000e+00 0.092
SOLVED in 0.092s, 42 iterations
Total time: 0.193s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Final RMSE: 0.07649858MLSolver interface
Under the hood, this is just a wrapper around the MLSolver interface. This interface is more general, and allows us to specify the per-sample loss used in the machine learning problem. Specifically, it solves problems with the form
\[\begin{array}{ll} \text{minimize} & \sum_{i=1}^N f(a_i^Tx - b_i) + \lambda_1 \|x\|_1 + (\lambda_2/2) \|x\|_2^2. \end{array}\]
It's easy to see that the lasso problem is a special case.
f(x) = 0.5*x^2
fconj(x) = 0.5*x^2
λ1 = λ
λ2 = 0.0
solver = GeNIOS.MLSolver(f, λ1, λ2, A, b; fconj=fconj)
res = solve!(solver; options=GeNIOS.SolverOptions(relax=true, use_dual_gap=true, dual_gap_tol=1e-3, verbose=true))
rmse = sqrt(1/m*norm(A*solver.zk - b, 2)^2)
println("Final RMSE: $(round(rmse, digits=8))")Starting setup...
Setup in 0.108s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Iteration Objective RMSE Dual Gap r_primal r_dual ρ Time
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0 1.995e+01 3.158e-01 9.256e+00 Inf Inf 1.000e+00 0.000
1 1.995e+01 3.158e-01 9.256e+00 0.000e+00 1.058e+01 1.000e+00 0.132
20 4.260e+00 5.491e-02 5.403e-02 2.000e-02 6.603e-02 1.000e+00 0.138
36 4.260e+00 5.408e-02 8.625e-04 4.108e-04 1.712e-03 1.000e+00 0.143
SOLVED in 0.143s, 36 iterations
Total time: 0.251s
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Final RMSE: 0.076475Note that we also defined the conjugate function of $f$, defined as
\[f^*(y) = \sup_x \{yx - f(x)\},\]
which allows us to use the dual gap as a stopping criterion (see our paper for a derivation). Specifying the conjugate function is optional, and the solver will fall back to using the primal and dual residuals if it is not specified.
This page was generated using Literate.jl.