Balam et al 2025a - Revision history

Gstinoco at 20:48, 27 December 2025

2025-12-27T20:48:55Z

Show changes

Gstinoco: Gstinoco moved page Review 960218438971 to Balam et al 2025a

2025-12-27T20:06:45Z

Gstinoco moved page Review 960218438971 to Balam et al 2025a

Reymundo.itza at 17:02, 27 December 2025

2025-12-27T17:02:29Z

Reymundo.itza at 17:01, 27 December 2025

2025-12-27T17:01:42Z

Reymundo.itza at 17:00, 27 December 2025

2025-12-27T17:00:05Z

Reymundo.itza at 16:58, 27 December 2025

2025-12-27T16:58:56Z

Reymundo.itza at 16:57, 27 December 2025

2025-12-27T16:57:41Z

Reymundo.itza at 06:05, 27 December 2025

2025-12-27T06:05:10Z

Reymundo.itza at 05:58, 27 December 2025

2025-12-27T05:58:23Z

Reymundo.itza at 05:57, 27 December 2025

2025-12-27T05:57:29Z

@@ Line 1,026: / Line 1,026: @@
 ==8 Conclusions==
-Fitting a curve to a given set of data is one of the most simple of the so called `''ill-posed''' problems. This is an example of a broad set of problems called least squares problems. This simple problem contains many of the ingredients, both theoretical and computational, of modern challenge and complex problems that are of great importance in computational modelling and applications, specially when computer solutions are obtained using finite precision machines. Commonly there is no `''best computational algorithm''' for general problems, but for a particular problem, like the one considered in this article, we can compare results obtained with different approaches or algorithms.
+Fitting a curve to a given set of data is one of the most simple of the so called ''ill-posed'' problems. This is an example of a broad set of problems called least squares problems. This simple problem contains many of the ingredients, both theoretical and computational, of modern challenge and complex problems that are of great importance in computational modelling and applications, specially when computer solutions are obtained using finite precision machines. Commonly there is no `''best computational algorithm''' for general problems, but for a particular problem, like the one considered in this article, we can compare results obtained with different approaches or algorithms.
 It is clear that the best fit to a 10th-degree polynomial is obtained with the QR algorithm, as it produces the smallest residual when compared to algorithms based on ''the normal equations'' and SVD. It is noteworthy that each method yields entirely different coefficients for this polynomial. Not only the sign of the coefficients but also the scale of the values differ drastically. These results demonstrate that even simple ill-posed problems must be studied and numerically solved with extreme care, employing stable state-of-the-art algorithms and tools that avoid the accumulation of rounding errors due to the finite arithmetic precision of computers.

@@ Line 1,012: / Line 1,012: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_08.png|100px|NN solution with the hyperbolic tangent as activation function.]]
+|[[Image:Review_960218438971-fig_08.png|450px|NN solution with the hyperbolic tangent as activation function.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 8:''' NN solution with the hyperbolic tangent as activation function.
@@ Line 1,019: / Line 1,019: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_09.png|100px|NN solution with the sigmoidal as activation function.]]
+|[[Image:Review_960218438971-fig_09.png|450px|NN solution with the sigmoidal as activation function.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 9:''' NN solution with the sigmoidal as activation function.

@@ Line 6: / Line 6: @@
 ==2 Introduction==
-In mathematics, the term `''least squares''' refers to an approach for “solving” overdetermined linear or nonlinear systems of equations. A common problem in science is to fit a model to noisy measurements or observations. Instead of solving the equations exactly, which in many problems is not possible, we seek only to minimize the sum of the squares of the residuals.
+In mathematics, the term ''least squares'' refers to an approach for “solving” overdetermined linear or nonlinear systems of equations. A common problem in science is to fit a model to noisy measurements or observations. Instead of solving the equations exactly, which in many problems is not possible, we seek only to minimize the sum of the squares of the residuals.
 The algebraic procedure of the method of least squares was first published by Legendre in 1805 <span id='citeF-1'></span>[[#cite-1|[1]]]. It was justified as a statistical procedure by Gauss in 1809 <span id='citeF-2'></span>[[#cite-2|[2]]], where he claimed to have discovered the method of least squares in 1795 <span id='citeF-3'></span>[[#cite-3|[3]]]. Robert Adrian had already published a work in 1808, according to <span id='citeF-4'></span>[[#cite-4|[4]]]. After Gauss, the method of least squares quickly became the standard procedure for analysis of astronomical and geodetic data. There are several good accounts of the history of the invention of least squares and the dispute between Gauss and Legendre, as shown in <span id='citeF-3'></span>[[#cite-3|[3]]] and references therein. Gauss gave the method a theoretical basis in two memoirs <span id='citeF-5'></span>[[#cite-5|[5]]], where he proves the optimality of the least squares estimate without any assumptions that the random variables follow a particular distribution. In an article by Yves <span id='citeF-6'></span>[[#cite-6|[6]]] there is a survey of the history, development, and applications of least squares, including ordinary, constrained, weighted, and total least squares, where he includes information about fitting curves and surfaces from ancient civilizations, with applications to astronomy and geodesy.

@@ Line 842: / Line 842: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_06.png|100px|Computed polynomial curve with QR and SVD.]]
+|[[Image:Review_960218438971-fig_06.png|450px|Computed polynomial curve with QR and SVD.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 6:''' Computed polynomial curve with QR and SVD.

@@ Line 6: / Line 6: @@
 ==2 Introduction==
-In mathematics, the term ''least squares'' refers to an approach for “solving” overdetermined linear or nonlinear systems of equations. A common problem in science is to fit a model to noisy measurements or observations. Instead of solving the equations exactly, which in many problems is not possible, we seek only to minimize the sum of the squares of the residuals.
+In mathematics, the term `''least squares''' refers to an approach for “solving” overdetermined linear or nonlinear systems of equations. A common problem in science is to fit a model to noisy measurements or observations. Instead of solving the equations exactly, which in many problems is not possible, we seek only to minimize the sum of the squares of the residuals.
 The algebraic procedure of the method of least squares was first published by Legendre in 1805 <span id='citeF-1'></span>[[#cite-1|[1]]]. It was justified as a statistical procedure by Gauss in 1809 <span id='citeF-2'></span>[[#cite-2|[2]]], where he claimed to have discovered the method of least squares in 1795 <span id='citeF-3'></span>[[#cite-3|[3]]]. Robert Adrian had already published a work in 1808, according to <span id='citeF-4'></span>[[#cite-4|[4]]]. After Gauss, the method of least squares quickly became the standard procedure for analysis of astronomical and geodetic data. There are several good accounts of the history of the invention of least squares and the dispute between Gauss and Legendre, as shown in <span id='citeF-3'></span>[[#cite-3|[3]]] and references therein. Gauss gave the method a theoretical basis in two memoirs <span id='citeF-5'></span>[[#cite-5|[5]]], where he proves the optimality of the least squares estimate without any assumptions that the random variables follow a particular distribution. In an article by Yves <span id='citeF-6'></span>[[#cite-6|[6]]] there is a survey of the history, development, and applications of least squares, including ordinary, constrained, weighted, and total least squares, where he includes information about fitting curves and surfaces from ancient civilizations, with applications to astronomy and geodesy.
@@ Line 156: / Line 156: @@
 The normal equations approach is a very simple procedure to solve the linear least squares problem. It is the most used approach in the scientific and engineering community, and very popular in statistical software. However, it must be used with precaution, specially when the design matrix <math display="inline">A</math> is ill-conditioned (or it is rank deficient) and finite precision arithmetic, in digital conventional devices, is employed. In order to understand this phenomenon, it is convenient to show an example and then discuss the results.
-Example 2: The National Institute of Standards and Technology (''NIST'') is a branch of the U.S. Department of Commerce responsible for establishing national and international standards. ''NIST'' maintains reference data sets for use in the calibration and certification of statistical software. On its website <span id='citeF-10'></span>[[#cite-10|[10]]] we can find the ``''Filip'''' data set, which consists of 82 observations of a variable <math display="inline">y</math> for different <math display="inline">t</math> values. The aim is to model this data set using a 10th-degree polynomial. This is part of exercise 5.10 in Cleve Molers' book <span id='citeF-11'></span>[[#cite-11|[11]]].  For this problem we have <math display="inline">m = 82</math> data points <math display="inline">(t_i, \hat{y}_i)</math>, and we want to compute <math display="inline">n = 11</math> coefficients <math display="inline">c_j</math> for the 10th-degree polynomial. The <math display="inline">m \times n</math> design matrix <math display="inline">A</math> has coefficients <math display="inline">a_{ij} = t_i^{j-1}</math>. In order to given an idea of the complexity of this matrix, we observe that its minimum coefficient is 1 and its maximum coefficient is a bit greater than <math display="inline">2.7\times 10^9</math>, while its condition number is <math display="inline">\kappa (A)\approx \mathcal{O}(10^{15})</math>. The matrix of the normal equations, <math display="inline">A^TA</math>, is a much smaller matrix of size <math display="inline">n\times n</math>, but `''more singular''', since its minimum and maximum coefficients (in absolute value) are close to 82 and <math display="inline">5.1\times 10^{19}</math>, respectively, with a very high condition number <math display="inline">\kappa (A^TA)\approx \mathcal{O}(10^{30})</math>. The matrix of the normal equations is highly ill-conditioned in this case because there are some clusters of data points very close to each other with almost identical <math display="inline">t_i</math>  values.  The computed coefficients <math display="inline">\widehat{c}_j</math> using the normal equations are shown in Table [[#table-1|1]], along with the certified values provided by ''NIST''. The ''NIST'' certified values were found solving the normal equations, but with multiple precision of 500 digits (which represents an idealization of what would be achieved if the calculations were made without rounding error). Our calculated values differ significantly from those of ''NIST'', even in the sign, the relative difference <math display="inline">\Vert \widehat{\mathbf{c}} - \mathbf{c}_{nist} \Vert / \Vert \mathbf{c}_{nist} \Vert </math> is about 118<math display="inline">%</math>. This dramatic difference is mainly because we are using finite arithmetic with 16-digit standard ''IEEE double precision'', and solving the normal equations with the Cholesky factorization yields a relative error amplified proportionately to the product of the condition number times the machine epsilon. The computed residual keeps reasonable, though. Figure [[#img-2|2]] shows ''Filip'' data along with the certified curve and our computed curve. The difference is most visible at the extremes, where our computed curve shows some pronounced oscillations.
+Example 2: The National Institute of Standards and Technology (''NIST'') is a branch of the U.S. Department of Commerce responsible for establishing national and international standards. ''NIST'' maintains reference data sets for use in the calibration and certification of statistical software. On its website <span id='citeF-10'></span>[[#cite-10|[10]]] we can find the ''Filip'' data set, which consists of 82 observations of a variable <math display="inline">y</math> for different <math display="inline">t</math> values. The aim is to model this data set using a 10th-degree polynomial. This is part of exercise 5.10 in Cleve Molers' book <span id='citeF-11'></span>[[#cite-11|[11]]].  For this problem we have <math display="inline">m = 82</math> data points <math display="inline">(t_i, \hat{y}_i)</math>, and we want to compute <math display="inline">n = 11</math> coefficients <math display="inline">c_j</math> for the 10th-degree polynomial. The <math display="inline">m \times n</math> design matrix <math display="inline">A</math> has coefficients <math display="inline">a_{ij} = t_i^{j-1}</math>. In order to given an idea of the complexity of this matrix, we observe that its minimum coefficient is 1 and its maximum coefficient is a bit greater than <math display="inline">2.7\times 10^9</math>, while its condition number is <math display="inline">\kappa (A)\approx \mathcal{O}(10^{15})</math>. The matrix of the normal equations, <math display="inline">A^TA</math>, is a much smaller matrix of size <math display="inline">n\times n</math>, but ''more singular'', since its minimum and maximum coefficients (in absolute value) are close to 82 and <math display="inline">5.1\times 10^{19}</math>, respectively, with a very high condition number <math display="inline">\kappa (A^TA)\approx \mathcal{O}(10^{30})</math>. The matrix of the normal equations is highly ill-conditioned in this case because there are some clusters of data points very close to each other with almost identical <math display="inline">t_i</math>  values.  The computed coefficients <math display="inline">\widehat{c}_j</math> using the normal equations are shown in Table [[#table-1|1]], along with the certified values provided by ''NIST''. The ''NIST'' certified values were found solving the normal equations, but with multiple precision of 500 digits (which represents an idealization of what would be achieved if the calculations were made without rounding error). Our calculated values differ significantly from those of ''NIST'', even in the sign, the relative difference <math display="inline">\Vert \widehat{\mathbf{c}} - \mathbf{c}_{nist} \Vert / \Vert \mathbf{c}_{nist} \Vert </math> is about 118<math display="inline">%</math>. This dramatic difference is mainly because we are using finite arithmetic with 16-digit standard ''IEEE double precision'', and solving the normal equations with the Cholesky factorization yields a relative error amplified proportionately to the product of the condition number times the machine epsilon. The computed residual keeps reasonable, though. Figure [[#img-2|2]] shows ''Filip'' data along with the certified curve and our computed curve. The difference is most visible at the extremes, where our computed curve shows some pronounced oscillations.
@@ Line 842: / Line 842: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_06.png|450px|Computed polynomial curve with QR and SVD.]]
+|[[Image:Review_960218438971-fig_06.png|100px|Computed polynomial curve with QR and SVD.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 6:''' Computed polynomial curve with QR and SVD.
@@ Line 1,012: / Line 1,012: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_08.png|450px|NN solution with the hyperbolic tangent as activation function.]]
+|[[Image:Review_960218438971-fig_08.png|100px|NN solution with the hyperbolic tangent as activation function.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 8:''' NN solution with the hyperbolic tangent as activation function.
@@ Line 1,019: / Line 1,019: @@
 {| class="floating_imageSCP" style="text-align: center; border: 1px solid #BBB; margin: 1em auto; width: 100%;max-width: 100%;"
 |-
-|[[Image:Review_960218438971-fig_09.png|450px|NN solution with the sigmoidal as activation function.]]
+|[[Image:Review_960218438971-fig_09.png|100px|NN solution with the sigmoidal as activation function.]]
 |- style="text-align: center; font-size: 75%;"
 | colspan="1" | '''Figure 9:''' NN solution with the sigmoidal as activation function.
@@ Line 1,026: / Line 1,026: @@
 ==8 Conclusions==
-Fitting a curve to a given set of data is one of the most simple of the so called ''ill-posed'' problems. This is an example of a broad set of problems called least squares problems. This simple problem contains many of the ingredients, both theoretical and computational, of modern challenge and complex problems that are of great importance in computational modelling and applications, specially when computer solutions are obtained using finite precision machines. Commonly there is no `''best computational algorithm''' for general problems, but for a particular problem, like the one considered in this article, we can compare results obtained with different approaches or algorithms.
+Fitting a curve to a given set of data is one of the most simple of the so called `''ill-posed''' problems. This is an example of a broad set of problems called least squares problems. This simple problem contains many of the ingredients, both theoretical and computational, of modern challenge and complex problems that are of great importance in computational modelling and applications, specially when computer solutions are obtained using finite precision machines. Commonly there is no `''best computational algorithm''' for general problems, but for a particular problem, like the one considered in this article, we can compare results obtained with different approaches or algorithms.
 It is clear that the best fit to a 10th-degree polynomial is obtained with the QR algorithm, as it produces the smallest residual when compared to algorithms based on ''the normal equations'' and SVD. It is noteworthy that each method yields entirely different coefficients for this polynomial. Not only the sign of the coefficients but also the scale of the values differ drastically. These results demonstrate that even simple ill-posed problems must be studied and numerically solved with extreme care, employing stable state-of-the-art algorithms and tools that avoid the accumulation of rounding errors due to the finite arithmetic precision of computers.

← Older revision	Revision as of 20:06, 27 December 2025
(No difference)

@@ Line 477: / Line 477: @@
 ===5.1 Symmetrizing===
-The key idea to achieving the SVD of a matrix <math display="inline">A</math> is `''symmetrizing'''. That is, if <math display="inline">A \in \mathbb{R}^{m\times n}</math>, we can consider the symmetric positive semidefinite matrices <math display="inline">A^T A \in \mathbb{R}^{n\times n}</math> and <math display="inline">A\,A^T \in \mathbb{R}^{m\times m}</math>. By the spectral theorem for symmetric matrices, these matrices are diagonalizable.  For instance, if <math display="inline">\lambda _1, \ldots ,\lambda _n</math> are the eigenvalues of <math display="inline">A^TA</math> with orthonormal eigenvectors <math display="inline">\mathbf{v}_1, \mathbf{v}_2, \ldots ,\mathbf{v}_n</math>, then
+The key idea to achieving the SVD of a matrix <math display="inline">A</math> is ''symmetrizing''. That is, if <math display="inline">A \in \mathbb{R}^{m\times n}</math>, we can consider the symmetric positive semidefinite matrices <math display="inline">A^T A \in \mathbb{R}^{n\times n}</math> and <math display="inline">A\,A^T \in \mathbb{R}^{m\times m}</math>. By the spectral theorem for symmetric matrices, these matrices are diagonalizable.  For instance, if <math display="inline">\lambda _1, \ldots ,\lambda _n</math> are the eigenvalues of <math display="inline">A^TA</math> with orthonormal eigenvectors <math display="inline">\mathbf{v}_1, \mathbf{v}_2, \ldots ,\mathbf{v}_n</math>, then
 {| class="formulaSCP" style="width: 100%; text-align: left;"