লজিস্টিক ফাংশনের হেসিয়ান

আমি উদ্দেশ্য ফাংশন, এর চট আহরণ করা অসুবিধা আছে , লজিস্টিক প্রত্যাবৃত্তি যেখানে হল: $l(\theta)$ $l(\theta)$

l (θ) = \sum_{i = 1}^{m} [y_{i} \log (h_{θ} (x_{i})) + (1 - y_{i}) \log (1 - h_{θ} (x_{i}))]

$l(\theta)=\sum_{i=1}^{m} \left[y_{i} \log(h_\theta(x_{i})) + (1- y_{i}) \log (1 - h_\theta(x_{i}))\right]$

$h_\theta(x)$ একটি লজিস্টিক ফাংশন। হেসিয়ানটি $X^T D X$ । আমি $\frac{\partial^2 l(\theta)}{\partial \theta_i \partial \theta_j}$ এটি , তবে তারপরে থেকে ম্যাট্রিক্স নোটেশনটি কীভাবে পাওয়া যায় তা আমার কাছে স্পষ্ট ছিল না $\frac{\partial^2 l(\theta)}{\partial \theta_i \partial \theta_j}$ ।

কোনও পরিষ্কার এবং সহজ উপায় কি কেউ জানেন ? $X^T D X$

logistic

— DSKim
সূত্র

আপনি কী ?

\frac{\partial^{2} l}{\partial θ_{i} \partial θ_{j}}

$\frac{\partial^2 l}{\partial \theta_i \partial \theta_j}$

— গ্লেন_বি -রিনস্টেট মনিকা

এখানে স্লাইডগুলির একটি ভাল সেট যা আপনি খুঁজছেন ঠিক সেই হিসাবটি দেখায়: সাইট. stat.psu.edu/~jiali/course/stat597e/notes2/logit.pdf

আমি একটি দুর্দান্ত ভিডিও পেয়েছি যা হেসিয়ানকে ধাপে ধাপে গণনা করে। লজিস্টিক রিগ্রেশন (বাইনারি) - হেসিয়ান গণনা

— নাওমি

$m$ $\{x_i,y_i\}$ $x_i\in\mathbb{R}^d$ $y_i\in\mathbb{R}$ $h_\theta$

h_{θ} (x_{i}) = σ (ω^{T} x_{i}) = σ (z_{i}) = \frac{1}{1 + e^{- z_{i}}},

$h_\theta(x_i)=\sigma(\omega^Tx_i)=\sigma(z_i)=\frac{1}{1+e^{-z_i}},$

$\omega\in\mathbb{R}^d$ $z_i=\omega^Tx_i$

l (ω) = \sum_{i = 1}^{m} - (y_{i} \log σ (z_{i}) + (1 - y_{i}) \log (1 - σ (z_{i})))

$l(\omega)=\sum_{i=1}^m -\Big( y_i\log\sigma(z_i)+(1-y_i)\log(1-\sigma(z_i))\Big)$

$1-\sigma(z)=1-1/(1+e^{-z})=e^{-z}/(1+e^{-z})=1/(1+e^z)=\sigma(-z)$

এটিও নোট করুন

\begin{aligned} \frac{\partial}{\partial z} σ (z) = \frac{\partial}{\partial z} (1 + e^{- z})^{- 1} = e^{- z} (1 + e^{- z})^{- 2} & = \frac{1}{1 + e^{- z}} \frac{e^{- z}}{1 + e^{- z}} = σ (z) (1 - σ (z)) \end{aligned}

$\begin{equation} \begin{aligned} \frac{\partial}{\partial z}\sigma(z)=\frac{\partial}{\partial z}(1+e^{-z})^{-1}=e^{-z}(1+e^{-z})^{-2}&=\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}} =\sigma(z)(1-\sigma(z)) \end{aligned} \end{equation}$

$l(\omega)$ $\vec{\nabla}^2l(\omega)$ $\frac{\partial z}{\partial \omega} = \frac{x^T\omega}{\partial \omega}=x^T$ $\frac{\partial z}{\partial \omega^T}=\frac{\partial \omega^Tx}{\partial \omega ^T} = x$

$l_i(\omega)=-y_i\log\sigma(z_i)-(1-y_i)\log(1-\sigma(z_i))$

\begin{aligned} \frac{\partial \log σ (z_{i})}{\partial ω^{T}} & = \frac{1}{σ (z_{i})} \frac{\partial σ (z_{i})}{\partial ω^{T}} = \frac{1}{σ (z_{i})} \frac{\partial σ (z_{i})}{\partial z_{i}} \frac{\partial z_{i}}{\partial ω^{T}} = (1 - σ (z_{i})) x_{i} \\ \frac{\partial \log (1 - σ (z_{i}))}{\partial ω^{T}} & = \frac{1}{1 - σ (z_{i})} \frac{\partial (1 - σ (z_{i}))}{\partial ω^{T}} = - σ (z_{i}) x_{i} \end{aligned}

$\begin{equation} \begin{aligned} \frac{\partial \log\sigma(z_i)}{\partial \omega^T} &= \frac{1}{\sigma(z_i)}\frac{\partial\sigma(z_i)}{\partial \omega^T} = \frac{1}{\sigma(z_i)}\frac{\partial\sigma(z_i)}{\partial z_i}\frac{\partial z_i}{\partial \omega^T}=(1-\sigma(z_i))x_i\\ \frac{\partial \log(1-\sigma(z_i))}{\partial \omega^T}&= \frac{1}{1-\sigma(z_i)}\frac{\partial(1-\sigma(z_i))}{\partial \omega^T} =-\sigma(z_i)x_i \end{aligned} \end{equation}$

এটা এখন তুচ্ছ দেখানোর জন্য

\vec{\nabla} l_{i} (ω) = \frac{\partial l_{i} (ω)}{\partial ω^{T}} = - y_{i} x_{i} (1 - σ (z_{i})) + (1 - y_{i}) x_{i} σ (z_{i}) = x_{i} (σ (z_{i}) - y_{i})

$\vec{\nabla}l_i(\omega)=\frac{\partial l_i(\omega)}{\partial \omega^T} =-y_ix_i(1-\sigma(z_i))+(1-y_i)x_i\sigma(z_i)=x_i(\sigma(z_i)-y_i)$

হালকা

আমাদের শেষ পদক্ষেপ হেসিয়ান গণনা করা

{\vec{\nabla}}^{2} l_{i} (ω) = \frac{\partial l_{i} (ω)}{\partial ω \partial ω^{T}} = x_{i} x_{i}^{T} σ (z_{i}) (1 - σ (z_{i}))

$\vec{\nabla}^2l_i(\omega)=\frac{\partial l_i(\omega)}{\partial \omega\partial \omega^T}=x_ix_i^T\sigma(z_i)(1-\sigma(z_i))$

$m$ $\vec{\nabla}^2l(\omega)=\sum_{i=1}^m x_ix_i^T\sigma(z_i)(1-\sigma(z_i))$ . This is equivalent to concatenating column vectors $x_i\in\mathbb{R}^d$ into a matrix $X$ of size $d\times m$ such that $\sum_{i=1}^m x_ix_i^T=XX^T$ . The scalar terms are combined in a diagonal matrix $D$ such that $D_{ii}=\sigma(z_i)(1-\sigma(z_i))$ . Finally, we conclude that

\vec{H} (ω) = {\vec{\nabla}}^{2} l (ω) = X D X^{T}

$\vec{H}(\omega)=\vec{\nabla}^2l(\omega)=XDX^T$

A faster approach can be derived by considering all samples at once from the beginning and instead work with matrix derivatives. As an extra note, with this formulation it's trivial to show that $l(\omega)$ is convex. Let $\delta$ be any vector such that $\delta\in\mathbb{R}^d$ . Then

δ^{T} \vec{H} (ω) δ = δ^{T} {\vec{\nabla}}^{2} l (ω) δ = δ^{T} X D X^{T} δ = δ^{T} X D (δ^{T} X)^{T} = ‖ δ^{T} D X ‖^{2} \geq 0

$\delta^T\vec{H}(\omega)\delta = \delta^T\vec{\nabla}^2l(\omega)\delta = \delta^TXDX^T\delta = \delta^TXD(\delta^TX)^T = \|\delta^TDX\|^2\geq 0$

since $D>0$ and $\|\delta^TX\|\geq 0$ . This implies $H$ is positive-semidefinite and therefore $l$ is convex (but not strongly convex).

— Manuel Morales
সূত্র

In the last equation, shouldn't it be

| | δ D^{1 / 2} X | |

$||\delta D^{1/2}X||$ since

X D X^{⊤}

$XDX^\top$ =

X D^{1 / 2} (X D^{1 / 2})^{⊤}

$XD^{1/2}(XD^{1/2})^\top$ ?

— appletree

Shouldn't it be

X^{T} D X

$X^T D X$ ?

— Chintan Shah