Sigmoid and Its Gradient

Implement the logistic sigmoid and its gradient from scratch with NumPy, in a way that does not overflow for large-magnitude inputs. The sigmoid squashes any real number into (0, 1) and shows up everywhere in logistic regression and neural nets, so interviewers use it to check that you can write numerically stable vectorized code.

Definition

The sigmoid and its derivative are:

s(x)  = 1 / (1 + exp(-x))
s'(x) = s(x) * (1 - s(x))

The naive form 1 / (1 + exp(-x)) overflows when x is a large negative number, because exp(-x) blows up. The stable trick is to branch on the sign of x:

x >= 0:  s = 1 / (1 + exp(-x))          # exp arg is <= 0
x <  0:  s = exp(x) / (1 + exp(x))      # exp arg is <  0

Either branch only ever feeds a non-positive value to exp, so it stays in [0, 1] and never overflows.

Task

Complete sigmoid_and_grad(x) so it returns a tuple (s, grad) of NumPy arrays with the same shape as x, where s is the numerically stable sigmoid and grad is its elementwise derivative s * (1 - s). Do not call scipy, sklearn, or any built-in sigmoid/expit helper.

Example

x = np.array([-1000.0, -1.0, 0.0, 1.0, 1000.0])
s, grad = sigmoid_and_grad(x)
s      # -> array([0.        , 0.26894142, 0.5       , 0.73105858, 1.        ])
grad   # -> array([0.        , 0.19661193, 0.25      , 0.19661193, 0.        ])

At x = 0 the sigmoid is exactly 0.5 and its slope is maximal at 0.25. At the extremes the output saturates to 0 or 1 with no overflow warning, and the gradient vanishes.

Constraints

• 1 <= n <= 10^6; use vectorized NumPy, not a Python loop.
• x may contain values with magnitude up to 1e3 or more; the code must not raise an overflow warning or return nan/inf.
• Values fit in float64.

Approach

Compute the sigmoid with a sign-dependent formula so exp never sees a positive argument, which is what causes overflow. For x >= 0 use 1 / (1 + exp(-x)); for x < 0 use exp(x) / (1 + exp(x)). Both branches are evaluated with np.where and stitched together elementwise. The derivative is then the closed form s * (1 - s), which reuses the value we just computed.

Solution

import numpy as np

def sigmoid_and_grad(x: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    x = np.asarray(x, dtype=np.float64)
    # Only ever exponentiate a non-positive number: no overflow.
    pos = 1.0 / (1.0 + np.exp(-np.abs(x)))          # stable branch for x >= 0
    neg = np.exp(-np.abs(x)) / (1.0 + np.exp(-np.abs(x)))  # value for x < 0
    s = np.where(x >= 0, pos, neg)
    grad = s * (1.0 - s)
    return s, grad

Here np.abs(x) guarantees the argument to exp is <= 0 in both expressions. For x >= 0, 1/(1+exp(-|x|)) is the direct formula. For x < 0, |x| = -x, so exp(-|x|)/(1+exp(-|x|)) = exp(x)/(1+exp(x)), the stable negative branch. np.where picks the correct one per element.

Walkthrough

On x = [-1000, -1, 0, 1, 1000]:

• np.abs(x) = [1000, 1, 0, 1, 1000], so exp(-np.abs(x)) = [~0, 0.3679, 1, 0.3679, ~0] — all finite, no overflow.
• pos = 1/(1+exp(-|x|)) = [1.0, 0.7311, 0.5, 0.7311, 1.0].
• neg = exp(-|x|)/(1+exp(-|x|)) = [~0, 0.2689, 0.5, 0.2689, ~0].
• x >= 0 is [False, False, True, True, True], so s = [neg, neg, pos, pos, pos] = [0, 0.2689, 0.5, 0.7311, 1.0].
• grad = s*(1-s) = [0, 0.1966, 0.25, 0.1966, 0].

The output saturates cleanly at the extremes and peaks at 0.25 where s = 0.5.

Complexity & notes

• Time O(n), space O(n) — a constant number of elementwise passes over the array, plus temporaries for the two branches.
• The stability trick is the whole point: 1/(1+exp(-x)) with x = -1000 computes exp(1000), which overflows to inf and warns. Folding through np.abs keeps every exp argument in [-inf, 0], where the result is safely in [0, 1].
• Both branches are computed for every element (then masked by np.where); this is the idiomatic vectorized approach and still O(n). If you must avoid the wasted branch, index with boolean masks instead, at the cost of more code.
• Using s * (1 - s) for the gradient reuses the forward value rather than recomputing exponentials, which is exactly how a real autograd/backprop implementation caches the sigmoid activation.