Dense Layer Forward Pass

Implement the forward pass of a fully-connected layer, the fundamental building block of every neural network. Given a batch of inputs X of shape (n, d_in), a weight matrix W of shape (d_in, d_out), and a bias vector b of shape (d_out,), compute the affine transform X @ W + b. Interviewers use this to check that you understand the shapes flowing through a layer and that you reach for a single vectorized matmul instead of looping over samples.

Definition

A dense layer maps each input row x of length d_in to an output row of length d_out:

out[i, j] = sum_k X[i, k] * W[k, j] + b[j]

Stacked over the whole batch, this is one matrix product plus a broadcast bias add:

Y = X @ W + b     # (n, d_in) @ (d_in, d_out) + (d_out,) -> (n, d_out)

Task

Complete dense_forward(X, W, b) so it returns the (n, d_out) output matrix, fully vectorized with NO Python loops. Take the matrix product X @ W, then add the bias b, which NumPy broadcasts across every row. Do not call any deep-learning framework (no PyTorch, TensorFlow, or Keras); use plain NumPy only.

Example

X = np.array([[1.0, 2.0],
              [3.0, 4.0]])
W = np.array([[1.0, 0.0, -1.0],
              [0.0, 1.0,  1.0]])
b = np.array([0.5, -0.5, 1.0])
dense_forward(X, W, b)
# -> array([[1.5, 1.5, 2.0],
#           [3.5, 3.5, 2.0]])

For row 0, [1, 2] @ W = [1, 2, 1], then adding b = [0.5, -0.5, 1.0] gives [1.5, 1.5, 2.0].

Constraints

• 1 <= n <= 10^4, 1 <= d_in, d_out <= 10^3; use vectorized NumPy, no Python loop over samples.
• X.shape[1] equals W.shape[0] (both d_in), and b.shape[0] equals W.shape[1] (d_out).
• Values fit in float64; the output is a dense (n, d_out) float array.

Approach

A dense layer is an affine transform: multiply the input batch by the weight matrix, then add the bias. The matmul X @ W contracts the shared d_in axis so each of the n rows becomes a d_out-length vector, and adding the (d_out,) bias broadcasts it across all rows. One matrix product plus one broadcast add does the whole batch at once, no loops.

Solution

import numpy as np

def dense_forward(X: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    X = np.asarray(X, dtype=np.float64)
    W = np.asarray(W, dtype=np.float64)
    b = np.asarray(b, dtype=np.float64)

    return X @ W + b        # (n, d_in) @ (d_in, d_out) + (d_out,) -> (n, d_out)

Walkthrough

On the example with X = [[1, 2], [3, 4]], W = [[1, 0, -1], [0, 1, 1]], b = [0.5, -0.5, 1.0]:

• X @ W contracts the length-2 d_in axis. Row 0: [1*1 + 2*0, 1*0 + 2*1, 1*(-1) + 2*1] = [1, 2, 1]. Row 1: [3*1 + 4*0, 3*0 + 4*1, 3*(-1) + 4*1] = [3, 4, 1]. The product is [[1, 2, 1], [3, 4, 1]], shape (2, 3).
• Adding b = [0.5, -0.5, 1.0] broadcasts the bias down every row. Row 0 becomes [1.5, 1.5, 2.0]; row 1 becomes [3.5, 3.5, 2.0].
• Result matches the expected [[1.5, 1.5, 2.0], [3.5, 3.5, 2.0]].

Complexity & notes

• Time O(n * d_in * d_out) for the matmul, which BLAS runs in optimized C; the bias add is O(n * d_out) and negligible. Space O(n * d_out) for the output.
• The bias broadcast is the one subtlety: b has shape (d_out,), and NumPy aligns it to the trailing axis of the (n, d_out) product, so it is added to every row without any manual tiling or reshape.
• Shape discipline matters in interviews: the inner dimensions must agree (X is (n, d_in), W is (d_in, d_out)), or the matmul raises a ValueError. If you were handed W transposed as (d_out, d_in), you would write X @ W.T instead.
• Casting to float64 keeps integer inputs from truncating and guards X @ W against overflow on large integer batches.
• This is only the affine part of a layer; a real network follows it with a nonlinearity such as relu or softmax. Keeping the linear step separate is exactly what frameworks do, since the affine output is what backprop needs cached for the weight gradient.