Ensure Strictly Positive Regression Outputs in a Keras ConvNet by Using Softplus in the Final Dense Layer

Ensure strictly positive regression outputs in a Keras Conv1D model: replace the final Dense layer with softplus to prevent negatives and stabilize training.

Predicting strictly positive parameters with a convolutional neural network is a common requirement in scientific and engineering workflows. If the output layer is unconstrained, negative predictions will inevitably slip through, even when targets are positive by design. That’s exactly what happens when the final layer is linear. Fortunately, the fix is minimal and robust.

Problem setup

You have many simulated datasets of real values, each shaped as 200 rows by 2 columns, and you use a ConvNet to map each dataset to three model parameters. All three outputs must be positive, and one of them is typically a small fraction. With the architecture below, that parameter occasionally goes negative.

net = models.Sequential()
net.add(layers.Conv1D(32, 3, activation='relu', input_shape=(200, 2)))
net.add(layers.MaxPooling1D(2))
net.add(layers.Conv1D(64, 3, activation='relu'))
net.add(layers.MaxPooling1D(2))
net.add(layers.Conv1D(128, 3, activation='relu'))
net.add(layers.Flatten())
net.add(layers.Dense(3))

Why negative predictions appear

The last layer is a plain Dense without an activation function. That makes it linear and unbounded over the real line, so it can output any real number. Even if training targets are positive, intermediate predictions can be negative, and there is nothing in the model that enforces non-negativity at inference time.

Fix: enforce positivity with softplus

To guarantee positive outputs, use a positivity-preserving activation at the output. The softplus activation is a standard choice here. It maps any real input to a strictly positive value and handles small magnitudes gracefully.

Replace the output layer with Dense(3, activation='softplus').

Here is the adjusted model with the only change applied to the final layer:

net = models.Sequential()
net.add(layers.Conv1D(32, 3, activation='relu', input_shape=(200, 2)))
net.add(layers.MaxPooling1D(2))
net.add(layers.Conv1D(64, 3, activation='relu'))
net.add(layers.MaxPooling1D(2))
net.add(layers.Conv1D(128, 3, activation='relu'))
net.add(layers.Flatten())
net.add(layers.Dense(3, activation='softplus'))

This change replaces the previous Dense(3) layer; it is not an additional layer on top. With softplus as the output activation, the network’s three predictions are strictly greater than zero, which fits the requirement for positive parameters, including those that are usually small fractions.

Why this matters

Respecting domain constraints at the model level prevents invalid predictions and reduces post-processing hacks. It improves training stability by aligning the hypothesis space with the target manifold and makes evaluation cleaner: you no longer need to clamp outputs or discard negatives after the fact.

Takeaway

If your regression targets must be positive, constrain the output layer accordingly. In this case, using softplus in the final Dense layer ensures all three predicted parameters are positive, including the one that tends to be a small fraction. The change is minimal, the behavior is reliable, and the resulting model respects the physics or business rules baked into your data.