2025, Dec 25 07:00
Fixing stalled LSTM training on sine-wave forecasting: use target history, seq2seq windows, and amplitude normalization
Learn why LSTM models stall on sine-wave forecasting and fix flat loss fast: feed target history, switch to seq2seq windows, and normalize amplitudes.
Why your LSTM stalls on a simple sine wave — and how to fix it
Training a neural network to predict a sine wave should be straightforward, yet it often stalls with flat loss when the input pipeline is wrong. The common pitfall is feeding the model a linearly increasing time index and asking it to predict future sin(x), while the model actually needs recent values of the target signal itself. There’s a second issue as well: large, unnormalized amplitudes make optimization harder. Let’s walk through a minimal failing setup, then switch it to a proper sequence-to-sequence forecasting pipeline that learns quickly.
Problem setup (what makes the model fail)
The input is a linear ramp x and the target is a scaled sine wave y. The dataset windows are built by combining x and y but the model effectively sees time steps, not the recent y values. With a large amplitude, the network struggles even more, and training does not improve loss across epochs.
import tensorflow as tf
import numpy as np
# Constants
HIST_STEPS = 100
PRED_STEPS = 100
# Synthetic series
x_vals = (np.arange(0, 2000, 0.5)).reshape(-1, 1)
y_vals = 20 * np.sin(x_vals)
# Window builder (inputs are time values, not the target history)
def assemble_windows(x_src, y_src, shuffle=False):
x_tensor = tf.convert_to_tensor(x_src, dtype=tf.float32)
y_tensor = tf.convert_to_tensor(y_src, dtype=tf.float32)
combo = tf.concat([x_tensor, y_tensor], axis=-1)
ds = tf.keras.preprocessing.timeseries_dataset_from_array(
data=combo,
targets=None,
sequence_length=HIST_STEPS + PRED_STEPS,
sequence_stride=1,
shuffle=shuffle,
batch_size=1,
)
def cut_xy(seq):
past = seq[:, :HIST_STEPS, : x_tensor.shape[-1]]
future = seq[:, HIST_STEPS:, x_tensor.shape[-1] :]
return past, future
return ds.map(cut_xy)
# Model (tries to predict multi-step future)
def make_seq_model(n_features, horizon, n_targets=1):
net = tf.keras.Sequential([
tf.keras.layers.Input(shape=(HIST_STEPS, n_features)),
tf.keras.layers.LSTM(128),
tf.keras.layers.Dense(horizon * n_targets),
tf.keras.layers.Reshape((horizon, n_targets)),
])
net.compile(loss="mse", optimizer=tf.keras.optimizers.RMSprop(0.001), metrics=["mse"])
return net
What’s actually wrong
The model is asked to forecast sin(x) using a sliding window of x (a monotonic ramp), not y. That strips the network of the very information it needs: the recent dynamics of the target signal. On top of that, the output’s amplitude is large, while models train more reliably on normalized targets. For time series forecasting, the correct input is the history of the variable you want to predict. In this case you should feed recent values of y and predict its future values. Reducing the amplitude to a bounded range, such as [-1, 1], further stabilizes learning and loss.
The fix: use target history as input and keep amplitudes bounded
Below is a generator that yields windows of y for inputs and y for future targets, a thin tf.data wrapper, a quick plotting helper, and a compact LSTM head. This converts the task into a standard sequence-to-sequence forecast. You can later rescale the predictions if you need a larger amplitude.
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Time series generator based on y history
def wave_stream(hist_len: int,
fut_len: int,
n_samples: int = 1000,
x_start: float = 0.0,
x_stop: float = 2000.0,
amp: float = 20.0,
dtype: str = 'float32'):
"""
Emits pairs (past_y, future_y) from y = amp * sin(x).
"""
for _ in range(n_samples):
begin = np.random.randint(low=x_start, high=x_stop - hist_len)
x_seq = np.arange(begin, begin + hist_len + fut_len, dtype=dtype)
y_seq = (amp * np.sin(x_seq)).astype(dtype)
yield y_seq[:hist_len], y_seq[hist_len:]
# Wrap generator as a tf.data.Dataset
def build_tf_stream(gen_fn, hist_len: int, fut_len: int, batch: int):
ds = tf.data.Dataset.from_generator(
gen_fn,
output_signature=(
tf.TensorSpec(shape=(hist_len,), dtype=tf.float32),
tf.TensorSpec(shape=(fut_len,), dtype=tf.float32),
),
)
return ds.batch(batch)
# Visual sanity check
def show_series_samples(tf_ds, max_examples=3):
for i, (past, fut) in enumerate(tf_ds.unbatch().take(max_examples)):
plt.figure(figsize=(10, 4))
n_hist = past.shape[0]
n_fut = fut.shape[0]
plt.plot(range(n_hist), past.numpy(), label="Input (Past)", color='blue')
plt.plot(range(n_hist, n_hist + n_fut), fut.numpy(), label="Target (Future)", color='orange')
plt.axvline(x=n_hist - 1, color="gray", linestyle="--")
plt.title(f"Sample {i+1}: Forecasting {n_fut} steps from {n_hist} past values")
plt.xlabel("Timestep")
plt.ylabel("Value")
plt.grid(True)
plt.legend()
plt.tight_layout()
plt.show()
# Minimal LSTM forecaster
def build_lstm_head(inp_shape, out_dim):
model = tf.keras.Sequential([
tf.keras.layers.LSTM(128, activation='tanh', input_shape=inp_shape),
tf.keras.layers.Dense(out_dim),
])
model.compile(optimizer='adam', loss='mse')
return model
# Example usage
if __name__ == "__main__":
batch_sz = 32
cfg = {
'hist_len': 100,
'fut_len': 100,
'n_samples': 1000,
'x_start': 0.0,
'x_stop': 2000.0,
'amp': 1.0,
'dtype': 'float32',
}
ds = build_tf_stream(
lambda: wave_stream(**cfg),
hist_len=cfg['hist_len'],
fut_len=cfg['fut_len'],
batch=batch_sz,
)
# Optional: visualize a few windows
show_series_samples(ds, max_examples=3)
net = build_lstm_head(inp_shape=(cfg['hist_len'], 1), out_dim=cfg['fut_len'])
net.summary()
net.fit(ds, epochs=10)
A training run on this setup converges smoothly when the amplitude is normalized. You can multiply predictions by the desired amplitude in post-processing if needed. An example training trace for the LSTM is shown below:
Model: "sequential"
┌─────────────────────────────────┬────────────────────────┬───────────────┐
│ Layer (type) │ Output Shape │ Param # │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ lstm (LSTM) │ (None, 128) │ 66,560 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 12,900 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 79,460 (310.39 KB)
Trainable params: 79,460 (310.39 KB)
Non-trainable params: 0 (0.00 B)
Epoch 1/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 3s 45ms/step - loss: 0.4757
Epoch 2/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 44ms/step - loss: 0.1000
Epoch 3/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 43ms/step - loss: 0.0011
Epoch 4/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 43ms/step - loss: 1.5433e-04
Epoch 5/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 45ms/step - loss: 8.2073e-05
Epoch 6/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 2s 47ms/step - loss: 6.9301e-05
Epoch 7/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 2s 47ms/step - loss: 6.1493e-05
Epoch 8/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 45ms/step - loss: 5.0608e-05
Epoch 9/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 2s 51ms/step - loss: 4.3731e-05
Epoch 10/10
32/32 ━━━━━━━━━━━━━━━━━━━━ 2s 48ms/step - loss: 3.9795e-05
Why this matters
Feeding a network the wrong modality forces it to learn what it cannot infer. For periodic signals, the structure the model needs is in the recent values of the target series, not in the raw time index. Casting the task as sequence-to-sequence forecasting and keeping amplitudes bounded unlocks the inductive bias of recurrent models like LSTM and makes optimization stable.
Takeaways
If the loss won’t budge on a seemingly trivial signal, verify that your inputs are the past values of the variable you want to predict, not unrelated covariates like a monotonically increasing index. Keep the target amplitude within a compact range such as [-1, 1]; rescale outputs afterward if necessary. With those two adjustments, even a small LSTM head fits a multi-step sine forecast quickly and reliably.