My loss won't go down from a certain range in my network

[Naiva805]

Outputs.csv
X_scaled.csv
Simple2.pdf

Hi guys.
I have these two datasets. X_scaled.csv is my Input and Outputs.csv is my output(Not scaled).
I have built a neural network that predict each Outputs column. My model has two outputs: first one is for predicting P,I,O,N,A(dense(5, softmax))(I have used softmax since the sum of the vector should be one) then I used this output with the Inputs to get 20 outputs that left in Outputs.csv(Their sum should also be equal to 1 so I have used softmax again)..I have uploaded my jupyternotebook codes ins simple2.pdf

I have used cosin decay restart learning-scheduling and Huber loss but my loss stucked as you can see in these pics:

Also I have calculated the average norm of gradients on each 100 batches as you can see they decay fast Also my prediction especially for 20 subsets are not that good: I don't know what am I doing wrong?? Or I just hit the noise floor on my dataset?? Is it possible to train the network further??

[Kxrma47]

The two softmax outputs make this a compositional prediction problem, and Huber loss is usually a poor match for that geometry. Huber treats each component as an independent regression target even though increasing one component necessarily reduces the others.

If the targets are probability-like distributions, try categorical cross-entropy or KL divergence and compare them with a simple baseline that always predicts the mean training composition. If these are proportions rather than class probabilities, also consider transforming them with a log-ratio method; plain softmax can struggle when many true components are near zero.

The second head also receives the predicted five-component output, so errors from the first head are propagated into it. A useful experiment is to train the second head once with the true five values as input. If performance improves sharply, the bottleneck is error propagation rather than optimizer capacity. You can then train jointly with weighted losses, or gradually replace true intermediate values with predicted ones.

Before extending training, I would run three sanity checks:

1. overfit 16–32 samples; failure means the architecture/loss pipeline has a problem;
2. report each head's loss separately instead of only their sum;
3. compare validation performance with the mean-composition baseline.

Cosine restarts will not help once gradients are tiny because the objective or information has reached a plateau. Those checks will tell you whether it is an optimization problem, error propagation, or simply the dataset's noise floor.