There are different ways of teaching non-networks. Most often, networks are trained on a fixed dataset that is predefined, for example, in problems of text or image classification.
Some networks can learn from a dynamically generated set of training examples that are updated depending on the current state of the network, for example, networks are trained to play games (for example, Atari games). The network is constantly improving its strategy by playing the same game, and new tutorials are generated as you progress through it.
And there are networks that are trained on the outputs of other networks, and thus can continuously train each other. For example, some image generation methods work this way. Also the well-known algorithm AlphaGo Zero is trained according to this scheme.
But all these different methods have something in common.
The loss function is a function that allows the network to calculate how well or poorly it responds to a specific training example. This function is used to update the model weights at each step. The weights are updated in such a way as to reduce the loss function.
The learning process - the way the network will receive new data for training is fixed and determined by the initial parameters and state of the network.
When the moment comes when the loss function reaches its minimum, convergence occurs, the network stops learning and changing its state, the set of training examples also stops changing significantly. Then we can assume that the network has been trained and, for the given initial parameters, it is no longer possible to train it better.
It also happens that the loss function continues to fall on the training set, but on the test set it has reached a minimum (overfitting effect). Then they also think that the network is trained and it is necessary to stop training, but this indicates that the architecture of the network is not suitable for the task, or insufficient regularization is chosen.
In order to answer this question, we must first discuss the problem of assessing the quality of the neural network. Depending on the task, we can evaluate the accuracy of the result in different ways. In addition, the accuracy requirements can also vary. In practice, different quantitative estimates are usually used to select the best network. During the learning process, these assessments are constantly monitored, and a conclusion is made about whether it is worth continuing learning. Learning can be stopped for two reasons. Either the network has trained to acceptable indicators and the quality of its work suits us, or its indicators stop improving. In the first case, we consider the network trained. In the second case, you need to think about changing the parameters of the network or the learning process.