How to configure early stopping?

[strubell]

The readme says to point to the train/best directory for decoding a trained model, but the code doesn't seem to be saving any models to that directory. It is saving models to the train directory, which I can successfully evaluate. How can I configure training to save the best model during training?

Here is the command I am running to train:

python main.py train \
    --data_path data2/ \
    --model_dir train \
    --model_name deepatt \
    --vocab_path data/word_dict data/label_dict \
    --emb_path data/glove.6B.100d.txt \
    --model_params=feature_size=100,hidden_size=200,filter_size=800,residual_dropout=0.2,num_hidden_layers=10,attention_dropout=0.1,relu_dropout=0.1 \
    --training_params=batch_size=4096,eval_batch_size=1024,optimizer=Adadelta,initializer=orthogonal,use_global_initializer=false,initializer_gain=1.0,train_steps=600000,learning_rate_decay=piecewise_constant,learning_rate_values=[1.0,0.5,0.25],learning_rate_boundaries=[400000,500000],device_list=[0],clip_grad_norm=1.0 \
    --validation_params=script=run.sh

[Playinf]

The validation is done by run.sh which is evaluated every 300 seconds by default. Make sure run.sh can work properly.

[strubell]

Oops, yes, that makes sense. Is there any specific form that run.sh needs to take in order for the best model to be saved (like writing the score somewhere)? If you could provide an example that would be ideal.

[Playinf]

Normally, there would be a file named log which records the F1 score of each validation. For example:

model.ckpt-2246: 24.330000
model.ckpt-4470: 49.570000
model.ckpt-6699: 59.640000
model.ckpt-8919: 65.980000

If this file is empty, it may indicate that the run.sh have failed to execute. For example, if you run out of GPU memory, then run.sh may fail.