BioHackathon_Lipids-tapt_ulmfit-LR_5e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0413
• Accuracy: 0.7657

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.1168	1.0	29	1.0696	0.7637
1.1267	2.0	58	1.0728	0.7618
1.1216	3.0	87	1.0138	0.7740
1.1144	4.0	116	1.0432	0.7676
1.1034	5.0	145	1.0598	0.7629
1.0808	6.0	174	1.0371	0.7715
1.0731	7.0	203	1.0582	0.7674
1.0745	8.0	232	1.0189	0.7685
1.086	9.0	261	1.0626	0.7630
1.0495	10.0	290	1.0152	0.7739
1.0604	11.0	319	1.0594	0.7627
1.0609	12.0	348	1.0114	0.7738
1.0446	13.0	377	1.0357	0.7673
1.0477	14.0	406	1.0101	0.7749
1.0164	15.0	435	1.0365	0.7687
1.0375	16.0	464	1.0249	0.7691
1.0229	17.0	493	1.0109	0.7724
1.0325	18.0	522	1.0390	0.7727
1.0029	19.0	551	1.0173	0.7720
1.0094	20.0	580	0.9962	0.7727
1.0047	21.0	609	1.0157	0.7687
1.0008	22.0	638	1.0162	0.7701
0.9801	23.0	667	1.0685	0.7645
0.9687	24.0	696	1.0477	0.7719
0.9864	25.0	725	1.0134	0.7722
0.983	26.0	754	1.0016	0.7719
0.971	27.0	783	1.0311	0.7670
0.9575	28.0	812	1.0171	0.7667
0.9556	29.0	841	1.0105	0.7734
0.9463	30.0	870	1.0300	0.7706
0.9483	31.0	899	1.0166	0.7667
0.9209	32.0	928	1.0287	0.7652
0.9239	33.0	957	1.0556	0.7651
0.9178	34.0	986	1.0388	0.7678
0.913	35.0	1015	1.0030	0.7709
0.897	36.0	1044	1.0188	0.7713
0.9113	37.0	1073	1.0326	0.7673
0.9096	38.0	1102	1.0181	0.7685
0.8987	39.0	1131	1.0742	0.7594
0.9019	40.0	1160	1.0270	0.7715
0.8975	41.0	1189	1.0298	0.7693
0.872	42.0	1218	1.0666	0.7607
0.8802	43.0	1247	1.0537	0.7634
0.8842	44.0	1276	1.0966	0.7543
0.8658	45.0	1305	1.0463	0.7688
0.8611	46.0	1334	1.0673	0.7682
0.8576	47.0	1363	1.0586	0.7664
0.8821	48.0	1392	1.0404	0.7639
0.8773	49.0	1421	1.0405	0.7642
0.8557	50.0	1450	1.0633	0.7666
0.8532	51.0	1479	1.0351	0.7693
0.8542	52.0	1508	1.0515	0.7633
0.8595	53.0	1537	1.0418	0.7664
0.8479	54.0	1566	1.0627	0.7634
0.8381	55.0	1595	1.0479	0.7653
0.8536	56.0	1624	1.0607	0.7653
0.8346	57.0	1653	1.0422	0.7696
0.8343	58.0	1682	1.0693	0.7621
0.8371	59.0	1711	1.0413	0.7685
0.8269	60.0	1740	1.0188	0.7712
0.8323	61.0	1769	1.0435	0.7637
0.8253	62.0	1798	1.0954	0.7613
0.8276	63.0	1827	1.0548	0.7607
0.8382	64.0	1856	1.0348	0.7705
0.8254	65.0	1885	1.0556	0.7691
0.8223	66.0	1914	1.0524	0.7624
0.8255	67.0	1943	1.0091	0.7719
0.8198	68.0	1972	1.0421	0.7676
0.8226	69.0	2001	1.0730	0.7594
0.8167	70.0	2030	1.0402	0.7645
0.8225	71.0	2059	1.0688	0.7610
0.8289	72.0	2088	1.0325	0.7674
0.8378	73.0	2117	1.0354	0.7682
0.8096	74.0	2146	1.0741	0.7669
0.8064	75.0	2175	1.0623	0.7674
0.8171	76.0	2204	1.0758	0.7679
0.8212	77.0	2233	1.0459	0.7739
0.804	78.0	2262	1.0472	0.7686
0.806	79.0	2291	1.0378	0.7699
0.8371	80.0	2320	1.0089	0.7723
0.8114	81.0	2349	1.1131	0.7638
0.8229	82.0	2378	1.0552	0.7644
0.8053	83.0	2407	1.0513	0.7613
0.8025	84.0	2436	1.0464	0.7631
0.8148	85.0	2465	1.0840	0.7611
0.8136	86.0	2494	1.0513	0.7649
0.8124	87.0	2523	1.0612	0.7620
0.803	88.0	2552	1.0832	0.7598
0.791	89.0	2581	1.0610	0.7654
0.7982	90.0	2610	1.0700	0.7639
0.8032	91.0	2639	1.0676	0.7669
0.8059	92.0	2668	1.0803	0.7589
0.7865	93.0	2697	1.0448	0.7699
0.8103	94.0	2726	1.0461	0.7689
0.7937	95.0	2755	1.0669	0.7638
0.8027	96.0	2784	1.0995	0.7573
0.8096	96.5614	2800	1.0413	0.7657

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0