Major performance differences when using the tokenizer in huggingface - huggingface-transformers

Question

I'm using the dataset
from transformers import AutoTokenizer, AutoModelForTokenClassification, set_seed, AutoModelForMultipleChoice, AutoModelForMaskedLM, BertForSequenceClassification, AutoTokenizer, AutoModel
from datasets import load_dataset
mrpc = load_dataset('glue', 'mrpc')
mrpc_test = mrpc['test']
and the following tokenizer and model
baseline_model_name = 'Intel/bert-base-uncased-mrpc'
baseline_model = BertForSequenceClassification.from_pretrained(baseline_model_name).to(device)
baseline_tokenizer = AutoTokenizer.from_pretrained(baseline_model_name, truncation=True)
The issue is that I get one performance using
successes = 0
batch_size = 15
for sample in range(0,len(mrpc_test), batch_size):
tok = baseline_tokenizer(mrpc_test[sample:sample+batch_size]['sentence1'], mrpc_test[sample:sample+batch_size]['sentence2'], return_tensors="pt", padding=True)
logits = baseline_model(**tok.to(device)).logits
results_softmax = torch.softmax(logits, dim=1)
prediction = torch.argmax(results_softmax, dim=1)
labels = mrpc_test[sample:sample+batch_size]['label']
successes += torch.sum(torch.tensor(labels).to(device)==prediction)
print(f'final accuracy: {successes/len(mrpc_test)}')
>>>final accuracy: 0.8307245969772339
But different performance if I run
successes = 0
batch_size = 15
for sample in range(0,len(mrpc_test), batch_size):
sentences = [f'{s1} [SEP] {s2}' for s1, s2 in zip(mrpc_test[sample:sample+batch_size]['sentence1'],mrpc_test[sample:sample+batch_size]['sentence2'])]
tok = baseline_tokenizer(sentences, return_tensors="pt", padding=True)
logits = baseline_model(**tok.to(device)).logits
results_softmax = torch.softmax(logits, dim=1)
prediction = torch.argmax(results_softmax, dim=1)
labels = mrpc_test[sample:sample+batch_size]['label']
successes += torch.sum(torch.tensor(labels).to(device)==prediction)
print(f'final accuracy: {successes/len(mrpc_test)}')
>>>final accuracy: 0.49159419536590576
Note that the only difference is not sending the 2 sentences together to the tokenizer, and instead doing
sentences = [f'{s1} [SEP] {s2}' for s1, s2 in zip(mrpc_test[sample:sample+batch_size]['sentence1'],mrpc_test[sample:sample+batch_size]['sentence2'])]