False negative from csv.Sniffer.has_header with only strings

[Midnighter]

Bug report

On the following CSV content, the csv.Sniffer.has_header method returns False although it clearly has a header.

sample,fastq_1,fastq_2
A1-35-8,/mnt/scratch/sarek/data/A1-35-8/A1-35-8_R1.fastq.gz,/mnt/scratch/sarek/data/A1-35-8/A1-35-8_R2.fastq.gz
A2-102-5,/mnt/scratch/sarek/data/A2-102-5/A2-102-5_R1.fastq.gz,/mnt/scratch/sarek/data/A2-102-5/A2-102-5_R2.fastq.gz
A5-35-17,/mnt/scratch/sarek/data/A5-35-17/A5-35-17_R1.fastq.gz,/mnt/scratch/sarek/data/A5-35-17/A5-35-17_R2.fastq.gz
AD1-7a,/mnt/scratch/sarek/data/AD1-7a/AD1-7a_R1.fastq.gz,/mnt/scratch/sarek/data/AD1-7a/AD1-7a_R2.fastq.gz
AD1-83a,/mnt/scratch/sarek/data/AD1-83a/AD1-83a_R1.fastq.gz,/mnt/scratch/sarek/data/AD1-83a/AD1-83a_R2.fastq.gz
AD2-60a,/mnt/scratch/sarek/data/AD2-60a/AD2-60a_R1.fastq.gz,/mnt/scratch/sarek/data/AD2-60a/AD2-60a_R2.fastq.gz
Arg1366,/mnt/scratch/sarek/data/Arg1366/Arg1366_R1.fastq.gz,/mnt/scratch/sarek/data/Arg1366/Arg1366_R2.fastq.gz
Br795,/mnt/scratch/sarek/data/Br795/Br795_R1.fastq.gz,/mnt/scratch/sarek/data/Br795/Br795_R2.fastq.gz
Bt100,/mnt/scratch/sarek/data/Bt100/Bt100_R1.fastq.gz,/mnt/scratch/sarek/data/Bt100/Bt100_R2.fastq.gz

I believe this is due to the following lines in the has_header method.

                if thisType != columnTypes[col]:
                    if columnTypes[col] is None: # add new column type
                        columnTypes[col] = thisType
                    else:
                        # type is inconsistent, remove column from
                        # consideration
                        del columnTypes[col]

When all columns are strings both thisType and columnTypes[col] are integers denoting their length. Since they are of different lengths all the column are removed and columnTypes ends up being an empty dictionary which leads to the false negative down the line.

I believe there needs to be a special case introduced to avoid this when comparing integers rather than types.

Your environment

• CPython versions tested on: Python 3.10.10
• Operating system and architecture: Linux helios 5.17.5-76051705-generic #202204271406~1653440576~20.04~6277a18-Ubuntu SMP PREEMPT Thu Ma x86_64 x86_64 x86_64 GNU/Linux

Linked PRs

• gh-102140: False neg csv header bug fix #102784
• pythongh-102140 : False neg csv header bug fix #102787
• gh-102140: fix false negative in csv.Sniffer.has_header #103341

[Drakariboo]

Hello.
Me and @Vanille-22 are french students in CS and we must contribute to open source project as a university project. We would be very happy to work on this issue.

[Drakariboo]

@Midnighter Hi ! I'm trying to resolve the problem but I don't really understand and I need more explanations about your CSV content. Can I have a representation in the form of a table please ? Like this :

|  Header 1 | Header 2 |
| Data     |  Data    |

I've tried this and I also have False.

import csv

with open("test_header.csv", "r") as file:
    print(csv.Sniffer().has_header(file.read()))

[Midnighter]

Sure, here's the CSV converted to markdown.

sample	fastq_1	fastq_2
A1-35-8	/mnt/scratch/sarek/data/A1-35-8/A1-35-8_R1.fastq.gz	/mnt/scratch/sarek/data/A1-35-8/A1-35-8_R2.fastq.gz
A2-102-5	/mnt/scratch/sarek/data/A2-102-5/A2-102-5_R1.fastq.gz	/mnt/scratch/sarek/data/A2-102-5/A2-102-5_R2.fastq.gz
A5-35-17	/mnt/scratch/sarek/data/A5-35-17/A5-35-17_R1.fastq.gz	/mnt/scratch/sarek/data/A5-35-17/A5-35-17_R2.fastq.gz
AD1-7a	/mnt/scratch/sarek/data/AD1-7a/AD1-7a_R1.fastq.gz	/mnt/scratch/sarek/data/AD1-7a/AD1-7a_R2.fastq.gz
AD1-83a	/mnt/scratch/sarek/data/AD1-83a/AD1-83a_R1.fastq.gz	/mnt/scratch/sarek/data/AD1-83a/AD1-83a_R2.fastq.gz
AD2-60a	/mnt/scratch/sarek/data/AD2-60a/AD2-60a_R1.fastq.gz	/mnt/scratch/sarek/data/AD2-60a/AD2-60a_R2.fastq.gz
Arg1366	/mnt/scratch/sarek/data/Arg1366/Arg1366_R1.fastq.gz	/mnt/scratch/sarek/data/Arg1366/Arg1366_R2.fastq.gz
Br795	/mnt/scratch/sarek/data/Br795/Br795_R1.fastq.gz	/mnt/scratch/sarek/data/Br795/Br795_R2.fastq.gz
Bt100	/mnt/scratch/sarek/data/Bt100/Bt100_R1.fastq.gz	/mnt/scratch/sarek/data/Bt100/Bt100_R2.fastq.gz

And yes, has_header returning false is exactly the problem.

[Drakariboo]

Hello. You have described the problem very well. The thing is, the length of a string has its limits. The function compares the length of the string to the previous one in the same column and if they are different, the column is deleted. As the comment suggests, everything should be of equal length except the first row. We must therefore find a better definition of what is a header. I was thinking about doing a weighted average of the whole lengths and comparing it to the header's length.
However, I'm not sure this is a good idea cause it could bring more problems...

Or, we can also compare the weighted average of the whole lengths to min and max length. (or calculate a similiraty rate)

Also, I think comparing columns between them could be interesting, instead of comparing each element in each column.

[Midnighter]

I've mainly used the has_header method because it exists. Working around this problem, I now load data assuming that a header exists and validate the columns. Most tools I know for parsing CSVs leave it up to the user to really figure this out.

[johnD18]

Hello, me and @Drakariboo improved the heuristic of thehas_header method. It does return True now for the CSV content used above. It works for this particular case, however, it's not doable in a generic way. We added more flexebility to the rules used ( because we wanted to respect the methodology), in order to take into account more informations. I suggest to make this method deprecated for the future versions.

[johnD18]

Hello, we've run the test_csv without our contribution first and it gave 4 errors and 7 fails. Here are the results:

========================================
ERROR: test_read_delimiter (__main__.Test_Csv)
reader = csv.reader(input, **kwargs)
TypeError: "delimiter" must be a 1-character string
========================================
ERROR: test_read_escape (__main__.Test_Csv)
result = list(reader)
_csv.Error: line contains NUL
========================================
ERROR: test_read_nul (__main__.Test_Csv)
result = list(reader)
_csv.Error: line contains NUL
========================================
ERROR: test_read_quoting (__main__.Test_Csv)
reader = csv.reader(input, **kwargs)
TypeError: quotechar must be set if quoting enabled
========================================
FAIL: test_escapechar (__main__.TestDialectValidity)
with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'):
AssertionError: Error not raised
========================================
FAIL: test_quoting (__main__.TestDialectValidity)
self.assertEqual(str(cm.exception),
AssertionError: 'quotechar must be set if quoting enabled' != '"quotechar" must be a 1-character string'
- quotechar must be set if quoting enabled
+ "quotechar" must be a 1-character string
========================================
FAIL: test_dict_reader_fieldnames_accepts_iter (__main__.TestDictFields)
self.assertEqual(reader.fieldnames, fieldnames)
AssertionError: <list_iterator object at 0x7fb9df1d9ea0> != ['a', 'b', 'c']
========================================
FAIL: test_dict_writer_fieldnames_rejects_iter (__main__.TestDictFields)
self.assertEqual(writer.fieldnames, fieldnames)
AssertionError: <list_iterator object at 0x7fb9df1dbf70> != ['a', 'b', 'c']
========================================
FAIL: test_write_field_not_in_field_names_raise (__main__.TestDictFields)
self.assertRaises(ValueError, csv.DictWriter.writerow, writer, dictrow)
AssertionError: ValueError not raised by writerow
========================================
FAIL: test_reader_arg_valid (__main__.Test_Csv)
self.assertRaises(TypeError, ctor, arg,
AssertionError: TypeError not raised by reader
========================================
FAIL: test_writer_arg_valid (__main__.Test_Csv)
self.assertRaises(TypeError, ctor, arg,
AssertionError: TypeError not raised by writer
========================================

[johnD18]

Hello, we're having some trouble with the pull request, we're not sure why it's not working and were hoping for some guidance. Would you be able to help us out?