Error tokenizing data. c error: buffer overflow caught - possible malformed input file.

Question

Problem :

I wanted to read all the csv files from my folder and after that concatenate them to create a big csv file. The structure of all my files was the same, After concatenating I want to save it and read it again. All this was done by me using Pandas. But I am facing the Error while reading.

Below is my csv reader code :-

import csv

with open("C:\mywork\DATA\Raw_data\\store.csv",'rb') as csv:

csvreader = csv. csvreader (csv)

l = list(csvreader)

My Error:-

Error                                     Traceback (most recent call last)

<ipython-input-36-9249469f31a6> in <module>()

1 with open('C:\ mywork \DATA\Raw_data\\store.csv', 'rb') as csv:

2     csvreader = csv. csvreader (csv)

----> 3     l = list(csvreader)

Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

Answer 1

I also faced this error in recent past, the root cause I found was that there were some carriage returns "\r" in my data that pandas was using as the line terminator similar to "\n". I guess I want to share it here as that may be a common reason behind this error

The solution which I found was to add the lineterminator='\n' into my read_csv function as follows:

df_clean = pd.read_csv('test_myerror.csv',lineterminator='\n')