How import large CSV file in Laravel?

Large files can be read fairly easily with PHP the problem is when you insert the data into a DB, even worse if you are first verifying if the record already exists. In that position, say you’re working with a 100 lined CSV file your end up doing 200 queries. You can’t prevent needing to do the queries but doing lots in one go will use up your application’s memory and will timeout when taking too long.

It’s easier to break up a large file into smaller files and process them in batches.

Let’s go through the steps achieved.

Let’s create two routes one for loading an import view and another to process the import CSV.

Now create a ContactsController.

In this ContactsController create an import method to load a import view.

Create a view called import.blade.php. This view will have a form to upload CSV file.

Next, in the ContactsController create a method called parseImport and Place validation rules to ensures only a CSV file will be accepted.

After the validation store a path of the file.

Read the file contents into an array.

Remove the first line as this is a header line containing the column titles.

Now using array_chunk spit the CSV file into parts every 1000 lines the loop over each part and create a new CSV file containing the part we’ve just extracted.

NOTE – for this work you will need to create a folder inside the resources folder called pendingcontacts. This is where the new CSV files will be stored before they are processed.

The full method looks like this:

At this point, a CSV file has been broken into smaller CSV file. In order to import the contents of these files, a table is needed.

Make a new migration:

In this migration, to keep it simple I’m going to have an id, email and timestamp columns.

Next, run your migrations

Now create a model:

I’ll make email fillable in the model:

Now we have a table to store the CSV records in, let’s make a command that Artisan can use to import the records.

It will create new file app/Console/Commands/ImportContacts.php. Here add artisan command signature and description.

This opens the files inside the pendingcontacts folder we use array_slice(globa($path),0,2) to only open 2 files at a time. Then extract a line into an array then using Laravel’s updateOrCreate method to update a record if the email matches otherwise import a new contact.

Now in order to run this command, we need to tell Artisan where to find it. We do this by registering it in AppServiceProvider.php.

Inside a boot method:

It would be useful to inform the user how many contacts are left to be imported. This can be done in the import method of the controller:

Pass $toImport to the view that in the view show the number as long as there is at least 1:

Leave a Reply

Your email address will not be published. Required fields are marked *