This website uses cookies to ensure you get the best possible experience. See our Cookies Policy.

PMG Digital Made for Humans

Autosplit Large CSV Files into Smaller Pieces

2 MINUTE READ | April 22, 2016

Autosplit Large CSV Files into Smaller Pieces

With the recent news that Windows 10 will introduce a UNIX shell, users will have access to a command-line environment on Linux, Mac, and Windows for the first time. And that means you can finally stop bugging the developers at your company to break up the enormous files that Microsoft Excel refuses to open. You don’t need to know anything about the Terminal to do this— although, if you’d like to learn more, check out these tutorials:

Okay, here’s how it works (for this article, we’ll be working on a Mac, but you can use this method elsewhere).

  1. Open Terminal (Applications/Utilities/Terminal)

    Screen Shot 2016-04-18 at 12.38.41 PM
  2. Create a new folder on your desktop. This is where you’ll save the file that needs to be split. My folder is called ‘split’ and I have a CSV file that contains a few copies of the English dictionary, or about 2.36 million rows. 

    Screen Shot 2016-04-18 at 12.53.02 PM
  3. In Terminal, navigate to the folder you just created using the ‘cd’ command, which stands for ‘change directory.’ 

    Screen Shot 2016-04-18 at 12.55.32 PM
  4. Now, you’ll use the ‘split’ command to break the original file into smaller files. To do this, you’ll type the following, where 250000 is the incremental number of rows at which you want the files to break.

    split -l 250000 words.csv

    Screen Shot 2016-04-18 at 12.59.35 PM

    You can see that the new files are way smaller and have been named in a series of three letters, beginning with ‘xaa.’ 

    Screen Shot 2016-04-18 at 1.29.20 PM

    These are still CSV files – they just don’t have the ‘.csv’ file extension yet. You can add it manually, but if you have a lot of new files it’s easier to do it through the Terminal.

  5. Make sure that you’re still in the ‘Split’ folder on your desktop using the ‘pwd’ (print working directory) command. Next, paste the following – no need to worry about how it works.

    for i in *; do mv "$i" "$i.csv"; done

    Screen Shot 2016-04-18 at 1.34.16 PM

    You should now see that all of the split files are saved as comma-separated and will open nicely in Excel. 

    Screen Shot 2016-04-18 at 1.34.13 PM

Stay in touch

Bringing news to you

Subscribe to our newsletter

Posted by Austin Denny

Related Content

thumbnail image

Get Informed

PMG’s Predictive Dashboard Wins Innovation Award

1 MINUTE READ | September 28, 2021

Get Informed

The Road to Recovery for the Travel Industry

5 MINUTES READ | November 19, 2020

thumbnail image

Get Informed

Facebook Details Approach to Mobile Industry Changes

4 MINUTES READ | August 31, 2020

Get Inspired

Considerations for Reengaging New Online Customers

1 MINUTE READ | April 29, 2020

Get Informed

EMEA Search Trends Amid COVID-19

8 MINUTES READ | April 28, 2020

thumbnail image

Get Informed

A Permanent Shift Into Retail Media

1 MINUTE READ | April 23, 2020

Get Informed

Social eCommerce is The Darling of Cyber Weekend

4 MINUTES READ | December 2, 2019

thumbnail image

Get Informed

Reports of Amazon Clean Room Draws Attention of Advertisers

2 MINUTES READ | September 4, 2019

Get Informed

PMG Shares Thoughts About Rumored Amazon Data Clean Room

1 MINUTE READ | August 28, 2019

Get Inspired

Working with an Automation Mindset

5 MINUTES READ | August 22, 2019