PMG Digital Made for Humans

Autosplit Large CSV Files into Smaller Pieces

2 MINUTE READ | April 22, 2016

Autosplit Large CSV Files into Smaller Pieces

Author's headshot

Austin Denny

Austin Denny has written this article. More details coming soon.

With the recent news that Windows 10 will introduce a UNIX shell, users will have access to a command-line environment on Linux, Mac, and Windows for the first time. And that means you can finally stop bugging the developers at your company to break up the enormous files that Microsoft Excel refuses to open. You don’t need to know anything about the Terminal to do this— although, if you’d like to learn more, check out these tutorials:

Okay, here’s how it works (for this article, we’ll be working on a Mac, but you can use this method elsewhere).

  1. Open Terminal (Applications/Utilities/Terminal)

    Screen Shot 2016-04-18 at 12.38.41 PM
  2. Create a new folder on your desktop. This is where you’ll save the file that needs to be split. My folder is called ‘split’ and I have a CSV file that contains a few copies of the English dictionary, or about 2.36 million rows. 

    Screen Shot 2016-04-18 at 12.53.02 PM
  3. In Terminal, navigate to the folder you just created using the ‘cd’ command, which stands for ‘change directory.’ 

    Screen Shot 2016-04-18 at 12.55.32 PM
  4. Now, you’ll use the ‘split’ command to break the original file into smaller files. To do this, you’ll type the following, where 250000 is the incremental number of rows at which you want the files to break.

    split -l 250000 words.csv

    Screen Shot 2016-04-18 at 12.59.35 PM

    You can see that the new files are way smaller and have been named in a series of three letters, beginning with ‘xaa.’ 

    Screen Shot 2016-04-18 at 1.29.20 PM

    These are still CSV files – they just don’t have the ‘.csv’ file extension yet. You can add it manually, but if you have a lot of new files it’s easier to do it through the Terminal.

  5. Make sure that you’re still in the ‘Split’ folder on your desktop using the ‘pwd’ (print working directory) command. Next, paste the following – no need to worry about how it works.

    for i in *; do mv "$i" "$i.csv"; done

    Screen Shot 2016-04-18 at 1.34.16 PM

    You should now see that all of the split files are saved as comma-separated and will open nicely in Excel. 

    Screen Shot 2016-04-18 at 1.34.13 PM

Stay in touch

Bringing news to you

Subscribe to our newsletter

By clicking and subscribing, you agree to our Terms of Service and Privacy Policy

Related Content

thumbnail image

AlliCampaigns & Client WorkData & TechnologyStrategyDigital MarketingCompany News

PMG’s Predictive Dashboard Wins Innovation Award

1 MINUTE READ | September 28, 2021

thumbnail image

Consumer TrendsSocial MediaPlatforms & MediaDigital Marketing

What You Need to Know About Facebook’s Latest Content Transparency Reports

4 MINUTES READ | August 30, 2021

thumbnail image

Consumer Trends

The Road to Recovery for the Travel Industry

5 MINUTES READ | November 19, 2020

thumbnail image

EMEA Search Trends Amid COVID-19

8 MINUTES READ | April 28, 2020

thumbnail image

A Permanent Shift Into Retail Media

1 MINUTE READ | April 23, 2020

thumbnail image

Social eCommerce is The Darling of Cyber Weekend

4 MINUTES READ | December 2, 2019

thumbnail image

Working with an Automation Mindset

5 MINUTES READ | August 22, 2019