Tutorial: How to Crawl a YouTube Channel

What you will need for this tutorial:

Scream Frog SEO Spider (free version is available)

Text Editor such as Notepad

Firefox, Chrome, or Opera Browser

Web Developer Extension 

Excel

If you’re ever doing an audit on your client’s YouTube channel, crawling it can really come in handy to check extract titles, title length, and description, especially if the channel consists of hundreds of videos. However, crawling a YouTube channel isn’t as easy as just entering the channel URL into your favorite web crawler since most of the videos are inserted into the page using JavaScript, so only a portion of the videos will be successfully crawled.

Crawling a YouTube channel makes it easier to review the video titles–after all, they are the number one driver of click through rate. You can also sort the titles based on character length to see which titles are too long and which titles could use some beefing up. Screaming frog will also pull in the video descriptions BUT only the first 160 characters. But that’s okay, since the most important information about your video should be within in the first few sentences any way. Lastly, the video tags are inserted into the meta keyword tag which can be picked up Screaming Frog. Crawling a competitors YouTube channel can be a great way to see which tags they are using for their videos that your videos may be missing.

Open preferred browser (Firefox, Chrome, or Opera) and download the Web Developer extension. After that is done, restart your browser and go to YouTube and pick a channel. On the channels main page, click videos.

Depending on whether or not the Channel has it’s videos divided into sections, you will either need select ‘upload’ or ‘all videos’ in the top left drop down menu. If the videos are being displayed in grid-view, you will need to change the display to list view.

Once in list view, you will need to scroll to the bottom of the page and click “load more” continuously until all of the videos have loaded on the page.

Now that all of your links are displayed on the page, in the web developer extension toolbar in your browser, click information > view link information.

Copy and paste the links into a text editor, such as Notepad. Save the list as a text document.

Open Screaming Frog. Go to Mode > List. Upload your list of URLs.

Go to Configuration > Spider > Limits and make sure ‘0’ is selected.

Click Start to begin the crawl.

Go forth and conquer. Export into excel and have fun sorting and filtering this information however you would like.

NOTE: You will end up with a few links that are not videos. If you export your crawl into Excel, you can easily get rid of these extraneous links by applying a text filter to the Address column of “does not contain:watch?v=“.