vxUI070xSUq1R5qoKnETxcDi+rmjMk6ernMZ657hMus=

Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.

Home

+fQFnIt2m7zOhUxAXCo9PeQRtnla4zeiD+GrBd1qars=

Start learning about web scraping, web crawling, data collection, and popular tools to start developing your own scraper.

Introduction

Hktg+OTfe4WNs3h3Ra5vCCrY6dmxPp44IPgqCnCG8G8=

Learn about browser DevTools, a valuable tool in the world of web scraping , and how you can use them to collect data from a website.

Browser DevTools

hmV1GEtpmtVfOScKodzzOGm93t2vQkG3WQTanMvxbXs=

Learn how to use browser DevTools, CSS selectors, and JavaScript via the DevTools console to collect data from a website.

Using DevTools

lC8uKRMnVcCNPYlR7coR9PRW+XkDzgpaTxf7gmN7xR8=

Continue learning how to collect data from a website using browser DevTools, CSS selectors, and JavaScript via the DevTools console.

DevTools (continued)

3hVW+b9Pv0mYeIyc++t0aubec9DrbG51BEFDeHoj/ZQ=

Set up your computer to be able to code scrapers with Node.js and JavaScript. Download Node.js and NPM and run a Hello World script.

Computer preparation

baxfab2KQTzPU7dh7sM5KG7XO/iO87bLUIZjwKVBjBo=

Create a new project with NPM and Node.js. Install necessary libraries, and test that everything works before starting the next modules.

Project Setup

j2oofsUUa/nZhi7DngMobi46aP4VIu2JijIbLAM+658=

Learn how to use JavaScript and Node.js to create a web scraper, plus take advantage of the cheerio and got-scraping libraries to make your job easier.

Node.js scraper

MZhjALwMgujz/SVPXAvGWZBIiUULH8rpCNQabObKb3A=

Continue learning how to create a web scraper with Node.js and Cheerio. Learn how to parse HTML and print the results of the data your scraper has collected.

Node.js (continued)

R7jk+gNtBl3Z7u0Nx3ubnnjYaIYtn/xbOsb9vLxyatc=

Learn how to save the results of your scraper's collected data to a CSV file that can be opened in Excel, Google Sheets, or any other spreadsheets program.

Saving results to CSV

cVeZyrHJHHm3PEYPo2bejK7QZCjI5KrmJHh8B8IHIFQ=

Learn about "hidden" data found within the JavaScript of certain pages, which can increase the scraper reliability and improve your development experience.

JavaScript objects within HTML

No85DDiwoeN/0dN87IXoyU9N1F/D8GGOOH2BW8nr7Po=

Learn about HTML, CSS, and JavaScript, the basic building blocks of a website, and how to use them in web scraping and data collection.

Basics of data collection

rw5y4x08/nzrZb9L3Gl3DGJtIxqfuUGZfkGzKcRgMQw=

Review our e-commerce website scraper and refresh our memory about its code and the programming techniques we used to collect and save the data.

Recap of data collection

kLK4R2uKdG9G0EVDDF4P81fArGNT6veIUt7YTLcIvt8=

Learn what a link looks like in HTML and how to find and collect their URLs when web scraping using both DevTools and Node.js.

Finding links

eFmijD16ZMX7P9n7CBTFfCBVs3f5CIm7UZLghHvSazw=

When you collect links from a web page, you often end up with a lot of irrelevant URLs. Learn how to filter the links to only keep the ones you need.

Filtering links

8kV474wQUPOPzNyiTD9LTOVrGisCNVm4iwDRnvcBI8A=

Learn about absolute and relative URLs used on web pages and how to work with them when parsing HTML with Cheerio in your scraper.

Relative URLs

2QEzd42TL70IHGxfWFSRRhh/M5LvzsMoeCw2sK/BtZ0=

Learn how to crawl the web using Node.js, Cheerio and an HTTP client. Collect URLs from pages and use them to visit more websites.

First crawl

lyPlOdM1nEDTR3mj6Ni/FTCAztDDZdAdIYCv6ONE9i0=

Learn how to add data collection logic to your crawler, which will allow you to extract data from all the websites you crawled.

Scraping the data

IyhH8y8XSIcUv6K0IEuSeDQZWcJxsUyrRgh41JQGtXY=

Learn how to build scrapers quicker and get better and more robust results by using the Apify SDK, an open-source library for scraping in Node.js.

Professional scraping

B1s1QyDy1GwcSV96v2jsanda0VpRyZlWrkt7f1EnMXE=

Learn how to scrape the web with a headless browser using only a few lines of code. Chrome, Firefox, Safari, Edge - all are supported.

Headless browsers

Eu0vbvmpQWb2fLroHscOq0/sbbhuNSFYrMIMiTUOoWI=

Learn how to process the data you scraped using the Apify SDK and how to convert JSON to Excel files using the Apify API.

Processing data

gwD/UG3sWdIWaMH+x7xaJ2bpI5CPyJGxMTyJglA+X3k=

Learn about dynamic pages and dynamic content. How can we find out if a page is dynamic? How do we programmatically scrape dynamic content?

Dealing with dynamic pages

c5wHa7kdwzwXzUtQhO5SD5TP2MO82QBJQ/27PHYyyj0=

Learn how to crawl the web with your scraper. How to extract links and URLs from web pages and how to manage the collected links to crawl the web.

Basics of crawling

ceJ1hsYuMWWA4CB9xflPGcqAVlftnaPzfcI/0YkHnck=

Understand the standards and best practices that we here at Apify abide by to write readable, scalable, and maintainable code.

Best practices

ot16QxaSAYd2oDM/vvt+ke0S+b7MqvaDO3hVgfKxY+w=

Learn how to develop web scrapers with this comprehensive and practical course. Go from beginner to expert, all in one place.

Web scraping for beginners

TGAtPSzyuQgSoOe4lcWgOK3/CfvT/iqrGqhFReVL5To=

Understand what the Browser object is in Puppeteer/Playwright, how to create one, and a bit about how to interact with one.

I - Launching a browser

yOsoTm5AB34JM+TwReFhsJWqqrhjyjXRnYIe5UqKiR4=

Learn how to programmatically do actions on a page such as clicking, typing, and pressing keys. Also, discover a common roadblock that comes up when automating.

Interacting with a page

fS3pmRuKWkrzRbFn0bkvL5qXCIFwBSyMcyFHWs8oG2w=

Learn the importance of waiting for content and events before running interaction/collection code, as well as the best practices for doing so.

Waiting for content & events

yYnvr3cKzntqAt04oNJqFrM+f7CAEgsK+lZVvrFT6IU=

Understand that the Page object has many different methods to offer, and learn how to use two of them to capture a page's title and take a screenshot.

Page methods

AigblrteDWz3GsLLR+0TMIPNjcD06eCx8Xb+lGdlCq0=

Learn how to create and open a Page with a Browser, and how to use it to visit and programmatically interact with a website.

II - Opening & controlling a page

7EWlAAqIoeqCz5XolF4SYAVdu+/eYJ8NQGUQFHBbBwo=

Learn how to inject scripts prior to a page's load (pre-injecting), as well as how to expose functions to be run at a later time on the page.

Injecting scripts

DpKWcisYTxV2ycicxR51dWLs5oKW20mVFXHU7MtGO4I=

Learn how to collect data from a page with evaluate functions, then how to safely collect it by using a second library called Cheerio.

Collecting data

RTGTtG/1SrrSyE6CCp0ysUT7ZdBSiRCPutj8Q/brSmo=

Understand the two different contexts which your code can be run in, and how to run custom scripts in the context of the browser.

III - Executing scripts

eAy8UorA1VlVKnBc5oxzJUGluuyUVRtf/is+QTMJDGs=

You can use DevTools, but did you know that you can do all the same stuff (plus more) programmatically? Read and intercept requests in Puppeteer/Playwright.

IV - Reading & intercepting requests

Y/1bAyMjPGbI+MRl3MGv1h1oIk9TnpRDDgXj7zVldOs=

Understand how to use proxies in your Puppeteer and Playwright requests, as well as a couple of the most common use cases for proxies.

V - Using proxies

uNJslD5WvKaOY+CC1/uQgGEI9QN6NZnWRf+nbD3UWfM=

Learn what a browser context is, how to create one, how to emulate devices, and how to use browser contexts to automate multiple sessions at one time.

VI - Creating multiple browser contexts

BVE0fiorTc/atZBwEoRiu9z4gNWDeOAqjVOqXDSmGHM=

Learn in-depth how to use two of the most popular Node.js libraries for controlling a headless browser - Puppeteer and Playwright.

Puppeteer & Playwright

SRgc+2BsMMfc5v/eViWsk62nl6rr3HcDR39t0OAWsIg=

Learn about rate-limiting, a common tactic used by websites to avoid a large and non-human rate of requests coming from a single IP address.

Rate-limiting

xtChBJd+8FDmrkGLsYr4EdZYX/03C+EcfBeoeGwExP0=

Understand browser fingerprinting, an advanced technique used by browsers to track user data and even block bots from accessing them.

Fingerprinting

Th6+Pkq+ZZWjruhVS+NH4ld9wGk6OnNCS5oJrQmzwoM=

Learn about the geolocation techniques to determine where requests are coming from, and a bit about how to avoid being blocked based on geolocation.

Geolocation

3Igiy8LANV8BjX1bzxf56C2pJVpo/+dzhfkwu1YgHng=

Understand what a web-application firewall is, how they work, and the various common techniques for avoiding them altogether.

Firewalls

+sj9CYeVjvcy1ZxRHW6cbs7tF9UIm3abB/qipsXF3EM=

Learn about the reasons a bot might be presented a captcha, the best ways to avoid captchas in the first place, and how to programmatically solve them.

Captchas

Eu4gUHZ9spDGrvgX4XWCd6YLd7LVZ4UiF4E4rwnc2ok=

Understand the various common (and obscure) anti-scraping techniques used by websites to prevent bots from accessing their content.

Anti-scraping techniques

fwBr5kgR/l3yx0frRr9rOlan77qNyv/tT3cU1zfeUJw=

Learn how to use and automagically rotate proxies in your scrapers by using the Apify SDK, and a bit about how to easily obtain pools of proxies.

Using proxies

b/nGstwVxw9AzpK75nKgFEEIZToLPycjD+7nXV4K/pU=

Learn all about proxies, how they work, and how they can be leveraged in a scraper to avoid blocking and other anti-scraping tactics.

Proxies

Apl7t74BumhpKEncdeSJ2GgmENyRe6Kbnv6J6UTaLMM=

Understand the various anti-scraping measures different sites use to prevent bots from accessing them, and how to appear more human to fix these issues.

Anti-scraping protections

5CWlAvqNeuAMzYqRnanMsdwYvGhM4XkDVnhiJZfsCtc=

Learn how to effectively locate a website's API endpoints, and learn how to use them to get the data you want faster and more reliably.

Locating API endpoints

hnMAo1VepbuvRd8yHr0DO5sO6/lKwIcDmZ8MY2O+mE4=

Learn about how some APIs require certain cookies, headers, and/or tokens to be present in a request in order for data to be received.

Cookies, headers, and tokens

d2pRNL2WITcm23+3LGP5JC9DkgexidGpU1MvLHl2wUQ=

Learn the benefits and drawbacks of API scraping, how to locate an API, how to utilize its features, and how to work around common roadblocks.

General API scraping

XqAjhm71yzx67juNOIj4sFtNNRM3+2txW7SYb4ZGVw4=

Learn how to modify the variables of a JSON format GraphQL query to use the API without needing to write any GraphQL language or create custom queries.

Modifying variables

P73NR6evGor0zWCY2SGNF70hyLpurL23IwFNsDMOvFQ=

Understand what introspection is, and how it can help you understand a GraphQL API to take advantage the features it has to offer before writing any code.

Introspection

ixLGYBFh9047FO5Qg7A8isafKBU0h7QiiqtpLj77T30=

Learn how to write custom GraphQL queries, how to pass input values into GraphQL requests as variables, and how to retrieve and output the data from a scraper.

Custom queries

Ru4eyeqe5ndURwbU55jbO486eHyizwusO8T1s4WCWKA=

Dig into the topic of scraping APIs which use the latest and greatest API technology - GraphQL. GraphQL APIs are very different from regular REST APIs.

GraphQL scraping

NxOuO9+htDobmcL5Uo+ASBQ+FL759ayC7OZ1jlsv4Gs=

Learn all about how the professionals scrape various types of APIs with various configurations, parameters, and requirements.

API scraping

q99e+KuQvAlnhCjIHcTJ+9pQ/ReWpCYCtyWenkDGNyA=

What is an actor? How do we create them? Learn the basics of what actors are, how they work, and try out an actor yourself right on the Apify platform!

Actors

H4ERRbMU1SRdorjXa59gZTZZ5LTy7KvgfaTi0dltvi4=

Build and run your very first actor right on the Apify platform from a template. This lesson provides a hands-on experience with building and running an actor.

Creating actors

g2iW/PN7HevOQAttw5Ym7dPDUf8Nx7q8PcI2FoP+1Gs=

Create an actor from scratch which takes an input, processes that input, then outputs a result that can be used elsewhere.

Inputs & outputs

evJcqSDX/zpJ5+SVHgXgE1KbOBDF2A+xopwngHYlF04=

Learn how to use the Apify API to programmatically call your actors, retrieve data stored on the platform, view actor logs, and more!

Apify API

CmZ+9jsHFokYywiD3m0RZFhUr0FAHBJEE4qTtmlY6wE=

Interact with the Apify API in your code by using the apify-client package, which is available for both JavaScript and Python.

Apify client

odHz6nngHcid/qM6lGPe5aXOkND71X+m6LKfKpilmwo=

Get started with the Apify platform by creating an account and learning about the Apify Console, which is where all Apify actors are born!

Getting started

J7p4Zx6/syfIla/aFtJpY1U3jvskvji+qFnxeE0+YJ8=

Learn to accept input into your actor, do something with it, then return output. Actors can be written in any language, so this concept is language agnostic.

TZzlYEqcpazwzpHIZm0G1W0w+IGwFVCliwggPs3JCgU=

Learn how to generate a user interface on the platform for your actor's input with a single file - the INPUT_SCHEMA.json file.

Input schema

HJuw8pTeTaQ/qCYgzvUICM1m3Q/fK1FSo/Si4flkv68=

Understand how to write a Dockerfile (Docker image blueprint) for your project so that it can be run within a Docker container on the Apify platform.

Dockerfile

9TRnvkhUP0aVdI5S+Nsny4bZDVaEe8ncP9hvjy8xAwg=

Push local code to the platform, or create a new actor on the console and integrate it with a Git repo to optionally automatically rebuild any new changes.

Deploying

ObhXg497A2RID956J3MJ8zh7L0lC5+82UMESBcSvWVU=

In this course learn how to take an existing project of yours and deploy it to the Apify platform as an actor in just a few minutes!

Deploying your code

IhKukePaWbCKaE7m89rUISzS2gYRUHykCcPL187WBEQ=

Learn all about the Apify platform, all of the tools it offers, and how it can improve your overall development experience.

Apify platform

jkV0Q/4SsgAsymvWNOSWM0vcoii2ZEbpUB4+bM9ZdE8=

Learn about the Apify SDK, as well as some semi-advanced topics which will be important throughout the next lessons in this section of the course.

I - Apify SDK introduction

Xr+wGt+O4o3OhrDXjDCYOjpSyE70MfiOiryFiws58EE=

Learn more advanced details about actors, how they work, and the default configurations they can take. Also learn how to integrate your actor with webhooks.

II - Webhooks & advanced actor overview

l7A63U1HwepBCW+E70d+u79Y09+Ia3FOvfl0WuGmatg=

After learning the basics of actors, learn to develop pro-level scrapers on the Apify platform with this advanced course.