Don't Start Web Scraping without Doing These First

Thanks John, I have 2 questions...First, how do you download the HTLM with requests? I tried looking it up and didn't find the solution. Second, looking at the source, what are we suppose to be seeing? I have dont that but not sure what I am looking for.Thanks

Tnx

it helped mate. what lib do you recommend for parsing lxml/html? and ofcourse for async request.get (only) and request.post(rarely). minimal libs just to get the work done. in one of your vids u talked about selectolax, and request-html in this one. I only need those two functionalities I mentioned above(parsing, requests). much appreciate it.🙏🏼

Nice video! What do you use for screen recording ?

nice video. i use bs4 because a lot of your videos use bs4 and i try to adapt your examples to my projects. Could you do future video with more complex selectors please :) because i have a lot problem to adapt with something like that lol <div id="ember2514" class="sb-accordion-item game-market sb-accordion-item__open ember-view">.

Hey John Can you make a short crash course on phantom js?

Thanks for the great content, your channel is an excellent learning resource. May I ask for a starting suggestion for a project that involves authentication and downloading CSV and Excel files.

Thank you, John, the tips were really helpful. and I would love it if you can share more of this in the future.

How would you recommend dealing with IFrames? Any tips to extract data from those easily?

Thank you for the great advice.

Bro if you're yanking 500k files saving them all in github is not ideal

Wonderful video. Do you have any on decision tree ?

Hey man, I have an e commerce site I'm trying to scrape and I found that one section of the page I'm trying to get calls an API post and it's paginated. With that said, will it be better to just go straight and call the data through the API on that part instead of scraping it off the page? Follow up, should I still use scrapy or in combination of bs4? One to load and scrape the page and the other one just for the post API call.

Great video! Thanks!!

I think we need a video where you talk about all the challenges that will face us when scraping like blocking ip or problems caused by sending too many requests.

Thanks.

Your channel is too good for us scrapers!!!

parsing locally...men....that was it!!!

Valuable advice from 50,000 ft, not the usual 500 ft.  
Don't just start coding. Stop, think, design, look harder.  
Do you really understand the specific details of the problem, or are you guessing?

Wonderful videos you have. How can I select the columns I want to scrape. Maybe the the information I need is in column 1,2 and 4. How do I don that? Thank you

Man!, I'm having so much fun learning from watching your videos.

Hey John,
Just recently sub'd...
These are great tips!

How about a separate vid for each one?
Looking over your shoulder,
The 1st one:
What will You be looking for? Keeping an eye out for?

Listening to your train of thought - while you're going through the motion/ process would be awesome!

Hope you consider this request.

You are the best man. Much respect, keep up the good work, I learn a ton from you as a beginner

When building my scraper, I love to do it on a jupyter-notebook first so that I could separate the request and parse part of the program.

Why can’t I scrape most amazon sites?

Hi John,can you please help to scrape the reviews from slicksdeals site for all the sublinks of a product..I have tried bit failed to do it... please help me

How to scrape site that have " Loade more " or "show more" Button.plaz show us example

We urgently need video about scraping from TripAdvisor using Selenium please 😀

Gold! Great channel.

My favorite tip: Parse Locally 👍🌹

John, that was a really helpful (top down) overview which I found very helpful. As a visual learner, I almost need a decision tree diagram to take me down the most appropriate route... thereby taking me to the right set of tools/ routines to use. It’s also helpful to have a video in the 7- 10 minute time range, to focus in on the particular topic in hand. 10 out of 10 from me! 👍

Thanks for the tips!

Scrap location wise loaded content.... Please

I'm looking for a market place where people hire scrapers?

Great video Sir.

hello john. i would like you make a video scraping linkedin without selenium. for search jobs. thanks

Hello John. Thanks for your tips.

Top content as usual

Thank you I just completed my first scrapy project today

bell gang! (2)

bell gang!