Always Check for the Hidden API when Web Scraping

wonderful. thank you.

Thanks John! You are a lifesaver sir!

excellent video. Subscribed!

This video is really amazing I learned web scraping from your videos
thanks

Excuse me, what tool is the "Dashboard/PLZSUB" in your operation interface, and where can I download it?

I'm having trouble seeing the requests and responses in the network tab. All I see is post requests (XHR). JS requests are gibberish too. This is only happening for a particular shopify-type site. Any recommendations?

Don't many pages protect themselves from this by requiring id's (api keys) for using the API? Keys you only get if you load the page in full, right?

What to do if API's response is in HTML .. and If the API is CORS configured ??

哇，牛逼的狠，希望您做更多这样的视频，谢谢！

Does this work with api requests that have auth credentials meaning specific data for the logged in user? To track their sales etc

That was incredibly helpful and exactly what I needed today. Your presentation is very clear. Thank you!

Greetings from Brazil! Thank you! I just had to adjust some of the quote marks on the header (there were some 'chained' double quotes (like ""windows"")), making some of the header's strings be interpreted by python as code, not text. Just had to change inner double quotes for single quotes (e.g. "'windows'") and it worked perfectly!). Can't wait to try your other tutorials! Once more, thank your very much!

John thank you for the videos.
How do you deal when in the network tab xhr you have a graphql object not a Json one?

What application you use in video?

Thanks Sir !!

John, what should I do if I get the error "msg": "token timed out or duplicated"? There is a "g-google-authorization" header which gets updated everytime I reload the page on the browser. I'm not logged in or anything, just entering the website as a random person. Is it possible to get this token through Python and use it in the request? Can you make a video about it?

what browser is that? What are you using to allow you to right-click a response and do copy as curl cmd ? What tool are you referring to?

what browser is this

What if the button establishes a websocket which is then used to retrieve all the data?

hi. what if the cookies are expiring. is there any way in python to get cookies automatically

I checked 2 website i need data but they are using websockets can i fetch data ?

This works in a lot of cases were the API is open. However, in cases like Social Media Platforms were you have to have an account to access the API or a Wordpress Websites were the API is turned off it wont work.

The best approach in these situations, is really just to use Selenium or anything close and try to crawl the pages with a delay.

Great content! I just have one question: Which web browser you are using in the video? The response tab on my browser is not structuring the received data. It just shows it in a single line.

very nice

Nice sir 🎉

what do I do if it says curl access denied?

really nice and helpful tips in an actual topic with a sight-pleasuring recording quality, thank you for your time and efforts.

Thanks John! I got 90% of the way there, but I’m not sure how to get the desired data separated into csv columns. I’m trying to gather product data for import into the BigCommerce e-commerce platform. I’d be so grateful if you could help with that.

Thanks so much for this video.
I sucessfully obtained the data I want via Insomnia, but when I try to retrieve it via Python it doesn't work. Any ideas?

Hi there! Great video. But I have a question. Have can I make the same thing but with website that has login? I was trying to get data like you with Insomia, but I'm getting 401 error. How can I add auth credentials correctly

It doesn't work for some websites 😢

Merci !

Simply amazing, but one question:
I tried the method and have some search queries in the payload (copying the cURL in Insomnia). But, the search query terms are basically the information I want to extract.
How can I solve this?

bro, you're a game changer and i love you. if i ever see you in person ill offer to buy you a beer, or lunch, coffee whatever

Best! Thank you!

Hi there, I found your channel where each and every video delicately made for web scrapping and automation which helps me a lot as work with web scraping and web automation.

I have a request, if possible then please make python data post methods on Stateful api v1 and how to mimic cookies and session to get the job done.

Thank you.

Great content. Thanks for this video

Dear John, I am really appreicate your work. I have an issue with scraping a page that information is hidden in an api with buttons ( each companys details is hidden in each button of website). Do you have any recommend for me ? Thank you for your consideration.

Love & peace

Thank you so much for the tutorial. I have a question, how to get a Authentication value that include the header, can I do automatically and without selenium?
In this moment, I get it manually in the network tab, further, the authentication value expire after of a time.

I didn't knew that is so simple.I think I make everything much harder for me hah. Thank you!

This is seriously high level content right here

Thank you so much - this is so insightful and educational. Really helped me understand so many things in so little time.

nice! thanks man!

Hey John there's this response that gets returned in this format.
[
"blah",
"blah",
{
"key1": "xxxxxx",
"key2": "xxxxxx",
"key3": "xxxxxx",
"key4": xxxxxx,
"key5": {
"blah": {
"key": "xxxxxx"
}
},
"key6": {),
}
]&&&[
{
"key1": "xxxxxx",
"key2": "xxxxxx",
"key3": "xxxxxx",
"key4": xxxxxx,
"key5": {
"client-side-metrics-info": {
"requestId": "xxxxxx"
}
},
"key6": {),
}
]&&&[
{
"key1": "xxxxxx",
"key2": "xxxxxx",
"key3": "xxxxxx",
"key4": xxxxxx,
"key5": {
"client-side-metrics-info": {
"requestId": "xxxxxx"
}
},
"key6": {),
}
]&&&...

What's your recommendation to parse something like this? My first thought was to use regex, but I don't know if that would be the most efficient to convert this to a better format.
The only things I need are key1 and key6.

Thanks for your work, I've learned way more with your videos than some of the books and tuts I've spent hours on in the past.

I tried this method on web page xhr. It says "unauthorized". I have username and password but I don't know how to use that in get request now.

I would like to use your method but I get error 401 meassage "Access denied due to missing subscription key. Make sure to include subscription key when making requests to an API." Is there some method to find it or use other way?

yu are the best bro