Latest Blogs
Stay up to date with the latest features, tutorials, and insights about social media scraping
New Endpoint: Scrape Instagram Comments from Any Public Post!
You asked for it. I needed it myself. And now it’s finally live, you can now scrape Instagram comments directly from a public post using the Scrape Creators unofficial Instagram comments API. No login. No shady tactics. Just good old public data. Why This Is a Big Deal Instagram comments are a goldmine of insight: Want to know what customers really think about a product? Looking to analyze sentiment on a viral post? Need to build a dataset of user-generated content, testimonials, or audience engagement? Now you can. Until now, there weren’t many ways to get IG comments without logging in or doing something sketchy. But this endpoint works entirely from public data. That’s how all Scrape Creators endpoints work, and that’s how we stay in business. How It Works All you have to do is submit the post URL and the amount of comments you want. I handle all the pagination and complexity behind the scenes. Let's say we wanted to get the comments for this Mr Beast post: https://www.instagram.com/p/DC7Y9vyyg_b First, sign up for an API Key: https://app.scrapecreators.com/ Don't worry, you get 100 free credits. Then just make a GET request to https://api.scrapecreators.com/v1/instagram/post/comments/simple?url=https://www.instagram.com/p/DC7Y9vyyg_b&amount=500 Make sure to include your API key as the x-api-key header. And we actually got 500 comments! It did take 4 minutes though, so if you want a lot of comments, be prepared to wait unfortunately. The response will look like this: A Few Things to Note You likely won't get *all* the comments You’ll get roughly 100–300 comments (although we did get 500 above), depending on how many IG will return publicly. For huge posts with thousands of comments, you probably won’t get all of them, that’s Instagram limiting what’s public, not me. Since I’m handling the pagination, this endpoint costs multiple credits. It’s about 1 credit per 15 comments. Most of my other endpoints cost 1 credit and return exactly what you request. This one’s different because I’m calling the IG API for each new page of results What You Can Do With This The use cases are wild: Sentiment analysis on influencer campaigns Pull UGC for DTC product pages (testimonials, feedback, reactions) Build datasets of real customer language Analyze virality, what kinds of comments do people leave on viral reels/posts? Find leads by scraping who’s engaging with posts in your niche Track reactions to a new product, feature, or campaign Seriously, if you’re in marketing, UGC, influencer research, or AI training, this is 🔥 Try It Out Now The endpoint is live: https://docs.scrapecreators.com/v1/instagram/post/comments/simple Just pass a public Instagram post URL and how many comments you want. Sit back. I’ll handle the rest. This has been one of my most requested features, and something I’ve personally wanted for a while. Now it’s here. Go crazy. Sign up for an API key and get 100 free credits.
Don't Want to Pay Outrageous Twitter (X) API Fees in 2025? Here Are Your Options for Scraping Twitter
If you're building anything that relies on Twitter (X) data, you've probably come face to face with their API pricing. In 2025, it’s downright brutal. $100/month gets you just 15,000 tweets, and $5,000/month caps you at 1 million tweets. Enterprise pricing? Don’t even ask. So what do you do if you’re not a Fortune 500 company but still need Twitter data? Here are the best options in 2025: Option 1: Scraping Behind the Login (Powerful but Risky) This is where all the good stuff lives: full user timelines, Twitter Search, replies, and more. But here's the deal: scraping behind the login is against X's terms of service. It's risky. Tools that do this could be shut down overnight, just like SocialData.tools was recently. Still, there’s one standout: Old Bird V2 on RapidAPI Old Bird v2 is a third-party API that mimics the old Twitter API and scrapes behind the login. Pros: Read access to user timelines, tweet search, tweet metadata Way cheaper than Twitter/X’s official API Been around for years and still working as of 2025 Pricing (as of July 2025): Pro: $24.99/month → 100k tweets Ultra: $69.99/month → 300k tweets Mega: $179.99/month → 1M tweets Cons: It scrapes behind the login — which means it could disappear anytime No write/post access (read-only) Verdict: If you want full-featured access at a reasonable price and can accept the risk, Old Bird is your best option today. 🌐 Option 2: Scraping Only Public Twitter Data (Compliant but Limited) Want to stay fully compliant and not worry about takedowns? Then stick to scraping only public data, the kind you can see when not logged in. BUT: You can’t access search You can’t see replies A user’s profile only shows their top ~100 tweets, not their most recent Test it yourself: open a profile in an incognito browser and you’ll see what we mean. If you're okay with those limits, I’ve made it super easy with my own tool: Scrape Creators. We only scrape public data, so it’s stable and safe. Here’s an example of how easy it is to get someone's public profile: GET https://api.scrapecreators.com/v1/twitter/profile?handle=elonmusk Bonus Trick: Get Recent Tweets via Google Google’s search results often show a user’s most recent tweets. Search for: twitter austen allred This will show a block of tweets in the results. Weirdly, if you search for: austen allred twitter …it won’t work. 🤷♂️ So the query is a little finicky. If you want to automate this, I’ve got a Google Search scraper you can use. I’d just need to tweak it to extract tweets from the result block. Email me if you want help with that: adrian@thewebscrapingguy.com Final Thoughts There’s no perfect solution to scraping Twitter in 2025, but there are smart options. Want full power? Use Old Bird. Want safety? Use Scrape Creators. Want the most recent tweets? Hack Google or use Old Bird As long as you stay out of the login wall, you’re golden. But if you go behind it, don’t be surprised if your API provider disappears overnight. To try Scrape Creators for free sign up here: https://app.scrapecreators.com/
Top SERP APIs for 2025 Compared: Which Google Search API Is Right for You?
If you're building a product that depends on Google search data, like an SEO tool, keyword analyzer, or lead scraper, you’re likely exploring SERP APIs. These tools let you extract structured search results from Google without worrying about proxies, headless browsers, or HTML parsing. But with multiple SERP APIs on the market, how do you choose? In this post, we’ll compare three leading search APIs in 2025: SerpApi SearchAPI.io Scrape Creators We’ll break down their pricing, features, and supported endpoints to help you choose the best fit for your use case. SERP API Pricing Overview Feature Comparison API Coverage: Search vs. Social Both SerpApi and SearchAPI.io offer rich SERP features and multiple Google-related endpoints, including: Google Maps News Images Shopping Autocomplete This makes them ideal for deep SEO platforms and location based queries. Scrape Creators, on the other hand, is built for high volume scraping across search and social, and includes powerful APIs you won’t find elsewhere at this price or scale: TikTok User, Video, and Hashtag APIs Meta Ad Library Scraping LinkedIn Company & Profile Data Instagram, Reddit, YouTube APIs Google Ad Transparency Data If your project spans across search engines and social media platforms, Scrape Creators offers a broader scraping toolkit than traditional SERP APIs. Choosing the Right API for You Final Thoughts The right SERP API depends on your goals. If you're building an SEO tool that needs full featured Google data (including maps, shopping, and knowledge panels), SerpApi and SearchAPI.io are strong candidates. If you're focused on scalable search scraping or want access to hard to access social media data, Scrape Creators offers more coverage at a fraction of the cost, and with no subscription required. Scrape Creators is built for developers who want to move fast, scrape at scale, and access both search and social platforms with one flexible API stack.
Scrape Creators x Databar AI Partnership 🤝
Scrape Creators x Databar AI: Enrich Any Lead List with Just a Few Clicks We just partnered with Databar AI, and it unlocks some seriously powerful workflows you can now run using Scrape Creators scraping APIs, all without writing a single line of code. Databar is like a spreadsheet on steroids, it lets you call APIs and chain them together visually. And now, they’ve plugged in our TikTok, Instagram, YouTube, and Ad Library scrapers directly into their platform. Here are two game-changing use cases you can run right now: Use Case #1: Competitor Ad Intelligence by Domain Want to know what ads a company is running across the internet? Just drop a company’s domain into Databar, and it’ll use our Scrape Creators ad endpoints to fetch: LinkedIn ads Meta (Facebook/Instagram) ads Google ads Each row includes ad copy, start/end date, targeting info, and ad creative, giving you a complete view of their ad strategy. Perfect for: Competitive research Finding ad angles that are working Reverse-engineering go-to-market strategies Check out the demo below Use Case #2: Discover TikTok Creators in Any Niche + Grab All Their Followers Let’s say you want to find TikTok influencers talking about web scraping. Search TikTok for creators for “web scraping” using our search endpoint Enrich each profile to pull: Username, follower count, profile bio, and more Use our TikTok followers endpoint to grab all of their followers, turning one profile into thousands of leads Perfect for: Building outreach lists Discovering niche communities Analyzing follower overlap between creators Check out the demo below Why This Is So Powerful Zero code: Just plug in keywords or domains and Databar takes care of the rest Scrape Creators power under the hood: You get battle-tested endpoints for social scraping, ads, and more Stackable workflows: Turn a single TikTok search into a 10,000-row lead list in seconds Try It Free You can test these workflows today Start using Scrape Creators on Databar AI → Explore our API docs →
The Ultimate Guide to Scraping LinkedIn Profiles in 2025
The Ultimate Guide to Scraping LinkedIn Profiles (2025 Edition) LinkedIn is one of the richest sources of professional data on the internet and everyone from recruiters to B2B founders to data analysts wants access to it. But scraping LinkedIn isn’t as simple as hitting an endpoint and calling it a day. There are a lot of changes, legal developments, and tradeoffs you should know about before you dive in. Here’s everything you need to know. TL;DR – Which Option is Right for You? The Big Shift: LinkedIn No Longer Shows Work History Publicly (for most profiles) Until recently, you could get a lot of value from scraping public LinkedIn profiles, even without being logged in. But LinkedIn has quietly removed work history from public profiles. That means if you’re scraping the open web version of LinkedIn, you're no longer getting the most important part of someone’s professional profile. So what’s the workaround? Option 1: Scraping Behind the Login (Full Data, Higher Risk) If you need full job history and detailed professional background, there’s only one way to get it: scrape LinkedIn while logged in. That means using buying accounts, authenticated sessions, or API services that go behind the login. Here are some options: Scrapers That Access Full Profile Data Fresh LinkedIn Profile Data (RapidAPI) Gives you job history and more. Scrapin.io Another scraper with detailed data. Like all such tools, it’s subject to takedown risk. These services work, and if you absolutely need that juicy data, they’re often the best option. Just keep in mind they could go down unexpectedly due to legal pushback. Legal Context: Why These Tools Can Disappear Several high profile legal cases explain why behind-the-login scrapers operate in a gray zone: HiQ vs. LinkedIn: The court ruled that scraping public LinkedIn data is legal, but didn’t grant protection for scraping behind login. Meta vs. Bright Data: Court rules in favor of Bright Data, stating public data can't be policed Case in point: Proxycurl, one of the most popular LinkedIn scraping APIs, recently shut down under legal pressure, because they were scraping behind the login. Read their shutdown post. Option 2: Scraping Public Profiles (Limited Data, Lower Risk) If you're looking for a stable, compliant, and cost effective solution, and you're okay with not having full job history, then public scraping is still a viable path. That’s where Scrape Creators comes in. We focus on publicly available LinkedIn data, and don’t require login or cookies. This means: No risk of your scraper being shut down overnight No need to manage sessions or proxies Affordable pricing for ongoing use You won’t get full job history, but for many use cases (light lead gen, contact enrichment, etc.), it’s more than enough. Final Thoughts There’s no “one size fits all” answer when it comes to scraping LinkedIn. If you need the full data, you'll have to go behind the login, just understand the risks. If you're looking for something lightweight and stable, a public scraper like Scrape Creators might be the better fit. Try Scrape Creators Free Want to try a compliant LinkedIn scraper that just works? Get 100 free requests here
Unlock Facebook Group Insights with Our Facebook Groups API (Posts + Comments)
Why You Need a Facebook Groups API Facebook groups are a gold mine for niche conversations online. Whether it's fitness, crypto, parenting, SaaS, or real estate there’s a Facebook group full of your exact target audience. But good luck getting data out of it. Until now. Introducing the Facebook Groups API Our Facebook Groups API lets you scrape public group posts and comments programmatically, no browser automation or sketchy workarounds. ✅ Get the latest posts from any public group ✅ Pull all comments from a specific post ✅ Extremely fast and easy to use ✅ Designed for high-volume scraping This is simple API built for devs and data teams. What You Can Do With It 1. Lead Generation Find Facebook posts where people are asking for recommendations, then offer your service or put leads into your CRM. 2. Market Research Analyze the language your customers use. Scrape posts + comments, summarize sentiment, identify recurring pain points, or categorize discussion topics with ChatGPT or another LLM. 3. Competitor Intel See what people are saying about your competitors in niche Facebook groups. Are people complaining? Loving certain features? You'll know. 4. Trendspotting Track what’s trending inside communities before it breaks out. See what new product ideas or tools people are talking about, straight from the source. How to Use the Facebook Groups API Using the API is dead simple, here’s how you can get started in just a few lines of code. Step 1: Get Your API Key Sign up and automatically get an API Key Step 2: Grab Posts from a Public Group Lets say we want to get the posts and some of the comments from the Dad Jokes Facebook Group. Just grab the link to the group, and make a GET request to the API (docs link) https://api.scrapecreators.com/v1/facebook/group/posts?url=https://www.facebook.com/groups/1158298182085866/ Make sure include your API key as a x-api-key header. The response will look something like this: Use the cursor in your next request to get additional posts (sadly, Facebook only returns 3 posts at a time). Try It Out. 100 Free Requests on Us We made this API because scraping Facebook groups was too damn hard. Now it’s easy. If you’ve been trying to: Extract insights from niche communities Do smarter lead gen Monitor trends or competitors Or just build something cool with real data... You’re gonna love this. 👉 Get Your API Key Now, includes 100 free requests so you can try it risk-free. No credit card. No rate limit headaches. Just clean data from public Facebook groups, ready to use.
The Best Unofficial Reddit API for Fast, Easy Access to Posts, Comments & Trends (2025)
The Unofficial Reddit API That Actually Works for You If you've ever tried using Reddit’s official API, you already know it’s hard to use, and you're severely rate limited. Whether you're a solo dev testing an idea or a data analyst building dashboards, Reddit’s official API makes it hard to move fast. That’s why we built an Unofficial Reddit API, designed to be simple, fast, and powerful, without the bloat or the headaches. What You Can Do With It Our API supports just the essentials you need: Search Reddit posts by keyword Get posts from any subreddit Get comments from any Reddit post No OAuth. No bloated response objects. No need to read a 20 page wiki just to get a list of posts. Even Reddit admits it: The Data API is not intended for use as a high-volume data source. Real World Use Case: How Notion Could Spy on ClickUp Let’s say you work on the growth team at Notion, and you want to better understand what users are saying about your competitor, ClickUp. Your goal? Find out: What frustrates users about ClickUp What features they rave about Where Notion could win over those users Here’s how you’d do it with our API: 1. Search Reddit for mentions of ClickUp First grab an API key from Scrape Creators (100 free requests) Here's the documentation if you want to follow along there. We're going to search all of Reddit for clickup, sorting by new and trimming the response to make it easier for us to read. Make sure to include a header called x-api-key and use your API key. https://api.scrapecreators.com/v1/reddit/search?query=clickup&sort=new&trim=true 2. Grab comments from high-engagement posts Let's take a look at this post: https://www.reddit.com/r/projectmanagement/comments/1lvfkgp/curious_what_people_are_doing_with_project/ https://api.scrapecreators.com/v1/reddit/post/comments?url=https://www.reddit.com/r/projectmanagement/comments/1lvfkgp/curious_what_people_are_doing_with_project&trim=true 3. Analyze with ChatGPT Paste the comments into GPT-4 and ask: “Summarize what Reddit users like and dislike about ClickUp. What problems do they mention? What features stand out? How could Notion position itself as a better alternative?” Or use the OpenAI API to automate this. Here's exactly what Chat GPT said: Example: Reddit + ChatGPT Insight Topic: ClickUp Findings: Users dislike weak automations, steep learning curve, and shallow AI features. They want tools that integrate with existing workflows (e.g. SmartSheets) and offer real AI value. Alternatives like Motion.io and custom setups with Make/N8N are preferred. Insight: Notion can win by being easier to train, more flexible, and offering deeper AI-powered workflows. Other Use Cases Reddit is a goldmine of unfiltered opinions and unmet needs. With our API, you can: Spot Trends Before They Explode Track keyword spikes across Reddit and get notified when new ideas go viral. Validate Product Ideas Just like GummySearch, you can mine niche subreddits for problems, questions, and tool requests that people are begging to solve. Monitor Your Own Brand Set up alerts to track sentiment and feedback as it happens, from product launches to bug complaints. Power Internal Tools & Automations Send Reddit threads to Slack, add quotes to your CRM, or feed your analytics dashboards. Simple, Flexible Pricing No subscriptions. No surprises. Just buy credits and each API call === 1 credit. Use them whenever you want. Scale up or down as needed. TL;DR If you want to: Skip Reddit’s API drama Get only the data you care about Build fast, iterate faster Do it all without breaking the bank Then our Unofficial Reddit API is your new secret weapon. Try it now
5 Best Web Scraping APIs to Use in 2025 (Based on Your Use Case)
Whether you’re scraping Google results, TikTok videos, or just trying to grab structured data from a JavaScript-heavy site, the scraping tool you choose makes a big difference. Here are the top 5 scraping APIs in 2025, what they’re best at, and when you should actually use them. 1. Scrape Creators - Best for Social Media Data (TikTok, IG, YouTube, etc.) Scrape Creators is built specifically for social media, including the social media Ad Libraries. With over 100+ endpoints across TikTok, Instagram, YouTube, Twitter/X, Reddit, the Meta Ad Library and more, it’s the most complete social media scraping API out there. Unlike general-purpose scraping tools that give you HTML, Scrape Creators gives you clean, structured JSON, already parsed and ready to use. Want TikTok trending videos, IG bios, YouTube transcripts, or TikTok Shop detection? It’s one API call away. ✅ 100+ social endpoints ✅ Real-time data from social platforms ✅ JSON out-of-the-box (no parsing needed) ✅ Available via API or prebuilt Apify actors Best for: Building influencer tools Monitoring creators Social media lead generation Anything social-related at scale Ad Spying Ad Inspiration 2. ScrapingBee - Best for Headless HTML Scraping (No Setup Required) ScrapingBee is a solid, general-purpose scraping API. It's the OG scraping API and was recently acquired by OxyLabs. It renders JavaScript, manages proxies, and returns the HTML of the page (you parse it yourself). It’s great when you want something quick and reliable for scraping content-heavy sites like blogs, ecommerce pages, or product listings. ✅ Headless browser ✅ Handles JavaScript & CAPTCHAs ✅ Simple HTML extraction Best for: Scraping non-social websites General purpose use Devs who want to write their own parsing logic 3. Firecrawl – Best for AI-Powered Extraction via Natural Language Firecrawl is the new kid on the block that’s building around natural language extraction, meaning you can ask for what you want using plain English, and it tries to give you back the structured data. Under the hood, it also uses headless browsing to fetch pages. ✅ Headless browser ✅ Handles JavaScript & CAPTCHAs ✅ Use AI to parse the page, or return straight up HTML Best for: AI agents or LLM tools that need structured data People who don't want to parse the html themselves and want AI to extract content for them General Purpose use 4. ScrapingDog – ScrapingBee Alternative With Built-In Endpoints ScrapingDog offers a very similar experience to ScrapingBee: pass it a URL, and it handles headless browsing, proxy rotation, and returns the raw HTML. Where it stands out is in pricing (generally cheaper) and having more built-in scrapers, like for Google Search, some social media, or even job boards, without needing to build your own logic from scratch. ✅ Headless + proxy support ✅ Built-in endpoints for Google, LinkedIn, etc. ✅ HTML response (you parse) Best for: Developers comparing against ScrapingBee Projects that benefit from built-in endpoints Anyone who wants general-purpose scraping at scale 5. Apify – Best for No-Code + Scraping Automation Apify isn’t just a scraping API, it’s a scraping platform. Meaning any dev can write a scraper and put it up on Apify. You can run bots ("actors") that scrape and automate workflows, then send the data to Airtable, Google Sheets, or an API. And yes, Scrape Creators APIs are also available as Apify actors, so you can connect social scraping to no-code tools like n8n, Zapier, Make, or Retool. ✅ Marketplace of prebuilt actors ✅ Great for non-coders and automation ✅ Powerful for teams Best for: No-code teams Workflow automation Connecting scraped data to apps, CRMs, or Airtable Final Thoughts Not all scraping APIs are built the same, and you’ll save hours (and dollars) if you pick the right tool for the job. Need structured social media data? → Scrape Creators Want AI to extract data with no selectors? → Firecrawl Need fast, cheap HTML scraping? → ScrapingDog Want no-code workflows? → Apify Just need a simple headless scraper? → ScrapingBee
How Trading Firms Can Get Real Time Alerts When Trump Posts on Truth Social
Donald Trump's tweets have moved markets before, and now, he's doing it on Truth Social. For hedge funds, quant teams, and high-frequency trading desks, reacting fast to political news is no longer a luxury, it’s alpha. That’s why we built a real-time webhook that notifies you instantly when Trump posts anything on Truth Social. No polling. No scraping. No noise. Delivered in under 4 seconds. Why Trading Firms Monitor Trump’s Posts Trump’s posts on Truth Social have proven to be high-impact market movers, especially when they concern tariffs. Markets React Sharply to Tariffs July 7, 2025: Trump announced 25% tariffs on Japan and South Korea, plus duties on South Africa, Malaysia, and more. The Dow plunged over 400 points, while the S&P 500 and Nasdaq each dropped nearly 1% in reaction. July 8, 2025: He posted on Truth Social a 50% tariff on copper imports, triggering record-high copper prices, up 17% in a single day, as traders rushed to reprice commodity sectors. The Dow and other equities pulled back further. April 2, 2025 (“Liberation Day”): He imposed sweeping tariffs, including up to 104% on Chinese goods, which led to the largest two-day market crash since 2020, with the Dow losing over 4,000 points, S&P down 10%, and Nasdaq dropping 11%. These moves highlight why quant firms and trading desks are glued to his posts: they’re not just news, they’re signals, capable of moving hundreds or even thousands of index points in minutes. Introducing the Trump Post Webhook We built a simple service that detects Trump’s posts on Truth Social in real time and sends a webhook to your server the moment a new post is live. You just give us a URL, and we do the rest. How Trading Teams Use It Trigger Real-Time Sentiment Analysis Feed Trump’s post into your NLP models immediately, and score the sentiment before it hits the news wires. Power News Based Trading Use the webhook as a direct input signal in your algorithmic trading logic. Human-In-The-Loop Alerts Pipe the webhook into Slack, Discord, or even Bloomberg Terminal via middleware. Instantly alert the trader on duty. Risk + Compliance Proactively flag posts that could impact your portfolio or raise compliance concerns around politically exposed persons (PEPs). How to Use It Check out the docs Sign up and get an API key: https://app.scrapecreators.com/ Send a POST request to this API: https://api.scrapecreators.com/v1/truthsocial/webhook You'll need to include your API key in a x-api-key header Then you'll need to set up a url that we can POST a response to once we have a new Trump truth social post. So the the body of the POST request should include your webhook url Feel free to test that it works by just simply making a GET request to this url: https://api.scrapecreators.com/v1/truthsocial/webhook/test Then, whenever Trump posts something new, you'll get a response to https://your-webhook-url.com/recieve-scrape-creators-webhook that will look like this: Why Not Just Scrape or Monitor Yourself? You can! Feel free to use our unofficial Truth Social API to do that. Try It Now This webhook has one job: Let you know , faster than anyone else, when Trump posts. Whether you’re automating trades, managing portfolio risk, or just watching political signals, you’ll never miss a post again. Sign up here to get access.
Track Influencers and Analyze Instagram Bios at Scale
Most people don’t realize this, but Instagram actually does offer an official API. The catch? You have to get your app reviewed and approved, have users log in and authorize your app, and deal with strict rate limits. Oh, and good luck if you’re just trying to look at public info like bios, follower counts, or recent posts, most of that isn’t even accessible unless the user authenticates. That’s why I built something better. The Scrape Creators Instagram Profile API gives you real-time access to public Instagram data, no login, no approval process, no rate limits. You give it a user's handle, it gives you the info you actually want. What You Get with One Call Here’s what the API returns when you pass in an Instagram handle: Bio text Follower count Following count Links Profile image URL Last 12 posts (with captions and media URLs) Related accounts (when Instagram suggests them) No friction. Just public facing profile data, at scale. Why This Works (And Keeps Working) We don’t rely on hacky workarounds like headless browsers or logging into fake accounts. Instead, this API pulls directly from Instagram’s public facing endpoints. That means: Real-time results: every time you call it, you’re getting fresh info No rate limits: make as many concurrent calls you want Sustainable: no sketchy tactics that get shut down in 3 months It’s built for devs, SaaS tools, marketers, anyone who needs creator intel, fast and reliably. Use Cases: How People Are Using It You can plug this API into a bunch of different workflows: Build internal influencer databases for outreach Enrich lead lists with bio info and follower counts Monitor creators over time to track growth or posting habits Discover related creators in a niche using Instagram’s “related profiles” Pull recent posts to analyze content themes or brand collabs Whether you’re a marketing agency, a B2B SaaS product, or just running your own research workflow, this unlocks a ton of value with very little setup. How to Use It You can either use this in code, or no code. Check out the instructions below if you are using code. If you are using n8n, check out our integration. Sign up for an API key View the documentation Find someone's IG page, in this case lets use Zucks. His handle is zuck Make a GET request to https://api.scrapecreators.com/v1/instagram/profile?handle=zuck Make sure you include your API key in a x-api-key And that's it! The response will look something like: Try It Out 👉 View the Docs 👉 Test the Endpoint Live 👉 Start for Free No upfront payment required. Pay as you go, scale as you need.
How to Spy on Your Competitors’ Facebook Ads with One API Call
Introduction If you're running Facebook ads, one of the easiest wins is just seeing what your competitors are doing. And yeah, you can go scroll through Meta’s Ad Library manually… but that sucks. It’s slow, clunky, and impossible to scale if you’re checking more than one or two pages. What you actually want is an API that just gives you all the ads, clean JSON, no login, no BS. That’s where the Scrape Creators Meta Ad Library API comes in. It lets you pull all the ads for any Facebook Page, even your competitors, in a few lines of code. Or you can use our Apify or n8n integrations. Let me walk you through what it gives you and how people are using it. Why It’s Worth Looking at Your Competitors’ Ads If you’re running ads and not keeping tabs on other brands in your space, you’re missing out. It’s not about copying, it’s about understanding: What kinds of offers are people pushing? What angles are they testing? How much are they spending (based on volume or how long the ads have been running)? What’s working right now in your niche? You can get a bunch of ideas for your own ads just by seeing what other brands are paying to promote. Especially if you’re doing DTC or working with clients. What You Can Actually See in the Facebook Ad Library Meta shows a lot of info in the Ad Library, most people just don’t realize how much you can get: Text + visuals (image or video) Start date (when the ad launched) Whether it’s still active Which countries it’s running in Page name and ID Categories (like political, credit, housing, etc.) It’s all public. Meta makes this data available for transparency reasons, but that doesn’t mean you have to sit there and click through it manually. The Problem with Meta’s Official API Meta has an official Ad Library API, but using it is kind of a pain: You need a developer account and access tokens You have to get app approval You’ll run into rate limits You can’t search all ad types And honestly, it’s just not that easy to use If you’re just trying to get ad data for research or to keep an eye on competitors, you probably don’t want to jump through all those hoops. A Better Way: The Scrape Creators Meta Ad Library API This API makes it dead simple to pull ad data for any Facebook Page, even if you don’t own it. Here’s what it does for you: Easy to sign up and get started No rate limits Clean JSON response Supports pagination if there are a ton of ads Works with cURL, JS, Python, whatever you want Or no code tools like Apify, n8n How It Works (Example) Let’s say you want to see what ads lululemon is running in the US right now. First sign up for an API key. You’d just hit the API like this: GET https://api.scrapecreators.com/v1/facebook/adLibrary/company/ads?companyName=lululemon&country=US&trim=true Make sure you include your API key in a x-api-key header. Response (simplified) If there are a lot of ads, you just paginate with the cursor. Easy. Check out the full documentation here: https://docs.scrapecreators.com/v1/facebook/adLibrary/company/ads Real World Use Cases Here’s how people are using it: DTC brands: tracking competitors messaging and product pushes Agencies: pulling ads for pitch decks or client reporting Growth hackers: studying ad angles before launching something new Political orgs: monitoring election ad spend AI tools: training on ad creative + messaging at scale Whether you’re doing manual research or building something automated, this saves a ton of time.
How to Find Out If a TikTok User Has a TikTok Shop
If you're targeting TikTok creators for outreach, sales, or influencer marketing, one of the most valuable data points you can get is whether they have a TikTok Shop. And now, with Scrape Creators, you can instantly tell if any TikTok user has a shop, plus get their seller_id for further research. Why This Matters TikTok Shop is where creators monetize directly. If someone has a shop, they're already trying to sell, which makes them an ideal target for: Cold outreach: these users are warm leads eCommerce service providers: agencies, UGC creators, designers Brand partnerships: partner with creators who’ve already sold something Market research: see who’s entering the space and what they’re selling How It Works: The Scrape Creators Endpoint We made it dead simple. Sign up for an API Key: https://app.scrapecreators.com/ Check out the endpoint in the documentation. Make a GET request to: https://api.scrapecreators.com/v1/tiktok/user/is-shop And you need a header called x-api-key, and the value will be your API key. The response will look like this: No headless browser. No proxies. No captcha headaches. It just works. Why Use Scrape Creators? Blazing fast API. Most responses return in 3 seconds or less No scraping headaches, we handle all that for you Pay-as-you-go pricing Use our Apify or n8n Works at scale
How To Tell If a TikTok Video Is an Ad (And Why It Matters for Your Brand)
Why This Matters: Ad Status = Performance Signal If a brand takes a creator’s TikTok and turns it into an ad, it’s a signal that the video performed well, and that the creator likely drove real results. For marketers, influencer agencies, and competitive researchers, identifying which TikToks are ads can help you: Find high-performing creators worth working with Discover creative styles that resonate enough to get paid promotion Track brand partnerships before they’re obvious Reverse-engineer what’s converting across your industry How To Check if a TikTok Video Is an Ad TikTok doesn't display ad status in the UI, but if you're savvy you can still find it. The raw HTML of a video page contains a field called isAd. If this is set to true, the video was promoted as a paid advertisement. Here’s how to find it: Visit the TikTok video URL in a browser (e.g., https://www.tiktok.com/@noah.rolette/video/7376380452136914218) Right-click and click Inspect Go to the Network Tab Refresh the page Click on the first request, which should be the HTML document In the Response tab, search (Cmd+F or Ctrl+F) for isAd You’ll see something like this inside a JSON blob: Scrape Creators If you don't want to build and manage your own scrapers to get this information, consider using Scrape Creators, which makes is super simple to get this information with a simple API call. (On Scrape Creators the field is actually is_ad. In addition to is_ad you can also see if the video is an affiliate for a product, get the images if the tiktok is a photo carousel, get the raw video without the watermark, and much more. And get access 100 other API's for Instagram, LinkedIn, YouTube, Twitter, TruthSocial, etc. All for a pay as you go plan, no monthly subscription required. And if you don't code, we also have integrations with n8n and Apify 🙌
How To Tell if a TikTok Video Is Promoting a TikTok Shop Product (With Examples + API Tips)
TikTok has become more than just a place for trends and dances, it’s now a full-blown e-commerce platform. With TikTok Shop, creators can earn commissions by promoting products directly in their videos. But how do you actually **tell** if a TikTok video is promoting a product through TikTok Shop? This post will show you exactly how to spot those videos both **visually** and **programmatically**, so you can analyze, track, or build tools around TikTok Shop content. Why this matters Identifying TikTok Shop videos is useful for: Marketers looking for creators who are actively promoting products Brands scouting for potential influencer partnerships Devs scraping TikTok to track product trends or build affiliate dashboards Analysts measuring which content converts into sales And they're actually pretty easy to spot. There’s one clear signal that gives them away. The Key Indicator: "Eligible for Commission" If you visit this video: https://www.tiktok.com/@thecutechiro_/video/7433747950766951723, you'll see that on the web, at the bottom of the video it will say "Eligible for commission" Check out how it looks below On the TikTok mobile app it will actually say "Creator earns commission" or "Commission paid" How it looks on mobile: How to Get This Info TikTok does not expose this data in their public API. But if you want this field at scale, you can get it using Scrape Creators, a powerful API that returns a is_eligible_for_commission field as part of every TikTok video object. Specifically you'll want to use the TikTok Profile Videos endpoint to get the users videos, or if you have an individual video you want to check, use the TikTok Video Info endpoint. Example response (using the tiktok from above) Scrape Creators helps you find videos that are monetized, allowing you to: Track product trends Build TikTok affiliate dashboards Discover top-performing promotional content Real Use Cases Affiliate Discovery: Find creators actively promoting TikTok Shop products Trend Analysis: Track what types of videos are most often linked to product sales Build Tools: Power search engines, databases, or dashboards for TikTok Shop Partnership Research: Identify affiliate-heavy creators for collabs Final Thoughts Knowing how to spot TikTok Shop content gives you a huge edge, whether you're building something, running campaigns, or just trying to understand what’s selling. Look for the “Eligible for commission” tag on videos, or tap into the is_eligible_for_commission field using Scrape Creators.
The Best Truth Social API for Tracking Trump’s Posts
What Is the Truth Social API? If you’re trying to monitor Trump’s posts on Truth Social or build anything programmatic with the platform, you’ve probably noticed something frustrating: there’s no official Truth Social API. That’s where Scrape Creators comes in. We built an unofficial Truth Social API that gives you real-time access to posts, perfect for alerts, automations, dashboards, or just staying ahead of the news cycle. Why No Official Truth Social API Exists Truth Social was never designed with developers in mind. There's no public documentation, no API, and no developer portal. As of now, there’s no official way to pull posts, user timelines, or search content via an API. This is why we built something fast, lightweight, and reliable. How the Scrape Creators Unoffical API Works Our API hits Truth Social directly and delivers clean JSON data in a REST format. We handle all the heavy lifting behind the scenes, rotating proxies, bypassing detection, and making sure the data stays fresh and accurate. You just hit the endpoint, and boom the latest posts are yours. Use Case: Monitor Trump Posts Want to get notified the second Trump posts something? After you sign up for an API key, here's how you would do that: API Features REST Endpoint Simple, fast GET requests with optional filters by username or date. Webhook Support (Coming Soon) Soon you’ll be able to register a URL and get new posts pushed to you in real time, no polling required. Clean JSON Response No HTML parsing, no fluff, just structured, developer-friendly data. Pay-As-You-Go Only pay for what you use. No monthly minimums, no surprise charges. No Rate Limits We don’t throttle. Use it as fast as your infrastructure allows.
How To Scrape Images from TikTok Slideshow
Why Slideshows? TikTok photo carousels, or slideshows, are helping creators get millions of views, with relatively little effort. If you've ever wanted to scrape the images from a TikTok, here's how you do it: How to: Let's take this account as an example: https://www.tiktok.com/@mens.guidance All of their posts are photo carousels and some of their posts are popping off. First sign up for this product named Scape Creators, it grabs public social media data in real time. It only take a couple clicks to sign up. Let's use this post as a example: https://www.tiktok.com/@mens.guidance/photo/7467226363544554770 We want to grab the images from this post. The endpoint we will be hitting will be TikTok Video Endpoint (docs link) You can test it out using the playground. Pro tip, enable trim to trim the response to make it a little more manageable. Look for the key `image_post_info`. That will have an array called `images` And we actually want to go into the `thumbnail` key to get the image that doesn't have a watermark. Key into the first url in `url_list`, and we have the url of the image! Go ahead and visit that url to verify. To get all the images, simply loop over the images array! In code it would look like this: We are gonna use JavaScript (Node.js) for this example. I'm using the http client library called axios here, but feel free to use the one you want. And that's it! Those url's will expire after a few hours, so if you want to save them to view later, you will want to download them and then upload them to a cloud storage provider like supabase storage, Amazon S3, or Cloudinary, etc. In node.js you would fs to download the image. Also, if you don't want to do this for each individual video, you can use Scrape Creators Profile Videos endpoint to grab the user's first 20 tiktoks (or however many you want), and then loop through all their tiktoks and grab the images (or videos) from all of them.
How to scrape YouTube transcripts with node.js in 2025
I’m going to show you how to scrape YouTube transcripts in node.js, but the technique can be used for any programming language. If you are just looking for a pre-built API, check out the scrape creators YouTube Transcript API. Scrape Creators also has transcript API’s for TikTok, Instagram, Facebook, and Twitter. Ok first, go to the youtube search page. In this case, I am going to search for “Charles Barkley Jussie Smollett” Next we want to see if we can find any API’s that YouTube is using to fetch the video and hopefully transcript, so open up the dev tools by Right clicking > Inspect Element Then go to the “Network Tab” To make things easier for us, filter by “Fetch/XHR” Now click on any video you want, and observe the requests. Notice the route: “next?prettyPrint=false” Click on that route and check out the Response. If you start searching for the video title or views, you’ll see them in this response: So cool, looks like we found the endpoint that YouTube is using to fetch video details. Now we want to actually call it in Node.js. Go to the next?prettyPrint=false endpoint and right click to Copy as fetch (Node.js) Make sure you have node-fetch installed with npm install node-fetch We’re going to be using async/await, so your code should look something like this: If you make the request, it should be successful. Whoohoo! 🥳 Nice job. But, we don’t want to fetch the same video over and over again, we want to dynamically fetch different videos. So if you notice in the request payload, there is a “videoId” field. And that’s pretty convenient for us, because it means we can just pass a different “videoId” and get the videos details. You can find the videoId easily because its always in the query params of the video. For example: https://www.youtube.com/watch?v=Y2Ah_DFr8cw So just make sure to pass whatever videoId you want to get, and that should be it. You can get rid of that params key. Cool, now how do we get the transcript? Well lets first see how YouTube is getting the transcript. Click the “...more” in the video description Clear the Network Requests so we can get an easier view of what happens when we click on the Show Transcript button Click on the Show transcript button Now they fire off the “get_transcript” endpoint, which doesn’t take a genius to figure out that that’s how they are getting the transcript. Click on the response, and you can see that the transcript is there in nice JSON for us. Excellent. Now lets look at how they’re calling it. So if you go to the Payload tab you’ll see an “externalVideoId”, which you can see is just the videoId, and there is this “params” value, which we did not need to fetch the video details, but spoiler alert, we will need it here. So where does it come from? Well, if you still have the video details response available, search for “getTranscriptEndpoint” If you copy the params value and search for it in the video details response, you’ll see that is identical to the params value YouTube is using to get the transcript 🙌 So now, to actually call the endpoint to get the transcripts, Copy the get_transcript request like we did for the video details request. Then make sure to pass the externalVideoId and the getTranscriptEndpoint param from the video details endpoint, and thats it!
Instagram Scraping in 2025: The Workarounds That Still Work
First, for all the lawyers out there reading this, we're only getting *public* data, so just relax. Aight lets get into it: Now, if you try to scrape via the browser, you're already cooked. Using tools like puppeteer or selenium are easy to detect. If you want to find the API that gets someone's public profile (their follower count, bio, links, etc), do this: Visit my IG page (adrianhorning) Next, we need to monitor the network requests as they're happening> Open the dev tools with right click > Inspect > Network (filter by Fetch/XHR) Scroll to the bottom, to the "Related Accounts"> Click on one Observe the requests Notice one that is named "web_profile_info"? That is probably the one we need 😅 Next, the http client you use is super important. Use got-scraping from Apify. It is incredible at getting around stuff. And lastly, you'll need residential proxies. There are a lot of providers out there, I have my personal favorites. The just pretty much do that for all the public information you want. Sometimes you need to include the headers that they are, so pay attention to that. And to get additional pages of results, they use cursor based pagination, so just look for "cursor" (most of the time), in the payload. If you want a solution that's already done for you, you can just call my IG profile endpoint, docs here: https://docs.scrapecreators.com/v1/instagram/profile
Instagram API Without Rate Limits: How to Easily Access Public Data
The Instagram Graph API is awesome, but it has a rate limit of 200 calls per hour * the number of users. So if you're doing any kind of volume, this really sucks and is a huge bottleneck for your project. To get around this, you could try scraping Instagram yourself, using proxies, running headless browsers, constantly dodging bans, etc. But unless you want to babysit all that headache 24/7, there’s a much easier way: Scrape Creators gives you direct access to public Instagram data, without rate limits, and it's really easy to use. Here’s how easy it is to pull public Instagram data using Scrape Creators: Example: Get Public Profile Data Let’s say you want to pull public info about an Instagram user like nike. You want some basic information like bio, website, follower count, recent posts, etc. By the way the documentation for this api is here: Instagram Profile Endpoint All you need to do is make a GET request to /v1/instagram/profile like so: To get an API key, just sign up at app.scrapecreators.com (free trial, no credit card required) That will return something like this, which has that basic information, and even the users recent posts also: If you want to get just the posts of the user, use the /v2/instagram/user/posts endpoint. And that's it! How to Get Started Sign up for a Scrape Creators account Get your API key Start making requests You can check out the full API docs here. If you want reliable, no-drama access to public Instagram data, Scrape Creators makes it stupid simple. Sign up here and get started.
How to scrape the Meta Ad Library
The Meta Ad Library is a goldmine for marketers, researchers, and growth, but it wasn’t built with automation in mind. That’s why I built the Scrape Creators API to pull real-time Meta Ad Library data using simple HTTP requests. In this post, I’ll show you exactly how to do that using **JavaScript with Axios**. But, you can use any no-code platform also like Clay, Make or Zapier. Anything where you can make API calls from. You’ll also find: - A video walkthrough on YouTube - A working code example on GitHub - Live demos for searching ads, finding companies, and more 🎥 Watch the Video 👉 Click here to watch on YouTube 🧠 What You'll Learn - How to search for ads by keyword - How to look up a company page - How to get all ads from a specific advertiser - How to retrieve a single ad by ID - Copy/paste-ready Axios code examples 🛠 Setup Before you start: - Sign up at [Scrape Creators](https://scrapecreators.com) and grab your API key - If you are using JavaScript (Node.js), install Axios in your project: 🔍 Search Ads by Keyword You know how you can search for ads by a keyword in the Meta Ad Library? Well, you can do the same thing programmatically! All you have to do is call the Search Ads endpoint with a `query` parameter, like so: And you can even search by Ad Type, Country, Status, and Start and End Date! The response will look something like this. Here are some of the valuable fields that will be returned: `ad_archive_id` -> this is the ad id `start_date_string` `end_date_string` `publisher_platform` -> to know if the ad is running on Instagram, Facebook, etc `snapshot` -> This is where all the ad copy and variations will be `body` -> body of the ad `cta_text` `link_url` -> where the user goes after they click on the ad `page_id` -> The company's ad library page id `videos` `images` And if you want more pages of results, just pass the `cursor` that is returned. Get Individual Ad Details Lets say we want to get the ad detail of a single ad. For that we just need the Library ID, in this case 1755312555404167. You will get the same details as above, including start_date_string, end_date_string, publisher_platform, snapshot, cta_text, link, etc. Search for a company's ads Now lets say we wanted to just get the ads from lululemon. First we need the lululemon page id. If you already know it, great, hold onto it. If you don't, you can find it by using this endpoint: That will return a `searchResults` array. You want to key into the first search result and return the page_id, like so: Then you can use that to fetch their ads using this endpoint: And the response will almost be identical to the Search endpoint. If you want additional pages, a `cursor` will be returned, and you need to pass that to subsequent requests like so: And that's it! 📦 Get the Full Code Want to see all this code in one file? 👉 View on GitHub ✅ Ready to Start? Create your free API key at Scrape Creators and start pulling Meta Ad Library data into your app. Questions or feedback? Hit me up at adrian@thewebscrapingguy.com Other APIs If you're looking to scrape other Ad Libraries like the LinkedIn Ad Library, or Google Ad Transparency Center, check out Scrape Creators. Also, if you want to scrape Instagram, YouTube, TikTok, Twitter, and more, you can also use scrape creators for that too!