Skip to main content

Web Scraper Tool

The Web Scraper Tool allows users to scrape data from a specified URL.

Steps to Use the Web Scraper Tool​

Step 1: Access the Tool​

  • Click on Add User Step.
  • From the available options, select the Web Scraper Tool.

Step 2: Input the URL​

You need to provide the URL of the webpage you want to scrape. You have two options for entering the URL:

  • Manual Entry: Type the URL of the webpage directly into the input field.
  • User Input Variable: Use a user input variable to dynamically provide the URL. To use input variables within your web scraper, use the syntax {{variable_name}} to access the value of an input variable. For example, if there is an input parameter named “webpageURL”, you can access its value using {{webpageURL}}.

Step 3: Configure Guardrails Validators (Optional)​

You can set up guardrails validators to ensure the scraped data meets certain criteria. Set up validators based on your specific needs. This may include rules for filtering out irrelevant data, ensuring data quality, or other criteria.

Step 4: Perform the Scraping​

Once you have entered the URL and configured any necessary validators, initiate the web scraping process:

  1. Run the tool: Click the Run button to start the web scraping process.
  2. Results: The tool will process your request and scrape data from the specified URL.

Step 5: View the Output​

The scraped data will be displayed in the results section. The results will be presented in a structured format, often in plain text or JSON, depending on the nature of the output.

Step 6: Utilize the Scraped Data​

You can now use the scraped data for further processing, analysis, or integration into your application.

Tips for Effective Use​

  • Correct URLs: Ensure the URL is correct and accessible to avoid errors during the scraping process.
  • Effective Guardrails: Apply guardrails validators to narrow down the scraped data to the most relevant and high-quality content.
  • Review Results: Carefully review the scraped data to ensure it meets your requirements.

If you have any questions or need further assistance, please contact the helpdesk([email protected]).