Live scraping html from a website using vue or javascript

Question

I need to scrape data from a website after the data is loaded,

There is a process that running in a loop between 1 to 200, I need to get the result after the process arrived to 200 on the HTML itself.

1. Is it possible? I know to work with cheerio but I didn't find the way to handle how to catch it live after the process ended.

2. How can I ignore CORS provision when I try to request using axios http get request.

I didn't understand how to use proxy in vue.config.js. I didn't find the full explanation for how to use it.

Here is my code, of course I changed some data for my safety:

  <div class="hello">
    <h1>{{ msg }}</h1>
    <ul>
      <li v-for="(message, index) in messages" :key="index">
        <b>{{ messages.ip }} [{{ message.type }}]:</b>
        {{ message.blocked }}
      </li>
    </ul>
  </div>
</template>

<script>
import axios from "axios";
import cheerio from "cheerio";
export default {
  name: "ScrapIP",
  props: {
    msg: String,
    messages: Array
  },
  methods: {
    fetchUrl() {
      for (let i = 0; i < 5; i++) {
        const ip = "192.168.0." + i;
        const url = "http://xxx/yyy.org/lookup/" + ip + ".html";
        axios.get(url).then(response => {
          const $ = cheerio.load(response.data);

          setTimeout(() => {
            if ($(".global_data_cnt_DNSBLBlacklistTest").text() == 243) {
              this.messages.push({
                ip: ip,
                type: "Blacklist Test",
                blocked: $(".global_data_cnt_DNSBLBlacklistTest").text()
              });
            }
          }, 10000);
        });
      }
    }
  },
  created() {
    this.fetchUrl();
  }
};
</script>

How can I ignore CORS. You cant. Make a simple serverside endpoint and pass it the ip, scrape it and return the result. — Lawrence Cherone
– Lawrence Cherone, Commented Jul 29, 2020 at 17:09
Thanks for the comment Lawrence, the important thing is that it must wait for the HTML/js process on the scraped website to be done first, and than present it on thr client, i thought there is a way to do that with cheerio or something like that. — Idoshin
– Idoshin, Commented Jul 29, 2020 at 17:23
it would be as vue is reactive, if you want a processing spinner/message, that would be implemented by setting a model like loading = true and once its returned from axios set it to false.. all common stuff — Lawrence Cherone
– Lawrence Cherone, Commented Jul 29, 2020 at 17:54

Korbin Williams · Accepted Answer · 2020-07-29 20:24:28Z

1

I'm not very familiar with cheerio, but puppeteer could be useful for what you're trying to do. It launches an instance of chromium in the background to do the tasks you're telling it to do, so it'd be easier to capture data after the loop. The only downside is speed and it is used in node.js.

answered Jul 29, 2020 at 20:24

Korbin Williams

112 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Idoshin Over a year ago

Yep, im on it right now, try to figure out it with vue.. im faces problem with it so ill try your suggestion to create the node server and work with this way. Many thanks

Collectives™ on Stack Overflow

Live scraping html from a website using vue or javascript

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related