Calgarypuck Forums - The Unofficial Calgary Flames Fan Community

Go Back   Calgarypuck Forums - The Unofficial Calgary Flames Fan Community > Main Forums > The Off Topic Forum > Tech Talk
Register Forum Rules FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Search this Thread
Old 12-09-2016, 06:11 PM   #1
DownhillGoat
Franchise Player
 
DownhillGoat's Avatar
 
Join Date: Jan 2010
Exp:
Default Pulling data from a website

Hoping the CP braintrust can help me out with something.

What I'd *like* to do is track precipitation and temperature (and possibly wind speed) over the course of the year from mountain-forecasts.com.

(example here)

Is it possible to (easily) automate something to pull that information daily into something like an excel sheet? Or would I be stuck grabbing it manually every day?

TIA.
DownhillGoat is offline   Reply With Quote
Old 12-09-2016, 07:31 PM   #2
photon
The new goggles also do nothing.
 
photon's Avatar
 
Join Date: Oct 2001
Location: Calgary
Exp:
Default

What you're talking about is called screen scraping, so that'd be what you'd want to search for. I've always done it myself using some app I wrote, but there's probably free apps or websites to do it for you.

Screen scraping is brittle in that changes to the pages can easily confuse a screen scraper.

You could contact them to see if they could give you some kind of access to the data as well.

Also there's often licenses for data involved, so screen scraping it might be illegal (i.e. they might have a right to display it, but not to give it to someone else). So depending on what you want to use it for this might be an issue or not.

Also consider the load you'd put on a server.. if it's one request a day then that's probably not a big deal, but one a minute might be significant to them. Or setting up a script to go through and get historical data as fast as possible can overload or crash some sites; so try and space out historical requests over a long span.

Getting permission to use the data is a good idea, and might even result in you getting more data or easier access.

EDIT: With Excel, there's probably tutorials on how to use VBA in Excel to screen scrape data off of websites
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
photon is offline   Reply With Quote
The Following User Says Thank You to photon For This Useful Post:
Old 12-09-2016, 08:39 PM   #3
DownhillGoat
Franchise Player
 
DownhillGoat's Avatar
 
Join Date: Jan 2010
Exp:
Default

Quote:
Originally Posted by photon View Post
Also there's often licenses for data involved, so screen scraping it might be illegal (i.e. they might have a right to display it, but not to give it to someone else). So depending on what you want to use it for this might be an issue or not.

Also consider the load you'd put on a server
Purely personal use for my own backcountry pursuits. Not posting it anywhere or sharing. One request a day, nothing historical.

Thanks for the info.
DownhillGoat is offline   Reply With Quote
Old 12-09-2016, 10:13 PM   #4
photon
The new goggles also do nothing.
 
photon's Avatar
 
Join Date: Oct 2001
Location: Calgary
Exp:
Default

Ah ok yeah then I wouldn't worry too much about licensing or anything like that.

A VBA script in Excel would seem like the easiest, unfortunately I've never done it so can't help with that, but searching vba screen scrape excel looks like it comes up with some useful stuff.

http://stackoverflow.com/questions/2...site-using-vba
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
photon is offline   Reply With Quote
The Following User Says Thank You to photon For This Useful Post:
Old 12-09-2016, 11:17 PM   #5
Skaloper
First Line Centre
 
Skaloper's Avatar
 
Join Date: Oct 2008
Location: BC
Exp:
Default

Quote:
Originally Posted by photon View Post
Ah ok yeah then I wouldn't worry too much about licensing or anything like that.

A VBA script in Excel would seem like the easiest, unfortunately I've never done it so can't help with that, but searching vba screen scrape excel looks like it comes up with some useful stuff.

http://stackoverflow.com/questions/2...site-using-vba
A VBA script to pull the data from the website in Excel would probably do the job but it might be a pain to remember to run the script manually or remember to have your computer online when it is set to run.

It probably is easier to generate a file with a webservice hosted in the cloud set to run at a specific time and email it. Then you could have a VBA script in Excel parse out the files after you download them all. The one catch that both methods have is if they update the format of their website it might break the code that generates (parses out the HTML of the Website) the file.
Skaloper is offline   Reply With Quote
The Following User Says Thank You to Skaloper For This Useful Post:
Old 12-10-2016, 12:38 AM   #6
kermitology
It's not easy being green!
 
kermitology's Avatar
 
Join Date: Oct 2001
Location: In the tubes to Vancouver Island
Exp:
Default

Easier is to run a script on your own computer. No need for a hosted solution. Or you could reach out to the website and ask if you could just use their data.
__________________
Who is in charge of this product and why haven't they been fired yet?
kermitology is offline   Reply With Quote
The Following User Says Thank You to kermitology For This Useful Post:
Old 12-10-2016, 11:18 AM   #7
Skaloper
First Line Centre
 
Skaloper's Avatar
 
Join Date: Oct 2008
Location: BC
Exp:
Default

Quote:
Originally Posted by kermitology View Post
Easier is to run a script on your own computer. No need for a hosted solution. Or you could reach out to the website and ask if you could just use their data.
If your computer is always connected to a network and is setup to always be on at the specific time you want the script to run I agree that a hosted solution probably isn't needed. For me though for reliability and because it would cost peanuts I would use a hosted solution.
Skaloper is offline   Reply With Quote
Old 12-11-2016, 10:32 AM   #8
Iceman90
Powerplay Quarterback
 
Iceman90's Avatar
 
Join Date: Apr 2004
Location: Behind the microphone
Exp:
Default

If this site offers an API, it would be very easy to do.
__________________
Fireside Chat - Official Podcast for the C of Red
New Episode Weekly! Listen Now: FiresideChat.ca
Iceman90 is offline   Reply With Quote
Old 12-11-2016, 12:16 PM   #9
DownhillGoat
Franchise Player
 
DownhillGoat's Avatar
 
Join Date: Jan 2010
Exp:
Default

Quote:
Originally Posted by Iceman90 View Post
If this site offers an API
I don't have a clue. Not my area of expertise at all.
DownhillGoat is offline   Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -6. The time now is 04:11 AM.

Calgary Flames
2023-24




Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright Calgarypuck 2021