Ignore False PWS data + Feature requests

To give everyone an update:
I’ve been testing how long I could delay updates over to pwsweather and found some issues with that.

What I am working on now: I’m adding a database support to delay just raindata. Meaning that your temperature, pressure and other measurements will be relayed in real-time, but your rainfall data may be D number of minutes delayed (you choose). (meaning that if rain started at 10pm, it may showup starting at 10pm + D minutes, 10:30 for example if Delay is set to 30 minutes).

Before the data is forwarded, I’ll add logic to filter out seemingly false data.

In case I’ll need wufyi users to update their configuration, I will trigger 5 response errors / day (a notification should than go through to your cron-job email account) with an otherwise constant message containing a new URL, just update URL within cron-job (copy+ paste) and you will not get any new errors. API errors will also be limited to 5 / day so that your cron-jobs do not get disabled.

2 Likes

First of all, thanks to @Gene for creating and hosting this script! It has saved me a fair bit of effort.

I’m seeing similar behavior to what is reported above when grabbing data from a nearby weather station. In this case, it is with the temperature and dew point measurements. Early this morning, my PWSWeather site shows a rogue data point of -999 degrees. This is obviously skewing the averages a bit. :slight_smile: This stray data point is not visible on the data table on WUnderground.

The anomaly seems to have occurred around 12:30AM Mountain on 5/8.

See https://www.wunderground.com/personal-weather-station/dashboard?ID=KCOLOVEL127#history
and
https://www.pwsweather.com/obs/WCMS80537.html

I assume this is related to the issue(s) being discussed above?

Thanks!

-Matt

@Gene keep going on this track and you may be taking someone’s job…

Thank you @goku3989 for bringing this to my attention, I will add this to the list of issues to be addressed, stay tuned. I’ve had the debug capture active to deal with the other issues and looked up the actual data that came from WU for the data-point in question:


[details=Raw Data from WU at 12:30AM]
{
“response”: {
“version”:“0.1”,
“termsofService”:“http://www.wunderground.com/weather/api/d/terms.html”,
“features”: {
“conditions”: 1
}
}
, “current_observation”: {
“image”: {
“url”:“http://icons.wxug.com/graphics/wu2/logo_130x80.png”,
“title”:“Weather Underground”,
“link”:“http://www.wunderground.com
},
“display_location”: {
“full”:“Loveland, CO”,
“city”:“Loveland”,
“state”:“CO”,
“state_name”:“Colorado”,
“country”:“US”,
“country_iso3166”:“US”,
“zip”:“80537”,
“magic”:“1”,
“wmo”:“99999”,
“latitude”:“40.384724”,
“longitude”:"-105.113541",
“elevation”:“1528.9”
},
“observation_location”: {
“full”:“Walt Clark Middle School, Loveland, Colorado”,
“city”:“Walt Clark Middle School, Loveland”,
“state”:“Colorado”,
“country”:“US”,
“country_iso3166”:“US”,
“latitude”:“40.384724”,
“longitude”:"-105.113541",
“elevation”:“4985 ft”
},
“estimated”: {
},
“station_id”:“KCOLOVEL127”,
“observation_time”:“Last Updated on May 8, 12:30 AM MDT”,
“observation_time_rfc822”:“Mon, 08 May 2017 00:30:09 -0600”,
“observation_epoch”:“1494225009”,
“local_time_rfc822”:“Mon, 08 May 2017 00:30:29 -0600”,
“local_epoch”:“1494225029”,
“local_tz_short”:“MDT”,
“local_tz_long”:“America/Denver”,
“local_tz_offset”:"-0600",
“weather”:“Clear”,
“temperature_string”:"-9999.0 F (-999.0 C)",
“temp_f”:-9999.0,
“temp_c”:-999.0,
“relative_humidity”:"-999%",
“wind_string”:“Calm”,
“wind_dir”:“North”,
“wind_degrees”:-9999,
“wind_mph”:-9999.0,
“wind_gust_mph”:0,
“wind_kph”:0,
“wind_gust_kph”:0,
“pressure_mb”:“1008”,
“pressure_in”:“29.77”,
“pressure_trend”:"+",
“dewpoint_string”:"-9999 F (0 C)",
“dewpoint_f”:-9999,
“dewpoint_c”:0,
“heat_index_string”:“NA”,
“heat_index_f”:“NA”,
“heat_index_c”:“NA”,
“windchill_string”:“NA”,
“windchill_f”:“NA”,
“windchill_c”:“NA”,
“feelslike_string”:"-9999.0 F (-999.0 C)",
“feelslike_f”:"-9999.0",
“feelslike_c”:"-999.0",
“visibility_mi”:“10.0”,
“visibility_km”:“16.1”,
“solarradiation”:"–",
“UV”:“0”,“precip_1hr_string”:“0.00 in ( 0 mm)”,
“precip_1hr_in”:“0.00”,
“precip_1hr_metric”:" 0",
“precip_today_string”:“0.00 in (0 mm)”,
“precip_today_in”:“0.00”,
“precip_today_metric”:“0”,
“icon”:“clear”,
“icon_url”:“http://icons.wxug.com/i/c/k/nt_clear.gif”,
“forecast_url”:“http://www.wunderground.com/US/CO/Loveland.html”,
“history_url”:“http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCOLOVEL127”,
“ob_url”:“http://www.wunderground.com/cgi-bin/findweather/getForecast?query=40.384724,-105.113541”,
“nowcast”:""
}
}[/details]


@mckynzee Don’t worry, it’s not easy to afford my services full time :wink:

2 Likes

Thanks, @Gene! If I were running the script on my own box, I might be tempted to do some value clamping based on some reasonable assumptions about this area. Wish I knew why WUnderground is reporting that rogue data point, yet not showing it on their own website. Like I think you mentioned before, they must do some post processing to filter stuff like that out.

-Matt

1 Like

Ah, looks like it happened again–just a little after 3:00PM Mountain. This time, I’m actually seeing a mostly blanked data line in the WUnderground data; not sure if it will get corrected on their side later, though.

For what it’s worth, that particular area just got hit with a barrage of hail right around that time. That might have temporarily borked the weather station.

-Matt

Looking at the data, I do not anticipate any issues about filtering this kind of data in the future, there are plenty of clearly wrong data to invalidate the whole data packet. I may expedite the fix for this particular problem since it looks pretty straight forward. Thank you @goku3989 for sharing your findings

P.S. Looks like WU didn’t yet filter out every bad data-point, will be interesting to check back alter to see if it is still there tomorrow.

2 Likes

Update: As of now, the negative readings should be filtered out of the data updates to PWS weather.
I’ve also improved error handling, so initial setup should be clearer. The site will now handle errors from others and help you figure out which of the parameters has an issue (such as that your PWS weather password is wrong in case you need to correct “pws” within the URL).
I’ve stripped HTML from the successful output so that it would show up clearer within cron-job output.
I’ve also added information on how old the data was during the transfer.

I’m still working on testing / debugging other false data filtering, current expectation that it will be ready next weekend.

As always, if you are using wufyi.com, updates were applied automatically. You do not have to do anything.

Cheers,
Gene

4 Likes

Excellent–this should make those troublesome spikes go away! Thanks, @Gene!

I took a peek at your GitHub. I wonder, for a possible future adjustment, it might make sense to allow some negative tolerance for values where is makes sense (e.g. the temperature values). Granted, it probably doesn’t have much impact with respect to sane times of year when we’d be running our Rachio devices, but rather in the spirit of an accurate data migration from WU->PWS. Indeed, we do get some subzero overnight lows here in Colorado in the winter! :wink:

Anyway, good stuff–thanks again!

-Matt

You are absolutely right, update is going up immediately. Sorry, being in Florida… :sunny:

Update is live on wufyi.
@goku3989 what do you think of the changes (link)?

2 Likes

@Gene --those changes seem reasonable to me. I think the ‘bad data’ temps usually get reported as -999, right? So using -459 as a floor seems pretty safe. I’ll keep an eye out to see if any spikes pop up going forward.

Cheers!

-Matt

Actually, something must have gone wrong. wufyi.com is now complaining about my PWS password being incorrect. Looks like the updates stopped around 8:10PM Mountain (I have my cron job set to run every 5 minutes).

I just double-checked and I’m able to sign in just fine.

Cheers,

-Matt

Ah, it looks like you are URL encoding the password. That would mess up the password in my particular case.

-Matt

Thanks, fixed. Next time I will not rush out the fix without testing it first. I guess it’s late :disappointed_relieved:

2 Likes

Brilliant–looks like that did the trick! No worries–I don’t mind helping with some QA. You are saving me having to write my own version of this in Python (and host it, too). :slight_smile:

-Matt

1 Like

As many of you may have noticed, I’m slowly rolling out changes I’ve been working on for wufyi.com.

Right now these changes are mostly visible via a better error reporting, though as I’ve started the roll-out I’ve run into unexpected issues (such as WU will sometimes report that API key is bad, even though it is not).

What this means is that many of you may have gotten an email notification from cron-job.org that your updates have failed, most of them are temporary (though I have about 3 users who are using WU stations which have gone offline), and I’ve implemented changes to filter those out (prevent unnecessary notifications).

Pardon the dust as improvements are made. I’ll post a followup after I’ve rolled out all of the changes and had time to test them. Some of the features to look forward to:

  1. Backup stations (if primary went offline, system will automatically start pulling data from the backup).
  2. Calibration (you can specify adjustments you would like to make to any parameter transferred).
  3. Better, step by step, setup procedure.
  4. Support for faster updates (run your cron-job every minute, wufyi will take care not to overtax your API).
  5. Almost forgot, better rain filtering! :wink:

Some of the features already live:

  1. Better error reporting (to cron-job, so you get an email notifications without disabling your cron-job)
  2. Detection when station went offline (rather then just filtering out “old” data which WU likes to provide sometimes)
  3. Added support for Solar Radiation (for stations which report it) and UV data. You will not see them at pwsweather, but the data is being transferred, if it is available.

Cheers,
Gene :cheers:

2 Likes

What do I need to do to get to use WU PWSs? I have Android, iPad and PC access capabilities. There is a station literally right down the road that is on WU.

I’ve setup a free service, which you can use, at wufyi.com.

First check that WU station you are considering provides good data. More on this here (link).
Then you can setup wufyi.com to transfer data. More on that here (link). Sorry updated setup is not ready yet.
When setting up pwsweather station (which would make WU data available to rachio), be careful about GPS coordinates when setting up pws weather station. Here is a guide (link).

That is about it. Setup takes about 10 minutes and you don’t need anything else.

Cheers,
Gene

1 Like

I swear it seems that WU is purposefully tweaking their API just to mess with data transfers.

Where is before all of the observation time-stamps where reported in UTC, now several are using a time zone offset.

I am aware of issues which are effecting a few of you. I’m working on a fix for it, if your station shows up offline due to a high delta even though it is reporting to weather underground, it will be fixed soon (famous last words).

I may need to start a new topic soon, just for the S :interrobang:T the Underground API puts me though…

Going to get some rest,
Gene

1 Like

I see that this morning, WU has gone back to reporting in UTC as normal… What are they up to over there?

1 Like