Tawhiri/noaa wind data - Revision history

EllieClifford: Changed redirect target from Tāwhirimātea to Tāwhirimātea#Idea: Getting data from the NOAA

2021-04-29T21:23:10Z

Changed redirect target from Tāwhirimātea to Tāwhirimātea#Idea: Getting data from the NOAA

← Older revision		Revision as of 21:23, 29 April 2021
Line 1:		Line 1:
	#REDIRECT [[Tāwhirimātea]]		#REDIRECT [[Tāwhirimātea#Idea: Getting data from the NOAA]]

EllieClifford: Redirected page to Tāwhirimātea

2021-04-29T18:04:39Z

Redirected page to Tāwhirimātea

← Older revision		Revision as of 18:04, 29 April 2021
Line 1:		Line 1:
	[[~~Category:Deleted~~]]		#REDIRECT [[Tāwhirimātea]]

EllieClifford: "Remove old Tawhiri pages"

2021-04-29T17:52:14Z

"Remove old Tawhiri pages"

← Older revision		Revision as of 17:52, 29 April 2021
Line 1:		Line 1:
	~~== Idea: Getting data from the NOAA ==~~		[[Category:Deleted]]

	~~Currently, we use PyDAP and talk to a NOAA Java OPENDAP server.~~

	~~This works well ish, but:~~

	* Often, just after dataset publication, we get "Server error 0: "/gfs/gfs20121225/gfs_06z is not an available dataset" - my unconfirmed theory is that of the two nodes on the round-robin DNS, one gets the dataset first. This then tells predict.py that it's available, and the next request goes to the other node, producing the above error
	* It's a bit slow
	* Caching is a [~~https~~:~~//github.com/jonsowman/cusf-standalone-predictor/blob/master/predict.py#L29-38 hack~~]

	~~Idea: drop PyDAP and write our own client~~

	~~==== Speaking OPENDAP ====~~

	On the one hand, we need not speak the whole of the OPENDAP protocol. This would simplify things, require less requests to get the format of the data and so on. We know what format the wind data is in, and if it did change, then it would affect other areas of the predictor that expect a certain format anyway.

	~~However: when it changes, if the expected form of wind data is hardcoded deep in the code, it may be a complete pain to change.~~

	~~==== DNS abuse ====~~

	We could perform a manual DNS look up, and get a list of IP addresses that we can talk to, and then maintain some information on each node (if it's up, which datasets it has, ...). Then if we want some data, we could hit all nodes simultaneously (for different portions of data) assuming that the limiting factor is not our network speed.

	~~==== Chunk sizes ====~~

	This requires more investigation; the NOAA OPENDAP servers have some strange behaviour depending on how much data you request in one go. Larger chunks seem to require it to think for a little while before it starts sending data over the network.

	~~If we download larger chunks, more data that we don't need is downloaded - wasted time Downloading lots of small chunks will slow us down due to the overhead of many small requests.~~

EllieClifford: "Add old wiki"

2021-04-19T18:58:43Z

"Add old wiki"

New page

== Idea: Getting data from the NOAA ==

Currently, we use PyDAP and talk to a NOAA Java OPENDAP server.

This works well ish, but:

* Often, just after dataset publication, we get "Server error 0: "/gfs/gfs20121225/gfs_06z is not an available dataset" - my unconfirmed theory is that of the two nodes on the round-robin DNS, one gets the dataset first. This then tells predict.py that it's available, and the next request goes to the other node, producing the above error
* It's a bit slow
* Caching is a [https://github.com/jonsowman/cusf-standalone-predictor/blob/master/predict.py#L29-38 hack]

Idea: drop PyDAP and write our own client

==== Speaking OPENDAP ====

On the one hand, we need not speak the whole of the OPENDAP protocol. This would simplify things, require less requests to get the format of the data and so on. We know what format the wind data is in, and if it did change, then it would affect other areas of the predictor that expect a certain format anyway.

However: when it changes, if the expected form of wind data is hardcoded deep in the code, it may be a complete pain to change.

==== DNS abuse ====

We could perform a manual DNS look up, and get a list of IP addresses that we can talk to, and then maintain some information on each node (if it's up, which datasets it has, ...). Then if we want some data, we could hit all nodes simultaneously (for different portions of data) assuming that the limiting factor is not our network speed.

==== Chunk sizes ====

This requires more investigation; the NOAA OPENDAP servers have some strange behaviour depending on how much data you request in one go. Larger chunks seem to require it to think for a little while before it starts sending data over the network.

If we download larger chunks, more data that we don't need is downloaded - wasted time Downloading lots of small chunks will slow us down due to the overhead of many small requests.