Glossary
This is a glossary of Unidata LDM-specific acronyms, terms, and phrases. If you don't find what you're
looking for here, try the more general On-line Dictionary of Computing, or the
Internet Users' Glossary (RFC 1392) from which we have
adapted some of these definitions.
A
- actions
- actions
- In the context of the Unidata LDM, one of the possible fates of a
data product when it is read out of the product queue by the
pqact program. If the
feed type and product ID of a data product match a list
maintained by pqact, the associated action is
invoked on the product. An action may have arguments that include substitution of matched subparts of the
product ID. The currently supported actions include FILE, STDIOFILE, PIPE, EXEC, DBFILE, and NOOP.
- AFOS
- An automated system (Automation of Field Operations and Services) used by the National Weather Service to
standardize and expedite meteorological services.
- Alden
- Alden Electronics, Inc., a company that provides meteorological
services.
- ANY
- In the context of the Unidata LDM, the psuedo-feed type that
designates any data feed.
- arrival time
- In the context of the Unidata LDM, old and now deprecated name for
injection time.
- ASOS
- Automated Surface Observing System
- automatic fail-over
- The ability of a server to automatically switch to use an alternate data source if the primary source
fails.
- AWIPS
- Advanced Weather Interactive Processing System, an NWS program, also known as the
"modernization program."
B
- BLKDATA
- The remote procedure call made by a data
source or upstream node to deliver another piece of a large
product.
- broadcast
- A special type of multicast packet that all nodes on the network are always willing
to receive. Also in the Unidata LDM context, refers to the satellite broadcast of
FOS and McIDAS data by Alden.
- BUFR
- Binary Universal Form for the Representation of meteorological data, a WMO standard for a
binary code for the exchange and storage of data. The format is documented in The WMO Manual on Codes; WMO
Publication No. 306; Volume I, Part B; 1988 Edition, plus Supplements 1, 2, and 3.
- bulletins
- bulletin
- A sequence of consecutive bytes within a data stream that comprises a single unit
that can be converted into a data product by supplying the necessary identifying
information.
C
- clients
- client
- A computer system or process that requests a service of another computer system or process (a
server).
- client-server model
- A model for software design in which programs use and provide distributed services. A task is split between
one or more server tasks that accept requests, according to some protocol, for
information or action from (distributed) client tasks. This model allows clients and servers to be placed
independently on nodes in a network. Examples are the name-server/name-resolver relationship in DNS, the
file-server/file-client relationship in NFS and the screen server/client application split in the X Window
System.
- COMINGSOON
- The remote procedure call made by a data
source or upstream node to announce the intent to deliver a
large product in pieces.
- connection-oriented
- The data communication method in which communication proceeds through three well-defined phases: connection
establishment, data transfer, connection release. TCP is a connection-oriented protocol.
- connectionless
- The data communication method in which communication occurs between hosts with no previous setup. Packets
between two hosts may take different routes, as each is independent of the other. UDP is a connectionless
protocol.
D
- daemon
- A program that is not invoked explicitly, but lies dormant waiting for some condition(s) to occur. The idea is
that the perpetrator of the condition need not be aware that a daemon is lurking (though often a program will
commit an action only because it knows that it will implicitly invoke a daemon). Daemons are usually spawned
automatically by the system, and may either live forever or be regenerated at intervals. Unix systems run many
daemons, chiefly to handle requests for services from other hosts on a network. Most of these are now started as
required by a single real daemon, inetd, rather than running continuously. Examples are cron (local timed
command execution), rshd (remote command execution), rlogind and telnetd (remote login), ftpd, nfsd (file
transfer), lpd (printing).
- data feeds
- data feed
- In the context of the Unidata LDM, a sequence of data
products, all of the same feed type, injected into the IDD from a single
data source.
- data products
- data product
- In the context of the Unidata LDM, a unit of data with an associated
feed type, product ID,
injection time, and origin, and
sequence number.
- data relays
- data relay
- In the context of the Unidata LDM, a network node that relays
data products from one or more upstream source nodes to one or more
downstream sink nodes. A relay node may provide data fan-out by receiving data
products from a single upstream source that it then transmits to multiple downstream sinks.
- data sinks
- data sink
- In the context of the Unidata LDM, a remote consumer of
data products that must be transmitted as they arrive.
- data sources
- data source
- In the context of the Unidata LDM, a remote provider of
data products.
- data streams
- data stream
- In the context of the Unidata LDM, a sequence of bulletins all of the same
feed type broadcast from a single data source. Once the
bulletins are converted into data products, the result is called a data feed.
- DCE
- Distributed Computing Environment, an architecture of programming interfaces, conventions, and server
functionalities (e.g. naming, distributed file system, remote procedure
call) for distributing applications transparently across networks of heterogeneous computers promoted and
controlled by the Open Software Foundation (OSF). The Unidata LDM does not use DCE RPCs.
- DDS
- Domestic Data Service, a data feed that is part of the NWS Family
of Services. DDS includes data products of many kinds for the U.S., Canada, Mexico.
- decoders
- decoder
- In the context of the Unidata LDM. a program that converts transmitted
data products into a different form more suitable for use by applications.
Decoders often combine multiple products into a single decoded file.
- DDPLUS
- DD+
- Domestic Data Plus, a data stream provided by Alden merging
the DDS and PPS data streams into one.
- DIFAX
- Digital facsimile maps, a data stream of digitized maps provided by
NWS.
- Disabling an ldmd.conf Entry
- An entry in the file ldmd.conf is disabled if it starts with the comment-character "#".
- downstream
- In the context of the Unidata IDD, the direction of data flow from
data sources to data sinks.
- downstream sink
- see data sink.
- domain name servers
- domain name server
- A server that provides translation of hostnames to IP addresses, as part of the Internet
Domain Name System.
- DNS
- Domain Name System
- A general-purpose distributed, replicated, data query service chiefly used on Internet for translating
hostnames into Internet addresses. DNS can be configured to use a sequence of name servers, based on the domains
in the name being looked for, until a match is found. DNS is a distributed database. It can be queried
interactively using the command nslookup.
- dot address
- Dot address refers to the common notation for IP addresses of the form A.B.C.D; where each letter represents,
in decimal, one byte of a four byte IP address. See also: IP address.
E
- ECMWF
- European Center for Medium-range Weather Forecasting, a source for some model output grids in the
HDS data stream.
- Enabling an ldmd.conf Entry
- An entry in the file ldmd.conf is enabled if it doesn't start with the comment-character
"#".
- event-driven
- A kind of program with a main loop that waits for events to occur. Each event has an associated handler that
is passed the details of the event as soon as it is detected. This is in contrast to a polling
program that frequently wakes up and checks for whether an event has happened yet, or a scheduled
system that periodically wakes up and handles pending events.
- EXP
- In the context of the Unidata LDM, the feed type for experimental
data. This feed type is unassigned, so it can be used for any purpose for data of low interest to the majority
of sites. This is much easier than adding a new feed type, which is technically a change to the protocol and
requires all sites to recompile.
F
- feed sets
- feed set
- In the context of the Unidata LDM, a combination of feed types that results from applying
set union and set difference operations to the basic feed types. For example,
DDS|PPS
designates the set of data products from either the
DDS or PPS data feed.
- feed types
- feed type
- In the context of the Unidata LDM, a category of data products
sharing a name space of product IDs and injected from a single
data source. The feed type is used for coarse selection of data products; finer
selection is provided by patterns matching a subset of product IDs. Primitive feed types supported by the
Unidata LDM include: DDS, IDS, PPS,
HDS, NLDN, WSI, EXP,
MCIDAS, ANY.
- FEEDME
- The remote procedure call made by a data sink or
downstream node to sign on and announce what feed types and
product IDs it wants to be fed.
- FOS
- Family of Services
- A collection of data streams from the National Weather Service, injected into the
IDD by the Alden Corporation under a contract with Unidata. This
currently includes DDS, PPS, IDS, and
HDS data streams.
- FSL
- Forecast Systems Laboratory, a NOAA program that is also a data source of some
experimental IDD data, such as MAPS FSL2 and
ISPAN.
- FSL2
- 60 minute Profiler Data, current source is FSL distributed via IDD.
- FTP
- File transfer protocol, a method of transferring files over TCP/IP networks. Anonymous FTP permits users to
obtain files from a remote system by logging in as anonymous.
G
- Gateway LDM
- The only LDM of an institution or one that serves as a top-level source of data for the institution's internal
LDM network.
- gdbm
- GNU database manager, a freely-available library that supports a simple key-contents database. The
pqact program of the Unidata
LDM can store data products in gdbm databases using the DBPUT
action.
- GDS
- GAI
- Global Atmospherics, Inc., provider of NLDN data.
- GEMPAK
- General Meteorological Package, an analysis and display package developed at NASA Goddard Space Flight Center.
More information is available here.
- Gopher
- Software developed by the University of Minnesota for easily providing and accessing public information on
computers linked by a network. There are two kinds of Gophers: servers, which provide a method of making files
(graphics, video, or text) publicly available, and clients, which are the tools for accessing information on a
Gopher server.
- GRIB
- GRidded Binary, a bit-oriented format approved by the WMO that has compaction features for
efficiently transmitting and storing large volumes of gridded data. A description of the format can be found
here.
- gribtonc
- An LDM in-line decoder, supported by the UPC, that decodes
GRIB products on the HDS data stream from
NMC and ECMWF and writes them into netCDF
files.
H
- HDS
- The feed type for the High-Resolution Data Service, a
data stream that is part of the NWS Family of Services. HDS includes model output
data from models run at the NMC and ECMWF.
- HEREIS
- The remote procedure call made by a
data source or upstream node to deliver a
small product.
- HIYA
- The remote procedure call made by a
data source or upstream node to sign on and announce what
feed types and product IDs it is willing to provide.
- HRS
- See HDS
- HTML
- HyperText Markup Language, the collection of styles, indicated by markup tags, that define the structure of a
World Wide Web document. More information is available
here.
- HTTP
- HyperText Transfer Protocol, the protocol used for the World-Wibe Web (WWW). Extensive
information on the protocol is available
here.
I
- IDD
- Internet Data Distribution, a system for disseminating real-time earth observations via the Internet. Unlike
other systems, which are based on data centers where the information can be accessed, the Unidata IDD is
designed so a university can request that certain data sets be delivered to computers at their site as soon as
they are available from the observing system. Moreover, any site with access to specialized observations of
interest to others can inject the dataset into the IDD for immediate delivery to interested sites.
- idd_monitor
- The idd_monitor is an X-based monitoring tool, using the MOTIF toolkit, that allows a user to monitor up to 14
IDD hosts. It provides audible and visual cues when one or more hosts is either not responding, or is suffering
from excessive packet loss.
- IDS
- The feed type for the International Data Service, a
data stream that is part of the NWS Family of Services. IDS includes international
data products analogous to the domestic products on DDS.
- ingester
- In the context of the Unidata LDM, a program such as
pqing that reads bytes from a data stream and creates
data products.
- injection time
- In the context of the Unidata LDM, time when a data product is
first injected into the IDD, an attribute of the data product that is preserved as it gets
distributed to other IDD nodes.
- IP
- Internet Protocol, the network layer for the TCP/IP protocol suite widely used on
Ethernet networks. IP is a connectionless, best-effort packet switching protocol. It provides packet routing,
fragmentation and re-assembly through the data link layer.
- IP address
- The 32-bit address defined by the Internet Protocol. It is usually represented in dotted decimal
notation.
- ISPAN
- Information Stream Project for AWIPS NOAAport, an experimental data stream available from
FSL.
J
K
L
- LAN
- Local area network, a data network intended to serve an area of only a few square kilometers or less. Since
such networks relatively small they can usually be directly controlled by the users and operate at relatively
high speeds (up to 100Mb/s) over inexpensive wiring.
- large product
- In the context of Unidata's LDM, a data product that is large enough to require multiple
remote procedure calls to send it. Currently, this means the product is
larger than about 16 Kb.
- latency
- In the context of the Unidata LDM, the delay in delivery of a
data product since it was first injected from a
data source site.
- LDMs
- LDM
- Local Data Manager, the Unidata software system for event-driven data distribution. The software allows
Unidata sites to select, capture, process, and distribute data products. The LDM is
implemented as a set of network client and server programs and their shared protocols. More information is
available here.
- ldmprods
- A perl script that monitors incoming data in realtime. Can be used to make sure that data is continuing to
flow. It will either sound an audible alarm, or send email, depending on how it is run, if data flow ceases
for any of the data streams it monitors.
- LDM servers
- LDM server
- The LDM program ldmd responsible for
servicing RPC requests, managing the reception, disposition, and injection of data to and from the
IDD.
- LDM-McIDAS
- a set of LDM-compatible programs used for decoding McIDAS products contained in the
Unidata-Wisconsin (McIDAS) data stream that is available to licensed sites. More information is available
here.
- leaf node
- In the context of the IDD, an LDM system that receives data products but does not relay
data products to other IDD nodes. If the IDD distribution is represented as a directed graph, the leaf nodes
have no outgoing edges. Technically, whether an IDD node is a data source,
data relay, or leaf node is relative to particular
data streams.
M
- MAPS
- Mesoscale Analysis and Prediction System (MAPS) MAPS is a data assimilation and mesoscale modeling system
developed by NOAA's Forecast Systems Laboratory; it incorporates asynoptic data such as wind profiler and ACARS
data into a mesoscale model with horizontal resolution of 30 kilometers.
- MCIDAS
- The feed type used for the Unidata/Wisconsin data stream
from SSEC.
- McIDAS
- Unidata McIDAS is a suite of applications,
developed at SSEC, for analyzing and displaying meteorological data for research and
education. The software can be used with conventional observational, satellite, and grid-point data. Unidata
distributes two versions of this software: McIDAS-X for UNIX platforms and McIDAS-OS2 for computers running
OS/2.
- McLDM
- OS/2 software to ingest the Unidata/Wisconsin data stream directly from an upstream LDM server.
- MD5 checksum
- a kind of checksum that is very likely to be unique for a sequence of bytes. In the context of the Unidata
LDM, it is an attribute of a data product that is used for detecting duplicate products.
- MRF
- Medium-Range Forecast, a forecasting model run by the NMC.
- multicast
- A packet with a special destination address that multiple nodes on the network may be willing to receive.
N
- namespace
- A commonly distributed set of names in which all names are unique.
- netCDF
- an interface for scientific data access and a library that provides an implementation of the interface. The
netCDF library also defines a machine-independent format for representing scientific data. Together, the
interface, library, and format support the creation, access, and sharing of scientific data. The netCDF
software was developed by Unidata. Some inline decoders that work with the LDM use
netCDF as the form of their output. More information is available from the
NetCDF Homepage.
- netcheck
- A perl script that is meant to be run out of cron. It monitors the network connection between IDD hosts. Can
be set up to provide email notification when the network connection is bad, based on packet loss.
- Netscape
- A widely-used WWW browser client for various operating systems, used for presenting this
Workshop. More information is available here.
- NEXRAD
- Next Generation Weather Radars, a NOAA radar system of WSR-88D radars and associated
processing equipment. NEXRAD data is made available at discounted prices to the IDD through
a contract with WSI.
- NFS
- Network File System, a protocol developed by Sun Microsystems, and defined in RFC 1094, which allows a
computer to access files over a network as if they were on its local disks. This protocol has been incorporated
in products by more than two hundred companies, and is now a de facto standard. NFS is implemented using a
connectionless protocol(UDP) in order to make it stateless.
- NIDS
- NEXRAD Information Dissemination System.
- NLDN
- National Lightning Detection Network, a data feed provided by SUNY Albany through a
cooperative agreement with Global Atmospherics, Inc.
- NMC
- National Meteorological Center, the source of most of the model output data that appears in
GRIB form on the HDS data stream. The NMC was
renamed the National Centers for Environmental Prediction after 1 October 1994.
- NOTIFYME
- The remote procedure call made by a data sink or
downstream node to sign on and announce what feed types and
product IDs it wants to be notified about when they arrive.
- NOAAport
- Data stream similar to the NWS FOS data stream but enhanced with other data products(Channel 3). It also
contains GOES (East|West) images and other experimental products on channels 1,2, and 4.
- NPS
- Numerical Products Service, a now-obsolete name for the HDS
data stream.
- NTP
- Network Time Protocol, an Internet Standard (12).
- NULLPROC
- The trivial remote procedure call supported by RPC
servers so that clients can test connectivity, timing, and flush
RPC buffers.
- NWS
- National Weather Service.
O
- ONC
- Open Network Computing, an architecture of programming interfaces, conventions, and server functionalities
(e.g. naming, distributed file system, remote procedure call) for
distributing applications transparently across networks of heterogeneous computers promoted by
Sun. The ONC architecture includes some widely
adopted ad hoc standards such as NFS and ONC RPCs. The latter are the
RPCs used by the Unidata LDM.
- origin
- In the context of the Unidata LDM, the host that first injected a
data product into the IDD, an attribute of the data product that
is preserved as it gets distributed to other IDD nodes.
P
- patterns
- pattern
- In the context of the Unidata LDM, a regular expression
that specifies the set of product IDs matching the pattern. For example, the pattern
`
^A
' specifies the set of all product IDs beginning with the character `A'.
- pattern-action file
- In the context of the Unidata LDM, the configuration file for the
pqact program that contains a list of entries
of the form feed type, product ID pattern, associated
action, and action arguments.
- PCWS
- The feed type for the PC DARE feed from FSL.
- perl
- A scripting language. More information is available here.
- ping
- Packet InterNet Groper, a program used to test reachability of destinations by sending them an ICMP echo
request and waiting for a reply. The term is used as a verb: "Ping host X to see if it is up!"
- port
- Client applications such as FTP must have a way to identify a corresponding server application on a remote
host. TCP/IP designates such remote servers with a port number. Most common Internet servers have specific,
reserved port numbers associated with them. For example, telnet uses port number 23. Port numbers less than
512 are assigned by the Internet Assigned Numbers Authority, and termed "well-known ports". The Unidata
LDM has been assigned port 388. Such ports are reserved for privileged processes.
- PPS
- Public Product Service, a data feed that is part of the NWS Family of Services. PPS includes watches,
warnings, and text.
- product exploder
- In the context of the Unidata LDM, a program such as surf_split
that receives a data product from a sender process, fragments it into multiple
smaller products with manufactured product IDs, and feeds them back to the
LDM server.
- product IDs
- product ID
- a short string assigned by a data source to uniquely identify each
data product.
- product queue
- A stored sequence of data products that is shared among LDM processes on a single
host. A receiver process gets products from
upstream sources and stores them at the back of the product queue. A
sender process takes products from the front of the product queue and transmits
them to a downstream sink, moving toward the back of the queue as fast as the
sink can accept the data. Other scanner processes read products out of the queue, moving from front to back, to
process the products in some way. Expire processes delete old products from the queue to make room for new
products.
- protocols
- protocol
- A set of formal rules describing how to transmit data, especially across networks. Low level protocols define
the electrical and physical standards to be observed, bit- and byte-ordering and the transmission and error
detection and correction of the bit stream. High level protocols deal with the data formatting, including the
syntax and sequencing of messages.
Q
- queue
- A data structure that permits inserting items at one end and reading and dequeueing items from the other end.
Also called a FIFO (First In, First Out) list. The LDM product queue is an
example.
R
- raw file
- In the context of the Unidata LDM, a file consisting of bytes as they appear in a
data stream, as a sequence of bulletins. A raw file is
acceptable as input to an LDM ingester.
- receiver
- receiver process
- In the context of the Unidata LDM, a process responsible for receiving
data products from one or more upstream sources and
storing the data products in the LDM product queue.
- regular expressions
- regular expression
- a notation for patterns used in many UNIX programs, e.g. ed, grep, sed, awk, lex, vi, emacs. The Unidata
LDM uses regular expressions as a notation for sets of product IDs
in the pattern-action file that serves as the configuration file for the
pqact program.
- relay node
- see data relay.
- RPCs
- RPC
- remote procedure calls
- remote procedure call
- An easy and popular paradigm for implementing the client-server model of distributed computing. In general,
a request is sent to a remote system to execute a designated procedure, using arguments supplied, and the result
returned to the caller. The LDM uses ONC RPC, version 2, documented as
Internet RFC 1050.
- RUC
- Rapid Update Cycle model, a mesoscale analysis/forecast system that operates on a 3-hour cycle to provide
frequent updates and short-range forecasts over the lower 48 United States. RUC output appears in the
HDS data stream.
S
- senders
- sender
- sender process
- In the context of the Unidata LDM, a process responsible for sending
data products or notifications of products matching specified criteria to a
downstream sink. The products to be sent are read from an LDM
product queue.
- sequence number
- In the context of the Unidata LDM, the sequence number of a
data product within a data feed, when a
data source provides it. Some data sources provide no useful information in this
field.
- servers
- server
- A computer system or program that provides a service to other client computer systems or
programs. The connection between client and server is normally by means of message passing, often over a
network, and uses some protocol to encode the client's requests and the server's
responses. The server may run continuously (as a daemon), waiting for requests to arrive
or, it may be invoked by some higher level daemon that controls a number of specific servers (inetd on Unix).
There are many servers associated with the Internet, such as those for Network File System, Network Information
Service(NIS), Domain Name System (DNS), FTP, news, finger, Network Time Protocol. On Unix, a long list can be
found in /etc/services or in the NIS database "services".
- small product
- In the context of Unidata's LDM, a data product that is small enough to be sent in a single
remote procedure call. Currently, this means the product should be smaller
than about 16 Kb.
- SSEC
- The Space Science and Engineering Center at the University of
Wisconsin in Madison. SSEC makes McIDAS data available to the IDD.
- SUNYA
- The State University of New York at Albany, the data source for
NLDN data through a cooperative agreement with GDS.
- syscheck
- A perl script that is made to run out of cron. It monitors various system parameters of the machine it runs
on. It is meant to be used to monitor machine load.
T
- TCP
- Transmission Control Protocol, the most common transport layer protocol used on Ethernet. It is built on top
of Internet Protocol and adds reliable communication, flow-control, multiplexing and connection-oriented
communication. It provides full-duplex, process-to-process connections. It is connection-oriented and
stream-oriented, as opposed to User Datagram Protocol (UDP).
- TCP/IP
- Transmission Control Protocol/Internet Protocol, named for two of the major communications protocols used
within the Internet (TCP and IP). These protocols (along with several
others) provide the basic foundation for communications between hosts in the Internet. The Unidata
LDM uses TCP/IP to transfer information.
- T1
- An AT&T term for a digital carrier facility used to transmit a digital signal at 1.544 megabits per
second.
- T3
- A term for a digital carrier facility used to transmit a digital signal at 44.746 megabits per second.
U
- UDP
- User Datagram Protocol, a connectionless protocol which, like TCP, is layered on top of
IP. UDP neither guarantees delivery nor does it require a connection. As a result it is
lightweight and efficient, but all error processing and retransmission must be taken care of by the application
program.
- UPC
- The Unidata Program Center.
- upstream
- In the context of the Unidata IDD, the direction from which data originates as it flows
from data sources to data sinks.
- upstream sources
- upstream source
- see data source.
- UTC
- Universal Coordinated Time.
V
W
- WMO
- World Meteorological Organization - an organization established under the auspices of the United Nations for
the orderly advancement of the science and practice of meteorology.
- An abbreviated heading used to identify meteorological products.
- WSI
- Provider of NIDS products to the IDD system. Also, a name of the feed
type for such products.
- WSR-88D
- A single radar from the national NEXRAD network.
- WWW
- The World Wide Web, or Web or W3 for short, a project to make the universe
of network-accessible information easily available to all who are suitably connected. This LDM Workshop is using
WWW documents for presentations, so they will be available remotely.
X
- XDR
- eXternal Data Representation, a standard for the description and encoding of data, useful for transferring
data between different computer architectures. XDR is documented in
Internet RFC 1014.
Y
Z
- Z
- an abbreviated symbol for Universal Coordinated Time (UTC) appended to a time.