Copyright © 2024 Staysail Systems, Inc.
San Marcos, California, United States of America

If you find this book or NNG useful, please consider sponsoring the project.

The MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Dedication

— For Jenichka

The sun always shines when you are home.

Preface

Preface for the First Edition

At the time of this writing, we are wrapping up NNG for its formal 1.0.0 release. It’s a good time for reflection on the road that we took to get here. Like the road on the cover of this book, it was windy (if quite a bit longer), but what we find at the end has made the journey worthwhile.

Originally the NNG project was conceived as a relatively modest effort to rewrite nanomsg based on threads, with a more readily extensible internal architecture so that we could more easily undertake projects like the ZeroTier and TLS transports.

It would not be incorrect to say that the initial NNG effort was started in “anger”, as we were frustrated with nanomsg’s very complex internal state machines. Looking back on it now, those complex state state machines don’t seem nearly as insane as they did just a year ago.

The simple, naïve, approach we would have preferred, and the one we originally started with, involved significant use of threads, inspired by the work we did in mangos, which uses Go’s goroutines heavily. Goroutines are excellent. Threads, it turns out, are not. Scalable, asynchronous, portable I/O is a lot harder than it looks.

Our experience with in-kernel threads on illumos and Solaris spoiled us, and left us utterly unprepared for cesspool that really is large amounts of userspace programming.

Instead, we have created our own, completely asynchronous core, giving us advanced multiprocessing and concurrency capabilities, without either sacrificing portability or settling for some unhappy least common denominator. This core is a robust foundation for NNG and handling the “Scalability Protocols”, but if we’re being completely honest, we think this core has braod applicability for beyond just the Scalability Protocols. It will be interesting to see if others come to the same conclusion.

Builting upon this robust foundation, we have engineered a substantial project, with capabilities far in exceess of the original nanomsg, while still preserving compatibility with the the network protocols that form the backbone of the nanomsg ecosystem, and even a compatible programming interface for nanomsg library users. In addition to compatibility with nanomsg, we find that NNG has greatly increased scalability, reliability, and usability (especially when developing concurrent applications).

NNG also has complete HTTP server and client implementations, support for TLS, and a plethora of other capabilities. Much of this is made possible by a the aforementioned asynchronous I/O framework.

We’ve tried to stay true to the core nanomsg goals about being light-weight, liberally licensed, and implemented in C. (After all, those were the things that drew us to nanomsg in the first place!) In addition we added a couple of new ones. Specifically, reliability, performance, and extensibility (in that order) were added as core goals for the project.

We believe that NNG represents a substantial step forward over other messaging frameworks, and have enjoyed creating it. We hope you find it useful. There is still a lot more we want to do, and future release of NNG will continue to expand it’s capabilities. We’re just getting started.

— Garrett D’Amore, May 30, 2018

Acknowledgements

We would like to thank Janjaap Bos, at Capitar IT Group BV. Without his patronage, neither NNG nor this book would be possible.

We would also like thank Martin Sústrik for creating the original nanomsg project, the foundation upon which all of this work is based.

And certainly not least of all, we would like to thank the various members of the community who have followed and supported the NNG project in so many different ways.

Conventions

Throughout this book there are occasional warnings, notices, and tips. These are visually distinguished as follows:

API Reference

This section is a reference guide for the NNG programming interfaces. It is meant to serve as a refernce, rather than as a tutorial.

The material here is organized by major areas of functionality.

Note that unless indicated otherwise, consumers of these interfaces must include the nng/nng.h header file like so:

#include <nng/nng.h>

Chapters

• Messages
• Threading and Synchronization
• Logging
• Utility Functions

Initialization & Finalization

This chapter details the function used to initialize the library before first use, and the funtion used to finalize the library and deallocate any resources used by the library.

Initialization

typedef struct {
    int16_t num_task_threads;
    int16_t max_task_threads;
    int16_t num_expire_threads;
    int16_t max_expire_threads;
    int16_t num_poller_threads;
    int16_t max_poller_threads;
    int16_t num_resolver_threads;
} nng_init_params;

extern int nng_init(nng_init_parms *params);

Before using other interfaces in this library, it is necessary to initialize the library. The nng_init function performs this initialization.

The function is idempotent, although on tear down, every call to nng_init must be paired with a call to nng_fini or no resources will be released. This allows for libraries consuming these interfaces to safely initialize and finalize the library without disrupting other consumers in the same process.

Further, only the first call to this function may have a value of params other than NULL. If params is not NULL, and the library has already been intiazed, then nng_init will return NNG_EBUSY.

In some cases it is desirable to tune certain runtime parameters for the library, which can be done by supplying a non-NULL params argument.

Parameters

The individual fields of the nng_init_params structure can be used to adjust certain runtime tunables for the library. There is no guarantee that these tunables are implemented, and applications should not depend upon them for correct operation.

Any member of nng_init_params that is set to zero will be ignored, and any built in default will be used instead for that value.

The following parameters are present:

• num_task_threads and max_task_threads
  Configures the number of threads to use for tasks, which are used principally for completion callbacks. The maximum value can be used to provide an upper limit while still allowing for a dynamically calculated value to be used, as long as it does not exceeed the maximum.

• num_expire_threads and max_expire_threads
  Configures the number of threads used for expiring operations. Using a larger value will reduce contention on some common locks, and may improve performance.

• num_poller_threads and max_poller_threads
  Configures the number of threads to be used for performing I/O. Not all configurations support changing these values.

• num_resolver_threads
  Changes the number of threads used for asynchronous DNS look ups.

Finalization

extern void nng_init(nng_init_parms *params);

When the consumer is ready to deallocate any resoures allocated by the library, it should call the nng_fini function. Each call to nng_fini should be paired with an earlier call to nng_init.

After calling nng_fini, the consuming application must not make any other calls to NNG functions, except that it may use nng_init to initialize the application for further use.

Messages

Messages in Scalability Protocols are the fundamental unit of transmission and reception, as these protocols are fundamentally message-oriented.

Messages have a body, containing the application-supplied payload, and a header, containing protocol specific routing and similar related information.

Message Structure

typedef struct nng_msg nng_msg;

The nng_msg structure represents a single message. It carries a body and a header.

Create a Message

int nng_msg_alloc(nng_msg **msgp, size_t size);

The nng_msg_alloc function allocates a new message. It takes a size argument, and returns a message with a preallocated body of that size in the msgp parameter.

If it succeeds, it returns zero, otherwise this function may return NNG_ENOMEM, indicating that insufficient memory is available to allocate a new message.

Destroy a Message

void nng_msg_free(nng_msg *msg);

The nng_msg_free function deallocates a message.

Duplicate a Message

int nng_msg_dup(nng_msg **dup, nng_msg *msg);

The nng_msg_dup function duplicates the message msg, storing a pointer to the new duplicate in dup. This function also returns zero on succes, or NNG_ENOMEM if memory is exhausted.

Message Size and Capacity

size_t nng_msg_capacity(nng_msg *msg);
int nng_msg_realloc(nng_msg *msg, size_t size);
int nng_msg_reserve(nng_msg *msg, size_t capacity);

Messages have a certain amount of pre-reserved space, which may exceed the total size of the message. This allows for content to be added to the message later, without necessarily performing a reallocation.

The nng_msg_capacity function returns the amount of prereserved space. If a message size change is required, and the new size will fit within the capacity reported by this function, then change will be done without a reallocation, and likely without a data copy as well.

The message size may be changed by use of the nng_msg_realloc function. This function will reallocate the underlying memory for the message msg, preserving contents while doing so. If the new size is smaller than the original message, it will truncate the message, but not perform any allocations. If reallocation fails due to insufficient memory, then the original is left intact.

The nng_msg_reserve function ensures that the total message capacity is at least capacity bytes. Use of this function to ensure the total anticipated capacity is present in the message may help prevent many small allocations.

Both nng_msg_realloc and nng_msg_reserve return zero on success, or may return NNG_ENOMEM if insufficient memory exists to preform allocation.

Message Body

void *nng_msg_body(nng_msg *msg);
size_t nng_msg_len(nng_msg *msg);

The body and body length of msg are returned by nng_msg_body and nng_msg_len, respectively.

Clear the Body

void *nng_msg_clear(nng_msg *msg);

The nng_msg_clear simply resets the total message body length to zero, but does not affect the capacity. It does not change the underlying bytes of the message.

Add to Body

int nng_msg_append(nng_msg *msg, const void *val, size_t size);
int nng_msg_append_u16(nng_msg *msg, uint16_t val16);
int nng_msg_append_u32(nng_msg *msg, uint32_t val32);
int nng_msg_append_u64(nng_msg *msg, uint64_t val64);

int nng_msg_insert(nng_msg *msg, const void *val, size_t size);
int nng_msg_insert_u16(nng_msg *msg, uint16_t val16);
int nng_msg_insert_u32(nng_msg *msg, uint32_t val32);
int nng_msg_insert_u64(nng_msg *msg, uint64_t val64);

Appending data to a message body is done by using the nng_msg_append functions. The base nng_msg_append function appends size bytes of untyped data to the end of the message.

Use of the typed versions, ending in suffixes _u16, _u32, and _u64 allows for unsigned integers to be appended directly. The integers are encoded in network byte order, with the most significant byte appearing first. The message body will by two, four, or eight bytes accordingly.

Data may inserted before the rest of the message body by using the nng_msg_insert functions. This will attempt to use “headroom” in the message to avoid a data copy. Otherwise they are like the nng_msg_append functions except that the put the data in front of the messages instead of at the end.

Consume From Body

int nng_msg_chop(nng_msg *msg, size_t size);
int nng_msg_chop_u16(nng_msg *msg, uint16_t *val16);
int nng_msg_chop_u32(nng_msg *msg, uint32_t *val32);
int nng_msg_chop_u64(nng_msg *msg, uint64_t *val64);

int nng_msg_trim(nng_msg *msg, size_t size);
int nng_msg_trim_u16(nng_msg *msg, uint16_t *val16);
int nng_msg_trim_u32(nng_msg *msg, uint32_t *val32);
int nng_msg_trim_u64(nng_msg *msg, uint64_t *val64);

The nng_msg_chop functions remove data from the end of the body of message msg, reducing the message length by either size, or the appropriate value size.

The nng_msg_trim functions remove data from the beginning of the message body of msg. but are otherwise just like the nng_msg_chop functions.

If the message is not big enough to remove requisite amount of bytes, these functions return NNG_EINVAL. Otherwise they return zero.

Additionally, functions with typed suffixes (_u16, _u32, _u64) decode the data and return it through the appropriate val pointer.

The data is assumed to have been in network byte order in the message, but is returned in the native machine byte order. The appropriate number of bytes is consumed for each of these types, so two bytes for _u16, four bytes for _u32, and eight bytes for _u64.

Message Header

void *nng_msg_header(nng_msg *msg);
size_t nng_msg_header_len(nng_msg *msg);

The header and header length of msg are returned by nng_msg_header and nng_msg_header_len, respectively.

The message headers are generally intended for limited use, to store protocol headers.

Clear the Header

void *nng_msg_header_clear(nng_msg *msg);

The nng_msg_header_clear simply resets the total message header length to zero.

Append or Insert Header

Appending data to a message header is done by using the nng_msg_header_append functions, and inserting data in the header is done using the nng_msg_header_insert functions.

These functions act just like the nng_msg_append and nng_msg_insert functions, except that they operate on the message header rather than the message body.

Consume from Header

The nng_msg_header_trim functions remove data from the beginning of the message header, and the nng_msg_header_chop functions remove data from the end of the message header.

These functions act just like the nng_msg_trim and nng_msg_chop functions, except that they operate the message header rather than the message body.

Message Pipe

nng_pipe nng_msg_get_pipe(nng_msg *msg);
void nng_msg_get_pipe(nng_msg *msg, nng_pipe p);

The nng_msg_set_pipe function sets the pipe associated with msg to p. This is most often useful when used with protocols that support directing a message to a specific peer. For example the PAIR version 1 protocol can do this when NNG_OPT_PAIR1_POLY mode is set.

The nng_msg_get_pipe function returns the pipe that was previously set on the message m, either directly by the application, or when the message was received by the protocol.

Examples

Example 1: Preparing a message for use

    #include <nng/nng.h>

    nng_msg *m;
    if (nng_msg_alloc(&m, strlen("content") + 1) != 0) {
       // handle error
    }
    strcpy(nng_msg_body(m), "content");

Example 2: Preallocating message content

    if (nng_msg_alloc(&m, 1024) != 0) {
        // handle error
    }
    while ((val64 = next_datum()) != 0) P
        if (nng_msg_append_u64(m, val64) != 0) {
            // handle error
        }
    }

Sockets

Sockets in Scalability Protocols provide the handle for communication between peers. Sockets also encapsulate protocol specific semantics, such as filtering subscriptions, or automatically retrying requests.

Socket Structure

#define NNG_SOCKET_INITIALIZER // opaque value

typedef struct nng_socket_s nng_socket;

The nng_socket structure represents socket. This is a handle, and the members of it are opaque. However, unlike a pointer, it is usually passed by value.

A socket may be initialized statically with the NNG_SOCKET_INITIALIZER macro, to ensure that it cannot be confused with a valid open socket.

Socket Identity

int nng_socket_id(nng_socket s);
int nng_socket_raw(nng_socket s, bool *raw);
int nng_socket_proto_id(nng_socket s, uint16_t *proto);
int nng_socket_peer_id(nng_socket s, uint16_t *proto);
int nng_socket_proto_name(nng_socket s, const char **name);
int nng_socket_peer_name(nng_socket s, const char **name);

These functions are used to provide fundamental information about the socket s. Most applications will not need to use these functions.

The nng_socket_id function returns the numeric id, which will be a non-negative value, associated with the socket. If the socket is uninitialized (has never been opened), then the return value may be -1.

The nng_socket_proto_id and nng_socket_peer_id functions provide the 16-bit protocol identifier for the socket’s protocol, and of the protocol peers will use when communicating with the socket.

The nng_socket_proto_name and nng_socket_peer_name functions provide the ASCII names of the socket’s protocol, and of the protocol peers of the socket use. The value stored in name is a fixed string located in program text, and must not be freed or altered. It is guaranteed to remain valid while this library is present.

The nng_socket_raw function determines whether the socket is in raw mode or not, storing true in raw if it is, or false if it is not.

Polling Socket Events

int nng_socket_get_recv_poll_fd(nng_socket s, int *fdp);
int nng_socket_get_send_poll_fd(nng_socket s, int *fdp);

Sometimes it is necessary to integrate a socket into a poll or select driven event loop. (Or, on Linux, epoll, or on BSD derived systems like macOS kqueue).

For these occasions, a suitable file descriptor for polling is provided by these two functions.

The nng_socket_get_recv_poll_fd function obtains a file descriptor that will poll as readable when a message is ready for receiving for the socket.

The nng_socket_get_send_poll_fd function obtains a file descriptor that will poll as readable when the socket can accept a message for sending.

These file descriptors should only be polled for readability, and no other operation performed on them. The socket will read from, or write to, these file descriptors to provide a level-signaled behavior automatically.

Additionally the socket will close these file descriptors when the socket itself is closed.

These functions replace the NNG_OPT_SENDFD and NNG_OPT_RECVFD socket options that were available in previous versions of NNG.

Examples

Example 1: Initializing a Socket

nng_socket s = NNG_SOCKET_INITIALIZER;

Memory

Managing memory and allocations is something that every C program has to deal with. In the case of NNG, it can be more complicated because the underlying platform code can provide different allocators that might not be compatible with the use system allocator used by malloc and free.

Allocate Memory

void *nng_alloc(size_t size);

The nng_alloc function allocates a contiguous memory region of at least size bytes, and returns a pointer to it. The memory will be 64-bit aligned. Note that the memory may have random data in it, just like with malloc.

If memory cannot be allocated for any reason, then NULL will be returned. Applications that experience this should treat this like NNG_ENOMEM.

Memory returned by nng_alloc can be used to hold message buffers, in which case it can be directly passed to nng_send using the flag NNG_FLAG_ALLOC. Alternatively, it can be freed when no longer needed using nng_free.

Deallocate Memory

void nng_free(void *ptr, size_t size);

The nng_free function deallocates memory previously allocated by nng_alloc.

The size argument must exactly match the size argument that was supplied to nng_alloc when the memory was allocated.

Duplicate String

char *nng_strdup(const char *src);

The nng_strdup duplicates the string src and returns it.

This is logically equivalent to using nng_alloc to allocate a region of memory of strlen(s) + 1 bytes, and then using strcpy to copy the string into the destination before returning it.

The returned string should be deallocated with nng_strfree, or may be deallocated using the nng_free using the length of the returned string plus one (for the NUL terminating byte).

Free String

void nng_strfree(char *str);

The nng_strfree function is a convenience function that can be used to deallocate strings allocated with nng_strdup.

It is effectively the same as nng_free(strlen(str) + 1).

Time

NNG supports has support for time in the form of access to a system clock, and supporting timeouts for certain operations.

Time Type

typedef uint64_t nng_time;

The nng_time type is used to represent a clock offset from a common base time, measured in milliseconds.

The reference, or zero value, is some arbitrary point in time, most often sytem boot, but can be process start time or any other convenient reference.

All threads within a process will use the same reference time, but be aware that different processes may use a different reference time.

Duration Type

typedef int64_t nng_duration;

#define NNG_DURATION_INFINITE (-1)
#define NNG_DURATION_DEFAULT  (-2)
#define NNG_DURATION_ZERO     (0)

The nng_duration time measures a time duration in milliseconds. Normally durations are positive, but some specific negative values are reserved.

• NNG_DURATION_INFINITE: The duration essentially means forever. This is most often used with a timeout to indicate that there is no timeout, the operation should wait until it is complete, even forever.

• NNG_DURATION_DEFAULT: This special value is used in some circumstances to prevent overriding a default timeout. Some operations have a default maximum time, and this value means that the previously established default should be used. The precise meaning is context-specific.

• NNG_DURATION_ZERO: A zero length duration is used to performan an immediate poll.

Get the Current Time

nng_time nng_clock(void);

The nng_clock function returns the number of elapsed milliseconds since some arbitrary time in the past. The resolution of the clock depends on the underlying timing facilities of the system. This function may be used for timing, but applications should not expect very fine-grained values.

Wait for Duration

void nng_msleep(nng_duration msec);

The nng_msleep function blocks the calling thread for at least the specified number of milliseconds.

Wait Asynchronously

void nng_sleep_aio(nng_duration msec, nng_aio *aio);

It is possible to wait as the action on an nng_aio, which in effect acts like scheduling a callback to run after a specified period of time.

The nng_sleep_aio function provides this capability. After msec milliseconds have passed, then aio’s callback will be executed. If this sleep waits without interruption, and then completes, the result from nng_aio_result will be zero.

See Also

Asynchronous Operations, Synchronization

URLs

Universal Resource Locators, or URLs for short, are a standardized way of representing a network resource, defined in RFC 1738, and RFC 3968.

In Scalability Protocols, this concept is extended, although it includes schemes that are not part of the IETF standards.

URL Structure

typedef struct nng_url {
    const char *u_scheme;
    char       *u_userinfo;
    char       *u_hostname;
    uint16_t   u_port;
    char       *u_path;
    char       *u_query;
    char       *u_fragment;
} nng_url;

URL Fields

Applications may access individual fields, but must not free or alter them, as the underlying memory is managed by the library.

Additionally applications must not depend on the size of this structure. Obtain one using nng_parse_url.

The fields of an nng_url object are as follows:

• u_scheme: The URL scheme, such as “http” or “inproc”. Always lower case. This will never be NULL.
• u_userinfo: This username and password if supplied in the URL string. Will be NULL when not present.
• u_hostname: The name of the host, and may be the empty string in some cases.
• u_port: The port. May be zero if irrelevant or not specified.
• u_path: The path, typically used with HTTP or WebSockets. Will be empty string if not specified.
• u_query: The query info (typically following ? in the URL.) Will be NULL if not present.
• u_fragment: This is used for specifying an anchor, the part after # in a URL. Will be NULL if not present.

Format a URL

int nng_url_sprintf(char *buf, size_t bufsz, const nng_url *url);

The nng_url_sprintf function formats the url to the buf, which must have bufsz bytes of free space associated with it.

This function returns the number of bytes formatted to buf, excludng the terminating zero byte, or if bufsz is too small, then it returns the number of bytes that would have been formatted if there was sufficient space. The semantics are similar to the snprintf function from C99.

Parse a URL

int nng_url_parse(nng_url **urlp, const char *str);

The nng_url_parse function parses a URL string (in str), and creates a dynamically allocated nng_url, returning it in urlp.

Clone a URL

int nng_url_clone(nng_url **dup, nng_url *url);

The nng_url_clone function creates a copy of url, and returns it in dup.

Destroy a URL

void nng_url_free(nng_url *url);

The nng_url_free function destroy an nng_url object created with either nng_url_parse or nng_url_free.

This is the only correct way to destroy an nng_url object.

See Also

More information about Universal Resource Locators can be found in RFC 3986.

Asynchronous Operations

In order to obtain significant scalability, with low-latency, and minimal overheads, NNG supports performing operations asynchronously.

One way that applications can perform work asynchronously and concurrently is by using threads, but threads carry significant resource overheads and frequently there are limits on the number that can easily be created.

Additionally, with most network applications, the flow of execution will spend the bulk of its time waiting for traffic from a peer.

For these kinds of applications, it is far more efficient to use asynchronous operations using the mechanisms described in this chapter.

Asynchronous I/O Handle

typedef struct nng_aio nng_aio;

An nng_aio is an opaque structure used in conjunction with asynchronous I/O operations. Every asynchronous operation uses one of these structures, each of which can only be used with a single operation at a time.

Asynchronous operations are performed without blocking calling application threads. Instead the application registers a callback function to be executed when the operation is complete (whether successfully or not). This callback will be executed exactly once.

The asynchronous I/O framework also supports cancellation of operations that are already in progress as well setting a maximum timeout for them to complete.

It is also possible to initiate an asynchronous operation, and wait for it to complete, creating a synchronous flow from an asynchronous one.

Create Handle

int nng_aio_alloc(nng_aio **aiop, void (*callb)(void *), void *arg);

The nng_aio_alloc function creates an nng_aio object, with the callback callb taking the argument arg, and returns it in aiop.

If this succeeds, the function returns zero. Otherwise it may return NNG_ENOMEM.

Destroy Handle

void nng_aio_free(nng_aio *aio);
void nng_aio_reap(nng_aio *aio);

The nng_aio_free handle destroys the handle aio, waiting for any operations and associated callbacks to complete before doing so.

The nng_aio_reap handle destroys the handle aio asynchronously, using a reaper thread to do so. It does not wait for the object to be destroyed. Thus this function is safe to call from aio’s own callback.

Cancellation

void nng_aio_abort(nng_aio *aio, int err);
void nng_aio_cancel(nng_aio *aio);
void nng_aio_stop(nng_aio *aio);

These functions are used to stop a previously submitted asynchronous I/O operation. The operation may be canceled, or may continue to completion. If no operation is in progress (perhaps because it has already completed), then these operations have no effect. If the operation is successfully canceled or aborted, then the callback will still be called.

The nng_aio_abort function aborts the operation associated with aio and returns immediately without waiting. If cancellation was successful, then nng_aio_result will return err.

The nng_aio_cancel function acts like nng_aio_abort, but uses the error code NNG_ECANCELED.

The nng_aio_stop function aborts the aio operation with NNG_ECANCELED, and then waits the operation and any associated callback to complete. This function also marks aio itself permanently stopped, so that any new operations scheduled by I/O providers using nng_aio_begin return false. Thus this function should be used to teardown operations.

Set Timeout

void nng_aio_set_timeout(nng_aio *aio, nng_duration timeout);
void nng_aio_set_expire(nng_aio *aio, nng_time expiration);

The nng_aio_set_timeout function sets a timeout for the asynchronous operation associated with aio. This causes a timer to be started when the operation is actually started. If the timer expires before the operation is completed, then it is aborted with an error of NNG_ETIMEDOUT. The timeout duration is specified as a relative number of milliseconds.

If the timeout is NNG_DURATION_INFINITE, then no timeout is used. If the timeout is NNG_DURATION_DEFAULT, then a “default” or socket-specific timeout is used. (This is frequently the same as NNG_DURATION_INFINITE.)

The nng_aio_set_expire function is similar to nng_aio_set_timeout, but sets an expiration time based on the system clock. The expiration time is a clock timestamp, such as would be returned by nng_clock.

Wait for Completion

void nng_aio_wait(nng_aio *aio);

The nng_aio_wait function waits for an asynchronous I/O operation to complete. If the operation has not been started, or has already completed, then it returns immediately.

If a callback was set with aio when it was allocated, then this function will not be called until the callback has completed.

Test for Completion

bool nng_aio_busy(nng_aio *aio);

The nng_aio_busy function returns true if the aio is currently busy performing an operation or is executing a completion callback. Otherwise it return false. This is the same test used internally by nng_aio_wait.

Result of Operation

int nng_aio_result(nng_aio *aio);
size_t nng_aio_count(nng_aio *aio);

The nng_aio_result function returns the result of the operation associated with the handle aio. If the operation was successful, then 0 is returned. Otherwise a non-zero error code, such as NNG_ECANCELED or NNG_ETIMEDOUT, is returned.

For operations that transfer data, nng_aio_count returns the number of bytes transferred by the operation associated with the handle aio. Operations that do not transfer data, or do not keep a count, may return zero for this function.

Messages

nng_msg *nng_aio_get_msg(nng_aio *aio);
void nng_aio_set_msg(nng_aio *aio, nng_msg *msg);

The nng_aio_get_msg and nng_aio_set_msg functions retrieve and store a message in aio. For example, if a function to receive data is called, that function can generally be expected to store a message on the asssociated aio, for the application to retrieve with nng_aio_get_msg. Conversely an application desiring to send a message msg will store it in the aio using nng_aio_set_msg. The function implementing the send operation will retrieve the message and arrange for it to be sent.

Message Ownership

For send or transmit operations, the rule of thumb is that implementation of the operation is responsible for taking ownership of the message (and releasing resources when it is complete), if it will return success. If the operation will end in error, then the message will be retained and it is the consuming application’s responsibility to dispose of the message. This allows an application the opportunity to reuse the message to try again, if it so desires.

For receive operations, the implementation of the operation will set the message on the aio on success, and the consuming application hasa responsibility to retrieve and dispose of the message. Failure to do so will leak the message. If the operation does not complete successfully, then no message is stored on the aio.

I/O Vector

typedef struct nng_iov {
    void * iov_buf;
    size_t iov_len;
};

int nng_aio_set_iov(nng_aio *aio, unsigned int niov, nng_iov *iov);

For some operations, the unit of data transferred is not a message, but rather a stream of bytes.

For these operations, an array of niov nng_iov structures can be passed to the nng_aio_set_iov function to provide a scatter/gather array of elements describing the location (iov_buf) and length (iov_len) of data, to transfer.

The iov vector is copied into storage in the aio itself, so that callers may use stack allocated nng_iov structures. The values pointed to by the iov_buf members are not copied by this function though.

A maximum of four (4) nng_iov members may be supplied.

Inputs and Outputs

void nng_aio_set_input(nng_aio *aio, unsigned int index, void *param);
void *nng_aio_get_output(nng_aio *aio, unsigned int index);

Asynchronous operations can take additional input parameters, and provide additional result outputs besides the result code.

The nng_aio_set_input function sets the input parameter at index to param for the operation associated with aio.

The nng_aio_get_output function returns the output result at index for the operation associated with aio.

The type and semantics of input parameters and output results are determined by specific operations. The documentation for the operation should provide details.

The valid values of index range from zero (0) to three (3), as no operation currently defined can accept more than four parameters or return more than four additional results.

See Also

Synchronization, Threads, Time

Synchronization Primitives

In order to allow safely accessing shared state, or to allow coordination between different threads, NNG provides synchronization primitives in the form of mutual exclusion locks and condition variables.

Correct use of these primitives will be needed when accessing shared state from threads, or from callback functions associated with asynchronous operations. (The need to do this in callbacks is because the callback may be executed under a task thread other than the submitting thread.)

Mutual Exclusion Lock

typedef struct nng_mtx nng_mtx;

Mutual exclusion locks, or mutex locks, represented by the nng_mtx structure, allow only a single thread to lock “own” the lock, acquired by nng_mtx_lock. Any other thread trying to acquire the same mutex will wait until the owner has released the mutex by calling nng_mtx_unlock.

Creating a Mutex

int nng_mutx_alloc(nng_mt **mtxp);

A mutex can be created by allocating one with nng_mtx_lock. On success, a pointer to the mutex is returned through mtxp. This function can fail due to insufficient memory or resources, in which case it will return NNG_ENOMEM. Otherwise it will succceed and return zero.

Destroying a Mutex

void nng_mtx_free(nng_mtx *mtx);

When no longer needed, a mutex can be deallocated and its resources returned to the caller, by calling nng_mtx_free. The mutex must not be locked by any thread when calling this function.

Acquiring a Mutex

void nng_mtx_lock(nng_mtx *mtx);

The nng_mtx_lock function acquires ownership of a mutex, waiting for it to unowned by any other threads if necessary.

Releasing a Mutex

void nng_mtx_unlock(nng_mtx *mtx);

The nng_mtx_unlock function releases a mutex that the calling thread has previously acquired with nng_mtx_lock.

Condition Variable

typedef struct nng_cv nng_cv;

The nng_cv structure implements a condition variable, associated with the the mutex mtx which was supplied when it was created.

Condition variables provide for a way to wait on an arbitrary condition, and to be woken when the condition is signaled. The mutex is dropped while the caller is asleep, and reacquired atomically when the caller is woken.

Creating a Condition Variable

int nng_cv_alloc(nng_cv **cvp, nng_mtx *mtx);

The nng_cv_alloc function allocates a condition variable, and associated with the mutex mtx, and returns a pointer to it in cvp.

Destroy a Condition Variable

void nng_cv_free(nng_cv *cv);

The nng_cv_free function deallocates the condition variable cv.

Waiting for the Condition

int nng_cv_until(nng_cv *cv, nng_time when);
void nng_cv_wait(nng_cv *cv);

The nng_cv_until and nng_cv_wait functions put the caller to sleep until the condition variable cv is signaled, or (in the case of nng_cv_until), the specified time when (as determined by nng_clock is reached.

While nng_cv_wait never fails and so has no return value, the nng_cv_until function can return NNG_ETIMEDOUT if the time is reached before condition cv is signaled by either nng_cv_wake or nng_cv_wake1.

Signaling the Condition

void nng_cv_wake(nng_cv *cv);
void nng_cv_wake1(nng_cv *cv);

The nng_cv_wake and nng_cv_wake1 functions wake threads waiting in nng_cv_until or nng_cv_wait. The difference between these functions is that nng_cv_wake will wake every thread, whereas nng_cv_wake1 will wake up exactly one thread (which may be chosen randomly).

Examples

Example 1: Allocating the condition variable

nng_mtx *m;
nng_cv *cv;
nng_mtx_alloc(&m); // error checks elided
nng_cv_alloc(&cv, m);

Example 2: Waiting for the condition

expire = nng_clock() + 1000; // 1 second in the future
nng_mtx_lock(m);  // assume cv was allocated using m
while (!condition_true) {
  if (nng_cv_until(cv, expire) == NNG_ETIMEDOUT) {
    printf("Time out reached!\n");
    break;
  }
}
// condition_true is true
nng_mtx_unlock(m);

Example 3: Signaling the condition

nng_mtx_lock(m);
condition_true = true;
nng_cv_wake(cv);
nng_mtx_unlock(m);

See Also

Threads, Time, Asynchronous I/O

Threads

Threads provide a means of representing multiple parallel execution contexts. NNG makes use of this concept internally, but also provides for user applications to utilize the same thread facilities. This allows one form of concurrency for applications.

Thread Structure

typedef struct nng_thread nng_thread;

The nng_thread structure represnts a thread, which is a single execution context. A given thread will have its own stack, and CPU registers. However global state, as well as values allocated on the heap, will be shared and accessible to all threads in the system (See the Synchronization chapter for functions to help with data sharing between different threads.)

Multiple threads can be thought of as running concurrently, even though they might not actually do so.

I/O operations that block (i.e. wait for completion) will block the thread, while allowing other threads to proceed.

Creating a Thread

int nng_thread_create(nng_thread **thrp, void (*func)(void *), void *arg);

The nng_thread_create function creates a thread, which will execute func, with the given argument arg, and returns a pointer to it in thrp.

The thread may begin execution immediately.

The thread will persist until func returns.

This function returns zero on success, but may return NNG_ENOMEM if insufficient resources to create a thread are available.

Destroying a Thread

void nng_thread_destroy(nng_thread *thr);

The nng_thread_destroy function waits for the thread thr to finish execution. This function should be called to reclaim any resources associated with the thread when done. It also has the effect of blocking execution in the caller until thr has completed executing.

Thread Names

void nng_thread_set_name(nng_thread *thr, const char *name);

In order to facilitate debugging, nng_thread_set_name may be called to provide a name for the thread. This may change how the thread is represented in debuggers. Not all platforms support setting the thread name.

See Also

Synchronization, Asynchronous Operations

Logging

This chapter describes the support for message logs. Both applications and NNG itself can emit logs, which can be useful for application field support and debugging. Additionally applications can customize the handling of this logging as needed.

Note that logging is disabled by default unless an application configures a suitable logger with nng_log_set_logger.

Submitting Logs

void nng_log_err(const char *msgid, const char *msg, ...);
void nng_log_warn(const char *msgid, const char *msg, ...);
void nng_log_notice(const char *msgid, const char *msg, ...);
void nng_log_info(const char *msgid, const char *msg, ...);
void nng_log_debug(const char *msgid, const char *msg, ...);

These functions inject a a message into the logging system, where it will be processed and potentially go to system logs, standard output, or procssed further.

The msgid is a short prefix that should uniquely identify the message, possibly also with some kind of category. It is recommended that strings between 8 and 16 charactes be used. As this may, but will not necessarily be displayed to the user, the content of the message should not appear solely in this field. A NULL value is permitted here, but that may make filtering the message or other automatic processing more difficult.

The msg is a printf-style format string, which is used to format the message content. The following arguments are consumed in the same manner as printf.

Auth Logs

void nng_log_auth(nng_log_level level, const char *msgid, const char *msg, ...);

The nng_log_auth function formats and injects a security related log message. (“Auth” can indicate either “authentication” or “authorization”.) The level is a log level. The msgid, msg, and any remaining arguments are processed in a fashion similar to the other logging functions, except that the logs may be are logged using the NNG_LOG_AUTH facility, and thus may be redirected or receive other special treatment.

Log Levels

typedef enum nng_log_level nng_log_level;

void nng_log_set_level(nng_log_level level);
nng_log_level nng_log_get_level(void);

The nng_log_level type represents a severity for logged messages. These levels correspond to those found in the UNIX syslog subsystem, although applications should not depend upon the values being identical.

The nng_log_set_level function sets the log level. Messages with a severity that is numerically greater than this (less-severe) will be discarded.

The nng_log_get_level function returns the log level most recently set by nng_log_set_level or the default if that function has not been called.

The log levels are defined as follows:

typedef enum nng_log_level {
    NNG_LOG_NONE   = 0, // used for filters only, NNG suppresses these
    NNG_LOG_ERR    = 3,
    NNG_LOG_WARN   = 4,
    NNG_LOG_NOTICE = 5,
    NNG_LOG_INFO   = 6,
    NNG_LOG_DEBUG  = 7
} nng_log_level;

The value NNG_LOG_NONE may be useful to suppress message logs altogether.

The default level is typically NNG_LOG_NOTICE, but applications should select a value rather than relying upon the default.

Log Facilities

typedef enum nng_log_facility

void nng_log_set_facility(nng_log_facility facility);

Logging facilities are used to indicate the source of a log message, and may be useful in routing and processing these logs. Traditionally these are used with the UNIX syslog system, and the values here represent some (but not all) of the values found there.

The following values are defined:

typedef enum nng_log_facility {
    NNG_LOG_USER   = 1,
    NNG_LOG_DAEMON = 3,
    NNG_LOG_AUTH   = 10,
    NNG_LOG_LOCAL0 = 16,
    NNG_LOG_LOCAL1 = 17,
    NNG_LOG_LOCAL2 = 18,
    NNG_LOG_LOCAL3 = 19,
    NNG_LOG_LOCAL4 = 20,
    NNG_LOG_LOCAL5 = 21,
    NNG_LOG_LOCAL6 = 22,
    NNG_LOG_LOCAL7 = 23,
} nng_log_facility;

The nng_log_set_facility function can be used to set the facility that the application will use when emitting log messages. This should be called as part of initialization of the application, if logging is to be used.

The default facility is typically NNG_LOG_USER, but applications should select a value rather than relying upon the default.

Log Handlers

typedef void (*nng_logger)(nng_log_level level, nng_log_facility facility,
    const char *msgid, const char *msg);

void nng_null_logger(nng_log_level, nng_log_facility, const char *, const char *);
void nng_stderr_logger(nng_log_level, nng_log_facility, const char *, const char *);
void nng_system_logger(nng_log_level, nng_log_facility, const char *, const char *);

void nng_log_set_logger(nng_logger logger);

Log handlers are responsible for actually processing the logged messages.

The nng_log_set_logger function installs the named logger, of type nng_logger, as the log handler. The function logger will be called when any message is meant to be processed. (Messages are first filtered by severity, then formatted, before calling the logger.)

Any previously installed logger is replaced by logger.

The nng_null_logger function is an implementation of nng_logger that simply discards the content. This is the default logger, so logging is disabled by default.

The nng_stderr_logger function is an implementation that logs messages to the standard error stream. It will attempt to colorize messages by the severity, if the standard error is a terminal device. This can be suppressed by setting either the NO_COLOR or NNG_LOG_NO_COLOR environment variables.

The nng_system_logger attempts to use an appropriate system facility to log messages. For POSIX systems, this means using syslog to process the messages. For other systems the defauilt behavior may be the same as nng_stderr_logger.

See Also

The Syslog Protocol upon which this is based is documented in the following two IETF RFCS,

• R. Gerhards, RFC 5424, The Syslog Protocol, March 2009
• C. Lonvick, RFC 3164, The BSD syslog Protocol, August 2001

Statistics

To facilitate debugging and support situations, the NNG library provides for collection and reporting of numerous statistics.

These statistics are organized in a tree, and include both values, and metadata describing the statistics. In order to be efficient and minimize the impact of maintaining statistics, an explicit snapshot of statistics must be taken, and that snapshot can then be processed.

Statistic Structure

typedef struct nng_stat nng_stat;

The nng_stat structure represents a statistic, which is a single value collected at a specific point in time.

This structure has meta-data describing the value, the value itself, and links to any sibling or child statistics.

Collecting a Snapshot

int nng_stats_get(nng_stat **statsp);

The nng_stats_get function takes a snapshot of the statistics for the system and returns it through the pointer statsp. This function may return NNG_ENOMEM if memory is exhausted, or NNG_ENOTSUP if the statistics support is not enabled in the build, but is otherwise expected to return zero.

Freeing a Snapshot

void nng_stats_free(nng_stat *stat);

The nng_stats_free function deallocates the snapshot referenced by stat.

Traversing the Tree

const nng_stat *nng_stat_child(const nng_stat *stat);
const nng_stat *nng_stat_next(const nng_stat *stat);

Traversing the tree of statistics is done using the nng_stat_child and nng_stat_next functions.

The nng_stat_child function returns either the first child of stat, or NULL if the stat has no children.

The nng_stat_next function returns the nearest sibling to the right of stat, or NULL if stat has no more siblings to the right.

Finding a Statistic

const nng_stat *nng_stat_find(const nng_stat *stat, const char *name);
const nng_stat *nng_stat_find_dialer(const nng_stat *stat, nng_dialer dialer);
const nng_stat *nng_stat_find_listener(const nng_stat *stat, nng_dialer listener);
const nng_stat *nng_stat_find_socket(const nng_stat *stat, nng_dialer socket);

Sometimes it is easiest to search for a specific statistic, matching by name, or possibly to find the tree of statistics associated iwth a specific socket, dialer, or listener.

The nng_stat_find functions are provided for this purpose.

The nng_stat_find function returns the first statistic within the subtree of statistics stat, with the given name. If no such statistic can be found, NULL is returned.

The nng_stat_find_dialer, nng_stat_find_listener, and nng_stat_find_socket return the statistics subtree for the given dialer, listener, or socket object. If no such statistic can be found, then they return NULL. These functions should be provided the root of the statistic tree, in order to ensure that they can find the desired object.

Statistic Identification

const char *nng_stat_name(const nng_stat *stat);
const char *nng_stat_desc(const nng_stat *stat);

Every statistic has a name, returned by nng_stat_name, and a description, returned by nng_stat_desc. Descriptions are human-readable text, which might be useful for display.

Statistic Type

int nng_stat_type(const nng_stat *stat);

The function nng_stat_type returns the type of the statistic. The type of a statistic determines the nature of the value, and which function can be used to obtain that value.

• NNG_STAT_SCOPE: The statistic does not carry any real value, but is used for grouping related statistics together. This is a nexus in the statistics tree.

• NNG_STAT_COUNTER: The statistic is a counter that only increments. Usually the change in the value of the statistic is more interesting (as a rate) than the absolute value at any given time. The value should be obtained using nng_stat_value. The units will be given by the value returned from nng_stat_unit.

• NNG_STAT_LEVEL: The statistic represnts a measured value which corresponds to a specific value at a specific time. For example, this may represent the number of messages currently queued for some operation, or the link speed of a network interface. Most often the absolute value is more interesting than the change in the value over time. Again the value can be obtained with nng_stat_value, and any appropriate unit of measurement with nng_stat_unit.

• NNG_STAT_STRING: The statistic is a string, such as a name. The value of the string can be obtained with nng_stat_string. The value of this string will remain valid until the snapshot is deallocated with nng_stats_free.

• NNG_STAT_BOOLEAN: The value of the statistic is a truth value (either true or false) and can be obtained with nng_stat_bool.

• NNG_STAT_ID: The value of the statistic is a numeric identifier, such as a socket identifier. The value can be obtained with nng_stat_value, and will be fixed for the life of the statistic.

Statistic Value

uint64_t nng_stat_value(const nng_stat *stat);
const char *nng_stat_string(const nng_stat *stat);
bool nng_stat_bool(const nng_stat *stat);

These functions return the value associated with the statistic.

The nng_stat_value function returns the the numeric value for the statistic stat of type NNG_STAT_COUNTER, NNG_STAT_LEVEL, or NNG_STAT_ID. If stat is not one of these types, then it returns zero.

The nng_stat_bool function returns the Boolean value (either true or false) for the statistic stat of type NNG_STAT_BOOLEAN. If the statistics is not of this type, then it returns false.

The nng_stat_string function returns a pointer to a string value for the statistic stat, of type NNG_STAT_STRING. This string will remain valud until the snapshot that stat was collected with is deallocated with nng_stats_free. If the statistic is not of type NNG_STAT_STRING, then NULL is returned.

Statistic Units

int nng_stat_unit(const nng_stat *stat);

For statistics of type NNG_STAT_COUNTER or NNG_STAT_LEVEL, it is often useful to know what that quantity being reported measures. The following units may be returned from nng_stat_unit for such a statistic:

• NNG_UNIT_NONE: No unit is known or applies.
• NNG_UNIT_BYTES: A count of bytes.
• NNG_UNIT_MESSAGES: A count of messages.
• NNG_UNIT_MILLIS: A count of milliseconds.
• NNG_UNIT_EVENTS: A count of events of some type.

Statistic Timestamp

uint64_t nng_stat_timestamp(const nng_stat *stat);

Statistics have a timestamp indicating when the value was sampled, obtained via nng_stat_timestamp. The timestamp is given in in milliseconds since a reference time, and the reference time used here is the same reference time used for nng_clock.

See Also

nng_clock

Errors

Many NNG functions can fail for a variety of reasons. These functions tend to return either zero on success, or a non-zero error code to indicate failure. ¹

All these error codes are int.

Not every possible error code is defined here, as sometimes an underlying system or library error code is “wrapped”.

Human Readable Error Message

const char *nng_strerror(int err);

The nng_strerror returns the human-readable description of the given error in err.

The error message returned is a fixed NUL-terminated string and may be located in read-only memory.

The returned error message is provided in US English, but in the future locale-specific strings may be presented instead.

List of Errors

1: This convention goes back to UNIX system calls, which behave the same way, but NNG does not use a separate errno variable.

Miscellaneous

This chapter discusses some interfaces that don’t really fit anywhere else.

Get Random Number

uint32_t nng_random(void);

The nng_random returns a random number. The value returned is suitable for use with cryptographic functions such as key generation, and is obtained using platform-specific cryptographically strong random number facilities when available.

Create Socket Pair

int nng_socket_pair(int fds[2]);

The nng_socket_pair function creates a pair of connected file descriptors. These file descriptors, which are returned in the fds array, are suitable for use with the Socket transport.

On POSIX platforms, this is a thin wrapper around the standard socketpair function, using the AF_UNIX family and the SOCK_STREAM socket type. ¹

This will return zero on success, or an error number. On platforms that lack this capability, such as Windows, it will return NNG_ENOTSUP.

Report Library Version

const char * nng_version(void);

The nng_version function returns a human readable version number for NNG, formatted as a NUL-terminated string.

Additionally, compile time version information is available via some predefined macros:

• NNG_MAJOR_VERSION: Major version number.
• NNG_MINOR_VERSION: Minor version number.
• NNG_PATCH_VERSION: Patch version number.

NNG is developed and released using Semantic Versioning 2.0, and the version numbers reported refer to both the API and the library itself. (The ABI – application binary interface – between the library and the application is controlled in a similar, but different manner depending upon the link options and how the library is built.)

1: At present only POSIX platforms implementing socketpair support this function.

ID Map

Internally, NNG uses a map of numeric identifiers to data structures. This feature is also exposed for application use, as a “supplemental” feature.

When using these functions, it is necessary to add the #include <nng/supplemental/util/idhash.h> include file to list of includes.

ID Map Structure

#include <nng/nng.h>
#include <nng/supplemental/util/idhash.h>

typedef struct nng_id_map_s nng_id_map;

The ID map structure, nng_id_map provides a table of identifiers mapping to user-supplied pointers (which must not be NULL). The identifiers can be thought of as indices into the table, with the pointers providing the reference for the user supplied data.

The values of identifiers can be supplied by the user, or can be allocated automatically by nng_id_map from a predefined range. The starting point for allocations can also be randomly within the range.

The identifiers are 64-bit unsigned integers and can be sparse; the structure will use space efficiently even if identifiers are very far apart. ¹

Create ID Map

#define NNG_MAP_RANDOM 1

int nng_id_map_alloc(nng_id_map **map_p, uint64_t lo, uint64_t hi, int flags);

The nng_id_map_alloc function allocates a map without any data in it, and returns a pointer to it in map_p. When allocating identifiers dynamically, the values will be chosen from the range defined by lo and hi, inclusive.

The flags argument is a bit mask of flags that can adjust behavior of the map. The only flag defined at present is NNG_MAP_RANDOM, which causes the first identifier allocation to start at a random point within the range. This is useful to reduce the odds of different instances of an application using the same identifiers at the same time.

If both lo and hi are zero, then the values 0 and 0xffffffff are substituted in their place, giving a full range of 32-bit identifiers.

This function can return NNG_ENOMEM if it is unable to allocate resources, otherwise it returns zero on success.

Destroy Map

void nng_id_map_free(nng_id_map *map);

The nng_id_map_free function destroys map, releasing any resources associated with it.

Store a Value

int nng_id_set(nng_id_map *map, uint64_t id, void *value);

The nng_id_map_set function is used to store the value in the map at index id.

If another value is already stored at that same location, then it is overwritten with value.

If the table has to grow to accommodate this value, it may fail if insufficient memory is available, returning NNG_ENOMEM. Otherwise it returns zero.

Lookup a Value

void *nng_id_get(nng_id_map *map, uint64_t id);

The nng_id_get function looks up the entry for id in map, returning the associated value if present, or NULL if no such entry exists.

Allocate an ID

int nng_id_alloc(nng_id_map *map, uint64_t *id_p, void *value);

The nng_id_alloc stores the value in the map, at a newly allocated index, and returns the index in id_p.

Identifiers are allocated in increasing order, without reusing old identifiers until the largest possible identifier is allocated. After wrapping, only identifiers that are no longer in use will be considered. No effort is made to order the availability of identifiers based on when they were freed.²

As with nng_id_set, this may need to allocate memory and can thus fail with NNG_ENOMEM.

Additionally, if there are no more free identifiers within the range specified when map was created, then it will return NNG_ENOSPC.

Otherwise it returns zero, indicating success.

Remove an ID

int nng_id_remove(nng_id_map *map, uint64_t id);

The nng_id_remove removes the entry at index id from map.

If no such entry exist, it will return NNG_ENOENT. Otherwise it returns zero.

Iterating IDs

bool nng_id_visit(nng_id_map *map, uint64_t *id_p, void **value_p, uint32_t *cursor);

The nng_id_visit function is used to iterate over all items in the table. The caller starts the iteration by setting the cursor to 0 before calling it. For each call, the associated key and value of the next item will be returned in id_p, and value_p and the cursor will be updated. When all items have been iterated, the function returns false. The order of items returned is not guaranteed to be sequential. The caller must not attempt to derive any value of the cursor as it refers to internal table indices.

Entries may be safely removed from map while iterating.

However, if new entries are added to the table while iterating, the result of iteration is undefined; entries may be repeated or omitted during such an iteration.

The caller must not attempt to derive any value of the cursor as it refers to internal table indices.

1: The ID map is capable of storing at most 2³² identifiers, even though the identifers may themselves be much larger than this.

2: The concern about possibly reusing a recently released identifier comes into consideration after the range has wrapped. Given a sufficiently large range, this is unlikely to be a concern.

Command Options

Some NNG utilities need to parse command line options, and the supplementary function here allows applications that need the same support to benefit from this.

To make use of this, the supplemental header <nng/supplemental/util/options.h> must be included.

Parse Command Line Options

typedef struct nng_optspec {
    const char *o_name;  // Long style name (may be NULL for short only)
    int         o_short; // Short option (no clustering!)
    int         o_val;   // Value stored on a good parse (>0)
    bool        o_arg;   // Option takes an argument if true
} nng_optspec;

int nng_opts_parse(int argc, char *const *argv,
                   const nng_optspec *spec, int *val, char **arg, int *idx);

The nng_opts_parse function is a intended to facilitate parsing command-line arguments. This function exists largely to stand in for getopt from POSIX systems, but it is available everywhere that NNG is, and it includes some capabilities missing from getopt.

The function parses arguments from main¹ (using argc and argv), starting at the index referenced by idx. (New invocations typically set the value pointed to by idx to 1.)

Options are parsed as specified by spec (see Option Specification.) The value of the parsed option will be stored at the address indicated by val, and the value of idx will be incremented to reflect the next option to parse.

If the option had an argument, a pointer to that is returned at the address referenced by arg.

This function should be called repeatedly, until it returns either -1 (indicating the end of options is reached) or a non-zero error code is returned.

This function may return the following errors:

• NNG_EAMBIGUOUS: Parsed option matches more than one specification.
• NNG_ENOARG: Option requires an argument, but one is not present.
• NNG_EINVAL: An invalid (unknown) argument is present.

Option Specification

The calling program must first create an array of nng_optspec structures describing the options to be supported. This structure has the following members:

• o_name:

  The long style name for the option, such as “verbose”. This will be parsed as a long option on the command line when it is prefixed with two dashes. It may be NULL if only a short option is to be supported.

• o_short:

  This is a single letter (at present only ASCII letters are supported). These options appear as just a single letter, and are prefixed with a single dash on the command line. The use of a slash in lieu of the dash is not supported, in order to avoid confusion with path name arguments. This value may be set to 0 if no short option is needed.

• o_val:

  This is a numeric value that is unique to this option. This value is assigned by the application program, and must be non-zero for a valid option. If this is zero, then it indicates the end of the specifications, and the rest of this structure is ignored. The value will be returned to the caller in val by nng_opts_parse when this option is parsed from the command line.

• o_arg:

  This value should be set to true if the option should take an argument.

Long Options

Long options are parsed from the argv array, and are indicated when the element being scanned starts with two dashes. For example, the “verbose” option would be specified as --verbose on the command line. If a long option takes an argument, it can either immediately follow the option as the next element in argv, or it can be appended to the option, separated from the option by an equals sign (=) or a colon (:).

Short Options

Short options appear by themselves in an argv element, prefixed by a dash (-). If the short option takes an argument, it can either be appended in the same element of argv, or may appear in the next argv element.

Prefix Matching

When using long options, the parser will match if it is equal to a prefix of the o_name member of a option specification, provided that it do so unambiguously (meaning it must not match any other option specification.)

Example

The following program fragment demonstrates this function.

    enum { OPT_LOGFILE, OPT_VERBOSE };
    char *logfile; // options to be set
    bool verbose;

    static nng_optspec specs[] = {
        {
            .o_name = "logfile",
            .o_short = 'D',
            .o_val = OPT_LOGFILE,
            .o_arg = true,
        }, {
            .o_name = "verbose",
            .o_short = 'V',
            .o_val = OPT_VERBOSE,
            .o_arg = false,
        }, {
            .o_val = 0; // Terminate array
        }
    };

    for (int idx = 1;;) {
        int rv, opt;
        char *arg;
        rv = nng_opts_parse(argc, argv, specs, &opt, &arg, &idx);
        if (rv != 0) {
            break;
        }
        switch (opt) {
        case OPT_LOGFILE:
            logfile = arg;
            break;
        case OPT_VERBOSE:
            verbose = true;
            break;
        }
    }
    if (rv != -1) {
        printf("Options error: %s\n", nng_strerror(rv));
        exit(1);
    }

1: Parsing argument strings from other sources can be done as well, although usually then idx will be initialized to zero.

Protocols

The Scalability Protocols are a principally a collection of common networking patterns found in applications.

The following patterns are included:

Request - Reply

The request/reply pattern is made up of the REQ and REP protocols. This most often used when implementing RPC-like services, where a given request is matched by a single reply.

Pipeline

The pipeline pattern is made up of the PUSH and PULL protocols.

In this pattern communication is half-duplex, in that one side sends data and another side receives.

This pattern is also characterized by its ability to solve distribution problems, and the fact that it has back-pressure, providing a measure of flow control to data production and consumption.

Publish - Subscribe

Bus

Pair

BUS Protocol

The BUS protocol provides for building mesh networks where every peer is connected to every other peer. In this protocol, each message sent by a node is sent to every one of its directly connected peers.

All message delivery in this pattern is best-effort, which means that peers may not receive messages. Furthermore, delivery may occur to some, all, or none of the directly connected peers. (Messages are not delivered when peer nodes are unable to receive.) Hence, send operations will never block; instead if the message cannot be delivered for any reason it is discarded.

Socket Operations

The nng_bus0_open functions create a bus socket. This socket may be used to send and receive messages. Sending messages will attempt to deliver to each directly connected peer.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The BUS protocol has no protocol-specific options.

Protocol Headers

When using a BUS socket in raw mode, received messages will contain the incoming pipe ID as the sole element in the header. If a message containing such a header is sent using a raw BUS socket, then, the message will be delivered to all connected pipes except the one identified in the header. This behavior is intended for use with device configurations consisting of just a single socket. Such configurations are useful in the creation of rebroadcasters, and this capability prevents a message from being routed back to its source. If no header is present, then a message is sent to all connected pipes.

When using normal (cooked mode) BUS sockets, no message headers are present.

PAIR protocol

The PAIR protocol implements a peer-to-peer pattern, where relationships between peers are one-to-one.

Socket Operations

The nng_pair_open functions create a PAIR socket.

Normally, this pattern will block when attempting to send a message if no peer is able to receive the message.

Protocol Versions

Version 0 is the legacy version of this protocol. It lacks any header information, and is suitable when building simple one-to-one topologies.

Version 1 of the protocol offers improved protection against loops when used with devices.

Polyamorous Mode

Normally pair sockets are for one-to-one communication, and a given peer will reject new connections if it already has an active connection to another peer.

Polyamorous changes this, to allow a socket to communicate with multiple directly-connected peers. This mode is enabled by opening a socket using nng_pair1_open_poly.

In polyamorous mode a socket can support many one-to-one connections. In this mode, the application must choose the remote peer to receive an outgoing message by setting the nng_pipe to use for the outgoing message using nng_msg_set_pipe.

If no remote peer is specified by the sender, then the protocol will select any available connected peer.

Most often the value of the outgoing pipe will be obtained from an incoming message using nng_msg_get_pipe, such as when replying to an incoming message.

In order to prevent head-of-line blocking, if the peer on the given pipe is not able to receive (or the pipe is no longer available, such as if the peer has disconnected), then the message will be discarded with no notification to the sender.

Protocol Options

The following protocol-specific options are available.

• NNG_OPT_MAXTTL: (int, version 1 only). Maximum time-to-live.

• NNG_OPT_PAIR1_POLY: (bool, version 1 only) This option is no longer supported. Formerly it was used to configure polyamorous mode, but that mode is now established by using the nng_pair1_open_poly function.

Protocol Headers

Version 0 of the pair protocol has no protocol-specific headers.

Version 1 of the pair protocol uses a single 32-bit unsigned value. The low-order (big-endian) byte of this value contains a “hop” count, and is used in conjunction with the NNG_OPT_MAXTTL option to guard against device forwarding loops. This value is initialized to 1, and incremented each time the message is received by a new node.

PUB Protocol

The PUB protocol is one half of a publisher/subscriber pattern. In this pattern, a publisher sends data, which is broadcast to all subscribers. The subscribing applications only see the data to which they have subscribed.

The PUB protocol is the publisher side, and the SUB protocol is the subscriber side.

The topics that subscribers subscribe to is just the first part of the message body. Applications should construct their messages accordingly.

Socket Operations

The nng_pub0_open functions create a publisher socket. This socket may be used to send messages, but is unable to receive them. Attempts to receive messages will result in NNG_ENOTSUP.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The PUB protocol has no protocol-specific options.

Protocol Headers

The PUB protocol has no protocol-specific headers.

PULL protocol

The PULL protocol is one half of a pipeline pattern. The other half is the PUSH protocol.

In the pipeline pattern, pushers distribute messages to pullers. Each message sent by a pusher will be sent to one of its peer pullers, chosen in a round-robin fashion from the set of connected peers available for receiving. This property makes this pattern useful in load-balancing scenarios.

Socket Operations

The nng_pull0_open functions create a PULL socket. This socket may be used to receive messages, but is unable to send them. Attempts to send messages will result in NNG_ENOTSUP.

When receiving messages, the PULL protocol accepts messages as they arrive from peers. If two peers both have a message ready, the order in which messages are handled is undefined.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The PULL protocol has no protocol-specific options.

Protocol Headers

The PULL protocol has no protocol-specific headers.

PUSH protocol

DESCRIPTION

The PUSH protocol is one half of a pipeline pattern. The other side is the PULL protocol.

In the pipeline pattern, pushers distribute messages to pullers. Each message sent by a pusher will be sent to one of its peer pullers, chosen in a round-robin fashion from the set of connected peers available for receiving. This property makes this pattern useful in load-balancing scenarios.

Socket Operations

The nng_push0_open call creates a PUSH socket. This socket may be used to send messages, but is unable to receive them. Attempts to receive messages will result in NNG_ENOTSUP.

Send operations will observe flow control (back-pressure), so that only peers capable of accepting a message will be considered. If no peer is available to receive a message, then the send operation will wait until one is available, or the operation times out.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

• NNG_OPT_SENDBUF: (int, 0 - 8192) Normally this is set to zero, indicating that send operations are unbuffered. In unbuffered operation, send operations will wait until a suitable peer is available to receive the message. If this is set to a positive value (up to 8192), then an intermediate buffer is provided for the socket with the specified depth (in messages).

Protocol Headers

The PUSH protocol has no protocol-specific headers.

REP Protocol

The REP protocol is one half of a request/reply pattern. In this pattern, a requester sends a message to one replier, who is expected to reply. The request is resent if no reply arrives, until a reply is received or the request times out.

The REP protocol is the replier side, and the REP protocol is the requester side.

Socket Operations

The nng_rep0_open functions create a replier socket. This socket may be used to receive messages (requests), and then to send replies.

Generally a reply can only be sent after receiving a request.

Send operations will result in NNG_ESTATE if no corresponding request was previously received.

Likewise, only one receive operation may be pending at a time. Any additional concurrent receive operations will result in NNG_ESTATE.

Raw mode sockets ignore all these restrictions.

Context Operations

This protocol supports the creation of contexts for concurrent use cases using nng_ctx_open.

Each context may have at most one outstanding request, and operates independently of the others. The restrictions for order of operations with sockets apply equally well for contexts, except that each context will be treated as if it were a separate socket.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The REP protocol has no protocol-specific options.

Protocol Headers

The REP protocol uses a backtrace in the header. This is more fully documented in the REQ chapter.

REQ protocol

The REQ protocol is one half of a request/reply pattern. In this pattern, a requester sends a message to one replier, who is expected to reply. The request is resent if no reply arrives, until a reply is received or the request times out.

The requester generally only has one outstanding request at a time unless in raw mode, and it will generally attempt to spread work requests to different peer repliers.

The REQ protocol is the requester side, and the REP protocol is the replier side.

Socket Operations

The nng_req0_open functions create a REQ socket. This socket may be used to send messages (requests), and then to receive replies.

Generally a reply can only be received after sending a request. (Attempts to receive a message will result in NNG_ESTATE if there is no outstanding request.)

Furthermore, only a single receive operation may be pending at a time. Attempts to post more receive operations concurrently will result in NNG_ESTATE.

Requests may be canceled by sending a different request. This will cause the requester to discard any reply from the earlier request, but it will not stop a replier from processing a request it has already received or terminate a request that has already been placed on the wire.

Raw mode sockets ignore all these restrictions.

Context Operations

This protocol supports the creation of contexts for concurrent use cases using nng_ctx_open.

The NNG_OPT_REQ_RESENDTIME value may be configured differently on contexts created this way.

Each context may have at most one outstanding request, and operates independently from the others.

The restrictions for order of operations with sockets apply equally well for contexts, except that each context will be treated as if it were a separate socket.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The following protocol-specific option is available.

• NNG_OPT_REQ_RESENDTIME:
  (nng_duration)
  When a new request is started, a timer of this duration is also started. If no reply is received before this timer expires, then the request will be resent.
  
  Requests are also automatically resent if the peer to whom the original request was sent disconnects.
  
  Resending may be deferred up to the value of the NNG_OPT_RESENDTICK parameter.
  
  If the value is set to NNG_DURATION_INFINITE, then resends are disabled altogether. This should be used when the request is not idemptoent.

• NNG_OPT_REQ_RESENDTICK:
  (nng_duration)
  This is the granularity of the clock that is used to check for resending. The default is a second. Setting this to a higher rate will allow for more timely resending to occur, but may incur significant additional overhead when the socket has many outstanding requests (contexts).
  
  When there are no requests outstanding that have a resend set, then the clock does not tick at all.
  
  This option is shared for all contexts on a socket, and is only available for the socket itself.

Protocol Headers

This protocol uses a backtrace in the header. This form uses a stack of 32-bit big-endian identifiers. There must be at least one identifier, the request ID, which will be the last element in the array, and must have the most significant bit set.

There may be additional peer IDs preceding the request ID. These will be distinguishable from the request ID by having their most significant bit clear.

When a request message is received by a forwarding node (such as a device), the forwarding node prepends a 32-bit peer ID (which must have the most significant bit clear), which is the forwarder’s way of identifying the directly connected peer from which it received the message. (This peer ID, except for the most significant bit, has meaning only to the forwarding node itself.)

It may help to think of prepending a peer ID as pushing a peer ID onto the front of the stack of headers for the message. (It will use the peer ID it popped from the front to determine the next intermediate destination for the reply.)

When a reply message is created, it is created using the same headers that the request contained.

A forwarding node can pop the peer ID it originally pushed on the message, stripping it from the front of the message as it does so.

When the reply finally arrives back at the initiating requester, it should have only a single element in the message, which will be the request ID it originally used for the request.

RESPONDENT protocol

The RESPONDENT protocol is one half of a survey pattern. In this pattern, a surveyor sends a survey, which is broadcast to all peer respondents. The respondents then have a chance to reply (but are not obliged to reply). The survey itself is a timed event, so that responses received after the survey has finished are discarded.

The RESPONDENT protocol is the respondent side, and the SURVEYOR protocol is the surveyor side.

Socket Operations

The nng_respondent0_open functions create a respondent socket. This socket may be used to receive messages, and then to send replies. A reply can only be sent after receiving a survey, and generally the reply will be sent to surveyor from whom the last survey was received.

Respondents may discard a survey by simply not replying to it.

Raw mode sockets ignore all these restrictions.

Context Operations

This protocol supports the creation of contexts for concurrent use cases using nng_ctx_open.

Incoming surveys will be routed to and received by only one context. Additional surveys may be received by other contexts in parallel. Replies made using a context will be returned to the the surveyor that issued the survey most recently received by that context. The restrictions for order of operations with sockets apply equally well for contexts, except that each context will be treated as if it were a separate socket.

Protocol Versions

Only version 0 of this protocol is supported. At the time of writing, no other versions of this protocol have been defined. ¹

Protocol Options

The respondent protocol has no protocol-specific options.

Protocol Headers

The RESPONDENT protocol uses a backtrace in the header. This is more fully documented in the SURVEYOR manual.

1: An earlier and incompatible version of the protocol was used in older pre-releases of nanomsg, but was not released in any production version.

SUB protocol

The SUB protocol is one half of a publisher/subscriber pattern. In this pattern, a publisher sends data, which is broadcast to all subscribers. The subscribing applications only see the data to which they have subscribed.

The SUB protocol is the subscriber side, and the PUB protocol is the publisher side.

The topics that subscribers subscribe to is compared to the leading bytes of the message body. Applications should construct their messages accordingly.

Socket Operations

The nng_sub0_open functions create a SUB socket. This socket may be used to receive messages, but is unable to send them. Attempts to send messages will result in NNG_ENOTSUP.

Protocol Versions

Only version 0 of this protocol is supported. (At the time of writing, no other versions of this protocol have been defined.)

Protocol Options

The following protocol-specific option is available.

• NNG_OPT_SUB_PREFNEW:
  (bool)
  
  This read/write option specifies the behavior of the subscriber when the queue is full. When true (the default), the subscriber will make room in the queue by removing the oldest message. When false, the subscriber will reject messages if the message queue does not have room.

Protocol Headers

The SUB protocol has no protocol-specific headers.

SURVEYOR protocol

The SURVEYOR protocol is one half of a survey pattern. In this pattern, a surveyor sends a survey, which is broadcast to all peer respondents. The respondents then have a chance, but are not obliged, to reply. The survey itself is a timed event, so that responses received after the survey has finished are discarded.

The SURVEYOR protocol is the surveyor side, and the RESPONDENT protocol is the respondent side.

Socket Operations

The nng_surveyor0_open functions create a surveyor socket. This socket may be used to send messages (surveys), and then to receive replies. A reply can only be received after sending a survey. A surveyor can normally expect to receive at most one reply from each responder. (Messages can be duplicated in some topologies, so there is no guarantee of this.)

Attempts to receive on a socket with no outstanding survey will result in NNG_ESTATE. If the survey times out while the surveyor is waiting for replies, then the result will be NNG_ETIMEDOUT.

Only one survey can be outstanding at a time; sending another survey will cancel the prior one, and any responses from respondents from the prior survey that arrive after this will be discarded.

Raw mode sockets ignore all these restrictions.

Context Operations

This protocol supports the creation of contexts for concurrent use cases using nng_ctx_open.

Each context can initiate its own surveys, and it will receive only responses to its own outstanding surveys. Other contexts on the same socket may have overlapping surveys operating at the same time.

Each of these may have their own timeouts established with NNG_OPT_SURVEYOR_SURVEYTIME.

Additionally, sending a survey on a context will only cancel an outstanding survey on the same context.

Protocol Versions

Only version 0 of this protocol is supported. At the time of writing, no other versions of this protocol have been defined. ¹

Protocol Options

The following protocol-specific option is available.

• NNG_OPT_SURVEYOR_SURVEYTIME:
  (nng_duration)
  
  When a new survey is started, a timer of this duration is started. Any responses arriving this time will be discarded. Attempts to receive after the timer expires with no other surveys started will result in NNG_ESTATE.
  
  If a receive is pending when this timer expires, it will result in NNG_ETIMEDOUT.

Protocol Headers

This form uses a stack of 32-bit big-endian identifiers. There must be at least one identifier, the survey ID, which will be the last element in the array, and must have the most significant bit set.

There may be additional peer IDs preceding the survey ID. These will be distinguishable from the survey ID by having their most significant bit clear.

When a survey message is received by a forwarding node (such as a device), the forwarding node prepends a 32-bit peer ID (which must have the most significant bit clear), which is the forwarder’s way of identifying the directly connected peer from which it received the message. (This peer ID, except for the most significant bit, has meaning only to the forwarding node itself.)

It may help to think of prepending a peer ID as pushing a peer ID onto the front of the stack of headers for the message. (It will use the peer ID it popped from the front to determine the next intermediate destination for the response.)

When a response message is created, it is created using the same headers that the survey contained.

A forwarding node can pop the peer ID it originally pushed on the message, stripping it from the front of the message as it does so.

When the response finally arrives back at the initiating surveyor, it should have only a single element in the message, which will be the survey ID it originally used for the request.

More detail can be found in the sp-surveyor-01 RFC document.

1: An earlier and incompatible version of the protocol was used in older pre-releases of nanomsg, but was not released in any production version.

Transports

This section documents transports for Scalabity Protocols implemented by NNG.

• BSD Socket
• Intra-Process Transport
• UDP

INPROC Transport

The inproc transportintra-process provides communication support between sockets within the same process. This may be used as an alternative to slower transports when data must be moved within the same process.

This transport tries hard to avoid copying data, and thus is very light-weight.

URI Format

This transport uses URIs using the scheme inproc://, followed by an arbitrary string of text, terminated by a NUL byte.

Multiple URIs can be used within the same application, and they will not interfere with one another.

Two applications may also use the same URI without interfering with each other, and they will be unable to communicate with each other using that URI.

Socket Address

When using an nng_sockaddr structure, the actual structure is of type nng_sockaddr_inproc.

Transport Options

The inproc transport has no special options.

Mixing Implementations

When mixing the NNG library with other implementations of these protocols in the same process (such as the mangos or libnanomsg implementations), it will not be possible to utilize the inproc transport to communicate across this boundary.

This limitation also extends to using different instances of the NNG library within the same process.

IPC Transport

DESCRIPTION

The ipc transport provides communication support between sockets within different processes on the same host. For POSIX platforms, this is implemented using UNIX domain sockets. For Windows, this is implemented using Windows named pipes. Other platforms may have different implementation strategies.

URI Formats

Traditional Names

This transport uses URIs using the scheme ipc://, followed by a path name in the file system where the socket or named pipe should be created.

UNIX Aliases

The unix:// scheme is an alias for ipc:// and can be used inter-changeably, but only on POSIX systems. ¹

Abstract Names

On Linux, this transport also can support abstract sockets. Abstract sockets use a URI-encoded name after the abstract:// scheme, which allows arbitrary values to be conveyed in the path, including embedded NUL bytes. For example, the name "a\0b" would be represented as abstract://a%00b.

Abstract names do not include the leading NUL byte used in the low-level socket address.

Abstract sockets do not have any representation in the file system, and are automatically freed by the system when no longer in use.

Abstract sockets ignore socket permissions, but it is still possible to determine the credentials of the peer with NNG_OPT_PEER_UID, and similar options. ²

Socket Address

When using an nng_sockaddr structure, the actual structure is of type nng_sockaddr_ipc, except for abstract sockets, which use nng_sockaddr_abstract.

Transport Options

The following transport options are supported by this transport, where supported by the underlying platform.

• NNG_OPT_IPC_PERMISSIONS
• NNG_OPT_IPC_SECURITY_DESCRIPTOR
• NNG_OPT_LOCADDR
• NNG_OPT_REMADDR
• NNG_OPT_PEER_GID
• NNG_OPT_PEER_PID
• NNG_OPT_PEER_UID
• NNG_OPT_PEER_ZONEID
• NNG_OPT_URL

1: The purpose of this scheme is to support a future transport making use of AF_UNIX on Windows systems, at which time it will be necessary to discriminate between the Named Pipes and the AF_UNIX based transports.

2: This property makes it important that names be chosen randomly to prevent unauthorized access, or that checks against the peer credentials are made, or ideally, both.

Socket Transport (Experimental)

Description

The socket transport supports communication between peers across arbitrary BSD sockets, such as those that are created with nng_socket_pair.

This transport only supports listeners, using nng_listener_create.

The socket file descriptor is passed to the listener using the NNG_OPT_SOCKET_FD option (as an integer). Setting this option will cause the listener to create a pipe backed by the file descriptor.

The protocol between peers using this transport is compatible with the protocol used for the tcp transport, but this is an implementation detail and subject to change without notice. ¹

Registration

No special action is necessary to register this transport.

URI Format

This transport uses the URL socket://, without further qualification.

Socket Address

The socket address will be of family NNG_AF_UNSPEC. There are no further socket details available.

Transport Options

The following transport option is available:

• NNG_OPT_SOCKET_FD:
  (int)
  
  This is a write-only option, that may be set multiple times on a listener. Each time this is set, the listener will create a pipe backed by the given file descriptor passed as an argument.

Additionally, the following options may be supported on pipes when the platform supports them:

• NNG_OPT_PEER_GID
• NNG_OPT_PEER_PID
• NNG_OPT_PEER_UID
• NNG_OPT_PEER_ZONEID

1: Specifically it is not compatible with the ipc transport.

2: Windows lacks a suitable socketpair equivalent function we could use.

UDP Transport (Experimental)

Description

The udp transport supports communication between peers using UDP.

UDP is a very light-weight connection-less, unreliable, unordered delivery mechanism.

Both IPv4 and IPv6 are supported when the underlying platform also supports it.

This transport adds an ordering guarantee, so that messages will always be received in the correct order. Messages that arrive out of order, or are duplicated, will be dropped. There may be gaps in the messages received, so applications should not assume that all messages sent will arrive.

URI Format

This transport uses URIs using the scheme udp://, followed by an IP address or hostname, followed by a colon and finally a UDP port number. For example, to contact port 8001 on the localhost either of the following URIs could be used: udp://127.0.0.1:8001 or udp://localhost:8001.

A URI may be restricted to IPv6 using the scheme udp6://, and may be restricted to IPv4 using the scheme udp4://.

When specifying IPv6 addresses, the address must be enclosed in square brackets ([]) to avoid confusion with the final colon separating the port.

For example, the same port 8001 on the IPv6 loopback address (::1) would be specified as udp://[::1]:8001.

The special value of 0 (INADDR_ANY) can be used for a listener to indicate that it should listen on all interfaces on the host. A short-hand for this form is to either omit the address, or specify the asterisk (*) character. For example, the following three URIs are all equivalent, and could be used to listen to port 9999 on the host:

1. udp://0.0.0.0:9999
2. udp://*:9999
3. udp://:9999

Socket Address

When using an nng_sockaddr structure, the actual structure is either of type nng_sockaddr_in (for IPv4) or nng_sockaddr_in6 (for IPv6).

Transport Options

The following transport options are supported by this transport, where supported by the underlying platform.

• NNG_OPT_LOCADDR
• NNG_OPT_REMADDR
• NNG_OPT_URL

TODO: Document other options.

Maximum Message Size

This transport maps each SP message to a single UDP packet. In order to allow room for network headers, we thus limit the maximum message size to 65000 bytes, minus the overhead for any SP protocol headers.

However, applications are strongly encouraged to only use this transport for very much smaller messages, ideally those that will fit within a single network packet without requiring fragmentation and reassembly.

For Ethernet without jumbo frames, this typically means an MTU of a little less than 1500 bytes. (Specifically, 1452, which allows 28 bytes for IP and UDP, and 20 bytes for the this transport). Other link layers may have different MTUs.

The maximum message size is negotiated as part of establishing a peering relationship, and oversize messages will be dropped by the sender before going to the network.

The maximum message size to receive can be configured with the NNG_OPT_RECVMAXSZ option.

Keep Alive

This transports maintains a logical “connection” with each peer, to provide a rough facsimile of a connection based semantic. This requires some resource on each peer. In order to ensure that resources are reclaimed when a peer vanishes unexpectedly, a keep-alive mechanism is implemented.

TODO: Document the tunables for this.

Migration Guides

This section provides some guides to aid in migrating software from earlier versions NNG or libnanomsg.

• Migrating from NNG 1.x
• Migrating from libnanomsg

Migrating from NNG 1.x

There are some incompatibities from NNG 1.x. This guide should help in migrating applications to use NNG 2.0.

Nanomsg Compatibility

Applications using the legacy libnanomsg API will have to be updated to native NNG interfaces. See the Migrating From libnanomsg chapter for details.

Library Initialization

It is now required for applications to initialize the library explicitly before using it. This is done using the nng_init function.

Transport Specific Functions

Transports have not needed to be registered for a long time now, and the functions for doing so have been removed. These functions can be simply removed from your application:

• nng_inproc_register
• nng_ipc_register
• nng_tls_register
• nng_tcp_register
• nng_ws_register
• nng_wss_register
• nng_zt_register

Additionally, the header files containing these functions have been removed, such as nng/transport/ipc/ipc.h. Simply remove #include references to those files.

(Special exception: The options for ZeroTier are still located in the nng/transport/zerotier/zerotier.h.)

The NNG_OPT_WSS_REQUEST_HEADERS and NNG_OPT_WSS_RESPONSE_HEADERS aliases for NNG_OPT_WS_OPT_WS_REQUEST_HEADERS and NNG_OPT_WS_RESPONSE_HEADERS have been removed. Just convert any use of them to NNG_OPT_WS_REQUEST_HEADERS or NNG_OPT_WS_RESPONSE_HEADERS as appropriate.

TLS Configuration

The support for configuring TLS via NNG_OPT_TLS_CONFIG, NNG_TLS_AUTH_MODE, NNG_OPT_TLS_CA_FILE, NNG_OPT_TLS_SERVER_NAME, and similar has been removed.

Instead configuration must be performed by allocating a nng_tls_config object, and then setting fields on it using the appropriate functions, after which it may be configured on a listener or dialer using the nng_listener_set_tls or nng_dialer_set_tls functions.

Likewise, when using the streams API, use the nng_stream_listener_set_tls or nng_stream_dialer_set_tls functions.

Note that the declarations needed for TLS configuration are now available in <nng/nng.h>, rather than the supplemental header.

Old TLS Versions Removed

Support for very old TLS versions 1.0 and 1.1 is removed. Further, the NNG_TLS_1_0 and NNG_TLS_1_1 constants are also removed. Applications should use NNG_TLS_1_2 or even NNG_TLS_1_3 instead.

Support for Local Addresses in Dial URLs Removed

NNG 1.x had an undocumented ability to specify the local address to bind to when dialing, by using the local address in front of the destination address separated by a semicolon. This was provided for legacy libnanomsg compatilibility, and is no longer offered. The correct way to specify a local address is by setting NNG_OPT_LOCADDR on the dialer.

Option Functions

The previously deprecated nng_pipe_getopt_xxx family of functions is removed. Applications should use nng_pipe_get and related functions instead.

The socket option function families for nng_getopt and nng_setopt have been removed as well. In this case, use the nng_socket_get and nng_socket_set functions as appropriate.

The _getopt and _setopt functions for contexts, listeners, and dialers are no longer present. Simply changing _getopt to _get or _setopt to _set in the function name should be sufficient in most cases.

Untyped Option Functions Removed

The following functions are removed. To access options, use a proper typed access function, such as one ending in a suffix like _bool (to access a bool typed option).

• nng_ctx_get
• nng_ctx_set
• nng_dialer_get
• nng_dialer_set
• nng_listener_get
• nng_listener_set
• nng_pipe_get
• nng_socket_get
• nng_socket_set
• nng_stream_get
• nng_stream_set
• nng_stream_dialer_get
• nng_stream_dialer_set
• nng_stream_listener_get
• nng_stream_listener_set

Stream Options

The ability to set options on streams after they have been created is no longer present. (It turns out that this was not very useful.) All functions nng_stream_set_xxx are removed. For tuning the NNG_OPT_TCP_NODELAY or similar properties, set the option on the listener or dialer that creates the stream instead.

Transport Options

A number of transport options can no longer be set on the socket. Instead these options must be set on the endpoint (dialer or listener) using the appropriate nng_dialer_set or nng_listener_set option. This likely means that it is necessary to allocate and configure the endpoint before attaching it to the socket. This will also afford a much more fine-grained level of control over transport options.

The following options are copied from the socket when creating a dialer or listener, but afterwards will not be changed on the dialer or listener if the socket changes. It is recommended to set them properly on the socket before creating dialers or listeners, or set them explicitly on the dialer or listener directly:

• NNG_OPT_RECONNMINT
• NNG_OPT_RECONNMAXT
• NNG_OPT_RECVMAXSZ

The latter option is a hint for transports and intended to facilitate early detection (and possibly avoidance of extra allocations) of oversize messages, before bringing them into the socket itself.

Socket Options

The NNG_OPT_PROTO, NNG_OPT_PROTONAME, NNG_OPT_PEER, and NNG_OPT_PEERNAME options have been replaced by functions instead of options. Use nng_socket_proto_id, nng_socket_peer_id, nng_socket_proto_name, and nng_socket_peer_name instead. Note that the new functions provide a reference to a static string, and thus do not require allocation, and the returned strings should not be freed. Also the IDs are provided as uint16_t, matching the actual wire protocol values, instead of int.

The NNG_OPT_RAW option has aso been replaced by a function, nng_socket_raw.

The NNG_OPT_SENDFD and NNG_OPT_RECVFD options have been replaced by nng_socket_get_send_poll_fd and nng_socket_get_recv_poll_fd respectively.

The NNG_OPT_SOCKNAME function is removed. This was provided for application use, and never used internally by NNG. Applications should keep track of this information separately.

Subscriptions

The NNG_OPT_SUB_SUBSCRIBE and NNG_OPT_SUB_UNSUBCRIBE options have been replaced by the following functions: nng_sub0_socket_subscribe, nng_sub0_socket_unsubscribe, nng_sub0_ctx_subscribe and nng_sub0_ctx_unsubscribe. These functions, like the options they replace, are only applicable to SUB sockets.

Statistics Use Constified Pointers

A number of the statistics functions take, or return, const nng_stat * instead of plain nng_stat *. The ABI has not changed, but it may be necessary to declare certain methods variables const to avoid warnings about misuse of const.

Wildcards Not Valid in URLs

The use of * to act as a wild card meaning all local interface addresses is removed. The empty string already performs this function, and unlike * is RFC compliant.

URL Structure Members

The details of nng_url have changed as follows:

• u_port is no longer a string, but a uint16_t
• u_scheme is a const char *
• u_requri is removed - it can be easily formulated from the other fields.
• u_host is removed - use u_hostname and u_port to construct if needed
• u_rawurl is removed - a “cooked” URL can be obtained from the new nng_url_sprintf function.

Migrating from libnanomsg

Previous version of NNG offered direct API compatibility with libnanomsg, but that support is no longer offered in this version.

If your application is still using legacy libnanomsg APIs, you will need to update it for this version of NNG.

Header Files

Most applications can replace all #include <nn/*.h> statements with #include <nng/nng.h>.

Link Libraries

Replace -lnanomsg with -lnng. It may be necessary to include additional system libraries, such as -lpthread, depending on your system.

Initialization

It is necessary to explicitly initialize the library, using the nng_init function. This must be called before any other function.

Types

Sockets, dialers, and listeners in libnanomsg are simple integers. In NNG, these are struct types.

Messages are quite different in NNG, with the absence of the POSIX message control headers.

The struct nn_msghdr structure has no equivalent. See nng_msg for the NNG approach to messages. Likewise there is no struct nn_cmsghdr equivalent.

API Conversions

Options

The following options are changed.

Error Codes

Most of the error codes have similar names in NNG, just prefixed with NNG_. There are some exceptions. Be aware that the numeric values are not the same.

Local Addresses for Dialing

The ability to specify the source address in the UROL,to use when using nn_dial inside the URL is not present in NNG. The correct way to specify the local address is using the NNG_OPT_LOCADDR option on the dialer before starting to dial.

Index

ABI, 1
abstract sockets, 1
abstract://, 1
AF_UNIX, 1
aio, 1
allocations, 1
application binary interface, 1
asynchronous I/O, 1
back-pressure, 1
backtrace, 1, 2, 3, 4
best-effort, 1
body, 1
BUS, 1
BUS protocol, 1
callback, 1
clock, 1
command-line arguments, 1
concurrency, 1
condition variable, 1
CPU-bound, 1
duration, 1
error message, 1
event loop, 1
flow control, 1
getopt, 1
half-duplex, 1
header, 1
idempotent, 1
INADDR_ANY, 1
inproc, 1
inproc transport, 1
inproc://, 1
intra-process, 1
ipc, 1
ipc transport, 1
ipc://, 1
IPv4, 1
IPv6, 1
leader election, 1
load-balancing, 1, 2, 3
Log handlers, 1
memory, 1
messages, 1
MTU, 1
mutex, 1
named pipes, 1
NNG_AF_UNSPEC, 1
nng_aio, 1
nng_aio_abort, 1
nng_aio_alloc, 1
nng_aio_busy, 1
nng_aio_cancel, 1
nng_aio_count, 1
nng_aio_free, 1
nng_aio_get_msg, 1
nng_aio_reap, 1
nng_aio_result, 1
nng_aio_set_expire, 1
nng_aio_set_iov, 1
nng_aio_set_msg, 1
nng_aio_stop, 1
nng_aio_wait, 1
nng_alloc, 1
nng_clock, 1
nng_cv, 1
nng_cv_alloc, 1
nng_cv_free, 1
nng_cv_until, 1
nng_cv_wait, 1
nng_cv_wake, 1
nng_cv_wake1, 1
nng_duration, 1
NNG_DURATION_DEFAULT, 1
NNG_DURATION_INFINITE, 1
NNG_DURATION_ZERO, 1
NNG_ECANCELED, 1
nng_fini, 1
nng_free, 1
nng_id_alloc, 1
nng_id_get, 1
nng_id_map, 1
nng_id_map_alloc, 1
nng_id_map_free, 1
nng_id_map_set, 1
nng_id_remove, 1
nng_id_visit, 1
nng_init, 1
nng_iov, 1
nng_log, 1
nng_log_auth, 1
nng_log_get_level, 1
nng_log_level, 1
nng_log_set_facility, 1
nng_log_set_level, 1
nng_log_set_logger, 1
nng_logger, 1
NNG_MAJOR_VERSION, 1
NNG_MINOR_VERSION, 1
nng_msg, 1
nng_msg_alloc, 1
nng_msg_append, 1
nng_msg_body, 1
nng_msg_capacity, 1
nng_msg_chop, 1
nng_msg_clear, 1
nng_msg_dup, 1
nng_msg_free, 1
nng_msg_get_pipe, 1
nng_msg_header, 1
nng_msg_header_append, 1
nng_msg_header_chop, 1
nng_msg_header_clear, 1
nng_msg_header_insert, 1
nng_msg_header_len, 1
nng_msg_header_trim, 1
nng_msg_insert, 1
nng_msg_len, 1
nng_msg_realloc, 1
nng_msg_reserve, 1
nng_msg_set_pipe, 1
nng_msg_trim, 1
nng_msleep, 1
nng_mtx, 1
nng_mtx_free, 1
nng_mtx_lock, 1, 2
nng_mtx_unlock, 1
nng_null_logger, 1
NNG_OPT_LOCADDR, 1
NNG_OPT_PEER_UID, 1
NNG_OPT_REQ_RESENDTICK, 1
NNG_OPT_REQ_RESENDTIME, 1
NNG_OPT_SOCKET_FD, 1, 2
NNG_OPT_SUB_PREFNEW, 1
NNG_OPT_SURVEYOR_SURVEYTIME, 1
nng_opts_parse, 1
nng_optspec, 1
NNG_PATCH_VERSION, 1
nng_random, 1
nng_sleep_aio, 1
nng_socket, 1
nng_socket_get_recv_poll_fd, 1
nng_socket_get_send_poll_fd, 1
nng_socket_id, 1
nng_socket_peer_id, 1
nng_socket_peer_name, 1
nng_socket_proto_id, 1
nng_socket_proto_name, 1
nng_socket_raw, 1
nng_stat, 1
nng_stat_bool, 1
NNG_STAT_BOOLEAN, 1
nng_stat_child, 1
NNG_STAT_COUNTER, 1
nng_stat_desc, 1
nng_stat_find, 1
nng_stat_find_dialer, 1
nng_stat_find_listener, 1
nng_stat_find_socket, 1
NNG_STAT_ID, 1
NNG_STAT_LEVEL, 1
nng_stat_name, 1
nng_stat_next, 1
NNG_STAT_SCOPE, 1
NNG_STAT_STRING, 1
nng_stat_string, 1
nng_stat_timestamp, 1
nng_stat_type, 1
nng_stat_unit, 1
nng_stat_value, 1
nng_stats_free, 1
nng_stats_get, 1
nng_stderr_logger, 1
nng_strdup, 1
nng_strerror, 1
nng_strfree, 1
nng_system_logger, 1
nng_thread, 1
nng_thread_create, 1
nng_thread_destroy, 1
nng_thread_set_name, 1
nng_time, 1
NNG_UNIT_BYTES, 1
NNG_UNIT_EVENTS, 1
NNG_UNIT_MESSAGES, 1
NNG_UNIT_MILLIS, 1
NNG_UNIT_NONE, 1
nng_url_clone, 1
nng_url_free, 1
nng_url_parse, 1
nng_url_sprintf, 1
nng_version, 1
PAIR, 1
PAIR protocol, 1
pipeline pattern, 1, 2, 3
polyamorous mode, 1
port number, 1
protocol, 1
PUB, 1
PUB protocol, 1
publisher, 1
PULL, 1
PULL protocol, 1
PUSH, 1
PUSH protocol, 1
random number, 1
REP, 1
REP protocol, 1
REQ, 1
REQ protocol, 1
request/reply pattern, 1, 2, 3
RESPONDENT, 1
RESPONDENT protocol, 1
scheduling, 1
service discovery, 1
socket, 1
socket transport, 1
socket://, 1
statistics, 1
SUB, 1
SUB protocol, 1
subscriber, 1
survey pattern, 1, 2
SURVEYOR, 1
SURVEYOR protocol, 1
synchronization primitives, 1
threads, 1
thundering herd, 1
timeout, 1
UDP, 1
udp transport, 1
udp://, 1
Universal Resource Locator, 1
UNIX domain sockets, 1
unix://, 1
URL, 1
version number, 1
voting, 1