Merge github.com:grpc/grpc into cleaner-posix2

CONTRIBUTING.md

@@ -13,7 +13,7 @@ In order to protect both you and ourselves, you will need to sign the
 ### Technical requirements
 
 You will need several tools to work with this repository. In addition to all of
-the packages described in the [INSTALL](INSTALL) file, you will also need
+the packages described in the [INSTALL](INSTALL.md) file, you will also need
 python, and the mako template renderer. To install the latter, using pip, one
 should simply be able to do `pip install mako`.
 
@@ -21,6 +21,15 @@ In order to run all of the tests we provide, you will need valgrind and clang.
 More specifically, under debian, you will need the package libc++-dev to
 properly run all the tests.
 
+Compiling and running grpc C++ tests depend on protobuf 3.0.0, gtest and gflags.
+Although gflags is provided in third_party, you will need to manually install
+that dependency on your system to run these tests. Under a Debian or Ubuntu
+system, you can install the gtests and gflags packages using apt-get:
+
+```sh
+ $ [sudo] apt-get install libgflags-dev libgtest-dev
+```
+
 If you are planning to work on any of the languages other than C and C++, you
 will also need their appropriate development environments.
 

INSTALL

@@ -1,217 +0,0 @@
-These instructions only cover building grpc C and C++ libraries under
-typical unix systems. If you need more information, please try grpc's
-wiki pages:
-
-  https://github.com/google/grpc/wiki
-
-
-*************************
-* If you are in a hurry *
-*************************
-
-On Linux (Debian):
-
- Note: you will need to add the Debian 'jessie-backports' distribution to your sources
- file first.
-
- Add the following line to your `/etc/apt/sources.list` file:
-
-   deb http://http.debian.net/debian jessie-backports main
-
- Install the gRPC library:
-
- $ [sudo] apt-get install libgrpc-dev
-
-OR
-
- $ git clone https://github.com/grpc/grpc.git
- $ cd grpc
- $ git submodule update --init
- $ make 
- $ [sudo] make install
-
-You don't need anything else than GNU Make, gcc and autotools. Under a Debian
-or Ubuntu system, this should boil down to the following packages:
-
- $ [sudo] apt-get install build-essential autoconf libtool
-
-Building the python wrapper requires the following:
-
- $ [sudo] apt-get install python-all-dev python-virtualenv
-
-If you want to install in a different directory than the default /usr/lib, you can
-override it on the command line:
-
- $ [sudo] make install prefix=/opt
-
-
-*******************************
-* More detailled instructions *
-*******************************
-
-Setting up dependencies
-=======================
-
-Dependencies to compile the libraries
--------------------------------------
-
-grpc libraries have few external dependencies. If you need to compile and
-install them, they are present in the third_party directory if you have
-cloned the github repository recursively. If you didn't clone recursively,
-you can still get them later by running the following command:
-
-  $ git submodule update --init
-
-Note that the Makefile makes it much easier for you to compile from sources
-if you were to clone recursively our git repository: it will automatically
-compile zlib and OpenSSL, which are core requirements for grpc. Note this
-creates grpc libraries that will have zlib and OpenSSL built-in inside of them,
-which significantly increases the libraries' size.
-
-In order to decrease that size, you can manually install zlib and OpenSSL on
-your system, so that the Makefile can use them instead.
-
-Under a Debian or Ubuntu system, one can acquire the development package
-for zlib this way:
-
-  # apt-get install zlib1g-dev
-
-To the best of our knowledge, no distribution has an OpenSSL package that
-supports ALPN yet, so you would still have to depend on installing from source
-for that particular dependency if you want to reduce the libraries' size.
-
-The recommended version of OpenSSL that provides ALPN support is available
-at this URL:
-
-  https://www.openssl.org/source/openssl-1.0.2.tar.gz
-
-
-Dependencies to compile and run the tests
------------------------------------------
-
-Compiling and running grpc plain-C tests dont't require any more dependency.
-
-
-Compiling and running grpc C++ tests depend on protobuf 3.0.0, gtest and
-gflags. Although gflags is provided in third_party, you will need to manually
-install that dependency on your system to run these tests.
-
-Under a Debian or Ubuntu system, you can install the gtests and gflags packages
-using apt-get:
-
-  # apt-get install libgflags-dev libgtest-dev
-
-However, protobuf 3.0.0 isn't in a debian package yet, but the Makefile will
-automatically try and compile the one present in third_party if you cloned the
-repository recursively, and that it detects your system is lacking it.
-
-Compiling and installing protobuf 3.0.0 requires a few more dependencies in
-itself, notably the autoconf suite. If you have apt-get, you can install
-these dependencies this way:
-
-  # apt-get install autoconf libtool
-
-If you want to run the tests using one of the sanitized configurations, you
-will need clang and its instrumented libc++:
-
-  # apt-get install clang libc++-dev
-
-Mac-specific notes:
--------------------
-
-For a Mac system, git is not available by default. You will first need to
-install Xcode from the Mac AppStore and then run the following command from a
-terminal:
-
-  $ sudo xcode-select --install
-
-You should also install "port" following the instructions at
-https://www.macports.org . This will reside in /opt/local/bin/port for
-most Mac installations. Do the "git submodule" command listed above.
-
-Then execute the following for all the needed build dependencies
-
-  $ sudo /opt/local/bin/port install autoconf automake libtool gflags cmake
-  $ mkdir ~/gtest-svn
-  $ svn checkout http://googletest.googlecode.com/svn/trunk/ gtest-svn
-  $ mkdir mybuild
-  $ cd mybuild
-  $ cmake ../gtest-svn
-  $ make
-  $ make gtest.a gtest_main.a
-  $ sudo cp libgtest.a libgtest_main.a /opt/local/lib
-  $ sudo mkdir /opt/local/include/gtest
-  $ sudo cp -pr ../gtest-svn/include/gtest /opt/local/include/gtest
-
-If you are going to make changes and need to regenerate the projects file,
-you will need to install certain modules for python.
-
-  $ sudo easy_install simplejson mako
-
-Mingw-specific notes:
----------------------
-
-While gRPC compiles properly under mingw, some more preparation work is needed.
-The recommendation is to use msys2. The installation instructions are available
-at that address: http://msys2.github.io/
-
-Once this is installed, make sure you are using the following: MinGW-w64 Win64.
-You'll be required to install a few more packages:
-
-  $ pacman -S make mingw-w64-x86_64-gcc mingw-w64-x86_64-zlib autoconf automake libtool
-
-Please also install OpenSSL from that website:
-
-  http://slproweb.com/products/Win32OpenSSL.html
-
-The package Win64 OpenSSL v1.0.2a should do. At that point you should be able
-to compile gRPC with the following:
-
-  $ export LDFLAGS="-L/mingw64/lib -L/c/OpenSSL-Win64"
-  $ export CPPFLAGS="-I/mingw64/include -I/c/OpenSSL-Win64/include"
-  $ make
-
-A word on OpenSSL
------------------
-
-Secure HTTP2 requires the TLS extension ALPN (see rfc 7301 and
-http://http2.github.io/http2-spec/ section 3.3). Our HTTP2 implementation
-relies on OpenSSL's implementation. OpenSSL 1.0.2 is the first released version
-of OpenSSL that has ALPN support, and this explains our dependency on it.
-
-Note that the Makefile supports compiling only the unsecure elements of grpc,
-and if you do not have OpenSSL and do not want it, you can still proceed
-with installing only the elements you require. However, we strongly recommend
-the use of encryption for all network traffic, and discourage the use of grpc
-without TLS.
-
-
-Compiling
-=========
-
-If you have all the dependencies mentioned above, you should simply be able
-to go ahead and run "make" to compile grpc's C and C++ libraries:
-
-  $ make
-
-
-Testing
-=======
-
-To build and run the tests, you can run the command:
-
-  $ make test
-
-If you want to be able to run them in parallel, and get better output, you can
-also use the python tool we have written:
-
-  $ ./tools/run_tests/run_tests.py
-
-
-Installing
-==========
-
-Once everything is compiled, you should be able to install grpc C and C++
-libraries and headers:
-
-  # make install

INSTALL.md

@@ -0,0 +1,46 @@
+#If you are in a hurry
+
+For language-specific installation instructions for gRPC runtime, please
+refer to these documents
+
+ * [C++](examples/cpp)
+ * [C#](src/csharp): NuGet package `Grpc`
+ * [Go](https://github.com/grpc/grpc-go): `go get google.golang.org/grpc`
+ * [Java](https://github.com/grpc/grpc-java)
+ * [Node](src/node): `npm install grpc`
+ * [Objective-C](src/objective-c)
+ * [PHP](src/php): `pecl install grpc-beta`
+ * [Python](src/python/grpcio): `pip install grpcio`
+ * [Ruby](src/ruby): `gem install grpc`
+
+
+#Pre-requisites
+
+##Linux
+
+```sh
+ $ [sudo] apt-get install build-essential autoconf libtool
+```
+
+##Mac OSX
+
+For a Mac system, git is not available by default. You will first need to
+install Xcode from the Mac AppStore and then run the following command from a
+terminal:
+
+```sh
+ $ [sudo] xcode-select --install
+```
+
+#Build from Source
+
+For developers who are interested to contribute, here is how to compile the
+gRPC C Core library.
+
+```sh
+ $ git clone https://github.com/grpc/grpc.git
+ $ cd grpc
+ $ git submodule update --init
+ $ make 
+ $ [sudo] make install
+```

README.md

@@ -13,7 +13,7 @@ You can find more detailed documentation and examples in the [doc](doc) and [exa
 
 #Installation
 
-See [grpc/INSTALL](INSTALL) for installation instructions for various platforms.
+See [INSTALL](INSTALL.md) for installation instructions for various platforms.
 
 #Repository Structure & Status
 

doc/interop-test-descriptions.md

@@ -2,9 +2,8 @@ Interoperability Test Case Descriptions
 =======================================
 
 Client and server use
-[test.proto](https://github.com/grpc/grpc/blob/master/test/proto/test.proto)
-and the [gRPC over HTTP/2 v2
-protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md).
+[test.proto](../src/proto/grpc/testing/test.proto)
+and the [gRPC over HTTP/2 v2 protocol](./PROTOCOL-HTTP2.md).
 
 Client
 ------

examples/README.md

@@ -447,4 +447,4 @@ $ greeter_client
 ## Read more!
 
 - You can find links to language-specific tutorials, examples, and other docs in each language's [quick start](#quickstart).
-- [gRPC Authentication Support](doc/grpc-auth-support.md) introduces authentication support in gRPC with supported mechanisms and examples.
+- [gRPC Authentication Support](http://www.grpc.io/docs/guides/auth.html) introduces authentication support in gRPC with supported mechanisms and examples.

examples/cpp/README.md

@@ -2,7 +2,7 @@
 
 ## Installation
 
-To install gRPC on your system, follow the instructions [here](../../INSTALL).
+To install gRPC on your system, follow the instructions [here](../../INSTALL.md).
 
 ## Hello C++ gRPC!
 
@@ -23,21 +23,6 @@ Change your current directory to examples/cpp/helloworld
 $ cd examples/cpp/helloworld/
 ```
 
-
-### Generating gRPC code
-
-To generate the client and server side interfaces:
-
-```sh
-$ make helloworld.grpc.pb.cc helloworld.pb.cc
-```
-Which internally invokes the proto-compiler as:
-
-```sh
-$ protoc -I ../../protos/ --grpc_out=. --plugin=protoc-gen-grpc=grpc_cpp_plugin ../../protos/helloworld.proto
-$ protoc -I ../../protos/ --cpp_out=. ../../protos/helloworld.proto
-```
-
 ### Client and server implementations
 
 The client implementation is at [greeter_client.cc](helloworld/greeter_client.cc).

examples/cpp/cpptutorial.md

@@ -91,7 +91,7 @@ message Point {
 
 Next we need to generate the gRPC client and server interfaces from our .proto service definition. We do this using the protocol buffer compiler `protoc` with a special gRPC C++ plugin.
 
-For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL) first):
+For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL.md) first):
 
 ```shell
 $ make route_guide.grpc.pb.cc route_guide.pb.cc

examples/cpp/helloworld/README.md

@@ -2,7 +2,7 @@
 
 ### Install gRPC
 Make sure you have installed gRPC on your system. Follow the instructions here:
-[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL).
+[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL.md).
 
 ### Get the tutorial source code
 

examples/node/README.md

@@ -20,7 +20,7 @@ TRY IT!
  - Run the server
 
    ```sh
-   $ # from this directory (grpc_common/node).
+   $ # from this directory
    $ node ./greeter_server.js &
    ```
 

examples/python/route_guide/route_guide_server.py

@@ -128,7 +128,7 @@ def serve():
     while True:
       time.sleep(_ONE_DAY_IN_SECONDS)
   except KeyboardInterrupt:
-    server.stop()
+    server.stop(0)
 
 if __name__ == '__main__':
   serve()

grpc.def

@@ -182,6 +182,7 @@ EXPORTS
     gpr_event_wait
     gpr_ref_init
     gpr_ref
+    gpr_ref_non_zero
     gpr_refn
     gpr_unref
     gpr_stats_init

include/grpc/census.h

@@ -80,18 +80,18 @@ CENSUSAPI int census_enabled(void);
   metrics will be recorded. Keys are unique within a context. */
 typedef struct census_context census_context;
 
-/* A tag is a key:value pair. The key is a non-empty, printable (UTF-8
-   encoded), nil-terminated string. The value is a binary string, that may be
-   printable. There are limits on the sizes of both keys and values (see
-   CENSUS_MAX_TAG_KB_LEN definition below), and the number of tags that can be
-   propagated (CENSUS_MAX_PROPAGATED_TAGS). Users should also remember that
-   some systems may have limits on, e.g., the number of bytes that can be
-   transmitted as metadata, and that larger tags means more memory consumed
-   and time in processing. */
+/* A tag is a key:value pair. Both keys and values are nil-terminated strings,
+   containing printable ASCII characters (decimal 32-126). Keys must be at
+   least one character in length. Both keys and values can have at most
+   CENSUS_MAX_TAG_KB_LEN characters (including the terminating nil). The
+   maximum number of tags that can be propagated is
+   CENSUS_MAX_PROPAGATED_TAGS. Users should also remember that some systems
+   may have limits on, e.g., the number of bytes that can be transmitted as
+   metadata, and that larger tags means more memory consumed and time in
+   processing. */
 typedef struct {
   const char *key;
   const char *value;
-  size_t value_len;
   uint8_t flags;
 } census_tag;
 
@@ -103,28 +103,25 @@ typedef struct {
 /* Tag flags. */
 #define CENSUS_TAG_PROPAGATE 1 /* Tag should be propagated over RPC */
 #define CENSUS_TAG_STATS 2     /* Tag will be used for statistics aggregation */
-#define CENSUS_TAG_BINARY 4    /* Tag value is not printable */
-#define CENSUS_TAG_RESERVED 8  /* Reserved for internal use. */
-/* Flag values 8,16,32,64,128 are reserved for future/internal use. Clients
+#define CENSUS_TAG_RESERVED 4  /* Reserved for internal use. */
+/* Flag values 4,8,16,32,64,128 are reserved for future/internal use. Clients
    should not use or rely on their values. */
 
 #define CENSUS_TAG_IS_PROPAGATED(flags) (flags & CENSUS_TAG_PROPAGATE)
 #define CENSUS_TAG_IS_STATS(flags) (flags & CENSUS_TAG_STATS)
-#define CENSUS_TAG_IS_BINARY(flags) (flags & CENSUS_TAG_BINARY)
 
 /* An instance of this structure is kept by every context, and records the
    basic information associated with the creation of that context. */
 typedef struct {
-  int n_propagated_tags;        /* number of propagated printable tags */
-  int n_propagated_binary_tags; /* number of propagated binary tags */
-  int n_local_tags;             /* number of non-propagated (local) tags */
-  int n_deleted_tags;           /* number of tags that were deleted */
-  int n_added_tags;             /* number of tags that were added */
-  int n_modified_tags;          /* number of tags that were modified */
-  int n_invalid_tags;           /* number of tags with bad keys or values (e.g.
-                                   longer than CENSUS_MAX_TAG_KV_LEN) */
-  int n_ignored_tags;           /* number of tags ignored because of
-                                   CENSUS_MAX_PROPAGATED_TAGS limit. */
+  int n_propagated_tags; /* number of propagated tags */
+  int n_local_tags;      /* number of non-propagated (local) tags */
+  int n_deleted_tags;    /* number of tags that were deleted */
+  int n_added_tags;      /* number of tags that were added */
+  int n_modified_tags;   /* number of tags that were modified */
+  int n_invalid_tags;    /* number of tags with bad keys or values (e.g.
+                            longer than CENSUS_MAX_TAG_KV_LEN) */
+  int n_ignored_tags;    /* number of tags ignored because of
+                            CENSUS_MAX_PROPAGATED_TAGS limit. */
 } census_context_status;
 
 /* Create a new context, adding and removing tags from an existing context.
@@ -132,10 +129,10 @@ typedef struct {
    to add as many tags in a single operation as is practical for the client.
    @param base Base context to build upon. Can be NULL.
    @param tags A set of tags to be added/changed/deleted. Tags with keys that
-   are in 'tags', but not 'base', are added to the tag set. Keys that are in
+   are in 'tags', but not 'base', are added to the context. Keys that are in
    both 'tags' and 'base' will have their value/flags modified. Tags with keys
-   in both, but with NULL or zero-length values, will be deleted from the tag
-   set. Tags with invalid (too long or short) keys or values will be ignored.
+   in both, but with NULL values, will be deleted from the context. Tags with
+   invalid (too long or short) keys or values will be ignored.
    If adding a tag will result in more than CENSUS_MAX_PROPAGATED_TAGS in either
    binary or non-binary tags, they will be ignored, as will deletions of
    tags that don't exist.
@@ -185,32 +182,19 @@ CENSUSAPI int census_context_get_tag(const census_context *context,
    for use by RPC systems only, for purposes of transmitting/receiving contexts.
    */
 
-/* Encode a context into a buffer. The propagated tags are encoded into the
-   buffer in two regions: one for printable tags, and one for binary tags.
+/* Encode a context into a buffer.
    @param context context to be encoded
-   @param buffer pointer to buffer. This address will be used to encode the
-                 printable tags.
+   @param buffer buffer into which the context will be encoded.
    @param buf_size number of available bytes in buffer.
-   @param print_buf_size Will be set to the number of bytes consumed by
-                         printable tags.
-   @param bin_buf_size Will be set to the number of bytes used to encode the
-                       binary tags.
-   @return A pointer to the binary tag's encoded, or NULL if the buffer was
-           insufficiently large to hold the encoded tags. Thus, if successful,
-           printable tags are encoded into
-           [buffer, buffer + *print_buf_size) and binary tags into
-           [returned-ptr, returned-ptr + *bin_buf_size) (and the returned
-           pointer should be buffer + *print_buf_size) */
-CENSUSAPI char *census_context_encode(const census_context *context,
-                                      char *buffer, size_t buf_size,
-                                      size_t *print_buf_size,
-                                      size_t *bin_buf_size);
-
-/* Decode context buffers encoded with census_context_encode(). Returns NULL
+   @return The number of buffer bytes consumed for the encoded context, or
+           zero if the buffer was of insufficient size. */
+CENSUSAPI size_t census_context_encode(const census_context *context,
+                                       char *buffer, size_t buf_size);
+
+/* Decode context buffer encoded with census_context_encode(). Returns NULL
    if there is an error in parsing either buffer. */
-CENSUSAPI census_context *census_context_decode(const char *buffer, size_t size,
-                                                const char *bin_buffer,
-                                                size_t bin_size);
+CENSUSAPI census_context *census_context_decode(const char *buffer,
+                                                size_t size);
 
 /* Distributed traces can have a number of options. */
 enum census_trace_mask_values {

include/grpc/impl/codegen/sync.h

@@ -182,6 +182,10 @@ GPRAPI void gpr_ref_init(gpr_refcount *r, int n);
 /* Increment the reference count *r.  Requires *r initialized. */
 GPRAPI void gpr_ref(gpr_refcount *r);
 
+/* Increment the reference count *r.  Requires *r initialized.
+   Crashes if refcount is zero */
+GPRAPI void gpr_ref_non_zero(gpr_refcount *r);
+
 /* Increment the reference count *r by n.  Requires *r initialized, n > 0. */
 GPRAPI void gpr_refn(gpr_refcount *r, int n);
 

package.xml

@@ -10,11 +10,11 @@
   <email>grpc-packages@google.com</email>
   <active>yes</active>
  </lead>
- <date>2016-02-24</date>
+ <date>2016-03-01</date>
  <time>16:06:07</time>
  <version>
-  <release>0.8.0</release>
-  <api>0.8.0</api>
+  <release>0.14.0</release>
+  <api>0.14.0</api>
  </version>
  <stability>
   <release>beta</release>
@@ -22,7 +22,7 @@
  </stability>
  <license>BSD</license>
  <notes>
-- Simplify gRPC PHP installation #4517
+- Increase unit test code coverage #5225
  </notes>
  <contents>
   <dir baseinstalldir="/" name="/">
@@ -965,5 +965,20 @@ Update to wrap gRPC C Core version 0.10.0
 - Simplify gRPC PHP installation #4517
    </notes>
   </release>
+  <release>
+   <version>
+    <release>0.14.0</release>
+    <api>0.14.0</api>
+   </version>
+   <stability>
+    <release>beta</release>
+    <api>beta</api>
+   </stability>
+   <date>2016-03-01</date>
+   <license>BSD</license>
+   <notes>
+- Increase unit test code coverage #5225
+   </notes>
+  </release>
  </changelog>
 </package>

setup.py

@@ -165,6 +165,7 @@ COMMAND_CLASS = {
     'build_tagged_ext': precompiled.BuildTaggedExt,
     'gather': commands.Gather,
     'run_interop': commands.RunInterop,
+    'test_lite': commands.TestLite
 }
 
 # Ensure that package data is copied over before any commands have been run:

src/core/census/context.c

@@ -60,10 +60,10 @@
 //   limit of 255 for both CENSUS_MAX_TAG_KV_LEN and CENSUS_MAX_PROPAGATED_TAGS.
 // * Keep all tag information (keys/values/flags) in a single memory buffer,
 //   that can be directly copied to the wire.
-// * Binary tags share the same structure as, but are encoded separately from,
-//   non-binary tags. This is primarily because non-binary tags are far more
-//   likely to be repeated across multiple RPC calls, so are more efficiently
-//   cached and compressed in any metadata schemes.
+
+// min and max valid chars in tag keys and values. All printable ASCII is OK.
+#define MIN_VALID_TAG_CHAR 32   // ' '
+#define MAX_VALID_TAG_CHAR 126  // '~'
 
 // Structure representing a set of tags. Essentially a count of number of tags
 // present, and pointer to a chunk of memory that contains the per-tag details.
@@ -77,7 +77,7 @@ struct tag_set {
   char *kvm;        // key/value memory. Consists of repeated entries of:
   //   Offset  Size  Description
   //     0      1    Key length, including trailing 0. (K)
-  //     1      1    Value length. (V)
+  //     1      1    Value length, including trailing 0 (V)
   //     2      1    Flags
   //     3      K    Key bytes
   //     3 + K  V    Value bytes
@@ -108,19 +108,36 @@ struct raw_tag {
 #define CENSUS_TAG_DELETED CENSUS_TAG_RESERVED
 #define CENSUS_TAG_IS_DELETED(flags) (flags & CENSUS_TAG_DELETED)
 
-// Primary (external) representation of a context. Composed of 3 underlying
-// tag_set structs, one for each of the binary/printable propagated tags, and
-// one for everything else. This is to efficiently support tag
-// encoding/decoding.
+// Primary representation of a context. Composed of 2 underlying tag_set
+// structs, one each for propagated and local (non-propagated) tags. This is
+// to efficiently support tag encoding/decoding.
+// TODO(aveitch): need to add tracing id's/structure.
 struct census_context {
-  struct tag_set tags[3];
+  struct tag_set tags[2];
   census_context_status status;
 };
 
 // Indices into the tags member of census_context
 #define PROPAGATED_TAGS 0
-#define PROPAGATED_BINARY_TAGS 1
-#define LOCAL_TAGS 2
+#define LOCAL_TAGS 1
+
+// Validate (check all characters are in range and size is less than limit) a
+// key or value string. Returns 0 if the string is invalid, or the length
+// (including terminator) if valid.
+static size_t validate_tag(const char *kv) {
+  size_t len = 1;
+  char ch;
+  while ((ch = *kv++) != 0) {
+    if (ch < MIN_VALID_TAG_CHAR || ch > MAX_VALID_TAG_CHAR) {
+      return 0;
+    }
+    len++;
+  }
+  if (len > CENSUS_MAX_TAG_KV_LEN) {
+    return 0;
+  }
+  return len;
+}
 
 // Extract a raw tag given a pointer (raw) to the tag header. Allow for some
 // extra bytes in the tag header (see encode/decode functions for usage: this
@@ -166,9 +183,7 @@ static bool context_delete_tag(census_context *context, const census_tag *tag,
                                size_t key_len) {
   return (
       tag_set_delete_tag(&context->tags[LOCAL_TAGS], tag->key, key_len) ||
-      tag_set_delete_tag(&context->tags[PROPAGATED_TAGS], tag->key, key_len) ||
-      tag_set_delete_tag(&context->tags[PROPAGATED_BINARY_TAGS], tag->key,
-                         key_len));
+      tag_set_delete_tag(&context->tags[PROPAGATED_TAGS], tag->key, key_len));
 }
 
 // Add a tag to a tag_set. Return true on success, false if the tag could
@@ -176,11 +191,11 @@ static bool context_delete_tag(census_context *context, const census_tag *tag,
 // not be called if the tag may already exist (in a non-deleted state) in
 // the tag_set, as that would result in two tags with the same key.
 static bool tag_set_add_tag(struct tag_set *tags, const census_tag *tag,
-                            size_t key_len) {
+                            size_t key_len, size_t value_len) {
   if (tags->ntags == CENSUS_MAX_PROPAGATED_TAGS) {
     return false;
   }
-  const size_t tag_size = key_len + tag->value_len + TAG_HEADER_SIZE;
+  const size_t tag_size = key_len + value_len + TAG_HEADER_SIZE;
   if (tags->kvm_used + tag_size > tags->kvm_size) {
     // allocate new memory if needed
     tags->kvm_size += 2 * CENSUS_MAX_TAG_KV_LEN + TAG_HEADER_SIZE;
@@ -191,13 +206,12 @@ static bool tag_set_add_tag(struct tag_set *tags, const census_tag *tag,
   }
   char *kvp = tags->kvm + tags->kvm_used;
   *kvp++ = (char)key_len;
-  *kvp++ = (char)tag->value_len;
+  *kvp++ = (char)value_len;
   // ensure reserved flags are not used.
-  *kvp++ = (char)(tag->flags & (CENSUS_TAG_PROPAGATE | CENSUS_TAG_STATS |
-                                CENSUS_TAG_BINARY));
+  *kvp++ = (char)(tag->flags & (CENSUS_TAG_PROPAGATE | CENSUS_TAG_STATS));
   memcpy(kvp, tag->key, key_len);
   kvp += key_len;
-  memcpy(kvp, tag->value, tag->value_len);
+  memcpy(kvp, tag->value, value_len);
   tags->kvm_used += tag_size;
   tags->ntags++;
   tags->ntags_alloc++;
@@ -207,30 +221,20 @@ static bool tag_set_add_tag(struct tag_set *tags, const census_tag *tag,
 // Add/modify/delete a tag to/in a context. Caller must validate that tag key
 // etc. are valid.
 static void context_modify_tag(census_context *context, const census_tag *tag,
-                               size_t key_len) {
+                               size_t key_len, size_t value_len) {
   // First delete the tag if it is already present.
   bool deleted = context_delete_tag(context, tag, key_len);
-  // Determine if we need to add it back.
-  bool call_add = tag->value != NULL && tag->value_len != 0;
   bool added = false;
-  if (call_add) {
-    if (CENSUS_TAG_IS_PROPAGATED(tag->flags)) {
-      if (CENSUS_TAG_IS_BINARY(tag->flags)) {
-        added = tag_set_add_tag(&context->tags[PROPAGATED_BINARY_TAGS], tag,
-                                key_len);
-      } else {
-        added = tag_set_add_tag(&context->tags[PROPAGATED_TAGS], tag, key_len);
-      }
-    } else {
-      added = tag_set_add_tag(&context->tags[LOCAL_TAGS], tag, key_len);
-    }
+  if (CENSUS_TAG_IS_PROPAGATED(tag->flags)) {
+    added = tag_set_add_tag(&context->tags[PROPAGATED_TAGS], tag, key_len,
+                            value_len);
+  } else {
+    added =
+        tag_set_add_tag(&context->tags[LOCAL_TAGS], tag, key_len, value_len);
   }
+
   if (deleted) {
-    if (call_add) {
-      context->status.n_modified_tags++;
-    } else {
-      context->status.n_deleted_tags++;
-    }
+    context->status.n_modified_tags++;
   } else {
     if (added) {
       context->status.n_added_tags++;
@@ -292,8 +296,6 @@ census_context *census_context_create(const census_context *base,
     memset(context, 0, sizeof(census_context));
   } else {
     tag_set_copy(&context->tags[PROPAGATED_TAGS], &base->tags[PROPAGATED_TAGS]);
-    tag_set_copy(&context->tags[PROPAGATED_BINARY_TAGS],
-                 &base->tags[PROPAGATED_BINARY_TAGS]);
     tag_set_copy(&context->tags[LOCAL_TAGS], &base->tags[LOCAL_TAGS]);
     memset(&context->status, 0, sizeof(context->status));
   }
@@ -301,22 +303,29 @@ census_context *census_context_create(const census_context *base,
   // the context to add/replace/delete as required.
   for (int i = 0; i < ntags; i++) {
     const census_tag *tag = &tags[i];
-    size_t key_len = strlen(tag->key) + 1;
-    // ignore the tag if it is too long/short.
-    if (key_len != 1 && key_len <= CENSUS_MAX_TAG_KV_LEN &&
-        tag->value_len <= CENSUS_MAX_TAG_KV_LEN) {
-      context_modify_tag(context, tag, key_len);
-    } else {
+    size_t key_len = validate_tag(tag->key);
+    // ignore the tag if it is invalid or too short.
+    if (key_len <= 1) {
       context->status.n_invalid_tags++;
+    } else {
+      if (tag->value != NULL) {
+        size_t value_len = validate_tag(tag->value);
+        if (value_len != 0) {
+          context_modify_tag(context, tag, key_len, value_len);
+        } else {
+          context->status.n_invalid_tags++;
+        }
+      } else {
+        if (context_delete_tag(context, tag, key_len)) {
+          context->status.n_deleted_tags++;
+        }
+      }
     }
   }
   // Remove any deleted tags, update status if needed, and return.
   tag_set_flatten(&context->tags[PROPAGATED_TAGS]);
-  tag_set_flatten(&context->tags[PROPAGATED_BINARY_TAGS]);
   tag_set_flatten(&context->tags[LOCAL_TAGS]);
   context->status.n_propagated_tags = context->tags[PROPAGATED_TAGS].ntags;
-  context->status.n_propagated_binary_tags =
-      context->tags[PROPAGATED_BINARY_TAGS].ntags;
   context->status.n_local_tags = context->tags[LOCAL_TAGS].ntags;
   if (status) {
     *status = &context->status;
@@ -331,7 +340,6 @@ const census_context_status *census_context_get_status(
 
 void census_context_destroy(census_context *context) {
   gpr_free(context->tags[PROPAGATED_TAGS].kvm);
-  gpr_free(context->tags[PROPAGATED_BINARY_TAGS].kvm);
   gpr_free(context->tags[LOCAL_TAGS].kvm);
   gpr_free(context);
 }
@@ -343,9 +351,6 @@ void census_context_initialize_iterator(const census_context *context,
   if (context->tags[PROPAGATED_TAGS].ntags != 0) {
     iterator->base = PROPAGATED_TAGS;
     iterator->kvm = context->tags[PROPAGATED_TAGS].kvm;
-  } else if (context->tags[PROPAGATED_BINARY_TAGS].ntags != 0) {
-    iterator->base = PROPAGATED_BINARY_TAGS;
-    iterator->kvm = context->tags[PROPAGATED_BINARY_TAGS].kvm;
   } else if (context->tags[LOCAL_TAGS].ntags != 0) {
     iterator->base = LOCAL_TAGS;
     iterator->kvm = context->tags[LOCAL_TAGS].kvm;
@@ -363,7 +368,6 @@ int census_context_next_tag(census_context_iterator *iterator,
   iterator->kvm = decode_tag(&raw, iterator->kvm, 0);
   tag->key = raw.key;
   tag->value = raw.value;
-  tag->value_len = raw.value_len;
   tag->flags = raw.flags;
   if (++iterator->index == iterator->context->tags[iterator->base].ntags) {
     do {
@@ -388,7 +392,6 @@ static bool tag_set_get_tag(const struct tag_set *tags, const char *key,
     if (key_len == raw.key_len && memcmp(raw.key, key, key_len) == 0) {
       tag->key = raw.key;
       tag->value = raw.value;
-      tag->value_len = raw.value_len;
       tag->flags = raw.flags;
       return true;
     }
@@ -403,8 +406,6 @@ int census_context_get_tag(const census_context *context, const char *key,
     return 0;
   }
   if (tag_set_get_tag(&context->tags[PROPAGATED_TAGS], key, key_len, tag) ||
-      tag_set_get_tag(&context->tags[PROPAGATED_BINARY_TAGS], key, key_len,
-                      tag) ||
       tag_set_get_tag(&context->tags[LOCAL_TAGS], key, key_len, tag)) {
     return 1;
   }
@@ -447,21 +448,9 @@ static size_t tag_set_encode(const struct tag_set *tags, char *buffer,
   return ENCODED_HEADER_SIZE + tags->kvm_used;
 }
 
-char *census_context_encode(const census_context *context, char *buffer,
-                            size_t buf_size, size_t *print_buf_size,
-                            size_t *bin_buf_size) {
-  *print_buf_size =
-      tag_set_encode(&context->tags[PROPAGATED_TAGS], buffer, buf_size);
-  if (*print_buf_size == 0) {
-    return NULL;
-  }
-  char *b_buffer = buffer + *print_buf_size;
-  *bin_buf_size = tag_set_encode(&context->tags[PROPAGATED_BINARY_TAGS],
-                                 b_buffer, buf_size - *print_buf_size);
-  if (*bin_buf_size == 0) {
-    return NULL;
-  }
-  return b_buffer;
+size_t census_context_encode(const census_context *context, char *buffer,
+                             size_t buf_size) {
+  return tag_set_encode(&context->tags[PROPAGATED_TAGS], buffer, buf_size);
 }
 
 // Decode a tag set.
@@ -506,8 +495,7 @@ static void tag_set_decode(struct tag_set *tags, const char *buffer,
   }
 }
 
-census_context *census_context_decode(const char *buffer, size_t size,
-                                      const char *bin_buffer, size_t bin_size) {
+census_context *census_context_decode(const char *buffer, size_t size) {
   census_context *context = gpr_malloc(sizeof(census_context));
   memset(&context->tags[LOCAL_TAGS], 0, sizeof(struct tag_set));
   if (buffer == NULL) {
@@ -515,16 +503,7 @@ census_context *census_context_decode(const char *buffer, size_t size,
   } else {
     tag_set_decode(&context->tags[PROPAGATED_TAGS], buffer, size);
   }
-  if (bin_buffer == NULL) {
-    memset(&context->tags[PROPAGATED_BINARY_TAGS], 0, sizeof(struct tag_set));
-  } else {
-    tag_set_decode(&context->tags[PROPAGATED_BINARY_TAGS], bin_buffer,
-                   bin_size);
-  }
   memset(&context->status, 0, sizeof(context->status));
   context->status.n_propagated_tags = context->tags[PROPAGATED_TAGS].ntags;
-  context->status.n_propagated_binary_tags =
-      context->tags[PROPAGATED_BINARY_TAGS].ntags;
-  // TODO(aveitch): check that BINARY flag is correct for each type.
   return context;
 }

src/core/channel/client_channel.c

@@ -251,7 +251,7 @@ static void cc_start_transport_op(grpc_exec_ctx *exec_ctx,
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
-  GPR_ASSERT(op->set_accept_stream == NULL);
+  GPR_ASSERT(op->set_accept_stream == false);
   if (op->bind_pollset != NULL) {
     grpc_pollset_set_add_pollset(exec_ctx, chand->interested_parties,
                                  op->bind_pollset);

src/core/channel/client_uchannel.c

@@ -107,7 +107,7 @@ static void cuc_start_transport_op(grpc_exec_ctx *exec_ctx,
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
-  GPR_ASSERT(op->set_accept_stream == NULL);
+  GPR_ASSERT(op->set_accept_stream == false);
   GPR_ASSERT(op->bind_pollset == NULL);
 
   if (op->on_connectivity_state_change != NULL) {

src/core/channel/subchannel_call_holder.c

@@ -168,21 +168,23 @@ retry:
 
 static void subchannel_ready(grpc_exec_ctx *exec_ctx, void *arg, bool success) {
   grpc_subchannel_call_holder *holder = arg;
-  grpc_subchannel_call *call;
   gpr_mu_lock(&holder->mu);
   GPR_ASSERT(holder->creation_phase ==
              GRPC_SUBCHANNEL_CALL_HOLDER_PICKING_SUBCHANNEL);
-  call = GET_CALL(holder);
-  GPR_ASSERT(call == NULL || call == CANCELLED_CALL);
   holder->creation_phase = GRPC_SUBCHANNEL_CALL_HOLDER_NOT_CREATING;
   if (holder->connected_subchannel == NULL) {
     fail_locked(exec_ctx, holder);
   } else {
-    gpr_atm_rel_store(
-        &holder->subchannel_call,
-        (gpr_atm)(uintptr_t)grpc_connected_subchannel_create_call(
-            exec_ctx, holder->connected_subchannel, holder->pollset));
-    retry_waiting_locked(exec_ctx, holder);
+    if (!gpr_atm_rel_cas(
+            &holder->subchannel_call, 0,
+            (gpr_atm)(uintptr_t)grpc_connected_subchannel_create_call(
+                exec_ctx, holder->connected_subchannel, holder->pollset))) {
+      GPR_ASSERT(gpr_atm_acq_load(&holder->subchannel_call) == 1);
+      /* if this cas fails, the call was cancelled before the pick completed */
+      fail_locked(exec_ctx, holder);
+    } else {
+      retry_waiting_locked(exec_ctx, holder);
+    }
   }
   gpr_mu_unlock(&holder->mu);
   GRPC_CALL_STACK_UNREF(exec_ctx, holder->owning_call, "pick_subchannel");

src/core/client_config/subchannel.c

@@ -395,7 +395,6 @@ void grpc_subchannel_notify_on_state_change(
     grpc_exec_ctx *exec_ctx, grpc_subchannel *c,
     grpc_pollset_set *interested_parties, grpc_connectivity_state *state,
     grpc_closure *notify) {
-  int do_connect = 0;
   external_state_watcher *w;
 
   if (state == NULL) {
@@ -425,17 +424,13 @@ void grpc_subchannel_notify_on_state_change(
     w->next->prev = w->prev->next = w;
     if (grpc_connectivity_state_notify_on_state_change(
             exec_ctx, &c->state_tracker, state, &w->closure)) {
-      do_connect = 1;
       c->connecting = 1;
       /* released by connection */
       GRPC_SUBCHANNEL_WEAK_REF(c, "connecting");
+      start_connect(exec_ctx, c);
     }
     gpr_mu_unlock(&c->mu);
   }
-
-  if (do_connect) {
-    start_connect(exec_ctx, c);
-  }
 }
 
 void grpc_connected_subchannel_process_transport_op(
@@ -635,11 +630,12 @@ static void on_alarm(grpc_exec_ctx *exec_ctx, void *arg, bool iomgr_success) {
   if (c->disconnected) {
     iomgr_success = 0;
   }
-  gpr_mu_unlock(&c->mu);
   if (iomgr_success) {
     update_reconnect_parameters(c);
     continue_connect(exec_ctx, c);
+    gpr_mu_unlock(&c->mu);
   } else {
+    gpr_mu_unlock(&c->mu);
     GRPC_SUBCHANNEL_WEAK_UNREF(exec_ctx, c, "connecting");
   }
 }

src/core/iomgr/ev_poll_and_epoll_posix.c

@@ -1323,6 +1323,7 @@ static void multipoll_with_poll_pollset_maybe_work_and_unlock(
     } else {
       h->fds[fd_count++] = h->fds[i];
       watchers[pfd_count].fd = h->fds[i];
+      GRPC_FD_REF(watchers[pfd_count].fd, "multipoller_start");
       pfds[pfd_count].fd = h->fds[i]->fd;
       pfds[pfd_count].revents = 0;
       pfd_count++;
@@ -1336,8 +1337,10 @@ static void multipoll_with_poll_pollset_maybe_work_and_unlock(
   gpr_mu_unlock(&pollset->mu);
 
   for (i = 2; i < pfd_count; i++) {
-    pfds[i].events = (short)fd_begin_poll(watchers[i].fd, pollset, worker,
-                                          POLLIN, POLLOUT, &watchers[i]);
+    grpc_fd *fd = watchers[i].fd;
+    pfds[i].events = (short)grpc_fd_begin_poll(fd, pollset, worker, POLLIN,
+                                               POLLOUT, &watchers[i]);
+    GRPC_FD_UNREF(fd, "multipoller_start");
   }
 
   /* TODO(vpai): Consider first doing a 0 timeout poll here to avoid

src/core/iomgr/iomgr.c

@@ -41,9 +41,11 @@
 #include <grpc/support/string_util.h>
 #include <grpc/support/sync.h>
 #include <grpc/support/thd.h>
+#include <grpc/support/useful.h>
 
 #include "src/core/iomgr/iomgr_internal.h"
 #include "src/core/iomgr/timer.h"
+#include "src/core/support/env.h"
 #include "src/core/support/string.h"
 
 static gpr_mu g_mu;
@@ -115,6 +117,9 @@ void grpc_iomgr_shutdown(void) {
                     "memory leaks are likely",
                     count_objects());
             dump_objects("LEAKED");
+            if (grpc_iomgr_abort_on_leaks()) {
+              abort();
+            }
           }
           break;
         }
@@ -152,3 +157,14 @@ void grpc_iomgr_unregister_object(grpc_iomgr_object *obj) {
   gpr_mu_unlock(&g_mu);
   gpr_free(obj->name);
 }
+
+bool grpc_iomgr_abort_on_leaks(void) {
+  char *env = gpr_getenv("GRPC_ABORT_ON_LEAKS");
+  if (env == NULL) return false;
+  static const char *truthy[] = {"yes",  "Yes",  "YES", "true",
+                                 "True", "TRUE", "1"};
+  for (size_t i = 0; i < GPR_ARRAY_SIZE(truthy); i++) {
+    if (0 == strcmp(env, truthy[i])) return true;
+  }
+  return false;
+}

src/core/iomgr/iomgr_internal.h

@@ -34,7 +34,8 @@
 #ifndef GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H
 #define GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H
 
-#include <grpc/support/sync.h>
+#include <stdbool.h>
+
 #include "src/core/iomgr/iomgr.h"
 
 typedef struct grpc_iomgr_object {
@@ -52,4 +53,6 @@ void grpc_iomgr_platform_flush(void);
 /** tear down all platform specific global iomgr structures */
 void grpc_iomgr_platform_shutdown(void);
 
+bool grpc_iomgr_abort_on_leaks(void);
+
 #endif /* GRPC_INTERNAL_CORE_IOMGR_IOMGR_INTERNAL_H */

src/core/support/sync.c

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -98,6 +98,11 @@ void gpr_ref_init(gpr_refcount *r, int n) { gpr_atm_rel_store(&r->count, n); }
 
 void gpr_ref(gpr_refcount *r) { gpr_atm_no_barrier_fetch_add(&r->count, 1); }
 
+void gpr_ref_non_zero(gpr_refcount *r) {
+  gpr_atm prior = gpr_atm_no_barrier_fetch_add(&r->count, 1);
+  GPR_ASSERT(prior > 0);
+}
+
 void gpr_refn(gpr_refcount *r, int n) {
   gpr_atm_no_barrier_fetch_add(&r->count, n);
 }

src/core/surface/server.c

@@ -407,8 +407,15 @@ static void destroy_channel(grpc_exec_ctx *exec_ctx, channel_data *chand) {
   maybe_finish_shutdown(exec_ctx, chand->server);
   chand->finish_destroy_channel_closure.cb = finish_destroy_channel;
   chand->finish_destroy_channel_closure.cb_arg = chand;
-  grpc_exec_ctx_enqueue(exec_ctx, &chand->finish_destroy_channel_closure, true,
-                        NULL);
+
+  grpc_transport_op op;
+  memset(&op, 0, sizeof(op));
+  op.set_accept_stream = true;
+  op.on_consumed = &chand->finish_destroy_channel_closure;
+  grpc_channel_next_op(exec_ctx,
+                       grpc_channel_stack_element(
+                           grpc_channel_get_channel_stack(chand->channel), 0),
+                       &op);
 }
 
 static void finish_start_new_rpc(grpc_exec_ctx *exec_ctx, grpc_server *server,
@@ -971,7 +978,8 @@ void grpc_server_setup_transport(grpc_exec_ctx *exec_ctx, grpc_server *s,
 
   GRPC_CHANNEL_INTERNAL_REF(channel, "connectivity");
   memset(&op, 0, sizeof(op));
-  op.set_accept_stream = accept_stream;
+  op.set_accept_stream = true;
+  op.set_accept_stream_fn = accept_stream;
   op.set_accept_stream_user_data = chand;
   op.on_connectivity_state_change = &chand->channel_connectivity_changed;
   op.connectivity_state = &chand->connectivity_state;

src/core/transport/chttp2/internal.h

@@ -358,6 +358,9 @@ struct grpc_chttp2_transport {
     /** connectivity tracking */
     grpc_connectivity_state_tracker state_tracker;
   } channel_callback;
+
+  /** Transport op to be applied post-parsing */
+  grpc_transport_op *post_parsing_op;
 };
 
 typedef struct {
@@ -417,7 +420,7 @@ typedef struct {
   /** HTTP2 stream id for this stream, or zero if one has not been assigned */
   uint32_t id;
   uint8_t fetching;
-  uint8_t sent_initial_metadata;
+  bool sent_initial_metadata;
   uint8_t sent_message;
   uint8_t sent_trailing_metadata;
   uint8_t read_closed;
@@ -509,7 +512,7 @@ void grpc_chttp2_publish_reads(grpc_exec_ctx *exec_ctx,
                                grpc_chttp2_transport_global *global,
                                grpc_chttp2_transport_parsing *parsing);
 
-void grpc_chttp2_list_add_writable_stream(
+bool grpc_chttp2_list_add_writable_stream(
     grpc_chttp2_transport_global *transport_global,
     grpc_chttp2_stream_global *stream_global);
 /** Get a writable stream
@@ -519,14 +522,13 @@ int grpc_chttp2_list_pop_writable_stream(
     grpc_chttp2_transport_writing *transport_writing,
     grpc_chttp2_stream_global **stream_global,
     grpc_chttp2_stream_writing **stream_writing);
-void grpc_chttp2_list_remove_writable_stream(
+bool grpc_chttp2_list_remove_writable_stream(
     grpc_chttp2_transport_global *transport_global,
-    grpc_chttp2_stream_global *stream_global);
+    grpc_chttp2_stream_global *stream_global) GRPC_MUST_USE_RESULT;
 
-/* returns 1 if stream added, 0 if it was already present */
-int grpc_chttp2_list_add_writing_stream(
+void grpc_chttp2_list_add_writing_stream(
     grpc_chttp2_transport_writing *transport_writing,
-    grpc_chttp2_stream_writing *stream_writing) GRPC_MUST_USE_RESULT;
+    grpc_chttp2_stream_writing *stream_writing);
 int grpc_chttp2_list_have_writing_streams(
     grpc_chttp2_transport_writing *transport_writing);
 int grpc_chttp2_list_pop_writing_stream(
@@ -770,4 +772,9 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx,
                           grpc_chttp2_transport_parsing *parsing,
                           const uint8_t *opaque_8bytes);
 
+/** add a ref to the stream and add it to the writable list;
+    ref will be dropped in writing.c */
+void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global,
+                                 grpc_chttp2_stream_global *stream_global);
+
 #endif

src/core/transport/chttp2/parsing.c

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -149,7 +149,7 @@ void grpc_chttp2_publish_reads(
   if (was_zero && !is_zero) {
     while (grpc_chttp2_list_pop_stalled_by_transport(transport_global,
                                                      &stream_global)) {
-      grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+      grpc_chttp2_become_writable(transport_global, stream_global);
     }
   }
 
@@ -178,7 +178,7 @@ void grpc_chttp2_publish_reads(
                                  outgoing_window);
     is_zero = stream_global->outgoing_window <= 0;
     if (was_zero && !is_zero) {
-      grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+      grpc_chttp2_become_writable(transport_global, stream_global);
     }
 
     stream_global->max_recv_bytes -= (uint32_t)GPR_MIN(

src/core/transport/chttp2/stream_lists.c

@@ -100,11 +100,14 @@ static void stream_list_remove(grpc_chttp2_transport *t, grpc_chttp2_stream *s,
   }
 }
 
-static void stream_list_maybe_remove(grpc_chttp2_transport *t,
+static bool stream_list_maybe_remove(grpc_chttp2_transport *t,
                                      grpc_chttp2_stream *s,
                                      grpc_chttp2_stream_list_id id) {
   if (s->included[id]) {
     stream_list_remove(t, s, id);
+    return true;
+  } else {
+    return false;
   }
 }
 
@@ -125,23 +128,24 @@ static void stream_list_add_tail(grpc_chttp2_transport *t,
   s->included[id] = 1;
 }
 
-static int stream_list_add(grpc_chttp2_transport *t, grpc_chttp2_stream *s,
-                           grpc_chttp2_stream_list_id id) {
+static bool stream_list_add(grpc_chttp2_transport *t, grpc_chttp2_stream *s,
+                            grpc_chttp2_stream_list_id id) {
   if (s->included[id]) {
-    return 0;
+    return false;
   }
   stream_list_add_tail(t, s, id);
-  return 1;
+  return true;
 }
 
 /* wrappers for specializations */
 
-void grpc_chttp2_list_add_writable_stream(
+bool grpc_chttp2_list_add_writable_stream(
     grpc_chttp2_transport_global *transport_global,
     grpc_chttp2_stream_global *stream_global) {
   GPR_ASSERT(stream_global->id != 0);
-  stream_list_add(TRANSPORT_FROM_GLOBAL(transport_global),
-                  STREAM_FROM_GLOBAL(stream_global), GRPC_CHTTP2_LIST_WRITABLE);
+  return stream_list_add(TRANSPORT_FROM_GLOBAL(transport_global),
+                         STREAM_FROM_GLOBAL(stream_global),
+                         GRPC_CHTTP2_LIST_WRITABLE);
 }
 
 int grpc_chttp2_list_pop_writable_stream(
@@ -159,20 +163,20 @@ int grpc_chttp2_list_pop_writable_stream(
   return r;
 }
 
-void grpc_chttp2_list_remove_writable_stream(
+bool grpc_chttp2_list_remove_writable_stream(
     grpc_chttp2_transport_global *transport_global,
     grpc_chttp2_stream_global *stream_global) {
-  stream_list_maybe_remove(TRANSPORT_FROM_GLOBAL(transport_global),
-                           STREAM_FROM_GLOBAL(stream_global),
-                           GRPC_CHTTP2_LIST_WRITABLE);
+  return stream_list_maybe_remove(TRANSPORT_FROM_GLOBAL(transport_global),
+                                  STREAM_FROM_GLOBAL(stream_global),
+                                  GRPC_CHTTP2_LIST_WRITABLE);
 }
 
-int grpc_chttp2_list_add_writing_stream(
+void grpc_chttp2_list_add_writing_stream(
     grpc_chttp2_transport_writing *transport_writing,
     grpc_chttp2_stream_writing *stream_writing) {
-  return stream_list_add(TRANSPORT_FROM_WRITING(transport_writing),
-                         STREAM_FROM_WRITING(stream_writing),
-                         GRPC_CHTTP2_LIST_WRITING);
+  GPR_ASSERT(stream_list_add(TRANSPORT_FROM_WRITING(transport_writing),
+                             STREAM_FROM_WRITING(stream_writing),
+                             GRPC_CHTTP2_LIST_WRITING));
 }
 
 int grpc_chttp2_list_have_writing_streams(
@@ -332,7 +336,7 @@ void grpc_chttp2_list_flush_writing_stalled_by_transport(
   while (stream_list_pop(transport, &stream,
                          GRPC_CHTTP2_LIST_WRITING_STALLED_BY_TRANSPORT)) {
     if (is_window_available) {
-      grpc_chttp2_list_add_writable_stream(&transport->global, &stream->global);
+      grpc_chttp2_become_writable(&transport->global, &stream->global);
     } else {
       grpc_chttp2_list_add_stalled_by_transport(transport_writing,
                                                 &stream->writing);

src/core/transport/chttp2/writing.c

@@ -83,7 +83,8 @@ int grpc_chttp2_unlocking_check_writes(
      (according to available window sizes) and add to the output buffer */
   while (grpc_chttp2_list_pop_writable_stream(
       transport_global, transport_writing, &stream_global, &stream_writing)) {
-    uint8_t sent_initial_metadata;
+    bool sent_initial_metadata = stream_writing->sent_initial_metadata;
+    bool become_writable = false;
 
     stream_writing->id = stream_global->id;
     stream_writing->read_closed = stream_global->read_closed;
@@ -92,16 +93,12 @@ int grpc_chttp2_unlocking_check_writes(
                                  outgoing_window, stream_global,
                                  outgoing_window);
 
-    sent_initial_metadata = stream_writing->sent_initial_metadata;
     if (!sent_initial_metadata && stream_global->send_initial_metadata) {
       stream_writing->send_initial_metadata =
           stream_global->send_initial_metadata;
       stream_global->send_initial_metadata = NULL;
-      if (grpc_chttp2_list_add_writing_stream(transport_writing,
-                                              stream_writing)) {
-        GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing");
-      }
-      sent_initial_metadata = 1;
+      become_writable = true;
+      sent_initial_metadata = true;
     }
     if (sent_initial_metadata) {
       if (stream_global->send_message != NULL) {
@@ -128,10 +125,7 @@ int grpc_chttp2_unlocking_check_writes(
            stream_writing->flow_controlled_buffer.length > 0) &&
           stream_writing->outgoing_window > 0) {
         if (transport_writing->outgoing_window > 0) {
-          if (grpc_chttp2_list_add_writing_stream(transport_writing,
-                                                  stream_writing)) {
-            GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing");
-          }
+          become_writable = true;
         } else {
           grpc_chttp2_list_add_stalled_by_transport(transport_writing,
                                                     stream_writing);
@@ -141,10 +135,7 @@ int grpc_chttp2_unlocking_check_writes(
         stream_writing->send_trailing_metadata =
             stream_global->send_trailing_metadata;
         stream_global->send_trailing_metadata = NULL;
-        if (grpc_chttp2_list_add_writing_stream(transport_writing,
-                                                stream_writing)) {
-          GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing");
-        }
+        become_writable = true;
       }
     }
 
@@ -153,10 +144,13 @@ int grpc_chttp2_unlocking_check_writes(
       GRPC_CHTTP2_FLOW_MOVE_STREAM("write", transport_global, stream_writing,
                                    announce_window, stream_global,
                                    unannounced_incoming_window_for_writing);
-      if (grpc_chttp2_list_add_writing_stream(transport_writing,
-                                              stream_writing)) {
-        GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing");
-      }
+      become_writable = true;
+    }
+
+    if (become_writable) {
+      grpc_chttp2_list_add_writing_stream(transport_writing, stream_writing);
+    } else {
+      GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2_writing");
     }
   }
 
@@ -310,10 +304,7 @@ static void finalize_outbuf(grpc_exec_ctx *exec_ctx,
          (stream_writing->send_message && !stream_writing->fetching)) &&
         stream_writing->outgoing_window > 0) {
       if (transport_writing->outgoing_window > 0) {
-        if (grpc_chttp2_list_add_writing_stream(transport_writing,
-                                                stream_writing)) {
-          /* do nothing - already reffed */
-        }
+        grpc_chttp2_list_add_writing_stream(transport_writing, stream_writing);
       } else {
         grpc_chttp2_list_add_writing_stalled_by_transport(transport_writing,
                                                           stream_writing);

src/core/transport/chttp2_transport.c

@@ -142,7 +142,7 @@ static void incoming_byte_stream_update_flow_control(
 static void fail_pending_writes(grpc_exec_ctx *exec_ctx,
                                 grpc_chttp2_stream_global *stream_global);
 
-/*
+/*******************************************************************************
  * CONSTRUCTION/DESTRUCTION/REFCOUNTING
  */
 
@@ -432,6 +432,14 @@ static void close_transport_locked(grpc_exec_ctx *exec_ctx,
     if (t->ep) {
       allow_endpoint_shutdown_locked(exec_ctx, t);
     }
+
+    /* flush writable stream list to avoid dangling references */
+    grpc_chttp2_stream_global *stream_global;
+    grpc_chttp2_stream_writing *stream_writing;
+    while (grpc_chttp2_list_pop_writable_stream(
+        &t->global, &t->writing, &stream_global, &stream_writing)) {
+      GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2_writing");
+    }
   }
 }
 
@@ -521,7 +529,6 @@ static void destroy_stream(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
                                            s->global.id) == NULL);
   }
 
-  grpc_chttp2_list_remove_writable_stream(&t->global, &s->global);
   grpc_chttp2_list_remove_unannounced_incoming_window_available(&t->global,
                                                                 &s->global);
   grpc_chttp2_list_remove_stalled_by_transport(&t->global, &s->global);
@@ -583,7 +590,7 @@ grpc_chttp2_stream_parsing *grpc_chttp2_parsing_accept_stream(
   return &accepting->parsing;
 }
 
-/*
+/*******************************************************************************
  * LOCK MANAGEMENT
  */
 
@@ -611,10 +618,18 @@ static void unlock(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t) {
   GPR_TIMER_END("unlock", 0);
 }
 
-/*
+/*******************************************************************************
  * OUTPUT PROCESSING
  */
 
+void grpc_chttp2_become_writable(grpc_chttp2_transport_global *transport_global,
+                                 grpc_chttp2_stream_global *stream_global) {
+  if (!TRANSPORT_FROM_GLOBAL(transport_global)->closed &&
+      grpc_chttp2_list_add_writable_stream(transport_global, stream_global)) {
+    GRPC_CHTTP2_STREAM_REF(stream_global, "chttp2_writing");
+  }
+}
+
 static void push_setting(grpc_chttp2_transport *t, grpc_chttp2_setting_id id,
                          uint32_t value) {
   const grpc_chttp2_setting_parameters *sp =
@@ -732,7 +747,7 @@ static void maybe_start_some_streams(
         stream_global->id, STREAM_FROM_GLOBAL(stream_global));
     stream_global->in_stream_map = 1;
     transport_global->concurrent_stream_count++;
-    grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+    grpc_chttp2_become_writable(transport_global, stream_global);
   }
   /* cancel out streams that will never be started */
   while (transport_global->next_stream_id >= MAX_CLIENT_STREAM_ID &&
@@ -821,7 +836,7 @@ static void perform_stream_op_locked(
         maybe_start_some_streams(exec_ctx, transport_global);
       } else {
         GPR_ASSERT(stream_global->id != 0);
-        grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+        grpc_chttp2_become_writable(transport_global, stream_global);
       }
     } else {
       grpc_chttp2_complete_closure_step(
@@ -838,7 +853,7 @@ static void perform_stream_op_locked(
           exec_ctx, &stream_global->send_message_finished, 0);
     } else if (stream_global->id != 0) {
       stream_global->send_message = op->send_message;
-      grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+      grpc_chttp2_become_writable(transport_global, stream_global);
     }
   }
 
@@ -858,7 +873,7 @@ static void perform_stream_op_locked(
     } else if (stream_global->id != 0) {
       /* TODO(ctiller): check if there's flow control for any outstanding
          bytes before going writable */
-      grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+      grpc_chttp2_become_writable(transport_global, stream_global);
     }
   }
 
@@ -944,12 +959,10 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx,
   unlock(exec_ctx, t);
 }
 
-static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
-                                 grpc_transport_op *op) {
-  grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt;
-  int close_transport = 0;
-
-  lock(t);
+static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx,
+                                        grpc_chttp2_transport *t,
+                                        grpc_transport_op *op) {
+  bool close_transport = false;
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
@@ -968,8 +981,8 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
     close_transport = !grpc_chttp2_has_streams(t);
   }
 
-  if (op->set_accept_stream != NULL) {
-    t->channel_callback.accept_stream = op->set_accept_stream;
+  if (op->set_accept_stream) {
+    t->channel_callback.accept_stream = op->set_accept_stream_fn;
     t->channel_callback.accept_stream_user_data =
         op->set_accept_stream_user_data;
   }
@@ -990,16 +1003,31 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
     close_transport_locked(exec_ctx, t);
   }
 
-  unlock(exec_ctx, t);
-
   if (close_transport) {
-    lock(t);
     close_transport_locked(exec_ctx, t);
-    unlock(exec_ctx, t);
   }
 }
 
-/*
+static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
+                                 grpc_transport_op *op) {
+  grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt;
+
+  lock(t);
+
+  /* If there's a set_accept_stream ensure that we're not parsing
+     to avoid changing things out from underneath */
+  if (t->parsing_active && op->set_accept_stream) {
+    GPR_ASSERT(t->post_parsing_op == NULL);
+    t->post_parsing_op = gpr_malloc(sizeof(*op));
+    memcpy(t->post_parsing_op, op, sizeof(*op));
+  } else {
+    perform_transport_op_locked(exec_ctx, t, op);
+  }
+
+  unlock(exec_ctx, t);
+}
+
+/*******************************************************************************
  * INPUT PROCESSING
  */
 
@@ -1064,7 +1092,6 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t,
   if (!s) {
     s = grpc_chttp2_stream_map_delete(&t->new_stream_map, id);
   }
-  grpc_chttp2_list_remove_writable_stream(&t->global, &s->global);
   GPR_ASSERT(s);
   s->global.in_stream_map = 0;
   if (t->parsing.incoming_stream == &s->parsing) {
@@ -1080,6 +1107,9 @@ static void remove_stream(grpc_exec_ctx *exec_ctx, grpc_chttp2_transport *t,
   if (grpc_chttp2_unregister_stream(t, s) && t->global.sent_goaway) {
     close_transport_locked(exec_ctx, t);
   }
+  if (grpc_chttp2_list_remove_writable_stream(&t->global, &s->global)) {
+    GRPC_CHTTP2_STREAM_UNREF(exec_ctx, &s->global, "chttp2_writing");
+  }
 
   new_stream_count = grpc_chttp2_stream_map_size(&t->parsing_stream_map) +
                      grpc_chttp2_stream_map_size(&t->new_stream_map);
@@ -1331,7 +1361,7 @@ static void update_global_window(void *args, uint32_t id, void *stream) {
   is_zero = stream_global->outgoing_window <= 0;
 
   if (was_zero && !is_zero) {
-    grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+    grpc_chttp2_become_writable(transport_global, stream_global);
   }
 }
 
@@ -1392,6 +1422,13 @@ static void recv_data(grpc_exec_ctx *exec_ctx, void *tp, bool success) {
     /* handle higher level things */
     grpc_chttp2_publish_reads(exec_ctx, transport_global, transport_parsing);
     t->parsing_active = 0;
+    /* handle delayed transport ops (if there is one) */
+    if (t->post_parsing_op) {
+      grpc_transport_op *op = t->post_parsing_op;
+      t->post_parsing_op = NULL;
+      perform_transport_op_locked(exec_ctx, t, op);
+      gpr_free(op);
+    }
     /* if a stream is in the stream map, and gets cancelled, we need to ensure
      * we are not parsing before continuing the cancellation to keep things in
      * a sane state */
@@ -1426,7 +1463,7 @@ static void recv_data(grpc_exec_ctx *exec_ctx, void *tp, bool success) {
   GPR_TIMER_END("recv_data", 0);
 }
 
-/*
+/*******************************************************************************
  * CALLBACK LOOP
  */
 
@@ -1440,7 +1477,7 @@ static void connectivity_state_set(
                               state, reason);
 }
 
-/*
+/*******************************************************************************
  * POLLSET STUFF
  */
 
@@ -1468,7 +1505,7 @@ static void set_pollset(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
   unlock(exec_ctx, t);
 }
 
-/*
+/*******************************************************************************
  * BYTE STREAM
  */
 
@@ -1508,7 +1545,7 @@ static void incoming_byte_stream_update_flow_control(
                                    add_max_recv_bytes);
     grpc_chttp2_list_add_unannounced_incoming_window_available(transport_global,
                                                                stream_global);
-    grpc_chttp2_list_add_writable_stream(transport_global, stream_global);
+    grpc_chttp2_become_writable(transport_global, stream_global);
   }
 }
 
@@ -1623,7 +1660,7 @@ grpc_chttp2_incoming_byte_stream *grpc_chttp2_incoming_byte_stream_create(
   return incoming_byte_stream;
 }
 
-/*
+/*******************************************************************************
  * TRACING
  */
 
@@ -1709,7 +1746,7 @@ void grpc_chttp2_flowctl_trace(const char *file, int line, const char *phase,
   gpr_free(prefix);
 }
 
-/*
+/*******************************************************************************
  * INTEGRATION GLUE
  */
 

src/core/transport/metadata.c

@@ -43,11 +43,13 @@
 #include <grpc/support/log.h>
 #include <grpc/support/string_util.h>
 #include <grpc/support/time.h>
+
 #include "src/core/profiling/timers.h"
 #include "src/core/support/murmur_hash.h"
 #include "src/core/support/string.h"
 #include "src/core/transport/chttp2/bin_encoder.h"
 #include "src/core/transport/static_metadata.h"
+#include "src/core/iomgr/iomgr_internal.h"
 
 /* There are two kinds of mdelem and mdstr instances.
  * Static instances are declared in static_metadata.{h,c} and
@@ -227,6 +229,9 @@ void grpc_mdctx_global_shutdown(void) {
     if (shard->count != 0) {
       gpr_log(GPR_DEBUG, "WARNING: %d metadata elements were leaked",
               shard->count);
+      if (grpc_iomgr_abort_on_leaks()) {
+        abort();
+      }
     }
     gpr_free(shard->elems);
   }
@@ -237,6 +242,9 @@ void grpc_mdctx_global_shutdown(void) {
     if (shard->count != 0) {
       gpr_log(GPR_DEBUG, "WARNING: %d metadata strings were leaked",
               shard->count);
+      if (grpc_iomgr_abort_on_leaks()) {
+        abort();
+      }
     }
     gpr_free(shard->strs);
   }

src/core/transport/transport.c

@@ -45,7 +45,7 @@ void grpc_stream_ref(grpc_stream_refcount *refcount, const char *reason) {
 #else
 void grpc_stream_ref(grpc_stream_refcount *refcount) {
 #endif
-  gpr_ref(&refcount->refs);
+  gpr_ref_non_zero(&refcount->refs);
 }
 
 #ifdef GRPC_STREAM_REFCOUNT_DEBUG

src/core/transport/transport.h

@@ -123,7 +123,7 @@ typedef struct grpc_transport_stream_op {
 
 /** Transport op: a set of operations to perform on a transport as a whole */
 typedef struct grpc_transport_op {
-  /** called when processing of this op is done */
+  /** Called when processing of this op is done. */
   grpc_closure *on_consumed;
   /** connectivity monitoring - set connectivity_state to NULL to unsubscribe */
   grpc_closure *on_connectivity_state_change;
@@ -138,9 +138,13 @@ typedef struct grpc_transport_op {
   grpc_status_code goaway_status;
   gpr_slice *goaway_message;
   /** set the callback for accepting new streams;
-      this is a permanent callback, unlike the other one-shot closures */
-  void (*set_accept_stream)(grpc_exec_ctx *exec_ctx, void *user_data,
-                            grpc_transport *transport, const void *server_data);
+      this is a permanent callback, unlike the other one-shot closures.
+      If true, the callback is set to set_accept_stream_fn, with its
+      user_data argument set to set_accept_stream_user_data */
+  bool set_accept_stream;
+  void (*set_accept_stream_fn)(grpc_exec_ctx *exec_ctx, void *user_data,
+                               grpc_transport *transport,
+                               const void *server_data);
   void *set_accept_stream_user_data;
   /** add this transport to a pollset */
   grpc_pollset *bind_pollset;

src/csharp/Grpc.Core/Internal/AsyncCallServer.cs

@@ -193,16 +193,6 @@ namespace Grpc.Core.Internal
             lock (myLock)
             {
                 finished = true;
-
-                if (cancelled)
-                {
-                    // Once we cancel, we don't have to care that much 
-                    // about reads and writes.
-
-                    // TODO(jtattermusch): is this still necessary?
-                    Cancel();
-                }
-
                 ReleaseResourcesIfPossible();
             }
             // TODO(jtattermusch): handle error

src/csharp/Grpc.IntegrationTesting/InteropClientServerTest.cs

@@ -1,6 +1,6 @@
 #region Copyright notice and license
 
-// Copyright 2015, Google Inc.
+// Copyright 2015-2016, Google Inc.
 // All rights reserved.
 //
 // Redistribution and use in source and binary forms, with or without
@@ -140,14 +140,12 @@ namespace Grpc.IntegrationTesting
         }
 
         [Test]
-        [Ignore("TODO: see #4427")]
         public async Task StatusCodeAndMessage()
         {
             await InteropClient.RunStatusCodeAndMessageAsync(client);
         }
 
         [Test]
-        [Ignore("TODO: see #4427")]
         public void UnimplementedMethod()
         {
             InteropClient.RunUnimplementedMethod(UnimplementedService.NewClient(channel));

src/csharp/Grpc.Tools.nuspec

@@ -4,18 +4,29 @@
     <id>Grpc.Tools</id>
     <title>gRPC C# Tools</title>
     <summary>Tools for C# implementation of gRPC - an RPC library and framework</summary>
-    <description>Precompiled Windows binary for generating gRPC client/server code</description>
+    <description>Precompiled protobuf compiler and gRPC protobuf compiler plugin for generating gRPC client/server C# code. Binaries are available for Windows, Linux and MacOS.</description>
     <version>$version$</version>
     <authors>Google Inc.</authors>
     <owners>grpc-packages</owners>
     <licenseUrl>https://github.com/grpc/grpc/blob/master/LICENSE</licenseUrl>
     <projectUrl>https://github.com/grpc/grpc</projectUrl>
     <requireLicenseAcceptance>false</requireLicenseAcceptance>
-    <releaseNotes>grpc_csharp_plugin.exe - gRPC C# protoc plugin version $version$</releaseNotes>
+    <releaseNotes>Release $version$</releaseNotes>
     <copyright>Copyright 2015, Google Inc.</copyright>
     <tags>gRPC RPC Protocol HTTP/2</tags>
   </metadata>
   <files>
-    <file src="..\..\vsprojects\Release\grpc_csharp_plugin.exe" target="tools" />
+    <file src="protoc_plugins\windows_x86\protoc.exe" target="tools\windows_x86\protoc.exe" />
+    <file src="protoc_plugins\windows_x86\grpc_csharp_plugin.exe" target="tools\windows_x86\grpc_csharp_plugin.exe" />
+    <file src="protoc_plugins\windows_x64\protoc.exe" target="tools\windows_x64\protoc.exe" />
+    <file src="protoc_plugins\windows_x64\grpc_csharp_plugin.exe" target="tools\windows_x64\grpc_csharp_plugin.exe" />
+    <file src="protoc_plugins\linux_x86\protoc" target="tools\linux_x86\protoc" />
+    <file src="protoc_plugins\linux_x86\grpc_csharp_plugin" target="tools\linux_x86\grpc_csharp_plugin" />
+    <file src="protoc_plugins\linux_x64\protoc" target="tools\linux_x64\protoc" />
+    <file src="protoc_plugins\linux_x64\grpc_csharp_plugin" target="tools\linux_x64\grpc_csharp_plugin" />
+    <file src="protoc_plugins\macosx_x86\protoc" target="tools\macosx_x86\protoc" />
+    <file src="protoc_plugins\macosx_x86\grpc_csharp_plugin" target="tools\macosx_x86\grpc_csharp_plugin" />
+    <file src="protoc_plugins\macosx_x64\protoc" target="tools\macosx_x64\protoc" />
+    <file src="protoc_plugins\macosx_x64\grpc_csharp_plugin" target="tools\macosx_x64\grpc_csharp_plugin" />
   </files>
 </package>

src/csharp/build_packages.bat

@@ -19,6 +19,14 @@ xcopy /Y /I ..\..\architecture=x64,language=csharp,platform=linux\artifacts\* gr
 xcopy /Y /I ..\..\architecture=x86,language=csharp,platform=macos\artifacts\* grpc.native.csharp\macosx_x86\
 xcopy /Y /I ..\..\architecture=x64,language=csharp,platform=macos\artifacts\* grpc.native.csharp\macosx_x64\
 
+@rem Collect protoc artifacts built by the previous build step
+xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=windows\artifacts\* protoc_plugins\windows_x86\
+xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=windows\artifacts\* protoc_plugins\windows_x64\
+xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=linux\artifacts\* protoc_plugins\linux_x86\
+xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=linux\artifacts\* protoc_plugins\linux_x64\
+xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=macos\artifacts\* protoc_plugins\macosx_x86\
+xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=macos\artifacts\* protoc_plugins\macosx_x64\
+
 @rem Fetch all dependencies
 %NUGET% restore ..\..\vsprojects\grpc_csharp_ext.sln || goto :error
 %NUGET% restore Grpc.sln || goto :error
@@ -27,24 +35,19 @@ setlocal
 
 @call "%VS120COMNTOOLS%\..\..\vc\vcvarsall.bat" x86
 
-@rem We won't use the native libraries from this step, but without this Grpc.sln will fail.  
+@rem We won't use the native libraries from this step, but without this Grpc.sln will fail.
 msbuild ..\..\vsprojects\grpc_csharp_ext.sln /p:Configuration=Release /p:PlatformToolset=v120 || goto :error
 
 msbuild Grpc.sln /p:Configuration=ReleaseSigned || goto :error
 
 endlocal
 
-@rem TODO(jtattermusch): re-enable protoc plugin building
-@rem @call ..\..\vsprojects\build_plugins.bat || goto :error
-
 %NUGET% pack grpc.native.csharp\grpc.native.csharp.nuspec -Version %VERSION% || goto :error
 %NUGET% pack Grpc.Auth\Grpc.Auth.nuspec -Symbols -Version %VERSION% || goto :error
 %NUGET% pack Grpc.Core\Grpc.Core.nuspec -Symbols -Version %VERSION% || goto :error
 %NUGET% pack Grpc.HealthCheck\Grpc.HealthCheck.nuspec -Symbols -Version %VERSION_WITH_BETA% -Properties ProtobufVersion=%PROTOBUF_VERSION% || goto :error
 %NUGET% pack Grpc.nuspec -Version %VERSION% || goto :error
-
-@rem TODO(jtattermusch): re-enable building Grpc.Tools package
-@rem %NUGET% pack Grpc.Tools.nuspec -Version %VERSION% || goto :error
+%NUGET% pack Grpc.Tools.nuspec -Version %VERSION% || goto :error
 
 @rem copy resulting nuget packages to artifacts directory
 xcopy /Y /I *.nupkg ..\..\artifacts\

src/node/interop/interop_client.js

@@ -290,6 +290,7 @@ function timeoutOnSleepingServer(client, done) {
   call.write({
     payload: {body: zeroBuffer(27182)}
   });
+  call.on('data', function() {});
   call.on('error', function(error) {
 
     assert(error.code === grpc.status.DEADLINE_EXCEEDED ||
@@ -336,6 +337,7 @@ function customMetadata(client, done) {
                      ['test_initial_metadata_value']);
     done();
   });
+  stream.on('data', function() {});
   stream.on('status', function(status) {
     var echo_trailer = status.metadata.get(ECHO_TRAILING_KEY);
     assert(echo_trailer.length > 0);
@@ -361,6 +363,7 @@ function statusCodeAndMessage(client, done) {
     done();
   });
   var duplex = client.fullDuplexCall();
+  duplex.on('data', function() {});
   duplex.on('status', function(status) {
     assert(status);
     assert.strictEqual(status.code, 2);

src/node/src/client.js

@@ -131,8 +131,68 @@ function ClientReadableStream(call, deserialize) {
   this.finished = false;
   this.reading = false;
   this.deserialize = common.wrapIgnoreNull(deserialize);
+  /* Status generated from reading messages from the server. Overrides the
+   * status from the server if not OK */
+  this.read_status = null;
+  /* Status received from the server. */
+  this.received_status = null;
 }
 
+/**
+ * Called when all messages from the server have been processed. The status
+ * parameter indicates that the call should end with that status. status
+ * defaults to OK if not provided.
+ * @param {Object!} status The status that the call should end with
+ */
+function _readsDone(status) {
+  /* jshint validthis: true */
+  if (!status) {
+    status = {code: grpc.status.OK, details: 'OK'};
+  }
+  this.finished = true;
+  this.read_status = status;
+  this._emitStatusIfDone();
+}
+
+ClientReadableStream.prototype._readsDone = _readsDone;
+
+/**
+ * Called to indicate that we have received a status from the server.
+ */
+function _receiveStatus(status) {
+  /* jshint validthis: true */
+  this.received_status = status;
+  this._emitStatusIfDone();
+}
+
+ClientReadableStream.prototype._receiveStatus = _receiveStatus;
+
+/**
+ * If we have both processed all incoming messages and received the status from
+ * the server, emit the status. Otherwise, do nothing.
+ */
+function _emitStatusIfDone() {
+  /* jshint validthis: true */
+  var status;
+  if (this.read_status && this.received_status) {
+    if (this.read_status.code !== grpc.status.OK) {
+      status = this.read_status;
+    } else {
+      status = this.received_status;
+    }
+    this.emit('status', status);
+    if (status.code !== grpc.status.OK) {
+      var error = new Error(status.details);
+      error.code = status.code;
+      error.metadata = status.metadata;
+      this.emit('error', error);
+      return;
+    }
+  }
+}
+
+ClientReadableStream.prototype._emitStatusIfDone = _emitStatusIfDone;
+
 /**
  * Read the next object from the stream.
  * @access private
@@ -150,6 +210,7 @@ function _read(size) {
     if (err) {
       // Something has gone wrong. Stop reading and wait for status
       self.finished = true;
+      self._readsDone();
       return;
     }
     var data = event.read;
@@ -157,8 +218,11 @@ function _read(size) {
     try {
       deserialized = self.deserialize(data);
     } catch (e) {
-      self.call.cancelWithStatus(grpc.status.INTERNAL,
-                                 'Failed to parse server response');
+      self._readsDone({code: grpc.status.INTERNAL,
+                       details: 'Failed to parse server response'});
+    }
+    if (data === null) {
+      self._readsDone();
     }
     if (self.push(deserialized) && data !== null) {
       var read_batch = {};
@@ -198,6 +262,11 @@ function ClientDuplexStream(call, serialize, deserialize) {
   this.serialize = common.wrapIgnoreNull(serialize);
   this.deserialize = common.wrapIgnoreNull(deserialize);
   this.call = call;
+  /* Status generated from reading messages from the server. Overrides the
+   * status from the server if not OK */
+  this.read_status = null;
+  /* Status received from the server. */
+  this.received_status = null;
   this.on('finish', function() {
     var batch = {};
     batch[grpc.opType.SEND_CLOSE_FROM_CLIENT] = true;
@@ -205,6 +274,9 @@ function ClientDuplexStream(call, serialize, deserialize) {
   });
 }
 
+ClientDuplexStream.prototype._readsDone = _readsDone;
+ClientDuplexStream.prototype._receiveStatus = _receiveStatus;
+ClientDuplexStream.prototype._emitStatusIfDone = _emitStatusIfDone;
 ClientDuplexStream.prototype._read = _read;
 ClientDuplexStream.prototype._write = _write;
 
@@ -487,22 +559,13 @@ function makeServerStreamRequestFunction(method, serialize, deserialize) {
     var status_batch = {};
     status_batch[grpc.opType.RECV_STATUS_ON_CLIENT] = true;
     call.startBatch(status_batch, function(err, response) {
-      response.status.metadata = Metadata._fromCoreRepresentation(
-          response.status.metadata);
-      stream.emit('status', response.status);
-      if (response.status.code !== grpc.status.OK) {
-        var error = new Error(response.status.details);
-        error.code = response.status.code;
-        error.metadata = response.status.metadata;
-        stream.emit('error', error);
+      if (err) {
+        stream.emit('error', err);
         return;
-      } else {
-        if (err) {
-          // Got a batch error, but OK status. Something went wrong
-          stream.emit('error', err);
-          return;
-        }
       }
+      response.status.metadata = Metadata._fromCoreRepresentation(
+          response.status.metadata);
+      stream._receiveStatus(response.status);
     });
     return stream;
   }
@@ -552,22 +615,13 @@ function makeBidiStreamRequestFunction(method, serialize, deserialize) {
     var status_batch = {};
     status_batch[grpc.opType.RECV_STATUS_ON_CLIENT] = true;
     call.startBatch(status_batch, function(err, response) {
-      response.status.metadata = Metadata._fromCoreRepresentation(
-          response.status.metadata);
-      stream.emit('status', response.status);
-      if (response.status.code !== grpc.status.OK) {
-        var error = new Error(response.status.details);
-        error.code = response.status.code;
-        error.metadata = response.status.metadata;
-        stream.emit('error', error);
+      if (err) {
+        stream.emit('error', err);
         return;
-      } else {
-        if (err) {
-          // Got a batch error, but OK status. Something went wrong
-          stream.emit('error', err);
-          return;
-        }
       }
+      response.status.metadata = Metadata._fromCoreRepresentation(
+          response.status.metadata);
+      stream._receiveStatus(response.status);
     });
     return stream;
   }

src/node/test/surface_test.js

@@ -1000,6 +1000,7 @@ describe('Call propagation', function() {
       proxy_impl.serverStream = function(parent) {
         var child = client.serverStream(parent.request, null,
                                         {parent: parent});
+        child.on('data', function() {});
         child.on('error', function(err) {
           assert(err);
           assert.strictEqual(err.code, grpc.status.CANCELLED);
@@ -1013,6 +1014,7 @@ describe('Call propagation', function() {
       var proxy_client = new Client('localhost:' + proxy_port,
                                     grpc.credentials.createInsecure());
       call = proxy_client.serverStream({});
+      call.on('data', function() {});
       call.on('error', function(err) {
         done();
       });
@@ -1022,6 +1024,7 @@ describe('Call propagation', function() {
       var call;
       proxy_impl.bidiStream = function(parent) {
         var child = client.bidiStream(null, {parent: parent});
+        child.on('data', function() {});
         child.on('error', function(err) {
           assert(err);
           assert.strictEqual(err.code, grpc.status.CANCELLED);
@@ -1035,6 +1038,7 @@ describe('Call propagation', function() {
       var proxy_client = new Client('localhost:' + proxy_port,
                                     grpc.credentials.createInsecure());
       call = proxy_client.bidiStream();
+      call.on('data', function() {});
       call.on('error', function(err) {
         done();
       });
@@ -1074,6 +1078,7 @@ describe('Call propagation', function() {
       proxy_impl.bidiStream = function(parent) {
         var child = client.bidiStream(
             null, {parent: parent, propagate_flags: deadline_flags});
+        child.on('data', function() {});
         child.on('error', function(err) {
           assert(err);
           assert(err.code === grpc.status.DEADLINE_EXCEEDED ||
@@ -1089,6 +1094,7 @@ describe('Call propagation', function() {
       var deadline = new Date();
       deadline.setSeconds(deadline.getSeconds() + 1);
       var call = proxy_client.bidiStream(null, {deadline: deadline});
+      call.on('data', function() {});
       call.on('error', function(err) {
         done();
       });
@@ -1130,6 +1136,7 @@ describe('Cancelling surface client', function() {
   });
   it('Should correctly cancel a server stream call', function(done) {
     var call = client.fib({'limit': 5});
+    call.on('data', function() {});
     call.on('error', function(error) {
       assert.strictEqual(error.code, surface_client.status.CANCELLED);
       done();
@@ -1138,6 +1145,7 @@ describe('Cancelling surface client', function() {
   });
   it('Should correctly cancel a bidi stream call', function(done) {
     var call = client.divMany();
+    call.on('data', function() {});
     call.on('error', function(error) {
       assert.strictEqual(error.code, surface_client.status.CANCELLED);
       done();

src/objective-c/GRPCClient/private/GRPCCompletionQueue.h

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

src/objective-c/GRPCClient/private/GRPCCompletionQueue.m

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

src/objective-c/tests/GRPCClientTests.m

@@ -103,6 +103,8 @@ static ProtoMethod *kUnaryCallMethod;
 @implementation GRPCClientTests
 
 - (void)setUp {
+  // Add a custom user agent prefix that will be used in test
+  [GRPCCall setUserAgentPrefix:@"Foo" forHost:kHostAddress];
   // Register test server as non-SSL.
   [GRPCCall useInsecureConnectionsForHost:kHostAddress];
 
@@ -257,6 +259,35 @@ static ProtoMethod *kUnaryCallMethod;
   [self waitForExpectationsWithTimeout:8 handler:nil];
 }
 
+- (void)testUserAgentPrefix {
+  __weak XCTestExpectation *response = [self expectationWithDescription:@"Empty response received."];
+  __weak XCTestExpectation *completion = [self expectationWithDescription:@"Empty RPC completed."];
+
+  GRPCCall *call = [[GRPCCall alloc] initWithHost:kHostAddress
+                                             path:kEmptyCallMethod.HTTPPath
+                                   requestsWriter:[GRXWriter writerWithValue:[NSData data]]];
+  // Setting this special key in the header will cause the interop server to echo back the
+  // user-agent value, which we confirm.
+  call.requestHeaders[@"x-grpc-test-echo-useragent"] = @"";
+
+  id<GRXWriteable> responsesWriteable = [[GRXWriteable alloc] initWithValueHandler:^(NSData *value) {
+    XCTAssertNotNil(value, @"nil value received as response.");
+    XCTAssertEqual([value length], 0, @"Non-empty response received: %@", value);
+    XCTAssertEqualObjects(call.responseHeaders[@"x-grpc-test-echo-useragent"],
+                          @"Foo grpc-objc/0.13.0 grpc-c/0.14.0-dev (ios)",
+                          @"Did not receive expected user agent %@",
+                          call.responseHeaders[@"x-grpc-test-echo-useragent"]);
+    [response fulfill];
+  } completionHandler:^(NSError *errorOrNil) {
+    XCTAssertNil(errorOrNil, @"Finished with unexpected error: %@", errorOrNil);
+    [completion fulfill];
+  }];
+
+  [call startWithWriteable:responsesWriteable];
+
+  [self waitForExpectationsWithTimeout:8 handler:nil];
+}
+
 // TODO(makarandd): Move to a different file that contains only unit tests
 - (void)testExceptions {
   // Try to set userAgentPrefix for host that is nil. This should cause

src/php/README.md

@@ -33,45 +33,12 @@ $ sudo mv phpunit.phar /usr/local/bin/phpunit
 
 ## Quick Install
 
-**Linux (Debian):**
-
-Add [Debian jessie-backports][] to your `sources.list` file. Example:
-
-```sh
-echo "deb http://http.debian.net/debian jessie-backports main" | \
-sudo tee -a /etc/apt/sources.list
-```
-
-Install the gRPC Debian package
-
-```sh
-sudo apt-get update
-sudo apt-get install libgrpc-dev
-```
-
 Install the gRPC PHP extension
 
 ```sh
 sudo pecl install grpc-beta
 ```
 
-**Mac OS X:**
-
-Install [homebrew][]. Example:
-
-```sh
-ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
-```
-
-Install the gRPC core library and the PHP extension in one step
-
-```sh
-$ curl -fsSL https://goo.gl/getgrpc | bash -s php
-```
-
-This will download and run the [gRPC install script][] and compile the gRPC PHP extension.
-
-
 ## Build from Source
 
 Clone this repository
@@ -297,7 +264,4 @@ Connect to `localhost/math_client.php` in your browser, or run this from command
 $ curl localhost/math_client.php
 ```
 
-[homebrew]:http://brew.sh
-[gRPC install script]:https://raw.githubusercontent.com/grpc/homebrew-grpc/master/scripts/install
 [Node]:https://github.com/grpc/grpc/tree/master/src/node/examples
-[Debian jessie-backports]:http://backports.debian.org/Instructions/

src/python/grpcio/README.rst

@@ -38,3 +38,17 @@ package named `python-dev`).
 
 Note that `$REPO_ROOT` can be assigned to whatever directory name floats your
 fancy.
+
+Troubleshooting
+~~~~~~~~~~~~~~~
+
+Help, I ...
+
+* **... see a** :code:`pkg_resources.VersionConflict` **when I try to install
+  grpc!**
+
+  This is likely because :code:`pip` doesn't own the offending dependency,
+  which in turn is likely because your operating system's package manager owns
+  it. You'll need to force the installation of the dependency:
+
+  :code:`pip install --ignore-installed $OFFENDING_DEPENDENCY`

src/python/grpcio/commands.py

@@ -264,6 +264,42 @@ class Gather(setuptools.Command):
       self.distribution.fetch_build_eggs(self.distribution.tests_require)
 
 
+class TestLite(setuptools.Command):
+  """Command to run tests without fetching or building anything."""
+
+  description = 'run tests without fetching or building anything.'
+  user_options = []
+
+  def initialize_options(self):
+    pass
+
+  def finalize_options(self):
+    # distutils requires this override.
+    pass
+
+  def run(self):
+    self._add_eggs_to_path()
+
+    import tests
+    loader = tests.Loader()
+    loader.loadTestsFromNames(['tests'])
+    runner = tests.Runner()
+    result = runner.run(loader.suite)
+    if not result.wasSuccessful():
+      sys.exit('Test failure')
+
+  def _add_eggs_to_path(self):
+    """Adds all egg files under .eggs to sys.path"""
+    # TODO(jtattemusch): there has to be a cleaner way to do this
+    import pkg_resources
+    eggs_dir = os.path.join(PYTHON_STEM, '../../../.eggs')
+    eggs = [os.path.join(eggs_dir, filename)
+            for filename in os.listdir(eggs_dir)
+            if filename.endswith('.egg')]
+    for egg in eggs:
+      sys.path.insert(0, pkg_resources.normalize_path(egg))
+
+
 class RunInterop(test.test):
 
   description = 'run interop test client/server'

src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -89,12 +89,13 @@ cdef class Channel:
 
   def check_connectivity_state(self, bint try_to_connect):
     return grpc_channel_check_connectivity_state(self.c_channel,
-                                                      try_to_connect)
+                                                 try_to_connect)
 
   def watch_connectivity_state(
-      self, last_observed_state, Timespec deadline not None,
-      CompletionQueue queue not None, tag):
+      self, grpc_connectivity_state last_observed_state,
+      Timespec deadline not None, CompletionQueue queue not None, tag):
     cdef OperationTag operation_tag = OperationTag(tag)
+    operation_tag.references = [self, queue]
     cpython.Py_INCREF(operation_tag)
     grpc_channel_watch_connectivity_state(
         self.c_channel, last_observed_state, deadline.c_time,

src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pxd.pxi

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -31,8 +31,9 @@
 cdef class CompletionQueue:
 
   cdef grpc_completion_queue *c_completion_queue
-  cdef object poll_condition
-  cdef bint is_polling
+  cdef object pluck_condition
+  cdef int num_plucking
+  cdef int num_polling
   cdef bint is_shutting_down
   cdef bint is_shutdown
 

src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -39,8 +39,9 @@ cdef class CompletionQueue:
     self.c_completion_queue = grpc_completion_queue_create(NULL)
     self.is_shutting_down = False
     self.is_shutdown = False
-    self.poll_condition = threading.Condition()
-    self.is_polling = False
+    self.pluck_condition = threading.Condition()
+    self.num_plucking = 0
+    self.num_polling = 0
 
   cdef _interpret_event(self, grpc_event event):
     cdef OperationTag tag = None
@@ -87,19 +88,15 @@ cdef class CompletionQueue:
       c_deadline = deadline.c_time
     cdef grpc_event event
 
-    # Poll within a critical section
-    # TODO(atash) consider making queue polling contention a hard error to
-    # enable easier bug discovery
-    with self.poll_condition:
-      while self.is_polling:
-        self.poll_condition.wait(float(deadline) - time.time())
-      self.is_polling = True
+    # Poll within a critical section to detect contention
+    with self.pluck_condition:
+      assert self.num_plucking == 0, 'cannot simultaneously pluck and poll'
+      self.num_polling += 1
     with nogil:
       event = grpc_completion_queue_next(
           self.c_completion_queue, c_deadline, NULL)
-    with self.poll_condition:
-      self.is_polling = False
-      self.poll_condition.notify()
+    with self.pluck_condition:
+      self.num_polling -= 1
     return self._interpret_event(event)
 
   def pluck(self, OperationTag tag, Timespec deadline=None):
@@ -111,19 +108,18 @@ cdef class CompletionQueue:
       c_deadline = deadline.c_time
     cdef grpc_event event
 
-    # Poll within a critical section
-    # TODO(atash) consider making queue polling contention a hard error to
-    # enable easier bug discovery
-    with self.poll_condition:
-      while self.is_polling:
-        self.poll_condition.wait(float(deadline) - time.time())
-      self.is_polling = True
+    # Pluck within a critical section to detect contention
+    with self.pluck_condition:
+      assert self.num_polling == 0, 'cannot simultaneously pluck and poll'
+      assert self.num_plucking < GRPC_MAX_COMPLETION_QUEUE_PLUCKERS, (
+          'cannot pluck more than {} times simultaneously'.format(
+              GRPC_MAX_COMPLETION_QUEUE_PLUCKERS))
+      self.num_plucking += 1
     with nogil:
       event = grpc_completion_queue_pluck(
           self.c_completion_queue, <cpython.PyObject *>tag, c_deadline, NULL)
-    with self.poll_condition:
-      self.is_polling = False
-      self.poll_condition.notify()
+    with self.pluck_condition:
+      self.num_plucking -= 1
     return self._interpret_event(event)
 
   def shutdown(self):
@@ -137,10 +133,14 @@ cdef class CompletionQueue:
       pass
 
   def __dealloc__(self):
+    cdef gpr_timespec c_deadline = gpr_inf_future(GPR_CLOCK_REALTIME)
     if self.c_completion_queue != NULL:
-      # Ensure shutdown, pump the queue
+      # Ensure shutdown
       if not self.is_shutting_down:
-        self.shutdown()
+        grpc_completion_queue_shutdown(self.c_completion_queue)
+      # Pump the queue
       while not self.is_shutdown:
-        self.poll()
+        event = grpc_completion_queue_next(
+            self.c_completion_queue, c_deadline, NULL)
+        self._interpret_event(event)
       grpc_completion_queue_destroy(self.c_completion_queue)

src/python/grpcio/grpc/_cython/_cygrpc/grpc.pxi

@@ -138,6 +138,8 @@ cdef extern from "grpc/_cython/loader.h":
   const int GRPC_WRITE_NO_COMPRESS
   const int GRPC_WRITE_USED_MASK
 
+  const int GRPC_MAX_COMPLETION_QUEUE_PLUCKERS
+
   ctypedef struct grpc_completion_queue:
     # We don't care about the internals (and in fact don't know them)
     pass

src/python/grpcio/grpc/_cython/_cygrpc/server.pxd.pxi

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -39,4 +39,5 @@ cdef class Server:
   cdef list references
   cdef list registered_completion_queues
 
+  cdef _c_shutdown(self, CompletionQueue queue, tag)
   cdef notify_shutdown_complete(self)

src/python/grpcio/grpc/_cython/_cygrpc/server.pyx.pxi

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -102,6 +102,16 @@ cdef class Server:
     else:
       return grpc_server_add_insecure_http2_port(self.c_server, address)
 
+  cdef _c_shutdown(self, CompletionQueue queue, tag):
+    self.is_shutting_down = True
+    operation_tag = OperationTag(tag)
+    operation_tag.shutting_down_server = self
+    operation_tag.references.extend([self, queue])
+    cpython.Py_INCREF(operation_tag)
+    grpc_server_shutdown_and_notify(
+        self.c_server, queue.c_completion_queue,
+        <cpython.PyObject *>operation_tag)
+
   def shutdown(self, CompletionQueue queue not None, tag):
     cdef OperationTag operation_tag
     if queue.is_shutting_down:
@@ -113,14 +123,7 @@ cdef class Server:
     elif queue not in self.registered_completion_queues:
       raise ValueError("expected registered completion queue")
     else:
-      self.is_shutting_down = True
-      operation_tag = OperationTag(tag)
-      operation_tag.shutting_down_server = self
-      operation_tag.references.extend([self, queue])
-      cpython.Py_INCREF(operation_tag)
-      grpc_server_shutdown_and_notify(
-          self.c_server, queue.c_completion_queue,
-          <cpython.PyObject *>operation_tag)
+      self._c_shutdown(queue, tag)
 
   cdef notify_shutdown_complete(self):
     # called only by a completion queue on receiving our shutdown operation tag
@@ -142,7 +145,7 @@ cdef class Server:
         pass
       elif not self.is_shutting_down:
         # the user didn't call shutdown - use our backup queue
-        self.shutdown(self.backup_shutdown_queue, None)
+        self._c_shutdown(self.backup_shutdown_queue, None)
         # and now we wait
         while not self.is_shutdown:
           self.backup_shutdown_queue.poll()

src/python/grpcio/grpc/_cython/imports.generated.c

@@ -220,6 +220,7 @@ gpr_event_get_type gpr_event_get_import;
 gpr_event_wait_type gpr_event_wait_import;
 gpr_ref_init_type gpr_ref_init_import;
 gpr_ref_type gpr_ref_import;
+gpr_ref_non_zero_type gpr_ref_non_zero_import;
 gpr_refn_type gpr_refn_import;
 gpr_unref_type gpr_unref_import;
 gpr_stats_init_type gpr_stats_init_import;
@@ -485,6 +486,7 @@ void pygrpc_load_imports(HMODULE library) {
   gpr_event_wait_import = (gpr_event_wait_type) GetProcAddress(library, "gpr_event_wait");
   gpr_ref_init_import = (gpr_ref_init_type) GetProcAddress(library, "gpr_ref_init");
   gpr_ref_import = (gpr_ref_type) GetProcAddress(library, "gpr_ref");
+  gpr_ref_non_zero_import = (gpr_ref_non_zero_type) GetProcAddress(library, "gpr_ref_non_zero");
   gpr_refn_import = (gpr_refn_type) GetProcAddress(library, "gpr_refn");
   gpr_unref_import = (gpr_unref_type) GetProcAddress(library, "gpr_unref");
   gpr_stats_init_import = (gpr_stats_init_type) GetProcAddress(library, "gpr_stats_init");

src/python/grpcio/grpc/_cython/imports.generated.h

@@ -91,10 +91,10 @@ extern census_context_next_tag_type census_context_next_tag_import;
 typedef int(*census_context_get_tag_type)(const census_context *context, const char *key, census_tag *tag);
 extern census_context_get_tag_type census_context_get_tag_import;
 #define census_context_get_tag census_context_get_tag_import
-typedef char *(*census_context_encode_type)(const census_context *context, char *buffer, size_t buf_size, size_t *print_buf_size, size_t *bin_buf_size);
+typedef size_t(*census_context_encode_type)(const census_context *context, char *buffer, size_t buf_size);
 extern census_context_encode_type census_context_encode_import;
 #define census_context_encode census_context_encode_import
-typedef census_context *(*census_context_decode_type)(const char *buffer, size_t size, const char *bin_buffer, size_t bin_size);
+typedef census_context *(*census_context_decode_type)(const char *buffer, size_t size);
 extern census_context_decode_type census_context_decode_import;
 #define census_context_decode census_context_decode_import
 typedef int(*census_trace_mask_type)(const census_context *context);
@@ -610,6 +610,9 @@ extern gpr_ref_init_type gpr_ref_init_import;
 typedef void(*gpr_ref_type)(gpr_refcount *r);
 extern gpr_ref_type gpr_ref_import;
 #define gpr_ref gpr_ref_import
+typedef void(*gpr_ref_non_zero_type)(gpr_refcount *r);
+extern gpr_ref_non_zero_type gpr_ref_non_zero_import;
+#define gpr_ref_non_zero gpr_ref_non_zero_import
 typedef void(*gpr_refn_type)(gpr_refcount *r, int n);
 extern gpr_refn_type gpr_refn_import;
 #define gpr_refn gpr_refn_import

src/python/grpcio/tests/_runner.py

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -143,10 +143,17 @@ class Runner(object):
 
   def run(self, suite):
     """See setuptools' test_runner setup argument for information."""
+    # only run test cases with id starting with given prefix
+    testcase_filter = os.getenv('GPRC_PYTHON_TESTRUNNER_FILTER')
+    filtered_cases = []
+    for case in _loader.iterate_suite_cases(suite):
+      if not testcase_filter or case.id().startswith(testcase_filter):
+        filtered_cases.append(case)
+
     # Ensure that every test case has no collision with any other test case in
     # the augmented results.
     augmented_cases = [AugmentedCase(case, uuid.uuid4())
-                       for case in _loader.iterate_suite_cases(suite)]
+                       for case in filtered_cases]
     case_id_by_case = dict((augmented_case.case, augmented_case.id)
                            for augmented_case in augmented_cases)
     result_out = StringIO.StringIO()

src/python/grpcio/tests/tests.json

@@ -0,0 +1,62 @@
+[
+  "_base_interface_test.AsyncEasyTest", 
+  "_base_interface_test.AsyncPeasyTest", 
+  "_base_interface_test.SyncEasyTest", 
+  "_base_interface_test.SyncPeasyTest", 
+  "_beta_features_test.BetaFeaturesTest", 
+  "_beta_features_test.ContextManagementAndLifecycleTest", 
+  "_channel_test.ChannelTest", 
+  "_connectivity_channel_test.ChannelConnectivityTest", 
+  "_core_over_links_base_interface_test.AsyncEasyTest", 
+  "_core_over_links_base_interface_test.AsyncPeasyTest", 
+  "_core_over_links_base_interface_test.SyncEasyTest", 
+  "_core_over_links_base_interface_test.SyncPeasyTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_implementations_test.ChannelCredentialsTest", 
+  "_insecure_interop_test.InsecureInteropTest", 
+  "_intermediary_low_test.CancellationTest", 
+  "_intermediary_low_test.EchoTest", 
+  "_intermediary_low_test.ExpirationTest", 
+  "_intermediary_low_test.LonelyClientTest", 
+  "_later_test.LaterTest", 
+  "_logging_pool_test.LoggingPoolTest", 
+  "_lonely_invocation_link_test.LonelyInvocationLinkTest", 
+  "_low_test.HangingServerShutdown", 
+  "_low_test.InsecureServerInsecureClient", 
+  "_not_found_test.NotFoundTest", 
+  "_sanity_test.Sanity", 
+  "_secure_interop_test.SecureInteropTest", 
+  "_transmission_test.RoundTripTest", 
+  "_transmission_test.TransmissionTest", 
+  "_utilities_test.ChannelConnectivityTest", 
+  "beta_python_plugin_test.PythonPluginTest", 
+  "cygrpc_test.InsecureServerInsecureClient", 
+  "cygrpc_test.SecureServerSecureClient", 
+  "cygrpc_test.TypeSmokeTest"
+]

src/python/grpcio/tests/unit/_sanity/__init__.py

@@ -0,0 +1,30 @@
+# Copyright 2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+

src/python/grpcio/tests/unit/_sanity/_sanity_test.py

@@ -0,0 +1,53 @@
+# Copyright 2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import json
+import unittest
+
+import tests
+
+
+class Sanity(unittest.TestCase):
+
+  def testTestsJsonUpToDate(self):
+    """Autodiscovers all test suites and checks that tests.json is up to date"""
+    loader = tests.Loader()
+    loader.loadTestsFromNames(['tests'])
+    test_suite_names = [
+        test_case_class.id().rsplit('.', 1)[0]
+        for test_case_class in tests._loader.iterate_suite_cases(loader.suite)]
+    test_suite_names = sorted(set(test_suite_names))
+
+    with open('src/python/grpcio/tests/tests.json') as tests_json_file:
+      tests_json = json.load(tests_json_file)
+    self.assertListEqual(test_suite_names, tests_json)
+
+
+if __name__ == '__main__':
+  unittest.main(verbosity=2)

src/ruby/ext/grpc/rb_grpc_imports.generated.c

@@ -220,6 +220,7 @@ gpr_event_get_type gpr_event_get_import;
 gpr_event_wait_type gpr_event_wait_import;
 gpr_ref_init_type gpr_ref_init_import;
 gpr_ref_type gpr_ref_import;
+gpr_ref_non_zero_type gpr_ref_non_zero_import;
 gpr_refn_type gpr_refn_import;
 gpr_unref_type gpr_unref_import;
 gpr_stats_init_type gpr_stats_init_import;
@@ -481,6 +482,7 @@ void grpc_rb_load_imports(HMODULE library) {
   gpr_event_wait_import = (gpr_event_wait_type) GetProcAddress(library, "gpr_event_wait");
   gpr_ref_init_import = (gpr_ref_init_type) GetProcAddress(library, "gpr_ref_init");
   gpr_ref_import = (gpr_ref_type) GetProcAddress(library, "gpr_ref");
+  gpr_ref_non_zero_import = (gpr_ref_non_zero_type) GetProcAddress(library, "gpr_ref_non_zero");
   gpr_refn_import = (gpr_refn_type) GetProcAddress(library, "gpr_refn");
   gpr_unref_import = (gpr_unref_type) GetProcAddress(library, "gpr_unref");
   gpr_stats_init_import = (gpr_stats_init_type) GetProcAddress(library, "gpr_stats_init");

src/ruby/ext/grpc/rb_grpc_imports.generated.h

@@ -91,10 +91,10 @@ extern census_context_next_tag_type census_context_next_tag_import;
 typedef int(*census_context_get_tag_type)(const census_context *context, const char *key, census_tag *tag);
 extern census_context_get_tag_type census_context_get_tag_import;
 #define census_context_get_tag census_context_get_tag_import
-typedef char *(*census_context_encode_type)(const census_context *context, char *buffer, size_t buf_size, size_t *print_buf_size, size_t *bin_buf_size);
+typedef size_t(*census_context_encode_type)(const census_context *context, char *buffer, size_t buf_size);
 extern census_context_encode_type census_context_encode_import;
 #define census_context_encode census_context_encode_import
-typedef census_context *(*census_context_decode_type)(const char *buffer, size_t size, const char *bin_buffer, size_t bin_size);
+typedef census_context *(*census_context_decode_type)(const char *buffer, size_t size);
 extern census_context_decode_type census_context_decode_import;
 #define census_context_decode census_context_decode_import
 typedef int(*census_trace_mask_type)(const census_context *context);
@@ -610,6 +610,9 @@ extern gpr_ref_init_type gpr_ref_init_import;
 typedef void(*gpr_ref_type)(gpr_refcount *r);
 extern gpr_ref_type gpr_ref_import;
 #define gpr_ref gpr_ref_import
+typedef void(*gpr_ref_non_zero_type)(gpr_refcount *r);
+extern gpr_ref_non_zero_type gpr_ref_non_zero_import;
+#define gpr_ref_non_zero gpr_ref_non_zero_import
 typedef void(*gpr_refn_type)(gpr_refcount *r, int n);
 extern gpr_refn_type gpr_refn_import;
 #define gpr_refn gpr_refn_import

summerofcode/ideas.md

@@ -19,7 +19,9 @@ gRPC C Core:
 
 1. Port gRPC to  one of the major BSD platforms ([FreeBSD](https://freebsd.org), [NetBSD](https://netbsd.org), and [OpenBSD](https://openbsd.org)) and create packages for them. Add [kqueue](https://www.freebsd.org/cgi/man.cgi?query=kqueue) support in the process.
  * **Required skills:** C programming language, BSD operating system.
- * **Likely mentors:** [Craig Tiller](https://github.com/ctiller), [Nicolas Noble](https://github.com/nicolasnoble).
+ * **Likely mentors:** [Craig Tiller](https://github.com/ctiller),
+ [Nicolas Noble](https://github.com/nicolasnoble),
+ [Vijay Pai](https://github.com/vjpai).
 1. Fix gRPC C-core's URI parser. The current parser does not qualify as a standard parser according to [RFC3986]( https://tools.ietf.org/html/rfc3986). Write test suites to verify this and make changes necessary to make the URI parser compliant.
  * **Required skills:** C programming language, HTTP standard compliance.
  * **Likely mentors:** [Craig Tiller](https://github.com/ctiller).

templates/README.md

@@ -6,78 +6,92 @@ was going to single handedly cover all of our usage cases.
 
 So instead we decided to work the following way:
 
-* A build.json file at the root is the source of truth for listing all of the
-target and files needed to build grpc and its tests, as well as basic system
-dependencies description.
+* A `build.yaml` file at the root is the source of truth for listing all the
+targets and files needed to build grpc and its tests, as well as a basic system
+for dependency description.
 
 * Each project file (Makefile, Visual Studio project files, Bazel's BUILD) is
-a plain-text template that uses the build.json file to generate the final
-output file.
+a [YAML](http://yaml.org) file used by the `build.yaml` file to generate the
+final output file.
 
 This way we can maintain as many project system as we see fit, without having
 to manually maintain them when we add or remove new code to the repository.
 Only the structure of the project file is relevant to the template. The actual
 list of source code and targets isn't.
 
-We currently have template files for GNU Make, Visual Studio 2010 to 2015,
-and Bazel. In the future, we would like to expand to generating gyp or cmake
-project files (or potentially both), XCode project files, and an Android.mk
-file to be able to compile gRPC using Android's NDK.
+We currently have template files for GNU Make, Visual Studio 2013,
+[Bazel](http://bazel.io) and [gyp](https://gyp.gsrc.io/) (albeit only for
+Node.js). In the future, we
+would like to expand to also generate [cmake](https://cmake.org)
+project files, XCode project files, and an Android.mk file allowing to compile
+gRPC using Android's NDK.
 
 We'll gladly accept contribution that'd create additional project files
 using that system.
 
-# Structure of build.json
+# Structure of `build.yaml`
 
-The build.json file has the following structure:
+The `build.yaml` file has the following structure:
 
 ```
-{
-  "settings": { ... },   # global settings, such as version number
-  "filegroups": [ ... ], # groups of file that is automatically expanded
-  "libs": [ ... ],       # list of libraries to build
-  "targets": [ ... ],    # list of targets to build
-}
+settings:  # global settings, such as version number
+  ...
+filegroups:  # groups of files that are automatically expanded
+  ...
+libs:  # list of libraries to build
+  ...
+target:   # list of targets to build
+  ...
 ```
 
 The `filegroups` are helpful to re-use a subset of files in multiple targets.
 One `filegroups` entry has the following structure:
 
 ```
-{
-  "name": "arbitrary string", # the name of the filegroup
-  "public_headers": [ ... ],  # list of public headers defined in that filegroup
-  "headers": [ ... ],         # list of headers defined in that filegroup
-  "src": [ ... ],             # list of source files defined in that filegroup
-}
+- name: "arbitrary string", # the name of the filegroup
+  public_headers: # list of public headers defined in that filegroup
+  - ...
+  headers: # list of headers defined in that filegroup
+  - ...
+  src: # list of source files defined in that filegroup
+  - ...
 ```
 
-The `libs` array contains the list of all the libraries we describe. Some may be
+The `libs` collection contains the list of all the libraries we describe. Some may be
 helper libraries for the tests. Some may be installable libraries. Some may be
 helper libraries for installable binaries.
 
 The `targets` array contains the list of all the binary targets we describe. Some may
 be installable binaries.
 
-One `libs` or `targets` entry has the following structure:
+One `libs` or `targets` entry has the following structure (see below for
+details):
 
 ```
-{
-  "name": "arbitrary string", # the name of the library
-  "build": "build type",      # in which situation we want that library to be
-                              # built and potentially installed
-  "language": "...",          # the language tag; "c" or "c++"
-  "public_headers": [ ... ],  # list of public headers to install
-  "headers": [ ... ],         # list of headers used by that target
-  "src": [ ... ],             # list of files to compile
-  "secure": "...",            # "yes", "no" or "check"
-  "baselib": boolean,         # this is a low level library that has system
-                              # dependencies
-  "vs_project_guid: "...",    # Visual Studio's unique guid for that project
-  "filegroups": [ ... ],      # list of filegroups to merge to that project
-                              # note that this will be expanded automatically
-  "deps": [ ... ],            # list of libraries this target depends on
-}
+name: "arbitrary string", # the name of the library
+build: "build type",      # in which situation we want that library to be
+                          # built and potentially installed (see below).
+language: "...",          # the language tag; "c" or "c++"
+public_headers:           # list of public headers to install
+headers:                  # list of headers used by that target
+src:                      # list of files to compile
+secure: boolean,          # see below
+baselib: boolean,         # this is a low level library that has system
+                          # dependencies
+vs_project_guid: '{...}', # Visual Studio's unique guid for that project
+filegroups:               # list of filegroups to merge to that project
+                          # note that this will be expanded automatically
+deps:                     # list of libraries this target depends on
+deps_linkage: "..."       # "static"  or "dynamic". Used by the Makefile only to
+                          # determine the way dependencies are linkned. Defaults
+                          # to "dynamic".
+dll: "..."                # see below.
+dll_def: "..."            # Visual Studio's dll definition file.
+vs_props:                 # List of property sheets to attach to that project.
+vs_config_type: "..."     # DynamicLibrary/StaticLibrary. Used only when
+                          # creating a library. Specifies if we're building a
+                          # static library or a dll. Use in conjunction with `dll_def`.
+vs_packages:              # List of nuget packages this project depends on.
 ```
 
 ## The `"build"` tag
@@ -86,8 +100,9 @@ Currently, the "`build`" tag have these meanings:
 
 * `"all"`: library to build on `"make all"`, and install on the system.
 * `"protoc"`: a protoc plugin to build on `"make all"` and install on the system.
-* `"priviate"`: a library to only build for tests.
+* `"private"`: a library to only build for tests.
 * `"test"`: a test binary to run on `"make test"`.
+* `"tool"`: a binary to be built upon `"make tools"`.
 
 All of the targets should always be present in the generated project file, if
 possible and applicable. But the build tag is what should group the targets
@@ -111,6 +126,18 @@ should merge OpenSSL, protobuf or zlib inside that library. That effect depends
 on the `"language"` tag. OpenSSL and zlib are for `"c"` libraries, while
 protobuf is for `"c++"` ones.
 
+## The `"dll"` tag
+
+Used only by Visual Studio's project files. "true" means the project will be
+built with both static and dynamic runtimes. "false" means it'll only be built
+with static runtime. "only" means it'll only be built with the dll runtime.
+
+## The `"dll_def"` tag
+
+Specifies the visual studio's dll definition file. When creating a DLL, you
+sometimes (not always) need a def file (see grpc.def).
+
+
 # The template system
 
 We're currently using the [mako templates](http://www.makotemplates.org/)

templates/package.xml.template

@@ -12,11 +12,11 @@
     <email>grpc-packages@google.com</email>
     <active>yes</active>
    </lead>
-   <date>2016-02-24</date>
+   <date>2016-03-01</date>
    <time>16:06:07</time>
    <version>
-    <release>0.8.0</release>
-    <api>0.8.0</api>
+    <release>0.14.0</release>
+    <api>0.14.0</api>
    </version>
    <stability>
     <release>beta</release>
@@ -24,7 +24,7 @@
    </stability>
    <license>BSD</license>
    <notes>
-  - Simplify gRPC PHP installation #4517
+  - Increase unit test code coverage #5225
    </notes>
    <contents>
     <dir baseinstalldir="/" name="/">
@@ -155,5 +155,20 @@
   - Simplify gRPC PHP installation #4517
      </notes>
     </release>
+    <release>
+     <version>
+      <release>0.14.0</release>
+      <api>0.14.0</api>
+     </version>
+     <stability>
+      <release>beta</release>
+      <api>beta</api>
+     </stability>
+     <date>2016-03-01</date>
+     <license>BSD</license>
+     <notes>
+  - Increase unit test code coverage #5225
+     </notes>
+    </release>
    </changelog>
   </package>

templates/src/csharp/build_packages.bat.template

@@ -21,6 +21,14 @@
   xcopy /Y /I ..\..\architecture=x86,language=csharp,platform=macos\artifacts\* grpc.native.csharp\macosx_x86${"\\"}
   xcopy /Y /I ..\..\architecture=x64,language=csharp,platform=macos\artifacts\* grpc.native.csharp\macosx_x64${"\\"}
   
+  @rem Collect protoc artifacts built by the previous build step
+  xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=windows\artifacts\* protoc_plugins\windows_x86${"\\"}
+  xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=windows\artifacts\* protoc_plugins\windows_x64${"\\"}
+  xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=linux\artifacts\* protoc_plugins\linux_x86${"\\"}
+  xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=linux\artifacts\* protoc_plugins\linux_x64${"\\"}
+  xcopy /Y /I ..\..\architecture=x86,language=protoc,platform=macos\artifacts\* protoc_plugins\macosx_x86${"\\"}
+  xcopy /Y /I ..\..\architecture=x64,language=protoc,platform=macos\artifacts\* protoc_plugins\macosx_x64${"\\"}
+  
   @rem Fetch all dependencies
   %%NUGET% restore ..\..\vsprojects\grpc_csharp_ext.sln || goto :error
   %%NUGET% restore Grpc.sln || goto :error
@@ -29,24 +37,19 @@
   
   @call "%VS120COMNTOOLS%\..\..\vc\vcvarsall.bat" x86
   
-  @rem We won't use the native libraries from this step, but without this Grpc.sln will fail.  
+  @rem We won't use the native libraries from this step, but without this Grpc.sln will fail.
   msbuild ..\..\vsprojects\grpc_csharp_ext.sln /p:Configuration=Release /p:PlatformToolset=v120 || goto :error
   
   msbuild Grpc.sln /p:Configuration=ReleaseSigned || goto :error
   
   endlocal
   
-  @rem TODO(jtattermusch): re-enable protoc plugin building
-  @rem @call ..\..\vsprojects\build_plugins.bat || goto :error
-  
   %%NUGET% pack grpc.native.csharp\grpc.native.csharp.nuspec -Version %VERSION% || goto :error
   %%NUGET% pack Grpc.Auth\Grpc.Auth.nuspec -Symbols -Version %VERSION% || goto :error
   %%NUGET% pack Grpc.Core\Grpc.Core.nuspec -Symbols -Version %VERSION% || goto :error
   %%NUGET% pack Grpc.HealthCheck\Grpc.HealthCheck.nuspec -Symbols -Version %VERSION_WITH_BETA% -Properties ProtobufVersion=%PROTOBUF_VERSION% || goto :error
   %%NUGET% pack Grpc.nuspec -Version %VERSION% || goto :error
-  
-  @rem TODO(jtattermusch): re-enable building Grpc.Tools package
-  @rem %NUGET% pack Grpc.Tools.nuspec -Version %VERSION% || goto :error
+  %%NUGET% pack Grpc.Tools.nuspec -Version %VERSION% || goto :error
   
   @rem copy resulting nuget packages to artifacts directory
   xcopy /Y /I *.nupkg ..\..\artifacts${"\\"}

test/core/census/context_test.c

@@ -42,60 +42,48 @@
 #include <string.h>
 #include "test/core/util/test_config.h"
 
-static uint8_t one_byte_val = 7;
-static uint32_t four_byte_val = 0x12345678;
-static uint64_t eight_byte_val = 0x1234567890abcdef;
-
-// A set of tags Used to create a basic context for testing. Each tag has a
-// unique set of flags. Note that replace_add_delete_test() relies on specific
-// offsets into this array - if you add or delete entries, you will also need
-// to change the test.
+// A set of tags Used to create a basic context for testing. Note that
+// replace_add_delete_test() relies on specific offsets into this array - if
+// you add or delete entries, you will also need to change the test.
 #define BASIC_TAG_COUNT 8
 static census_tag basic_tags[BASIC_TAG_COUNT] = {
-    /* 0 */ {"key0", "printable", 10, 0},
-    /* 1 */ {"k1", "a", 2, CENSUS_TAG_PROPAGATE},
-    /* 2 */ {"k2", "longer printable string", 24, CENSUS_TAG_STATS},
-    /* 3 */ {"key_three", (char *)&one_byte_val, 1, CENSUS_TAG_BINARY},
-    /* 4 */ {"really_long_key_4", "random", 7,
+    /* 0 */ {"key0", "tag value", 0},
+    /* 1 */ {"k1", "a", CENSUS_TAG_PROPAGATE},
+    /* 2 */ {"k2", "a longer tag value supercalifragilisticexpialiadocious",
+             CENSUS_TAG_STATS},
+    /* 3 */ {"key_three", "", 0},
+    /* 4 */ {"a_really_really_really_really_long_key_4", "random",
              CENSUS_TAG_PROPAGATE | CENSUS_TAG_STATS},
-    /* 5 */ {"k5", (char *)&four_byte_val, 4,
-             CENSUS_TAG_PROPAGATE | CENSUS_TAG_BINARY},
-    /* 6 */ {"k6", (char *)&eight_byte_val, 8,
-             CENSUS_TAG_STATS | CENSUS_TAG_BINARY},
-    /* 7 */ {"k7", (char *)&four_byte_val, 4,
-             CENSUS_TAG_PROPAGATE | CENSUS_TAG_STATS | CENSUS_TAG_BINARY}};
+    /* 5 */ {"k5", "v5", CENSUS_TAG_PROPAGATE},
+    /* 6 */ {"k6", "v6", CENSUS_TAG_STATS},
+    /* 7 */ {"k7", "v7", CENSUS_TAG_PROPAGATE | CENSUS_TAG_STATS}};
 
 // Set of tags used to modify the basic context. Note that
 // replace_add_delete_test() relies on specific offsets into this array - if
 // you add or delete entries, you will also need to change the test. Other
 // tests that rely on specific instances have XXX_XXX_OFFSET definitions (also
 // change the defines below if you add/delete entires).
-#define MODIFY_TAG_COUNT 11
+#define MODIFY_TAG_COUNT 10
 static census_tag modify_tags[MODIFY_TAG_COUNT] = {
 #define REPLACE_VALUE_OFFSET 0
-    /* 0 */ {"key0", "replace printable", 18, 0},  // replaces tag value only
+    /* 0 */ {"key0", "replace key0", 0},  // replaces tag value only
 #define ADD_TAG_OFFSET 1
-    /* 1 */ {"new_key", "xyzzy", 6, CENSUS_TAG_STATS},  // new tag
+    /* 1 */ {"new_key", "xyzzy", CENSUS_TAG_STATS},  // new tag
 #define DELETE_TAG_OFFSET 2
-    /* 2 */ {"k5", NULL, 5,
-             0},  // should delete tag, despite bogus value length
-    /* 3 */ {"k6", "foo", 0, 0},  // should delete tag, despite bogus value
-    /* 4 */ {"k6", "foo", 0, 0},  // try deleting already-deleted tag
-    /* 5 */ {"non-existent", NULL, 0, 0},  // another non-existent tag
-#define REPLACE_FLAG_OFFSET 6
-    /* 6 */ {"k1", "a", 2, 0},                   // change flags only
-    /* 7 */ {"k7", "bar", 4, CENSUS_TAG_STATS},  // change flags and value
-    /* 8 */ {"k2", (char *)&eight_byte_val, 8,
-             CENSUS_TAG_BINARY | CENSUS_TAG_PROPAGATE},  // more flags change
-                                                         // non-binary -> binary
-    /* 9 */ {"k6", "bar", 4, 0},  // add back tag, with different value
-    /* 10 */ {"foo", "bar", 4, CENSUS_TAG_PROPAGATE},  // another new tag
+    /* 2 */ {"k5", NULL, 0},            // should delete tag
+    /* 3 */ {"k5", NULL, 0},            // try deleting already-deleted tag
+    /* 4 */ {"non-existent", NULL, 0},  // delete non-existent tag
+#define REPLACE_FLAG_OFFSET 5
+    /* 5 */ {"k1", "a", 0},                    // change flags only
+    /* 6 */ {"k7", "bar", CENSUS_TAG_STATS},   // change flags and value
+    /* 7 */ {"k2", "", CENSUS_TAG_PROPAGATE},  // more value and flags change
+    /* 8 */ {"k5", "bar", 0},  // add back tag, with different value
+    /* 9 */ {"foo", "bar", CENSUS_TAG_PROPAGATE},  // another new tag
 };
 
 // Utility function to compare tags. Returns true if all fields match.
 static bool compare_tag(const census_tag *t1, const census_tag *t2) {
-  return (strcmp(t1->key, t2->key) == 0 && t1->value_len == t2->value_len &&
-          memcmp(t1->value, t2->value, t1->value_len) == 0 &&
+  return (strcmp(t1->key, t2->key) == 0 && strcmp(t1->value, t2->value) == 0 &&
           t1->flags == t2->flags);
 }
 
@@ -111,7 +99,7 @@ static void empty_test(void) {
   struct census_context *context = census_context_create(NULL, NULL, 0, NULL);
   GPR_ASSERT(context != NULL);
   const census_context_status *status = census_context_get_status(context);
-  census_context_status expected = {0, 0, 0, 0, 0, 0, 0, 0};
+  census_context_status expected = {0, 0, 0, 0, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_destroy(context);
 }
@@ -121,7 +109,7 @@ static void basic_test(void) {
   const census_context_status *status;
   struct census_context *context =
       census_context_create(NULL, basic_tags, BASIC_TAG_COUNT, &status);
-  census_context_status expected = {2, 2, 4, 0, 8, 0, 0, 0};
+  census_context_status expected = {4, 4, 0, 8, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_iterator it;
   census_context_initialize_iterator(context, &it);
@@ -161,15 +149,18 @@ static void invalid_test(void) {
   memset(key, 'k', 299);
   key[299] = 0;
   char value[300];
-  memset(value, 'v', 300);
-  census_tag tag = {key, value, 3, CENSUS_TAG_BINARY};
+  memset(value, 'v', 299);
+  value[299] = 0;
+  census_tag tag = {key, value, 0};
   // long keys, short value. Key lengths (including terminator) should be
   // <= 255 (CENSUS_MAX_TAG_KV_LEN)
+  value[3] = 0;
+  GPR_ASSERT(strlen(value) == 3);
   GPR_ASSERT(strlen(key) == 299);
   const census_context_status *status;
   struct census_context *context =
       census_context_create(NULL, &tag, 1, &status);
-  census_context_status expected = {0, 0, 0, 0, 0, 0, 1, 0};
+  census_context_status expected = {0, 0, 0, 0, 0, 1, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_destroy(context);
   key[CENSUS_MAX_TAG_KV_LEN] = 0;
@@ -180,24 +171,44 @@ static void invalid_test(void) {
   key[CENSUS_MAX_TAG_KV_LEN - 1] = 0;
   GPR_ASSERT(strlen(key) == CENSUS_MAX_TAG_KV_LEN - 1);
   context = census_context_create(NULL, &tag, 1, &status);
-  census_context_status expected2 = {0, 0, 1, 0, 1, 0, 0, 0};
+  census_context_status expected2 = {0, 1, 0, 1, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected2, sizeof(expected2)) == 0);
   census_context_destroy(context);
   // now try with long values
-  tag.value_len = 300;
+  value[3] = 'v';
+  GPR_ASSERT(strlen(value) == 299);
   context = census_context_create(NULL, &tag, 1, &status);
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_destroy(context);
-  tag.value_len = CENSUS_MAX_TAG_KV_LEN + 1;
+  value[CENSUS_MAX_TAG_KV_LEN] = 0;
+  GPR_ASSERT(strlen(value) == CENSUS_MAX_TAG_KV_LEN);
   context = census_context_create(NULL, &tag, 1, &status);
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_destroy(context);
-  tag.value_len = CENSUS_MAX_TAG_KV_LEN;
+  value[CENSUS_MAX_TAG_KV_LEN - 1] = 0;
+  GPR_ASSERT(strlen(value) == CENSUS_MAX_TAG_KV_LEN - 1);
   context = census_context_create(NULL, &tag, 1, &status);
   GPR_ASSERT(memcmp(status, &expected2, sizeof(expected2)) == 0);
   census_context_destroy(context);
   // 0 length key.
   key[0] = 0;
+  GPR_ASSERT(strlen(key) == 0);
+  context = census_context_create(NULL, &tag, 1, &status);
+  GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
+  census_context_destroy(context);
+  // invalid key character
+  key[0] = 31;  // 32 (' ') is the first valid character value
+  key[1] = 0;
+  GPR_ASSERT(strlen(key) == 1);
+  context = census_context_create(NULL, &tag, 1, &status);
+  GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
+  census_context_destroy(context);
+  // invalid value character
+  key[0] = ' ';
+  value[5] = 127;  // 127 (DEL) is ('~' + 1)
+  value[8] = 0;
+  GPR_ASSERT(strlen(key) == 1);
+  GPR_ASSERT(strlen(value) == 8);
   context = census_context_create(NULL, &tag, 1, &status);
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_context_destroy(context);
@@ -210,7 +221,7 @@ static void copy_test(void) {
   const census_context_status *status;
   struct census_context *context2 =
       census_context_create(context, NULL, 0, &status);
-  census_context_status expected = {2, 2, 4, 0, 0, 0, 0, 0};
+  census_context_status expected = {4, 4, 0, 0, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   for (int i = 0; i < BASIC_TAG_COUNT; i++) {
     census_tag tag;
@@ -228,7 +239,7 @@ static void replace_value_test(void) {
   const census_context_status *status;
   struct census_context *context2 = census_context_create(
       context, modify_tags + REPLACE_VALUE_OFFSET, 1, &status);
-  census_context_status expected = {2, 2, 4, 0, 0, 1, 0, 0};
+  census_context_status expected = {4, 4, 0, 0, 1, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_tag tag;
   GPR_ASSERT(census_context_get_tag(
@@ -245,7 +256,7 @@ static void replace_flags_test(void) {
   const census_context_status *status;
   struct census_context *context2 = census_context_create(
       context, modify_tags + REPLACE_FLAG_OFFSET, 1, &status);
-  census_context_status expected = {1, 2, 5, 0, 0, 1, 0, 0};
+  census_context_status expected = {3, 5, 0, 0, 1, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_tag tag;
   GPR_ASSERT(census_context_get_tag(
@@ -262,7 +273,7 @@ static void delete_tag_test(void) {
   const census_context_status *status;
   struct census_context *context2 = census_context_create(
       context, modify_tags + DELETE_TAG_OFFSET, 1, &status);
-  census_context_status expected = {2, 1, 4, 1, 0, 0, 0, 0};
+  census_context_status expected = {3, 4, 1, 0, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_tag tag;
   GPR_ASSERT(census_context_get_tag(
@@ -278,7 +289,7 @@ static void add_tag_test(void) {
   const census_context_status *status;
   struct census_context *context2 =
       census_context_create(context, modify_tags + ADD_TAG_OFFSET, 1, &status);
-  census_context_status expected = {2, 2, 5, 0, 1, 0, 0, 0};
+  census_context_status expected = {4, 5, 0, 1, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   census_tag tag;
   GPR_ASSERT(census_context_get_tag(context2, modify_tags[ADD_TAG_OFFSET].key,
@@ -295,24 +306,24 @@ static void replace_add_delete_test(void) {
   const census_context_status *status;
   struct census_context *context2 =
       census_context_create(context, modify_tags, MODIFY_TAG_COUNT, &status);
-  census_context_status expected = {2, 1, 6, 2, 3, 4, 0, 2};
+  census_context_status expected = {3, 7, 1, 3, 4, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   // validate context contents. Use specific indices into the two arrays
   // holding tag values.
   GPR_ASSERT(validate_tag(context2, &basic_tags[3]));
   GPR_ASSERT(validate_tag(context2, &basic_tags[4]));
+  GPR_ASSERT(validate_tag(context2, &basic_tags[6]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[0]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[1]));
+  GPR_ASSERT(validate_tag(context2, &modify_tags[5]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[6]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[7]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[8]));
   GPR_ASSERT(validate_tag(context2, &modify_tags[9]));
-  GPR_ASSERT(validate_tag(context2, &modify_tags[10]));
   GPR_ASSERT(!validate_tag(context2, &basic_tags[0]));
   GPR_ASSERT(!validate_tag(context2, &basic_tags[1]));
   GPR_ASSERT(!validate_tag(context2, &basic_tags[2]));
   GPR_ASSERT(!validate_tag(context2, &basic_tags[5]));
-  GPR_ASSERT(!validate_tag(context2, &basic_tags[6]));
   GPR_ASSERT(!validate_tag(context2, &basic_tags[7]));
   census_context_destroy(context);
   census_context_destroy(context2);
@@ -325,21 +336,15 @@ static void encode_decode_test(void) {
   char buffer[BUF_SIZE];
   struct census_context *context =
       census_context_create(NULL, basic_tags, BASIC_TAG_COUNT, NULL);
-  size_t print_bsize;
-  size_t bin_bsize;
   // Test with too small a buffer
-  GPR_ASSERT(census_context_encode(context, buffer, 2, &print_bsize,
-                                   &bin_bsize) == NULL);
-  char *b_buffer = census_context_encode(context, buffer, BUF_SIZE,
-                                         &print_bsize, &bin_bsize);
-  GPR_ASSERT(b_buffer != NULL && print_bsize > 0 && bin_bsize > 0 &&
-             print_bsize + bin_bsize <= BUF_SIZE &&
-             b_buffer == buffer + print_bsize);
-  census_context *context2 =
-      census_context_decode(buffer, print_bsize, b_buffer, bin_bsize);
+  GPR_ASSERT(census_context_encode(context, buffer, 2) == 0);
+  // Test with sufficient buffer
+  size_t buf_used = census_context_encode(context, buffer, BUF_SIZE);
+  GPR_ASSERT(buf_used != 0);
+  census_context *context2 = census_context_decode(buffer, buf_used);
   GPR_ASSERT(context2 != NULL);
   const census_context_status *status = census_context_get_status(context2);
-  census_context_status expected = {2, 2, 0, 0, 0, 0, 0, 0};
+  census_context_status expected = {4, 0, 0, 0, 0, 0, 0};
   GPR_ASSERT(memcmp(status, &expected, sizeof(expected)) == 0);
   for (int i = 0; i < BASIC_TAG_COUNT; i++) {
     census_tag tag;

test/core/iomgr/tcp_client_posix_test.c

@@ -179,98 +179,6 @@ void test_fails(void) {
   grpc_exec_ctx_finish(&exec_ctx);
 }
 
-void test_times_out(void) {
-  struct sockaddr_in addr;
-  socklen_t addr_len = sizeof(addr);
-  int svr_fd;
-#define NUM_CLIENT_CONNECTS 100
-  int client_fd[NUM_CLIENT_CONNECTS];
-  int i;
-  int r;
-  int connections_complete_before;
-  gpr_timespec connect_deadline;
-  grpc_closure done;
-  grpc_exec_ctx exec_ctx = GRPC_EXEC_CTX_INIT;
-
-  gpr_log(GPR_DEBUG, "test_times_out");
-
-  memset(&addr, 0, sizeof(addr));
-  addr.sin_family = AF_INET;
-
-  /* create a dummy server */
-  svr_fd = socket(AF_INET, SOCK_STREAM, 0);
-  GPR_ASSERT(svr_fd >= 0);
-  GPR_ASSERT(0 == bind(svr_fd, (struct sockaddr *)&addr, addr_len));
-  GPR_ASSERT(0 == listen(svr_fd, 1));
-  /* Get its address */
-  GPR_ASSERT(getsockname(svr_fd, (struct sockaddr *)&addr, &addr_len) == 0);
-
-  /* tie up the listen buffer, which is somewhat arbitrarily sized. */
-  for (i = 0; i < NUM_CLIENT_CONNECTS; ++i) {
-    client_fd[i] = socket(AF_INET, SOCK_STREAM, 0);
-    grpc_set_socket_nonblocking(client_fd[i], 1);
-    do {
-      r = connect(client_fd[i], (struct sockaddr *)&addr, addr_len);
-    } while (r == -1 && errno == EINTR);
-    GPR_ASSERT(r < 0);
-    GPR_ASSERT(errno == EWOULDBLOCK || errno == EINPROGRESS);
-  }
-
-  /* connect to dummy server address */
-
-  connect_deadline = GRPC_TIMEOUT_SECONDS_TO_DEADLINE(1);
-
-  gpr_mu_lock(g_mu);
-  connections_complete_before = g_connections_complete;
-  gpr_mu_unlock(g_mu);
-
-  grpc_closure_init(&done, must_fail, NULL);
-  grpc_tcp_client_connect(&exec_ctx, &done, &g_connecting, g_pollset_set,
-                          (struct sockaddr *)&addr, addr_len, connect_deadline);
-
-  /* Make sure the event doesn't trigger early */
-  gpr_mu_lock(g_mu);
-  for (;;) {
-    grpc_pollset_worker *worker = NULL;
-    gpr_timespec now = gpr_now(connect_deadline.clock_type);
-    gpr_timespec continue_verifying_time =
-        gpr_time_from_seconds(5, GPR_TIMESPAN);
-    gpr_timespec grace_time = gpr_time_from_seconds(3, GPR_TIMESPAN);
-    gpr_timespec finish_time =
-        gpr_time_add(connect_deadline, continue_verifying_time);
-    gpr_timespec restart_verifying_time =
-        gpr_time_add(connect_deadline, grace_time);
-    int is_after_deadline = gpr_time_cmp(now, connect_deadline) > 0;
-    if (gpr_time_cmp(now, finish_time) > 0) {
-      break;
-    }
-    gpr_log(GPR_DEBUG, "now=%lld.%09d connect_deadline=%lld.%09d",
-            (long long)now.tv_sec, (int)now.tv_nsec,
-            (long long)connect_deadline.tv_sec, (int)connect_deadline.tv_nsec);
-    if (is_after_deadline && gpr_time_cmp(now, restart_verifying_time) <= 0) {
-      /* allow some slack before insisting that things be done */
-    } else {
-      GPR_ASSERT(g_connections_complete ==
-                 connections_complete_before + is_after_deadline);
-    }
-    gpr_timespec polling_deadline = GRPC_TIMEOUT_MILLIS_TO_DEADLINE(10);
-    if (!grpc_timer_check(&exec_ctx, now, &polling_deadline)) {
-      grpc_pollset_work(&exec_ctx, g_pollset, &worker, now, polling_deadline);
-    }
-    gpr_mu_unlock(g_mu);
-    grpc_exec_ctx_flush(&exec_ctx);
-    gpr_mu_lock(g_mu);
-  }
-  gpr_mu_unlock(g_mu);
-
-  grpc_exec_ctx_finish(&exec_ctx);
-
-  close(svr_fd);
-  for (i = 0; i < NUM_CLIENT_CONNECTS; ++i) {
-    close(client_fd[i]);
-  }
-}
-
 static void destroy_pollset(grpc_exec_ctx *exec_ctx, void *p, bool success) {
   grpc_pollset_destroy(p);
 }
@@ -288,7 +196,6 @@ int main(int argc, char **argv) {
   test_succeeds();
   gpr_log(GPR_ERROR, "End of first test");
   test_fails();
-  test_times_out();
   grpc_pollset_set_destroy(g_pollset_set);
   grpc_closure_init(&destroyed, destroy_pollset, g_pollset);
   grpc_pollset_shutdown(&exec_ctx, g_pollset, &destroyed);

test/cpp/end2end/end2end_test.cc

@@ -437,9 +437,10 @@ class End2endServerTryCancelTest : public End2endTest {
         break;
 
       case CANCEL_AFTER_PROCESSING:
-        // Server cancelled after writing all messages. Client must have read
-        // all messages
-        EXPECT_EQ(num_msgs_read, kNumResponseStreamsMsgs);
+        // Even though the Server cancelled after writing all messages, the RPC
+        // may be cancelled before the Client got a chance to read all the
+        // messages.
+        EXPECT_LE(num_msgs_read, kNumResponseStreamsMsgs);
         break;
 
       default: {
@@ -519,7 +520,11 @@ class End2endServerTryCancelTest : public End2endTest {
 
       case CANCEL_AFTER_PROCESSING:
         EXPECT_EQ(num_msgs_sent, num_messages);
-        EXPECT_EQ(num_msgs_read, num_msgs_sent);
+
+        // The Server cancelled after reading the last message and after writing
+        // the message to the client. However, the RPC cancellation might have
+        // taken effect before the client actually read the response.
+        EXPECT_LE(num_msgs_read, num_msgs_sent);
         break;
 
       default:

test/cpp/end2end/test_service_impl.cc

@@ -326,7 +326,11 @@ void TestServiceImpl::ServerTryCancel(ServerContext* context) {
   EXPECT_FALSE(context->IsCancelled());
   context->TryCancel();
   gpr_log(GPR_INFO, "Server called TryCancel() to cancel the request");
-  EXPECT_TRUE(context->IsCancelled());
+  // Now wait until it's really canceled
+  while (!context->IsCancelled()) {
+    gpr_sleep_until(gpr_time_add(gpr_now(GPR_CLOCK_REALTIME),
+                                 gpr_time_from_micros(1000, GPR_TIMESPAN)));
+  }
 }
 
 }  // namespace testing

test/cpp/interop/metrics_client.cc

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -37,39 +37,45 @@
 #include <gflags/gflags.h>
 #include <grpc++/grpc++.h>
 
-#include "test/cpp/util/metrics_server.h"
-#include "test/cpp/util/test_config.h"
 #include "src/proto/grpc/testing/metrics.grpc.pb.h"
 #include "src/proto/grpc/testing/metrics.pb.h"
+#include "test/cpp/util/metrics_server.h"
+#include "test/cpp/util/test_config.h"
 
 DEFINE_string(metrics_server_address, "",
               "The metrics server addresses in the fomrat <hostname>:<port>");
+DEFINE_bool(total_only, false,
+            "If true, this prints only the total value of all gauges");
+
+int kDeadlineSecs = 10;
 
 using grpc::testing::EmptyMessage;
 using grpc::testing::GaugeResponse;
 using grpc::testing::MetricsService;
 using grpc::testing::MetricsServiceImpl;
 
-void PrintMetrics(const grpc::string& server_address) {
-  gpr_log(GPR_INFO, "creating a channel to %s", server_address.c_str());
-  std::shared_ptr<grpc::Channel> channel(
-      grpc::CreateChannel(server_address, grpc::InsecureChannelCredentials()));
-
-  std::unique_ptr<MetricsService::Stub> stub(MetricsService::NewStub(channel));
-
+// Prints the values of all Gauges (unless total_only is set to 'true' in which
+// case this only prints the sum of all gauge values).
+bool PrintMetrics(std::unique_ptr<MetricsService::Stub> stub, bool total_only) {
   grpc::ClientContext context;
   EmptyMessage message;
 
+  std::chrono::system_clock::time_point deadline =
+      std::chrono::system_clock::now() + std::chrono::seconds(kDeadlineSecs);
+
+  context.set_deadline(deadline);
+
   std::unique_ptr<grpc::ClientReader<GaugeResponse>> reader(
       stub->GetAllGauges(&context, message));
 
   GaugeResponse gauge_response;
   long overall_qps = 0;
-  int idx = 0;
   while (reader->Read(&gauge_response)) {
     if (gauge_response.value_case() == GaugeResponse::kLongValue) {
-      gpr_log(GPR_INFO, "Gauge: %d (%s: %ld)", ++idx,
-              gauge_response.name().c_str(), gauge_response.long_value());
+      if (!total_only) {
+        gpr_log(GPR_INFO, "%s: %ld", gauge_response.name().c_str(),
+                gauge_response.long_value());
+      }
       overall_qps += gauge_response.long_value();
     } else {
       gpr_log(GPR_INFO, "Gauge %s is not a long value",
@@ -77,12 +83,14 @@ void PrintMetrics(const grpc::string& server_address) {
     }
   }
 
-  gpr_log(GPR_INFO, "OVERALL: %ld", overall_qps);
+  gpr_log(GPR_INFO, "%ld", overall_qps);
 
   const grpc::Status status = reader->Finish();
   if (!status.ok()) {
     gpr_log(GPR_ERROR, "Error in getting metrics from the client");
   }
+
+  return status.ok();
 }
 
 int main(int argc, char** argv) {
@@ -97,7 +105,12 @@ int main(int argc, char** argv) {
     return 1;
   }
 
-  PrintMetrics(FLAGS_metrics_server_address);
+  std::shared_ptr<grpc::Channel> channel(grpc::CreateChannel(
+      FLAGS_metrics_server_address, grpc::InsecureChannelCredentials()));
+
+  if (!PrintMetrics(MetricsService::NewStub(channel), FLAGS_total_only)) {
+    return 1;
+  }
 
   return 0;
 }

test/cpp/interop/reconnect_interop_client.cc

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

test/cpp/util/metrics_server.cc

@@ -57,7 +57,7 @@ long Gauge::Get() {
 grpc::Status MetricsServiceImpl::GetAllGauges(
     ServerContext* context, const EmptyMessage* request,
     ServerWriter<GaugeResponse>* writer) {
-  gpr_log(GPR_INFO, "GetAllGauges called");
+  gpr_log(GPR_DEBUG, "GetAllGauges called");
 
   std::lock_guard<std::mutex> lock(mu_);
   for (auto it = gauges_.begin(); it != gauges_.end(); it++) {

test/cpp/util/test_credentials_provider.cc

@@ -34,6 +34,8 @@
 
 #include "test/cpp/util/test_credentials_provider.h"
 
+#include <unordered_map>
+
 #include <grpc/support/sync.h>
 #include <grpc++/impl/sync.h>
 
@@ -48,12 +50,36 @@ using grpc::InsecureServerCredentials;
 using grpc::ServerCredentials;
 using grpc::SslCredentialsOptions;
 using grpc::SslServerCredentialsOptions;
-using grpc::testing::CredentialsProvider;
+using grpc::testing::CredentialTypeProvider;
+
+// Provide test credentials. Thread-safe.
+class CredentialsProvider {
+ public:
+  virtual ~CredentialsProvider() {}
+
+  virtual void AddSecureType(
+      const grpc::string& type,
+      std::unique_ptr<CredentialTypeProvider> type_provider) = 0;
+  virtual std::shared_ptr<ChannelCredentials> GetChannelCredentials(
+      const grpc::string& type, ChannelArguments* args) = 0;
+  virtual std::shared_ptr<ServerCredentials> GetServerCredentials(
+      const grpc::string& type) = 0;
+  virtual std::vector<grpc::string> GetSecureCredentialsTypeList() = 0;
+};
 
 class DefaultCredentialsProvider : public CredentialsProvider {
  public:
   ~DefaultCredentialsProvider() override {}
 
+  void AddSecureType(
+      const grpc::string& type,
+      std::unique_ptr<CredentialTypeProvider> type_provider) override {
+    // This clobbers any existing entry for type, except the defaults, which
+    // can't be clobbered.
+    grpc::unique_lock<grpc::mutex> lock(mu_);
+    added_secure_types_[type] = std::move(type_provider);
+  }
+
   std::shared_ptr<ChannelCredentials> GetChannelCredentials(
       const grpc::string& type, ChannelArguments* args) override {
     if (type == grpc::testing::kInsecureCredentialsType) {
@@ -63,9 +89,14 @@ class DefaultCredentialsProvider : public CredentialsProvider {
       args->SetSslTargetNameOverride("foo.test.google.fr");
       return SslCredentials(ssl_opts);
     } else {
-      gpr_log(GPR_ERROR, "Unsupported credentials type %s.", type.c_str());
+      grpc::unique_lock<grpc::mutex> lock(mu_);
+      auto it(added_secure_types_.find(type));
+      if (it == added_secure_types_.end()) {
+        gpr_log(GPR_ERROR, "Unsupported credentials type %s.", type.c_str());
+        return nullptr;
+      }
+      return it->second->GetChannelCredentials(args);
     }
-    return nullptr;
   }
 
   std::shared_ptr<ServerCredentials> GetServerCredentials(
@@ -80,33 +111,38 @@ class DefaultCredentialsProvider : public CredentialsProvider {
       ssl_opts.pem_key_cert_pairs.push_back(pkcp);
       return SslServerCredentials(ssl_opts);
     } else {
-      gpr_log(GPR_ERROR, "Unsupported credentials type %s.", type.c_str());
+      grpc::unique_lock<grpc::mutex> lock(mu_);
+      auto it(added_secure_types_.find(type));
+      if (it == added_secure_types_.end()) {
+        gpr_log(GPR_ERROR, "Unsupported credentials type %s.", type.c_str());
+        return nullptr;
+      }
+      return it->second->GetServerCredentials();
     }
-    return nullptr;
   }
   std::vector<grpc::string> GetSecureCredentialsTypeList() override {
     std::vector<grpc::string> types;
     types.push_back(grpc::testing::kTlsCredentialsType);
+    grpc::unique_lock<grpc::mutex> lock(mu_);
+    for (const auto& type_pair : added_secure_types_) {
+      types.push_back(type_pair.first);
+    }
     return types;
   }
+
+ private:
+  grpc::mutex mu_;
+  std::unordered_map<grpc::string, std::unique_ptr<CredentialTypeProvider> >
+      added_secure_types_;
 };
 
-gpr_once g_once_init_provider_mu = GPR_ONCE_INIT;
-grpc::mutex* g_provider_mu = nullptr;
+gpr_once g_once_init_provider = GPR_ONCE_INIT;
 CredentialsProvider* g_provider = nullptr;
 
-void InitProviderMu() { g_provider_mu = new grpc::mutex; }
-
-grpc::mutex& GetMu() {
-  gpr_once_init(&g_once_init_provider_mu, &InitProviderMu);
-  return *g_provider_mu;
-}
+void CreateDefaultProvider() { g_provider = new DefaultCredentialsProvider; }
 
 CredentialsProvider* GetProvider() {
-  grpc::unique_lock<grpc::mutex> lock(GetMu());
-  if (g_provider == nullptr) {
-    g_provider = new DefaultCredentialsProvider;
-  }
+  gpr_once_init(&g_once_init_provider, &CreateDefaultProvider);
   return g_provider;
 }
 
@@ -115,15 +151,9 @@ CredentialsProvider* GetProvider() {
 namespace grpc {
 namespace testing {
 
-// Note that it is not thread-safe to set a provider while concurrently using
-// the previously set provider, as this deletes and replaces it. nullptr may be
-// given to reset to the default.
-void SetTestCredentialsProvider(std::unique_ptr<CredentialsProvider> provider) {
-  grpc::unique_lock<grpc::mutex> lock(GetMu());
-  if (g_provider != nullptr) {
-    delete g_provider;
-  }
-  g_provider = provider.release();
+void AddSecureType(const grpc::string& type,
+                   std::unique_ptr<CredentialTypeProvider> type_provider) {
+  GetProvider()->AddSecureType(type, std::move(type_provider));
 }
 
 std::shared_ptr<ChannelCredentials> GetChannelCredentials(

test/cpp/util/test_credentials_provider.h

@@ -46,20 +46,21 @@ namespace testing {
 const char kInsecureCredentialsType[] = "INSECURE_CREDENTIALS";
 const char kTlsCredentialsType[] = "TLS_CREDENTIALS";
 
-class CredentialsProvider {
+// Provide test credentials of a particular type.
+class CredentialTypeProvider {
  public:
-  virtual ~CredentialsProvider() {}
+  virtual ~CredentialTypeProvider() {}
 
   virtual std::shared_ptr<ChannelCredentials> GetChannelCredentials(
-      const grpc::string& type, ChannelArguments* args) = 0;
-  virtual std::shared_ptr<ServerCredentials> GetServerCredentials(
-      const grpc::string& type) = 0;
-  virtual std::vector<grpc::string> GetSecureCredentialsTypeList() = 0;
+      ChannelArguments* args) = 0;
+  virtual std::shared_ptr<ServerCredentials> GetServerCredentials() = 0;
 };
 
-// Set the CredentialsProvider used by the other functions in this file. If this
-// is not set, a default provider will be used.
-void SetTestCredentialsProvider(std::unique_ptr<CredentialsProvider> provider);
+// Add a secure type in addition to the defaults above
+// (kInsecureCredentialsType, kTlsCredentialsType) that can be returned from the
+// functions below.
+void AddSecureType(const grpc::string& type,
+                   std::unique_ptr<CredentialTypeProvider> type_provider);
 
 // Provide channel credentials according to the given type. Alter the channel
 // arguments if needed.

test/distrib/csharp/DistribTest.sln

@@ -8,13 +8,19 @@ EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
+		Debug|x64 = Debug|x64
 		Release|Any CPU = Release|Any CPU
+		Release|x64 = Release|x64
 	EndGlobalSection
 	GlobalSection(ProjectConfigurationPlatforms) = postSolution
 		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
 		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Debug|x64.ActiveCfg = Debug|x64
+		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Debug|x64.Build.0 = Debug|x64
 		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Release|Any CPU.Build.0 = Release|Any CPU
+		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Release|x64.ActiveCfg = Release|x64
+		{A3E61CC3-3710-49A3-A830-A0066EDBCE2F}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE

test/distrib/csharp/DistribTest/DistribTest.csproj

@@ -32,6 +32,26 @@
     <ErrorReport>prompt</ErrorReport>
     <WarningLevel>4</WarningLevel>
   </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Debug|x64'">
+    <DebugSymbols>true</DebugSymbols>
+    <OutputPath>bin\x64\Debug\</OutputPath>
+    <DefineConstants>DEBUG;TRACE</DefineConstants>
+    <DebugType>full</DebugType>
+    <PlatformTarget>x64</PlatformTarget>
+    <ErrorReport>prompt</ErrorReport>
+    <CodeAnalysisRuleSet>MinimumRecommendedRules.ruleset</CodeAnalysisRuleSet>
+    <Prefer32Bit>true</Prefer32Bit>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x64'">
+    <OutputPath>bin\x64\Release\</OutputPath>
+    <DefineConstants>TRACE</DefineConstants>
+    <Optimize>true</Optimize>
+    <DebugType>pdbonly</DebugType>
+    <PlatformTarget>x64</PlatformTarget>
+    <ErrorReport>prompt</ErrorReport>
+    <CodeAnalysisRuleSet>MinimumRecommendedRules.ruleset</CodeAnalysisRuleSet>
+    <Prefer32Bit>true</Prefer32Bit>
+  </PropertyGroup>
   <ItemGroup>
     <Reference Include="BouncyCastle.Crypto">
       <HintPath>..\packages\BouncyCastle.1.7.0\lib\Net40-Client\BouncyCastle.Crypto.dll</HintPath>

test/distrib/csharp/run_distrib_test.bat

@@ -0,0 +1,49 @@
+@rem Copyright 2016, Google Inc.
+@rem All rights reserved.
+@rem
+@rem Redistribution and use in source and binary forms, with or without
+@rem modification, are permitted provided that the following conditions are
+@rem met:
+@rem
+@rem     * Redistributions of source code must retain the above copyright
+@rem notice, this list of conditions and the following disclaimer.
+@rem     * Redistributions in binary form must reproduce the above
+@rem copyright notice, this list of conditions and the following disclaimer
+@rem in the documentation and/or other materials provided with the
+@rem distribution.
+@rem     * Neither the name of Google Inc. nor the names of its
+@rem contributors may be used to endorse or promote products derived from
+@rem this software without specific prior written permission.
+@rem
+@rem THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+@rem "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+@rem LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+@rem A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+@rem OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+@rem SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+@rem LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+@rem DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+@rem THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+@rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+@rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+@rem enter this directory
+cd /d %~dp0
+
+@rem extract input artifacts
+powershell -Command "Add-Type -Assembly 'System.IO.Compression.FileSystem'; [System.IO.Compression.ZipFile]::ExtractToDirectory('../../../input_artifacts/csharp_nugets.zip', 'TestNugetFeed');"
+
+update_version.sh auto
+
+set NUGET=C:\nuget\nuget.exe
+%NUGET% restore || goto :error
+
+@call build_vs2015.bat DistribTest.sln %MSBUILD_EXTRA_ARGS% || goto :error
+
+%DISTRIBTEST_OUTPATH%\DistribTest.exe || goto :error
+
+goto :EOF
+
+:error
+echo Failed!
+exit /b %errorlevel%

test/distrib/csharp/run_distrib_test.sh

@@ -34,9 +34,7 @@ cd $(dirname $0)
 
 unzip -o "$EXTERNAL_GIT_ROOT/input_artifacts/csharp_nugets.zip" -d TestNugetFeed
 
-# Extract the version number from Grpc nuget package name.
-CSHARP_VERSION=$(ls TestNugetFeed | grep '^Grpc\.[0-9].*\.nupkg$' | sed s/^Grpc\.// | sed s/\.nupkg$//)
-./update_version.sh $CSHARP_VERSION
+./update_version.sh auto
 
 nuget restore
 

test/distrib/csharp/update_version.sh

@@ -32,5 +32,13 @@ set -e
 
 cd $(dirname $0)
 
+CSHARP_VERSION="$1"
+if [ "$CSHARP_VERSION" == "auto" ]
+then
+  # autodetect C# version
+  CSHARP_VERSION=$(ls TestNugetFeed | grep '^Grpc\.[0-9].*\.nupkg$' | sed s/^Grpc\.// | sed s/\.nupkg$//)
+  echo "Autodetected nuget ${CSHARP_VERSION}"
+fi
+
 # Replaces version placeholder with value provided as first argument.
-sed -ibak "s/__GRPC_NUGET_VERSION__/$1/g" DistribTest/packages.config DistribTest/DistribTest.csproj
+sed -ibak "s/__GRPC_NUGET_VERSION__/${CSHARP_VERSION}/g" DistribTest/packages.config DistribTest/DistribTest.csproj

tools/README.md

@@ -1,7 +1,5 @@
 buildgen: template renderer for our build system.
 
-distpackages: script to generate debian packages.
-
 distrib: scripts to distribute language-specific packages.
 
 dockerfile: Docker files to test gRPC.
@@ -12,6 +10,4 @@ gce: scripts to help setup testing infrastructure on GCE.
 
 jenkins: support for running tests on Jenkins.
 
-profile_analyzer: pretty printer for gRPC profiling data.
-
 run_tests: scripts to run gRPC tests in parallel.

tools/dockerfile/grpc_interop_stress_cxx/Dockerfile

@@ -59,6 +59,8 @@ RUN apt-get update && apt-get install -y \
   wget \
   zip && apt-get clean
 
+RUN easy_install -U pip
+
 # Prepare ccache
 RUN ln -s /usr/bin/ccache /usr/local/bin/gcc
 RUN ln -s /usr/bin/ccache /usr/local/bin/g++
@@ -71,5 +73,8 @@ RUN ln -s /usr/bin/ccache /usr/local/bin/clang++
 # C++ dependencies
 RUN apt-get update && apt-get -y install libgflags-dev libgtest-dev libc++-dev clang
 
+# Google Cloud platform API libraries (for BigQuery)
+RUN pip install --upgrade google-api-python-client
+
 # Define the default command.
 CMD ["bash"]

tools/dockerfile/grpc_interop_stress_cxx/build_interop_stress.sh

@@ -42,4 +42,4 @@ cd /var/local/git/grpc
 make install-certs
 
 # build C++ interop stress client, interop client and server
-make stress_test interop_client interop_server
+make stress_test metrics_client interop_client interop_server

tools/gcp/stress_test/run_client.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python2.7
+# Copyright 2015-2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import datetime
+import os
+import re
+import select
+import subprocess
+import sys
+import time
+
+from stress_test_utils import EventType
+from stress_test_utils import BigQueryHelper
+
+
+# TODO (sree): Write a python grpc client to directly query the metrics instead
+# of calling metrics_client
+def _get_qps(metrics_cmd):
+  qps = 0
+  try:
+    # Note: gpr_log() writes even non-error messages to stderr stream. So it is 
+    # important that we set stderr=subprocess.STDOUT
+    p = subprocess.Popen(args=metrics_cmd,
+                         stdout=subprocess.PIPE,
+                         stderr=subprocess.STDOUT)
+    retcode = p.wait()
+    (out_str, err_str) = p.communicate()
+    if retcode != 0:
+      print 'Error in reading metrics information'
+      print 'Output: ', out_str
+    else:
+      # The overall qps is printed at the end of the line
+      m = re.search('\d+$', out_str)
+      qps = int(m.group()) if m else 0
+  except Exception as ex:
+    print 'Exception while reading metrics information: ' + str(ex)
+  return qps
+
+
+def run_client():
+  """This is a wrapper around the stress test client and performs the following:
+      1) Create the following two tables in Big Query:
+         (i) Summary table: To record events like the test started, completed
+                            successfully or failed
+        (ii) Qps table: To periodically record the QPS sent by this client
+      2) Start the stress test client and add a row in the Big Query summary
+         table
+      3) Once every few seconds (as specificed by the poll_interval_secs) poll
+         the status of the stress test client process and perform the
+         following:
+          3.1) If the process is still running, get the current qps by invoking
+               the metrics client program and add a row in the Big Query
+               Qps table. Sleep for a duration specified by poll_interval_secs
+          3.2) If the process exited successfully, add a row in the Big Query
+               Summary table and exit
+          3.3) If the process failed, add a row in Big Query summary table and
+               wait forever.
+               NOTE: This script typically runs inside a GKE pod which means
+               that the pod gets destroyed when the script exits. However, in
+               case the stress test client fails, we would not want the pod to
+               be destroyed (since we might want to connect to the pod for
+               examining logs). This is the reason why the script waits forever
+               in case of failures
+  """
+  env = dict(os.environ)
+  image_type = env['STRESS_TEST_IMAGE_TYPE']
+  image_name = env['STRESS_TEST_IMAGE']
+  args_str = env['STRESS_TEST_ARGS_STR']
+  metrics_client_image = env['METRICS_CLIENT_IMAGE']
+  metrics_client_args_str = env['METRICS_CLIENT_ARGS_STR']
+  run_id = env['RUN_ID']
+  pod_name = env['POD_NAME']
+  logfile_name = env.get('LOGFILE_NAME')
+  poll_interval_secs = float(env['POLL_INTERVAL_SECS'])
+  project_id = env['GCP_PROJECT_ID']
+  dataset_id = env['DATASET_ID']
+  summary_table_id = env['SUMMARY_TABLE_ID']
+  qps_table_id = env['QPS_TABLE_ID']
+
+  bq_helper = BigQueryHelper(run_id, image_type, pod_name, project_id,
+                             dataset_id, summary_table_id, qps_table_id)
+  bq_helper.initialize()
+
+  # Create BigQuery Dataset and Tables: Summary Table and Metrics Table
+  if not bq_helper.setup_tables():
+    print 'Error in creating BigQuery tables'
+    return
+
+  start_time = datetime.datetime.now()
+
+  logfile = None
+  details = 'Logging to stdout'
+  if logfile_name is not None:
+    print 'Opening logfile: %s ...' % logfile_name
+    details = 'Logfile: %s' % logfile_name
+    logfile = open(logfile_name, 'w')
+
+  # Update status that the test is starting (in the status table)
+  bq_helper.insert_summary_row(EventType.STARTING, details)
+
+  metrics_cmd = [metrics_client_image
+                ] + [x for x in metrics_client_args_str.split()]
+  stress_cmd = [image_name] + [x for x in args_str.split()]
+
+  print 'Launching process %s ...' % stress_cmd
+  stress_p = subprocess.Popen(args=stress_cmd,
+                              stdout=logfile,
+                              stderr=subprocess.STDOUT)
+
+  qps_history = [1, 1, 1]  # Maintain the last 3 qps readings
+  qps_history_idx = 0  # Index into the qps_history list
+
+  is_error = False
+  while True:
+    # Check if stress_client is still running. If so, collect metrics and upload
+    # to BigQuery status table
+    if stress_p.poll() is not None:
+      end_time = datetime.datetime.now().isoformat()
+      event_type = EventType.SUCCESS
+      details = 'End time: %s' % end_time
+      if stress_p.returncode != 0:
+        event_type = EventType.FAILURE
+        details = 'Return code = %d. End time: %s' % (stress_p.returncode,
+                                                      end_time)
+        is_error = True
+      bq_helper.insert_summary_row(event_type, details)
+      print details
+      break
+
+    # Stress client still running. Get metrics
+    qps = _get_qps(metrics_cmd)
+    qps_recorded_at = datetime.datetime.now().isoformat()
+    print 'qps: %d at %s' % (qps, qps_recorded_at)
+
+    # If QPS has been zero for the last 3 iterations, flag it as error and exit
+    qps_history[qps_history_idx] = qps
+    qps_history_idx = (qps_history_idx + 1) % len(qps_history)
+    if sum(qps_history) == 0:
+      details = 'QPS has been zero for the last %d seconds - as of : %s' % (
+          poll_interval_secs * 3, qps_recorded_at)
+      is_error = True
+      bq_helper.insert_summary_row(EventType.FAILURE, details)
+      print details
+      break
+
+    # Upload qps metrics to BiqQuery
+    bq_helper.insert_qps_row(qps, qps_recorded_at)
+
+    time.sleep(poll_interval_secs)
+
+  if is_error:
+    print 'Waiting indefinitely..'
+    select.select([], [], [])
+
+  print 'Completed'
+  return
+
+
+if __name__ == '__main__':
+  run_client()

tools/gcp/stress_test/run_server.py

@@ -0,0 +1,120 @@
+#!/usr/bin/env python2.7
+# Copyright 2015-2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import datetime
+import os
+import select
+import subprocess
+import sys
+import time
+
+from stress_test_utils import BigQueryHelper
+from stress_test_utils import EventType
+
+
+def run_server():
+  """This is a wrapper around the interop server and performs the following:
+      1) Create a 'Summary table' in Big Query to record events like the server
+         started, completed successfully or failed. NOTE: This also creates
+         another table called the QPS table which is currently NOT needed on the
+         server (it is needed on the stress test clients)
+      2) Start the server process and add a row in Big Query summary table
+      3) Wait for the server process to terminate. The server process does not
+         terminate unless there is an error.
+         If the server process terminated with a failure, add a row in Big Query
+         and wait forever.
+         NOTE: This script typically runs inside a GKE pod which means that the
+         pod gets destroyed when the script exits. However, in case the server
+         process fails, we would not want the pod to be destroyed (since we
+         might want to connect to the pod for examining logs). This is the
+         reason why the script waits forever in case of failures.
+  """
+
+  # Read the parameters from environment variables
+  env = dict(os.environ)
+
+  run_id = env['RUN_ID']  # The unique run id for this test
+  image_type = env['STRESS_TEST_IMAGE_TYPE']
+  image_name = env['STRESS_TEST_IMAGE']
+  args_str = env['STRESS_TEST_ARGS_STR']
+  pod_name = env['POD_NAME']
+  project_id = env['GCP_PROJECT_ID']
+  dataset_id = env['DATASET_ID']
+  summary_table_id = env['SUMMARY_TABLE_ID']
+  qps_table_id = env['QPS_TABLE_ID']
+
+  logfile_name = env.get('LOGFILE_NAME')
+
+  print('pod_name: %s, project_id: %s, run_id: %s, dataset_id: %s, '
+        'summary_table_id: %s, qps_table_id: %s') % (
+            pod_name, project_id, run_id, dataset_id, summary_table_id,
+            qps_table_id)
+
+  bq_helper = BigQueryHelper(run_id, image_type, pod_name, project_id,
+                             dataset_id, summary_table_id, qps_table_id)
+  bq_helper.initialize()
+
+  # Create BigQuery Dataset and Tables: Summary Table and Metrics Table
+  if not bq_helper.setup_tables():
+    print 'Error in creating BigQuery tables'
+    return
+
+  start_time = datetime.datetime.now()
+
+  logfile = None
+  details = 'Logging to stdout'
+  if logfile_name is not None:
+    print 'Opening log file: ', logfile_name
+    logfile = open(logfile_name, 'w')
+    details = 'Logfile: %s' % logfile_name
+
+  # Update status that the test is starting (in the status table)
+  bq_helper.insert_summary_row(EventType.STARTING, details)
+
+  stress_cmd = [image_name] + [x for x in args_str.split()]
+
+  print 'Launching process %s ...' % stress_cmd
+  stress_p = subprocess.Popen(args=stress_cmd,
+                              stdout=logfile,
+                              stderr=subprocess.STDOUT)
+
+  returncode = stress_p.wait()
+  if returncode != 0:
+    end_time = datetime.datetime.now().isoformat()
+    event_type = EventType.FAILURE
+    details = 'Returncode: %d; End time: %s' % (returncode, end_time)
+    bq_helper.insert_summary_row(event_type, details)
+    print 'Waiting indefinitely..'
+    select.select([], [], [])
+  return returncode
+
+
+if __name__ == '__main__':
+  run_server()

tools/gcp/stress_test/stress_test_utils.py

@@ -0,0 +1,197 @@
+#!/usr/bin/env python2.7
+# Copyright 2015-2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import datetime
+import json
+import os
+import re
+import select
+import subprocess
+import sys
+import time
+
+# Import big_query_utils module
+bq_utils_dir = os.path.abspath(os.path.join(
+    os.path.dirname(__file__), '../utils'))
+sys.path.append(bq_utils_dir)
+import big_query_utils as bq_utils
+
+
+class EventType:
+  STARTING = 'STARTING'
+  SUCCESS = 'SUCCESS'
+  FAILURE = 'FAILURE'
+
+
+class BigQueryHelper:
+  """Helper class for the stress test wrappers to interact with BigQuery.
+  """
+
+  def __init__(self, run_id, image_type, pod_name, project_id, dataset_id,
+               summary_table_id, qps_table_id):
+    self.run_id = run_id
+    self.image_type = image_type
+    self.pod_name = pod_name
+    self.project_id = project_id
+    self.dataset_id = dataset_id
+    self.summary_table_id = summary_table_id
+    self.qps_table_id = qps_table_id
+
+  def initialize(self):
+    self.bq = bq_utils.create_big_query()
+
+  def setup_tables(self):
+    return bq_utils.create_dataset(self.bq, self.project_id, self.dataset_id) \
+        and self.__create_summary_table() \
+        and self.__create_qps_table()
+
+  def insert_summary_row(self, event_type, details):
+    row_values_dict = {
+        'run_id': self.run_id,
+        'image_type': self.image_type,
+        'pod_name': self.pod_name,
+        'event_date': datetime.datetime.now().isoformat(),
+        'event_type': event_type,
+        'details': details
+    }
+    # row_unique_id is something that uniquely identifies the row (BigQuery uses
+    # it for duplicate detection).
+    row_unique_id = '%s_%s_%s' % (self.run_id, self.pod_name, event_type)
+    row = bq_utils.make_row(row_unique_id, row_values_dict)
+    return bq_utils.insert_rows(self.bq, self.project_id, self.dataset_id,
+                                self.summary_table_id, [row])
+
+  def insert_qps_row(self, qps, recorded_at):
+    row_values_dict = {
+        'run_id': self.run_id,
+        'pod_name': self.pod_name,
+        'recorded_at': recorded_at,
+        'qps': qps
+    }
+
+    # row_unique_id is something that uniquely identifies the row (BigQuery uses
+    # it for duplicate detection).
+    row_unique_id = '%s_%s_%s' % (self.run_id, self.pod_name, recorded_at)
+    row = bq_utils.make_row(row_unique_id, row_values_dict)
+    return bq_utils.insert_rows(self.bq, self.project_id, self.dataset_id,
+                                self.qps_table_id, [row])
+
+  def check_if_any_tests_failed(self, num_query_retries=3):
+    query = ('SELECT event_type FROM %s.%s WHERE run_id = \'%s\' AND '
+             'event_type="%s"') % (self.dataset_id, self.summary_table_id,
+                                   self.run_id, EventType.FAILURE)
+    query_job = bq_utils.sync_query_job(self.bq, self.project_id, query)
+    page = self.bq.jobs().getQueryResults(**query_job['jobReference']).execute(
+        num_retries=num_query_retries)
+    num_failures = int(page['totalRows'])
+    print 'num rows: ', num_failures
+    return num_failures > 0
+
+  def print_summary_records(self, num_query_retries=3):
+    line = '-' * 120
+    print line
+    print 'Summary records'
+    print 'Run Id: ', self.run_id
+    print 'Dataset Id: ', self.dataset_id
+    print line
+    query = ('SELECT pod_name, image_type, event_type, event_date, details'
+             ' FROM %s.%s WHERE run_id = \'%s\' ORDER by event_date;') % (
+                 self.dataset_id, self.summary_table_id, self.run_id)
+    query_job = bq_utils.sync_query_job(self.bq, self.project_id, query)
+
+    print '{:<25} {:<12} {:<12} {:<30} {}'.format(
+        'Pod name', 'Image type', 'Event type', 'Date', 'Details')
+    print line
+    page_token = None
+    while True:
+      page = self.bq.jobs().getQueryResults(
+          pageToken=page_token,
+          **query_job['jobReference']).execute(num_retries=num_query_retries)
+      rows = page.get('rows', [])
+      for row in rows:
+        print '{:<25} {:<12} {:<12} {:<30} {}'.format(
+            row['f'][0]['v'], row['f'][1]['v'], row['f'][2]['v'],
+            row['f'][3]['v'], row['f'][4]['v'])
+      page_token = page.get('pageToken')
+      if not page_token:
+        break
+
+  def print_qps_records(self, num_query_retries=3):
+    line = '-' * 80
+    print line
+    print 'QPS Summary'
+    print 'Run Id: ', self.run_id
+    print 'Dataset Id: ', self.dataset_id
+    print line
+    query = (
+        'SELECT pod_name, recorded_at, qps FROM %s.%s WHERE run_id = \'%s\' '
+        'ORDER by recorded_at;') % (self.dataset_id, self.qps_table_id,
+                                    self.run_id)
+    query_job = bq_utils.sync_query_job(self.bq, self.project_id, query)
+    print '{:<25} {:30} {}'.format('Pod name', 'Recorded at', 'Qps')
+    print line
+    page_token = None
+    while True:
+      page = self.bq.jobs().getQueryResults(
+          pageToken=page_token,
+          **query_job['jobReference']).execute(num_retries=num_query_retries)
+      rows = page.get('rows', [])
+      for row in rows:
+        print '{:<25} {:30} {}'.format(row['f'][0]['v'], row['f'][1]['v'],
+                                       row['f'][2]['v'])
+      page_token = page.get('pageToken')
+      if not page_token:
+        break
+
+  def __create_summary_table(self):
+    summary_table_schema = [
+        ('run_id', 'STRING', 'Test run id'),
+        ('image_type', 'STRING', 'Client or Server?'),
+        ('pod_name', 'STRING', 'GKE pod hosting this image'),
+        ('event_date', 'STRING', 'The date of this event'),
+        ('event_type', 'STRING', 'STARTED/SUCCESS/FAILURE'),
+        ('details', 'STRING', 'Any other relevant details')
+    ]
+    desc = ('The table that contains START/SUCCESS/FAILURE events for '
+            ' the stress test clients and servers')
+    return bq_utils.create_table(self.bq, self.project_id, self.dataset_id,
+                                 self.summary_table_id, summary_table_schema,
+                                 desc)
+
+  def __create_qps_table(self):
+    qps_table_schema = [
+        ('run_id', 'STRING', 'Test run id'),
+        ('pod_name', 'STRING', 'GKE pod hosting this image'),
+        ('recorded_at', 'STRING', 'Metrics recorded at time'),
+        ('qps', 'INTEGER', 'Queries per second')
+    ]
+    desc = 'The table that cointains the qps recorded at various intervals'
+    return bq_utils.create_table(self.bq, self.project_id, self.dataset_id,
+                                 self.qps_table_id, qps_table_schema, desc)

tools/gcp/utils/big_query_utils.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python2.7
+# Copyright 2015-2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import argparse
+import json
+import uuid
+import httplib2
+
+from apiclient import discovery
+from apiclient.errors import HttpError
+from oauth2client.client import GoogleCredentials
+
+NUM_RETRIES = 3
+
+
+def create_big_query():
+  """Authenticates with cloud platform and gets a BiqQuery service object
+  """
+  creds = GoogleCredentials.get_application_default()
+  return discovery.build('bigquery', 'v2', credentials=creds)
+
+
+def create_dataset(biq_query, project_id, dataset_id):
+  is_success = True
+  body = {
+      'datasetReference': {
+          'projectId': project_id,
+          'datasetId': dataset_id
+      }
+  }
+
+  try:
+    dataset_req = biq_query.datasets().insert(projectId=project_id, body=body)
+    dataset_req.execute(num_retries=NUM_RETRIES)
+  except HttpError as http_error:
+    if http_error.resp.status == 409:
+      print 'Warning: The dataset %s already exists' % dataset_id
+    else:
+      # Note: For more debugging info, print "http_error.content"
+      print 'Error in creating dataset: %s. Err: %s' % (dataset_id, http_error)
+      is_success = False
+  return is_success
+
+
+def create_table(big_query, project_id, dataset_id, table_id, table_schema,
+                 description):
+  is_success = True
+
+  body = {
+      'description': description,
+      'schema': {
+          'fields': [{
+              'name': field_name,
+              'type': field_type,
+              'description': field_description
+          } for (field_name, field_type, field_description) in table_schema]
+      },
+      'tableReference': {
+          'datasetId': dataset_id,
+          'projectId': project_id,
+          'tableId': table_id
+      }
+  }
+
+  try:
+    table_req = big_query.tables().insert(projectId=project_id,
+                                          datasetId=dataset_id,
+                                          body=body)
+    res = table_req.execute(num_retries=NUM_RETRIES)
+    print 'Successfully created %s "%s"' % (res['kind'], res['id'])
+  except HttpError as http_error:
+    if http_error.resp.status == 409:
+      print 'Warning: Table %s already exists' % table_id
+    else:
+      print 'Error in creating table: %s. Err: %s' % (table_id, http_error)
+      is_success = False
+  return is_success
+
+
+def insert_rows(big_query, project_id, dataset_id, table_id, rows_list):
+  is_success = True
+  body = {'rows': rows_list}
+  try:
+    insert_req = big_query.tabledata().insertAll(projectId=project_id,
+                                                 datasetId=dataset_id,
+                                                 tableId=table_id,
+                                                 body=body)
+    print body
+    res = insert_req.execute(num_retries=NUM_RETRIES)
+    print res
+  except HttpError as http_error:
+    print 'Error in inserting rows in the table %s' % table_id
+    is_success = False
+  return is_success
+
+
+def sync_query_job(big_query, project_id, query, timeout=5000):
+  query_data = {'query': query, 'timeoutMs': timeout}
+  query_job = None
+  try:
+    query_job = big_query.jobs().query(
+        projectId=project_id,
+        body=query_data).execute(num_retries=NUM_RETRIES)
+  except HttpError as http_error:
+    print 'Query execute job failed with error: %s' % http_error
+    print http_error.content
+  return query_job
+
+  # List of (column name, column type, description) tuples
+def make_row(unique_row_id, row_values_dict):
+  """row_values_dict is a dictionary of column name and column value.
+  """
+  return {'insertId': unique_row_id, 'json': row_values_dict}

tools/gke/kubernetes_api.py → tools/gcp/utils/kubernetes_api.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python2.7
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -33,8 +33,9 @@ import json
 
 _REQUEST_TIMEOUT_SECS = 10
 
+
 def _make_pod_config(pod_name, image_name, container_port_list, cmd_list,
-                    arg_list):
+                     arg_list, env_dict):
   """Creates a string containing the Pod defintion as required by the Kubernetes API"""
   body = {
       'kind': 'Pod',
@@ -48,20 +49,23 @@ def _make_pod_config(pod_name, image_name, container_port_list, cmd_list,
               {
                   'name': pod_name,
                   'image': image_name,
-                  'ports': []
+                  'ports': [{'containerPort': port,
+                             'protocol': 'TCP'}
+                            for port in container_port_list],
+                  'imagePullPolicy': 'Always'
               }
           ]
       }
   }
-  # Populate the 'ports' list
-  for port in container_port_list:
-    port_entry = {'containerPort': port, 'protocol': 'TCP'}
-    body['spec']['containers'][0]['ports'].append(port_entry)
+
+  env_list = [{'name': k, 'value': v} for (k, v) in env_dict.iteritems()]
+  if len(env_list) > 0:
+    body['spec']['containers'][0]['env'] = env_list
 
   # Add the 'Command' and 'Args' attributes if they are passed.
   # Note:
   #  - 'Command' overrides the ENTRYPOINT in the Docker Image
-  #  - 'Args' override the COMMAND in Docker image (yes, it is confusing!)
+  #  - 'Args' override the CMD in Docker image (yes, it is confusing!)
   if len(cmd_list) > 0:
     body['spec']['containers'][0]['command'] = cmd_list
   if len(arg_list) > 0:
@@ -70,7 +74,7 @@ def _make_pod_config(pod_name, image_name, container_port_list, cmd_list,
 
 
 def _make_service_config(service_name, pod_name, service_port_list,
-                        container_port_list, is_headless):
+                         container_port_list, is_headless):
   """Creates a string containing the Service definition as required by the Kubernetes API.
 
   NOTE:
@@ -124,6 +128,7 @@ def _print_connection_error(msg):
   print('ERROR: Connection failed. Did you remember to run Kubenetes proxy on '
         'localhost (i.e kubectl proxy --port=<proxy_port>) ?. Error: %s' % msg)
 
+
 def _do_post(post_url, api_name, request_body):
   """Helper to do HTTP POST.
 
@@ -135,7 +140,9 @@ def _do_post(post_url, api_name, request_body):
   """
   is_success = True
   try:
-    r = requests.post(post_url, data=request_body, timeout=_REQUEST_TIMEOUT_SECS)
+    r = requests.post(post_url,
+                      data=request_body,
+                      timeout=_REQUEST_TIMEOUT_SECS)
     if r.status_code == requests.codes.conflict:
       print('WARN: Looks like the resource already exists. Api: %s, url: %s' %
             (api_name, post_url))
@@ -143,7 +150,8 @@ def _do_post(post_url, api_name, request_body):
       print('ERROR: %s API returned error. HTTP response: (%d) %s' %
             (api_name, r.status_code, r.text))
       is_success = False
-  except(requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
+  except (requests.exceptions.Timeout,
+          requests.exceptions.ConnectionError) as e:
     is_success = False
     _print_connection_error(str(e))
   return is_success
@@ -165,7 +173,8 @@ def _do_delete(del_url, api_name):
       print('ERROR: %s API returned error. HTTP response: %s' %
             (api_name, r.text))
       is_success = False
-  except(requests.exceptions.Timeout, requests.exceptions.ConnectionError) as e:
+  except (requests.exceptions.Timeout,
+          requests.exceptions.ConnectionError) as e:
     is_success = False
     _print_connection_error(str(e))
   return is_success
@@ -179,12 +188,12 @@ def create_service(kube_host, kube_port, namespace, service_name, pod_name,
   post_url = 'http://%s:%d/api/v1/namespaces/%s/services' % (
       kube_host, kube_port, namespace)
   request_body = _make_service_config(service_name, pod_name, service_port_list,
-                                     container_port_list, is_headless)
+                                      container_port_list, is_headless)
   return _do_post(post_url, 'Create Service', request_body)
 
 
 def create_pod(kube_host, kube_port, namespace, pod_name, image_name,
-               container_port_list, cmd_list, arg_list):
+               container_port_list, cmd_list, arg_list, env_dict):
   """Creates a Kubernetes Pod.
 
   Note that it is generally NOT considered a good practice to directly create
@@ -200,7 +209,7 @@ def create_pod(kube_host, kube_port, namespace, pod_name, image_name,
   post_url = 'http://%s:%d/api/v1/namespaces/%s/pods' % (kube_host, kube_port,
                                                          namespace)
   request_body = _make_pod_config(pod_name, image_name, container_port_list,
-                                 cmd_list, arg_list)
+                                  cmd_list, arg_list, env_dict)
   return _do_post(post_url, 'Create Pod', request_body)
 
 
@@ -214,3 +223,47 @@ def delete_pod(kube_host, kube_port, namespace, pod_name):
   del_url = 'http://%s:%d/api/v1/namespaces/%s/pods/%s' % (kube_host, kube_port,
                                                            namespace, pod_name)
   return _do_delete(del_url, 'Delete Pod')
+
+
+def create_pod_and_service(kube_host, kube_port, namespace, pod_name,
+                           image_name, container_port_list, cmd_list, arg_list,
+                           env_dict, is_headless_service):
+  """A helper function that creates a pod and a service (if pod creation was successful)."""
+  is_success = create_pod(kube_host, kube_port, namespace, pod_name, image_name,
+                          container_port_list, cmd_list, arg_list, env_dict)
+  if not is_success:
+    print 'Error in creating Pod'
+    return False
+
+  is_success = create_service(
+      kube_host,
+      kube_port,
+      namespace,
+      pod_name,  # Use pod_name for service
+      pod_name,
+      container_port_list,  # Service port list same as container port list
+      container_port_list,
+      is_headless_service)
+  if not is_success:
+    print 'Error in creating Service'
+    return False
+
+  print 'Successfully created the pod/service %s' % pod_name
+  return True
+
+
+def delete_pod_and_service(kube_host, kube_port, namespace, pod_name):
+  """ A helper function that calls delete_pod and delete_service """
+  is_success = delete_pod(kube_host, kube_port, namespace, pod_name)
+  if not is_success:
+    print 'Error in deleting pod %s' % pod_name
+    return False
+
+  # Note: service name assumed to the the same as pod name
+  is_success = delete_service(kube_host, kube_port, namespace, pod_name)
+  if not is_success:
+    print 'Error in deleting service %s' % pod_name
+    return False
+
+  print 'Successfully deleted the Pod/Service: %s' % pod_name
+  return True

tools/jenkins/build_interop_stress_image.sh

@@ -35,6 +35,8 @@ set -x
 
 # Params:
 #  INTEROP_IMAGE - name of tag of the final interop image
+#  INTEROP_IMAGE_TAG - Optional. If set, the created image will be tagged using
+#    the command: 'docker tag $INTEROP_IMAGE $INTEROP_IMAGE_REPOSITORY_TAG'
 #  BASE_NAME - base name used to locate the base Dockerfile and build script
 #  TTY_FLAG - optional -t flag to make docker allocate tty
 #  BUILD_INTEROP_DOCKER_EXTRA_ARGS - optional args to be passed to the
@@ -77,6 +79,7 @@ CONTAINER_NAME="build_${BASE_NAME}_$(uuidgen)"
   $BASE_IMAGE \
   bash -l /var/local/jenkins/grpc/tools/dockerfile/$BASE_NAME/build_interop_stress.sh \
   && docker commit $CONTAINER_NAME $INTEROP_IMAGE \
+  && ( if [ -n "$INTEROP_IMAGE_REPOSITORY_TAG" ]; then docker tag -f $INTEROP_IMAGE $INTEROP_IMAGE_REPOSITORY_TAG ; fi ) \
   && echo "Successfully built image $INTEROP_IMAGE")
 EXITCODE=$?
 

tools/run_tests/build_node.bat

@@ -27,4 +27,16 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-npm install --build-from-source
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
+
+del /f /q BUILD || rmdir build /s /q
+
+call npm install --build-from-source
+
+@rem delete the redundant openssl headers
+for /f "delims=v" %%v in ('node --version') do (
+  rmdir "%USERPROFILE%\.node-gyp\%%v\include\node\openssl" /S /Q
+)
+
+@rem rebuild, because it probably failed the first time
+call npm install --build-from-source

tools/run_tests/build_python.sh

@@ -45,3 +45,6 @@ export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1
 tox --notest
 
 $ROOT/.tox/py27/bin/python $ROOT/setup.py build
+$ROOT/.tox/py27/bin/python $ROOT/setup.py build_py
+$ROOT/.tox/py27/bin/python $ROOT/setup.py build_ext --inplace
+$ROOT/.tox/py27/bin/python $ROOT/setup.py gather --test

tools/run_tests/distribtest_targets.py

@@ -96,6 +96,15 @@ class CSharpDistribTest(object):
       return create_jobspec(self.name,
           ['test/distrib/csharp/run_distrib_test.sh'],
           environ={'EXTERNAL_GIT_ROOT': '../../..'})
+    elif self.platform == 'windows':
+      if self.arch == 'x64':
+        environ={'MSBUILD_EXTRA_ARGS': '/p:Platform=x64',
+                 'DISTRIBTEST_OUTPATH': 'DistribTest\\bin\\x64\\Debug'}
+      else:
+        environ={'DISTRIBTEST_OUTPATH': 'DistribTest\\bin\\\Debug'}
+      return create_jobspec(self.name,
+          ['test\\distrib\\csharp\\run_distrib_test.bat'],
+          environ=environ)
     else:
       raise Exception("Not supported yet.")
 
@@ -240,6 +249,8 @@ def targets():
           CSharpDistribTest('linux', 'x64', 'ubuntu1510'),
           CSharpDistribTest('linux', 'x64', 'ubuntu1604'),
           CSharpDistribTest('macos', 'x86'),
+          CSharpDistribTest('windows', 'x86'),
+          CSharpDistribTest('windows', 'x64'),
           PythonDistribTest('linux', 'x64', 'wheezy'),
           PythonDistribTest('linux', 'x64', 'jessie'),
           PythonDistribTest('linux', 'x86', 'jessie'),

tools/run_tests/jobset.py

@@ -384,7 +384,8 @@ class Jobset(object):
                 self._travis,
                 self._add_env)
       self._running.add(job)
-      self.resultset[job.GetSpec().shortname] = []
+      if not self.resultset.has_key(job.GetSpec().shortname):
+        self.resultset[job.GetSpec().shortname] = []
     return True
 
   def reap(self):

tools/run_tests/pre_build_node.bat

@@ -27,13 +27,8 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-@rem Expire cache after 1 week
-npm update --cache-min 604800
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
 
-npm install node-gyp-install
-.\node_modules\.bin\node-gyp-install.cmd
+@rem Expire cache after 1 week
+call npm update --cache-min 604800
 
-@rem delete the redundant openssl headers
-for /f "delims=v" %%v in ('node --version') do (
-  rmdir "%HOMEDRIVE%%HOMEPATH%\.node-gyp\%%v\include\node\openssl" /S /Q
-)

tools/run_tests/run_interop_tests.py

@@ -60,6 +60,8 @@ _SKIP_COMPRESSION = ['large_compressed_unary',
 _SKIP_ADVANCED = ['custom_metadata', 'status_code_and_message',
                   'unimplemented_method']
 
+_TEST_TIMEOUT = 3*60
+
 class CXXLanguage:
 
   def __init__(self):
@@ -459,7 +461,7 @@ def cloud_to_prod_jobspec(language, test_case, server_host_name,
           environ=environ,
           shortname='%s:%s:%s:%s' % (suite_name, server_host_name, language,
                                      test_case),
-          timeout_seconds=90,
+          timeout_seconds=_TEST_TIMEOUT,
           flake_retries=5 if args.allow_flakes else 0,
           timeout_retries=2 if args.allow_flakes else 0,
           kill_handler=_job_kill_handler)
@@ -495,7 +497,7 @@ def cloud_to_cloud_jobspec(language, test_case, server_name, server_host,
           environ=environ,
           shortname='cloud_to_cloud:%s:%s_server:%s' % (language, server_name,
                                                         test_case),
-          timeout_seconds=90,
+          timeout_seconds=_TEST_TIMEOUT,
           flake_retries=5 if args.allow_flakes else 0,
           timeout_retries=2 if args.allow_flakes else 0,
           kill_handler=_job_kill_handler)

tools/run_tests/run_node.bat

@@ -27,6 +27,7 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
 set JUNIT_REPORT_PATH=src\node\report.xml
 set JUNIT_REPORT_STACK=1
 .\node_modules\.bin\mocha.cmd --reporter mocha-jenkins-reporter --timeout 8000 src\node\test

tools/run_tests/run_python.sh

@@ -42,7 +42,12 @@ export LDFLAGS="-L$ROOT/libs/$CONFIG"
 export GRPC_PYTHON_BUILD_WITH_CYTHON=1
 export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1
 
-tox
+if [ "$CONFIG" = "gcov" ]
+then
+  tox
+else
+  $ROOT/.tox/py27/bin/python $ROOT/setup.py test_lite
+fi
 
 mkdir -p $ROOT/reports
 rm -rf $ROOT/reports/python-coverage

tools/run_tests/run_tests.py

@@ -350,15 +350,27 @@ class PythonLanguage(object):
     _check_compiler(self.args.compiler, ['default'])
 
   def test_specs(self):
+    # load list of known test suites
+    with open('src/python/grpcio/tests/tests.json') as tests_json_file:
+      tests_json = json.load(tests_json_file)
     environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS)
     environment['PYVER'] = '2.7'
-    return [self.config.job_spec(
-        ['tools/run_tests/run_python.sh'],
-        None,
-        environ=environment,
-        shortname='py.test',
-        timeout_seconds=15*60
-    )]
+    if self.config.build_config != 'gcov':
+      return [self.config.job_spec(
+          ['tools/run_tests/run_python.sh'],
+          None,
+          environ=dict(environment.items() +
+                       [('GPRC_PYTHON_TESTRUNNER_FILTER', suite_name)]),
+          shortname='py.test.%s' % suite_name,
+          timeout_seconds=5*60)
+          for suite_name in tests_json]
+    else:
+      return [self.config.job_spec(['tools/run_tests/run_python.sh'],
+                                   None,
+                                   environ=environment,
+                                   shortname='py.test.coverage',
+                                   timeout_seconds=15*60)]
+
 
   def pre_build_steps(self):
     return []

tools/run_tests/stress_test/run_stress_tests_on_gke.py

@@ -0,0 +1,556 @@
+#!/usr/bin/env python2.7
+# Copyright 2015-2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+import argparse
+import datetime
+import os
+import subprocess
+import sys
+import time
+
+stress_test_utils_dir = os.path.abspath(os.path.join(
+    os.path.dirname(__file__), '../../gcp/stress_test'))
+sys.path.append(stress_test_utils_dir)
+from stress_test_utils import BigQueryHelper
+
+kubernetes_api_dir = os.path.abspath(os.path.join(
+    os.path.dirname(__file__), '../../gcp/utils'))
+sys.path.append(kubernetes_api_dir)
+
+import kubernetes_api
+
+_GRPC_ROOT = os.path.abspath(os.path.join(
+    os.path.dirname(sys.argv[0]), '../../..'))
+os.chdir(_GRPC_ROOT)
+
+# num of seconds to wait for the GKE image to start and warmup
+_GKE_IMAGE_WARMUP_WAIT_SECS = 60
+
+_SERVER_POD_NAME = 'stress-server'
+_CLIENT_POD_NAME_PREFIX = 'stress-client'
+_DATASET_ID_PREFIX = 'stress_test'
+_SUMMARY_TABLE_ID = 'summary'
+_QPS_TABLE_ID = 'qps'
+
+_DEFAULT_DOCKER_IMAGE_NAME = 'grpc_stress_test'
+
+# The default port on which the kubernetes proxy server is started on localhost
+# (i.e kubectl proxy --port=<port>)
+_DEFAULT_KUBERNETES_PROXY_PORT = 8001
+
+# How frequently should the stress client wrapper script (running inside a GKE
+# container) poll the health of the stress client (also running inside the GKE
+# container) and upload metrics to BigQuery
+_DEFAULT_STRESS_CLIENT_POLL_INTERVAL_SECS = 60
+
+# The default setting for stress test server and client
+_DEFAULT_STRESS_SERVER_PORT = 8080
+_DEFAULT_METRICS_PORT = 8081
+_DEFAULT_TEST_CASES_STR = 'empty_unary:1,large_unary:1,client_streaming:1,server_streaming:1,empty_stream:1'
+_DEFAULT_NUM_CHANNELS_PER_SERVER = 5
+_DEFAULT_NUM_STUBS_PER_CHANNEL = 10
+_DEFAULT_METRICS_COLLECTION_INTERVAL_SECS = 30
+
+# Number of stress client instances to launch
+_DEFAULT_NUM_CLIENTS = 3
+
+# How frequently should this test monitor the health of Stress clients and
+# Servers running in GKE
+_DEFAULT_TEST_POLL_INTERVAL_SECS = 60
+
+# Default run time for this test (2 hour)
+_DEFAULT_TEST_DURATION_SECS = 7200
+
+# The number of seconds it would take a GKE pod to warm up (i.e get to 'Running'
+# state from the time of creation). Ideally this is something the test should
+# automatically determine by using Kubernetes API to poll the pods status.
+_DEFAULT_GKE_WARMUP_SECS = 60
+
+
+class KubernetesProxy:
+  """ Class to start a proxy on localhost to the Kubernetes API server """
+
+  def __init__(self, api_port):
+    self.port = api_port
+    self.p = None
+    self.started = False
+
+  def start(self):
+    cmd = ['kubectl', 'proxy', '--port=%d' % self.port]
+    self.p = subprocess.Popen(args=cmd)
+    self.started = True
+    time.sleep(2)
+    print '..Started'
+
+  def get_port(self):
+    return self.port
+
+  def is_started(self):
+    return self.started
+
+  def __del__(self):
+    if self.p is not None:
+      print 'Shutting down Kubernetes proxy..'
+      self.p.kill()
+
+
+class TestSettings:
+
+  def __init__(self, build_docker_image, test_poll_interval_secs,
+               test_duration_secs, kubernetes_proxy_port):
+    self.build_docker_image = build_docker_image
+    self.test_poll_interval_secs = test_poll_interval_secs
+    self.test_duration_secs = test_duration_secs
+    self.kubernetes_proxy_port = kubernetes_proxy_port
+
+
+class GkeSettings:
+
+  def __init__(self, project_id, docker_image_name):
+    self.project_id = project_id
+    self.docker_image_name = docker_image_name
+    self.tag_name = 'gcr.io/%s/%s' % (project_id, docker_image_name)
+
+
+class BigQuerySettings:
+
+  def __init__(self, run_id, dataset_id, summary_table_id, qps_table_id):
+    self.run_id = run_id
+    self.dataset_id = dataset_id
+    self.summary_table_id = summary_table_id
+    self.qps_table_id = qps_table_id
+
+
+class StressServerSettings:
+
+  def __init__(self, server_pod_name, server_port):
+    self.server_pod_name = server_pod_name
+    self.server_port = server_port
+
+
+class StressClientSettings:
+
+  def __init__(self, num_clients, client_pod_name_prefix, server_pod_name,
+               server_port, metrics_port, metrics_collection_interval_secs,
+               stress_client_poll_interval_secs, num_channels_per_server,
+               num_stubs_per_channel, test_cases_str):
+    self.num_clients = num_clients
+    self.client_pod_name_prefix = client_pod_name_prefix
+    self.server_pod_name = server_pod_name
+    self.server_port = server_port
+    self.metrics_port = metrics_port
+    self.metrics_collection_interval_secs = metrics_collection_interval_secs
+    self.stress_client_poll_interval_secs = stress_client_poll_interval_secs
+    self.num_channels_per_server = num_channels_per_server
+    self.num_stubs_per_channel = num_stubs_per_channel
+    self.test_cases_str = test_cases_str
+
+    # == Derived properties ==
+    # Note: Client can accept a list of server addresses (a comma separated list
+    # of 'server_name:server_port'). In this case, we only have one server
+    # address to pass
+    self.server_addresses = '%s.default.svc.cluster.local:%d' % (
+        server_pod_name, server_port)
+    self.client_pod_names_list = ['%s-%d' % (client_pod_name_prefix, i)
+                                  for i in range(1, num_clients + 1)]
+
+
+def _build_docker_image(image_name, tag_name):
+  """ Build the docker image and add tag it to the GKE repository """
+  print 'Building docker image: %s' % image_name
+  os.environ['INTEROP_IMAGE'] = image_name
+  os.environ['INTEROP_IMAGE_REPOSITORY_TAG'] = tag_name
+  # Note that 'BASE_NAME' HAS to be 'grpc_interop_stress_cxx' since the script
+  # build_interop_stress_image.sh invokes the following script:
+  #   tools/dockerfile/$BASE_NAME/build_interop_stress.sh
+  os.environ['BASE_NAME'] = 'grpc_interop_stress_cxx'
+  cmd = ['tools/jenkins/build_interop_stress_image.sh']
+  retcode = subprocess.call(args=cmd)
+  if retcode != 0:
+    print 'Error in building docker image'
+    return False
+  return True
+
+
+def _push_docker_image_to_gke_registry(docker_tag_name):
+  """Executes 'gcloud docker push <docker_tag_name>' to push the image to GKE registry"""
+  cmd = ['gcloud', 'docker', 'push', docker_tag_name]
+  print 'Pushing %s to GKE registry..' % docker_tag_name
+  retcode = subprocess.call(args=cmd)
+  if retcode != 0:
+    print 'Error in pushing docker image %s to the GKE registry' % docker_tag_name
+    return False
+  return True
+
+
+def _launch_server(gke_settings, stress_server_settings, bq_settings,
+                   kubernetes_proxy):
+  """ Launches a stress test server instance in GKE cluster """
+  if not kubernetes_proxy.is_started:
+    print 'Kubernetes proxy must be started before calling this function'
+    return False
+
+  # This is the wrapper script that is run in the container. This script runs
+  # the actual stress test server
+  server_cmd_list = ['/var/local/git/grpc/tools/gcp/stress_test/run_server.py']
+
+  # run_server.py does not take any args from the command line. The args are
+  # instead passed via environment variables (see server_env below)
+  server_arg_list = []
+
+  # The parameters to the script run_server.py are injected into the container
+  # via environment variables
+  server_env = {
+      'STRESS_TEST_IMAGE_TYPE': 'SERVER',
+      'STRESS_TEST_IMAGE': '/var/local/git/grpc/bins/opt/interop_server',
+      'STRESS_TEST_ARGS_STR': '--port=%s' % stress_server_settings.server_port,
+      'RUN_ID': bq_settings.run_id,
+      'POD_NAME': stress_server_settings.server_pod_name,
+      'GCP_PROJECT_ID': gke_settings.project_id,
+      'DATASET_ID': bq_settings.dataset_id,
+      'SUMMARY_TABLE_ID': bq_settings.summary_table_id,
+      'QPS_TABLE_ID': bq_settings.qps_table_id
+  }
+
+  # Launch Server
+  is_success = kubernetes_api.create_pod_and_service(
+      'localhost',
+      kubernetes_proxy.get_port(),
+      'default',  # Use 'default' namespace
+      stress_server_settings.server_pod_name,
+      gke_settings.tag_name,
+      [stress_server_settings.server_port],  # Port that should be exposed
+      server_cmd_list,
+      server_arg_list,
+      server_env,
+      True  # Headless = True for server. Since we want DNS records to be created by GKE
+  )
+
+  return is_success
+
+
+def _launch_client(gke_settings, stress_server_settings, stress_client_settings,
+                   bq_settings, kubernetes_proxy):
+  """ Launches a configurable number of stress test clients on GKE cluster """
+  if not kubernetes_proxy.is_started:
+    print 'Kubernetes proxy must be started before calling this function'
+    return False
+
+  stress_client_arg_list = [
+      '--server_addresses=%s' % stress_client_settings.server_addresses,
+      '--test_cases=%s' % stress_client_settings.test_cases_str,
+      '--num_stubs_per_channel=%d' %
+      stress_client_settings.num_stubs_per_channel
+  ]
+
+  # This is the wrapper script that is run in the container. This script runs
+  # the actual stress client
+  client_cmd_list = ['/var/local/git/grpc/tools/gcp/stress_test/run_client.py']
+
+  # run_client.py takes no args. All args are passed as env variables (see
+  # client_env)
+  client_arg_list = []
+
+  metrics_server_address = 'localhost:%d' % stress_client_settings.metrics_port
+  metrics_client_arg_list = [
+      '--metrics_server_address=%s' % metrics_server_address,
+      '--total_only=true'
+  ]
+
+  # The parameters to the script run_client.py are injected into the container
+  # via environment variables
+  client_env = {
+      'STRESS_TEST_IMAGE_TYPE': 'CLIENT',
+      'STRESS_TEST_IMAGE': '/var/local/git/grpc/bins/opt/stress_test',
+      'STRESS_TEST_ARGS_STR': ' '.join(stress_client_arg_list),
+      'METRICS_CLIENT_IMAGE': '/var/local/git/grpc/bins/opt/metrics_client',
+      'METRICS_CLIENT_ARGS_STR': ' '.join(metrics_client_arg_list),
+      'RUN_ID': bq_settings.run_id,
+      'POLL_INTERVAL_SECS':
+          str(stress_client_settings.stress_client_poll_interval_secs),
+      'GCP_PROJECT_ID': gke_settings.project_id,
+      'DATASET_ID': bq_settings.dataset_id,
+      'SUMMARY_TABLE_ID': bq_settings.summary_table_id,
+      'QPS_TABLE_ID': bq_settings.qps_table_id
+  }
+
+  for pod_name in stress_client_settings.client_pod_names_list:
+    client_env['POD_NAME'] = pod_name
+    is_success = kubernetes_api.create_pod_and_service(
+        'localhost',  # Since proxy is running on localhost
+        kubernetes_proxy.get_port(),
+        'default',  # default namespace
+        pod_name,
+        gke_settings.tag_name,
+        [stress_client_settings.metrics_port
+        ],  # Client pods expose metrics port
+        client_cmd_list,
+        client_arg_list,
+        client_env,
+        False  # Client is not a headless service
+    )
+    if not is_success:
+      print 'Error in launching client %s' % pod_name
+      return False
+
+  return True
+
+
+def _launch_server_and_client(gke_settings, stress_server_settings,
+                              stress_client_settings, bq_settings,
+                              kubernetes_proxy_port):
+  # Start kubernetes proxy
+  print 'Kubernetes proxy'
+  kubernetes_proxy = KubernetesProxy(kubernetes_proxy_port)
+  kubernetes_proxy.start()
+
+  print 'Launching server..'
+  is_success = _launch_server(gke_settings, stress_server_settings, bq_settings,
+                              kubernetes_proxy)
+  if not is_success:
+    print 'Error in launching server'
+    return False
+
+  # Server takes a while to start.
+  # TODO(sree) Use Kubernetes API to query the status of the server instead of
+  # sleeping
+  print 'Waiting for %s seconds for the server to start...' % _GKE_IMAGE_WARMUP_WAIT_SECS
+  time.sleep(_GKE_IMAGE_WARMUP_WAIT_SECS)
+
+  # Launch client
+  client_pod_name_prefix = 'stress-client'
+  is_success = _launch_client(gke_settings, stress_server_settings,
+                              stress_client_settings, bq_settings,
+                              kubernetes_proxy)
+
+  if not is_success:
+    print 'Error in launching client(s)'
+    return False
+
+  print 'Waiting for %s seconds for the client images to start...' % _GKE_IMAGE_WARMUP_WAIT_SECS
+  time.sleep(_GKE_IMAGE_WARMUP_WAIT_SECS)
+  return True
+
+
+def _delete_server_and_client(stress_server_settings, stress_client_settings,
+                              kubernetes_proxy_port):
+  kubernetes_proxy = KubernetesProxy(kubernetes_proxy_port)
+  kubernetes_proxy.start()
+
+  # Delete clients first
+  is_success = True
+  for pod_name in stress_client_settings.client_pod_names_list:
+    is_success = kubernetes_api.delete_pod_and_service(
+        'localhost', kubernetes_proxy_port, 'default', pod_name)
+    if not is_success:
+      return False
+
+  # Delete server
+  is_success = kubernetes_api.delete_pod_and_service(
+      'localhost', kubernetes_proxy_port, 'default',
+      stress_server_settings.server_pod_name)
+  return is_success
+
+
+def run_test_main(test_settings, gke_settings, stress_server_settings,
+                  stress_client_clients):
+  is_success = True
+
+  if test_settings.build_docker_image:
+    is_success = _build_docker_image(gke_settings.docker_image_name,
+                                     gke_settings.tag_name)
+    if not is_success:
+      return False
+
+    is_success = _push_docker_image_to_gke_registry(gke_settings.tag_name)
+    if not is_success:
+      return False
+
+  # Create a unique id for this run (Note: Using timestamp instead of UUID to
+  # make it easier to deduce the date/time of the run just by looking at the run
+  # run id. This is useful in debugging when looking at records in Biq query)
+  run_id = datetime.datetime.now().strftime('%Y_%m_%d_%H_%M_%S')
+  dataset_id = '%s_%s' % (_DATASET_ID_PREFIX, run_id)
+
+  # Big Query settings (common for both Stress Server and Client)
+  bq_settings = BigQuerySettings(run_id, dataset_id, _SUMMARY_TABLE_ID,
+                                 _QPS_TABLE_ID)
+
+  bq_helper = BigQueryHelper(run_id, '', '', args.project_id, dataset_id,
+                             _SUMMARY_TABLE_ID, _QPS_TABLE_ID)
+  bq_helper.initialize()
+
+  try:
+    is_success = _launch_server_and_client(gke_settings, stress_server_settings,
+                                           stress_client_settings, bq_settings,
+                                           test_settings.kubernetes_proxy_port)
+    if not is_success:
+      return False
+
+    start_time = datetime.datetime.now()
+    end_time = start_time + datetime.timedelta(
+        seconds=test_settings.test_duration_secs)
+    print 'Running the test until %s' % end_time.isoformat()
+
+    while True:
+      if datetime.datetime.now() > end_time:
+        print 'Test was run for %d seconds' % test_settings.test_duration_secs
+        break
+
+      # Check if either stress server or clients have failed
+      if bq_helper.check_if_any_tests_failed():
+        is_success = False
+        print 'Some tests failed.'
+        break
+
+      # Things seem to be running fine. Wait until next poll time to check the
+      # status
+      print 'Sleeping for %d seconds..' % test_settings.test_poll_interval_secs
+      time.sleep(test_settings.test_poll_interval_secs)
+
+    # Print BiqQuery tables
+    bq_helper.print_summary_records()
+    bq_helper.print_qps_records()
+
+  finally:
+    # If is_success is False at this point, it means that the stress tests were
+    # started successfully but failed while running the tests. In this case we
+    # do should not delete the pods (since they contain all the failure
+    # information)
+    if is_success:
+      _delete_server_and_client(stress_server_settings, stress_client_settings,
+                                test_settings.kubernetes_proxy_port)
+
+  return is_success
+
+
+argp = argparse.ArgumentParser(
+    description='Launch stress tests in GKE',
+    formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+argp.add_argument('--project_id',
+                  required=True,
+                  help='The Google Cloud Platform Project Id')
+argp.add_argument('--num_clients',
+                  default=1,
+                  type=int,
+                  help='Number of client instances to start')
+argp.add_argument('--docker_image_name',
+                  default=_DEFAULT_DOCKER_IMAGE_NAME,
+                  help='The name of the docker image containing stress client '
+                  'and stress servers')
+argp.add_argument('--build_docker_image',
+                  dest='build_docker_image',
+                  action='store_true',
+                  help='Build a docker image and push to Google Container '
+                  'Registry')
+argp.add_argument('--do_not_build_docker_image',
+                  dest='build_docker_image',
+                  action='store_false',
+                  help='Do not build and push docker image to Google Container '
+                  'Registry')
+argp.set_defaults(build_docker_image=True)
+
+argp.add_argument('--test_poll_interval_secs',
+                  default=_DEFAULT_TEST_POLL_INTERVAL_SECS,
+                  type=int,
+                  help='How frequently should this script should monitor the '
+                  'health of stress clients and servers running in the GKE '
+                  'cluster')
+argp.add_argument('--test_duration_secs',
+                  default=_DEFAULT_TEST_DURATION_SECS,
+                  type=int,
+                  help='How long should this test be run')
+argp.add_argument('--kubernetes_proxy_port',
+                  default=_DEFAULT_KUBERNETES_PROXY_PORT,
+                  type=int,
+                  help='The port on which the kubernetes proxy (on localhost)'
+                  ' is started')
+argp.add_argument('--stress_server_port',
+                  default=_DEFAULT_STRESS_SERVER_PORT,
+                  type=int,
+                  help='The port on which the stress server (in GKE '
+                  'containers) listens')
+argp.add_argument('--stress_client_metrics_port',
+                  default=_DEFAULT_METRICS_PORT,
+                  type=int,
+                  help='The port on which the stress clients (in GKE '
+                  'containers) expose metrics')
+argp.add_argument('--stress_client_poll_interval_secs',
+                  default=_DEFAULT_STRESS_CLIENT_POLL_INTERVAL_SECS,
+                  type=int,
+                  help='How frequently should the stress client wrapper script'
+                  ' running inside GKE should monitor health of the actual '
+                  ' stress client process and upload the metrics to BigQuery')
+argp.add_argument('--stress_client_metrics_collection_interval_secs',
+                  default=_DEFAULT_METRICS_COLLECTION_INTERVAL_SECS,
+                  type=int,
+                  help='How frequently should metrics be collected in-memory on'
+                  ' the stress clients (running inside GKE containers). Note '
+                  'that this is NOT the same as the upload-to-BigQuery '
+                  'frequency. The metrics upload frequency is controlled by the'
+                  ' --stress_client_poll_interval_secs flag')
+argp.add_argument('--stress_client_num_channels_per_server',
+                  default=_DEFAULT_NUM_CHANNELS_PER_SERVER,
+                  type=int,
+                  help='The number of channels created to each server from a '
+                  'stress client')
+argp.add_argument('--stress_client_num_stubs_per_channel',
+                  default=_DEFAULT_NUM_STUBS_PER_CHANNEL,
+                  type=int,
+                  help='The number of stubs created per channel. This number '
+                  'indicates the max number of RPCs that can be made in '
+                  'parallel on each channel at any given time')
+argp.add_argument('--stress_client_test_cases',
+                  default=_DEFAULT_TEST_CASES_STR,
+                  help='List of test cases (with weights) to be executed by the'
+                  ' stress test client. The list is in the following format:\n'
+                  '  <testcase_1:w_1,<test_case2:w_2>..<testcase_n:w_n>\n'
+                  ' (Note: The weights do not have to add up to 100)')
+
+if __name__ == '__main__':
+  args = argp.parse_args()
+
+  test_settings = TestSettings(
+      args.build_docker_image, args.test_poll_interval_secs,
+      args.test_duration_secs, args.kubernetes_proxy_port)
+
+  gke_settings = GkeSettings(args.project_id, args.docker_image_name)
+
+  stress_server_settings = StressServerSettings(_SERVER_POD_NAME,
+                                                args.stress_server_port)
+  stress_client_settings = StressClientSettings(
+      args.num_clients, _CLIENT_POD_NAME_PREFIX, _SERVER_POD_NAME,
+      args.stress_server_port, args.stress_client_metrics_port,
+      args.stress_client_metrics_collection_interval_secs,
+      args.stress_client_poll_interval_secs,
+      args.stress_client_num_channels_per_server,
+      args.stress_client_num_stubs_per_channel, args.stress_client_test_cases)
+
+  run_test_main(test_settings, gke_settings, stress_server_settings,
+                stress_client_settings)