Merge branch 'master' into debug_qps_stream

CONTRIBUTING.md

@@ -13,7 +13,7 @@ In order to protect both you and ourselves, you will need to sign the
 ### Technical requirements
 
 You will need several tools to work with this repository. In addition to all of
-the packages described in the [INSTALL](INSTALL) file, you will also need
+the packages described in the [INSTALL](INSTALL.md) file, you will also need
 python, and the mako template renderer. To install the latter, using pip, one
 should simply be able to do `pip install mako`.
 

INSTALL

@@ -1,217 +0,0 @@
-These instructions only cover building grpc C and C++ libraries under
-typical unix systems. If you need more information, please try grpc's
-wiki pages:
-
-  https://github.com/google/grpc/wiki
-
-
-*************************
-* If you are in a hurry *
-*************************
-
-On Linux (Debian):
-
- Note: you will need to add the Debian 'jessie-backports' distribution to your sources
- file first.
-
- Add the following line to your `/etc/apt/sources.list` file:
-
-   deb http://http.debian.net/debian jessie-backports main
-
- Install the gRPC library:
-
- $ [sudo] apt-get install libgrpc-dev
-
-OR
-
- $ git clone https://github.com/grpc/grpc.git
- $ cd grpc
- $ git submodule update --init
- $ make 
- $ [sudo] make install
-
-You don't need anything else than GNU Make, gcc and autotools. Under a Debian
-or Ubuntu system, this should boil down to the following packages:
-
- $ [sudo] apt-get install build-essential autoconf libtool
-
-Building the python wrapper requires the following:
-
- $ [sudo] apt-get install python-all-dev python-virtualenv
-
-If you want to install in a different directory than the default /usr/lib, you can
-override it on the command line:
-
- $ [sudo] make install prefix=/opt
-
-
-*******************************
-* More detailled instructions *
-*******************************
-
-Setting up dependencies
-=======================
-
-Dependencies to compile the libraries
--------------------------------------
-
-grpc libraries have few external dependencies. If you need to compile and
-install them, they are present in the third_party directory if you have
-cloned the github repository recursively. If you didn't clone recursively,
-you can still get them later by running the following command:
-
-  $ git submodule update --init
-
-Note that the Makefile makes it much easier for you to compile from sources
-if you were to clone recursively our git repository: it will automatically
-compile zlib and OpenSSL, which are core requirements for grpc. Note this
-creates grpc libraries that will have zlib and OpenSSL built-in inside of them,
-which significantly increases the libraries' size.
-
-In order to decrease that size, you can manually install zlib and OpenSSL on
-your system, so that the Makefile can use them instead.
-
-Under a Debian or Ubuntu system, one can acquire the development package
-for zlib this way:
-
-  # apt-get install zlib1g-dev
-
-To the best of our knowledge, no distribution has an OpenSSL package that
-supports ALPN yet, so you would still have to depend on installing from source
-for that particular dependency if you want to reduce the libraries' size.
-
-The recommended version of OpenSSL that provides ALPN support is available
-at this URL:
-
-  https://www.openssl.org/source/openssl-1.0.2.tar.gz
-
-
-Dependencies to compile and run the tests
------------------------------------------
-
-Compiling and running grpc plain-C tests dont't require any more dependency.
-
-
-Compiling and running grpc C++ tests depend on protobuf 3.0.0, gtest and
-gflags. Although gflags is provided in third_party, you will need to manually
-install that dependency on your system to run these tests.
-
-Under a Debian or Ubuntu system, you can install the gtests and gflags packages
-using apt-get:
-
-  # apt-get install libgflags-dev libgtest-dev
-
-However, protobuf 3.0.0 isn't in a debian package yet, but the Makefile will
-automatically try and compile the one present in third_party if you cloned the
-repository recursively, and that it detects your system is lacking it.
-
-Compiling and installing protobuf 3.0.0 requires a few more dependencies in
-itself, notably the autoconf suite. If you have apt-get, you can install
-these dependencies this way:
-
-  # apt-get install autoconf libtool
-
-If you want to run the tests using one of the sanitized configurations, you
-will need clang and its instrumented libc++:
-
-  # apt-get install clang libc++-dev
-
-Mac-specific notes:
--------------------
-
-For a Mac system, git is not available by default. You will first need to
-install Xcode from the Mac AppStore and then run the following command from a
-terminal:
-
-  $ sudo xcode-select --install
-
-You should also install "port" following the instructions at
-https://www.macports.org . This will reside in /opt/local/bin/port for
-most Mac installations. Do the "git submodule" command listed above.
-
-Then execute the following for all the needed build dependencies
-
-  $ sudo /opt/local/bin/port install autoconf automake libtool gflags cmake
-  $ mkdir ~/gtest-svn
-  $ svn checkout http://googletest.googlecode.com/svn/trunk/ gtest-svn
-  $ mkdir mybuild
-  $ cd mybuild
-  $ cmake ../gtest-svn
-  $ make
-  $ make gtest.a gtest_main.a
-  $ sudo cp libgtest.a libgtest_main.a /opt/local/lib
-  $ sudo mkdir /opt/local/include/gtest
-  $ sudo cp -pr ../gtest-svn/include/gtest /opt/local/include/gtest
-
-If you are going to make changes and need to regenerate the projects file,
-you will need to install certain modules for python.
-
-  $ sudo easy_install simplejson mako
-
-Mingw-specific notes:
----------------------
-
-While gRPC compiles properly under mingw, some more preparation work is needed.
-The recommendation is to use msys2. The installation instructions are available
-at that address: http://msys2.github.io/
-
-Once this is installed, make sure you are using the following: MinGW-w64 Win64.
-You'll be required to install a few more packages:
-
-  $ pacman -S make mingw-w64-x86_64-gcc mingw-w64-x86_64-zlib autoconf automake libtool
-
-Please also install OpenSSL from that website:
-
-  http://slproweb.com/products/Win32OpenSSL.html
-
-The package Win64 OpenSSL v1.0.2a should do. At that point you should be able
-to compile gRPC with the following:
-
-  $ export LDFLAGS="-L/mingw64/lib -L/c/OpenSSL-Win64"
-  $ export CPPFLAGS="-I/mingw64/include -I/c/OpenSSL-Win64/include"
-  $ make
-
-A word on OpenSSL
------------------
-
-Secure HTTP2 requires the TLS extension ALPN (see rfc 7301 and
-http://http2.github.io/http2-spec/ section 3.3). Our HTTP2 implementation
-relies on OpenSSL's implementation. OpenSSL 1.0.2 is the first released version
-of OpenSSL that has ALPN support, and this explains our dependency on it.
-
-Note that the Makefile supports compiling only the unsecure elements of grpc,
-and if you do not have OpenSSL and do not want it, you can still proceed
-with installing only the elements you require. However, we strongly recommend
-the use of encryption for all network traffic, and discourage the use of grpc
-without TLS.
-
-
-Compiling
-=========
-
-If you have all the dependencies mentioned above, you should simply be able
-to go ahead and run "make" to compile grpc's C and C++ libraries:
-
-  $ make
-
-
-Testing
-=======
-
-To build and run the tests, you can run the command:
-
-  $ make test
-
-If you want to be able to run them in parallel, and get better output, you can
-also use the python tool we have written:
-
-  $ ./tools/run_tests/run_tests.py
-
-
-Installing
-==========
-
-Once everything is compiled, you should be able to install grpc C and C++
-libraries and headers:
-
-  # make install

INSTALL.md

@@ -0,0 +1,46 @@
+#If you are in a hurry
+
+For language-specific installation instructions for gRPC runtime, please
+refer to these documents
+
+ * [C++](examples/cpp)
+ * [C#](src/csharp): NuGet package `Grpc`
+ * [Go](https://github.com/grpc/grpc-go): `go get google.golang.org/grpc`
+ * [Java](https://github.com/grpc/grpc-java)
+ * [Node](src/node): `npm install grpc`
+ * [Objective-C](src/objective-c)
+ * [PHP](src/php): `pecl install grpc-beta`
+ * [Python](src/python/grpcio): `pip install grpcio`
+ * [Ruby](src/ruby): `gem install grpc`
+
+
+#Pre-requisites
+
+##Linux
+
+```sh
+ $ [sudo] apt-get install build-essential autoconf libtool
+```
+
+##Mac OSX
+
+For a Mac system, git is not available by default. You will first need to
+install Xcode from the Mac AppStore and then run the following command from a
+terminal:
+
+```sh
+ $ [sudo] xcode-select --install
+```
+
+#Build from Source
+
+For developers who are interested to contribute, here is how to compile the
+gRPC C Core library.
+
+```sh
+ $ git clone https://github.com/grpc/grpc.git
+ $ cd grpc
+ $ git submodule update --init
+ $ make 
+ $ [sudo] make install
+```

README.md

@@ -13,7 +13,7 @@ You can find more detailed documentation and examples in the [doc](doc) and [exa
 
 #Installation
 
-See [grpc/INSTALL](INSTALL) for installation instructions for various platforms.
+See [INSTALL](INSTALL.md) for installation instructions for various platforms.
 
 #Repository Structure & Status
 

doc/interop-test-descriptions.md

@@ -2,9 +2,8 @@ Interoperability Test Case Descriptions
 =======================================
 
 Client and server use
-[test.proto](https://github.com/grpc/grpc/blob/master/test/proto/test.proto)
-and the [gRPC over HTTP/2 v2
-protocol](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md).
+[test.proto](../src/proto/grpc/testing/test.proto)
+and the [gRPC over HTTP/2 v2 protocol](./PROTOCOL-HTTP2.md).
 
 Client
 ------

examples/README.md

@@ -447,4 +447,4 @@ $ greeter_client
 ## Read more!
 
 - You can find links to language-specific tutorials, examples, and other docs in each language's [quick start](#quickstart).
-- [gRPC Authentication Support](doc/grpc-auth-support.md) introduces authentication support in gRPC with supported mechanisms and examples.
+- [gRPC Authentication Support](http://www.grpc.io/docs/guides/auth.html) introduces authentication support in gRPC with supported mechanisms and examples.

examples/cpp/README.md

@@ -2,7 +2,7 @@
 
 ## Installation
 
-To install gRPC on your system, follow the instructions [here](../../INSTALL).
+To install gRPC on your system, follow the instructions [here](../../INSTALL.md).
 
 ## Hello C++ gRPC!
 

examples/cpp/cpptutorial.md

@@ -91,7 +91,7 @@ message Point {
 
 Next we need to generate the gRPC client and server interfaces from our .proto service definition. We do this using the protocol buffer compiler `protoc` with a special gRPC C++ plugin.
 
-For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL) first):
+For simplicity, we've provided a [makefile](route_guide/Makefile) that runs `protoc` for you with the appropriate plugin, input, and output (if you want to run this yourself, make sure you've installed protoc and followed the gRPC code [installation instructions](../../INSTALL.md) first):
 
 ```shell
 $ make route_guide.grpc.pb.cc route_guide.pb.cc

examples/cpp/helloworld/README.md

@@ -2,7 +2,7 @@
 
 ### Install gRPC
 Make sure you have installed gRPC on your system. Follow the instructions here:
-[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL).
+[https://github.com/grpc/grpc/blob/master/INSTALL](../../../INSTALL.md).
 
 ### Get the tutorial source code
 

examples/node/README.md

@@ -20,7 +20,7 @@ TRY IT!
  - Run the server
 
    ```sh
-   $ # from this directory (grpc_common/node).
+   $ # from this directory
    $ node ./greeter_server.js &
    ```
 

src/core/channel/client_channel.c

@@ -251,7 +251,7 @@ static void cc_start_transport_op(grpc_exec_ctx *exec_ctx,
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
-  GPR_ASSERT(op->set_accept_stream == NULL);
+  GPR_ASSERT(op->set_accept_stream == false);
   if (op->bind_pollset != NULL) {
     grpc_pollset_set_add_pollset(exec_ctx, chand->interested_parties,
                                  op->bind_pollset);

src/core/channel/client_uchannel.c

@@ -107,7 +107,7 @@ static void cuc_start_transport_op(grpc_exec_ctx *exec_ctx,
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
-  GPR_ASSERT(op->set_accept_stream == NULL);
+  GPR_ASSERT(op->set_accept_stream == false);
   GPR_ASSERT(op->bind_pollset == NULL);
 
   if (op->on_connectivity_state_change != NULL) {

src/core/surface/server.c

@@ -407,8 +407,15 @@ static void destroy_channel(grpc_exec_ctx *exec_ctx, channel_data *chand) {
   maybe_finish_shutdown(exec_ctx, chand->server);
   chand->finish_destroy_channel_closure.cb = finish_destroy_channel;
   chand->finish_destroy_channel_closure.cb_arg = chand;
-  grpc_exec_ctx_enqueue(exec_ctx, &chand->finish_destroy_channel_closure, true,
-                        NULL);
+
+  grpc_transport_op op;
+  memset(&op, 0, sizeof(op));
+  op.set_accept_stream = true;
+  op.on_consumed = &chand->finish_destroy_channel_closure;
+  grpc_channel_next_op(exec_ctx,
+                       grpc_channel_stack_element(
+                           grpc_channel_get_channel_stack(chand->channel), 0),
+                       &op);
 }
 
 static void finish_start_new_rpc(grpc_exec_ctx *exec_ctx, grpc_server *server,
@@ -971,7 +978,8 @@ void grpc_server_setup_transport(grpc_exec_ctx *exec_ctx, grpc_server *s,
 
   GRPC_CHANNEL_INTERNAL_REF(channel, "connectivity");
   memset(&op, 0, sizeof(op));
-  op.set_accept_stream = accept_stream;
+  op.set_accept_stream = true;
+  op.set_accept_stream_fn = accept_stream;
   op.set_accept_stream_user_data = chand;
   op.on_connectivity_state_change = &chand->channel_connectivity_changed;
   op.connectivity_state = &chand->connectivity_state;

src/core/transport/chttp2/internal.h

@@ -358,6 +358,9 @@ struct grpc_chttp2_transport {
     /** connectivity tracking */
     grpc_connectivity_state_tracker state_tracker;
   } channel_callback;
+
+  /** Transport op to be applied post-parsing */
+  grpc_transport_op *post_parsing_op;
 };
 
 typedef struct {

src/core/transport/chttp2_transport.c

@@ -432,6 +432,14 @@ static void close_transport_locked(grpc_exec_ctx *exec_ctx,
     if (t->ep) {
       allow_endpoint_shutdown_locked(exec_ctx, t);
     }
+
+    /* flush writable stream list to avoid dangling references */
+    grpc_chttp2_stream_global *stream_global;
+    grpc_chttp2_stream_writing *stream_writing;
+    while (grpc_chttp2_list_pop_writable_stream(
+        &t->global, &t->writing, &stream_global, &stream_writing)) {
+      GRPC_CHTTP2_STREAM_UNREF(exec_ctx, stream_global, "chttp2_writing");
+    }
   }
 }
 
@@ -951,12 +959,10 @@ void grpc_chttp2_ack_ping(grpc_exec_ctx *exec_ctx,
   unlock(exec_ctx, t);
 }
 
-static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
-                                 grpc_transport_op *op) {
-  grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt;
-  int close_transport = 0;
-
-  lock(t);
+static void perform_transport_op_locked(grpc_exec_ctx *exec_ctx,
+                                        grpc_chttp2_transport *t,
+                                        grpc_transport_op *op) {
+  bool close_transport = false;
 
   grpc_exec_ctx_enqueue(exec_ctx, op->on_consumed, true, NULL);
 
@@ -975,8 +981,8 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
     close_transport = !grpc_chttp2_has_streams(t);
   }
 
-  if (op->set_accept_stream != NULL) {
-    t->channel_callback.accept_stream = op->set_accept_stream;
+  if (op->set_accept_stream) {
+    t->channel_callback.accept_stream = op->set_accept_stream_fn;
     t->channel_callback.accept_stream_user_data =
         op->set_accept_stream_user_data;
   }
@@ -997,15 +1003,30 @@ static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
     close_transport_locked(exec_ctx, t);
   }
 
-  unlock(exec_ctx, t);
-
   if (close_transport) {
-    lock(t);
     close_transport_locked(exec_ctx, t);
-    unlock(exec_ctx, t);
   }
 }
 
+static void perform_transport_op(grpc_exec_ctx *exec_ctx, grpc_transport *gt,
+                                 grpc_transport_op *op) {
+  grpc_chttp2_transport *t = (grpc_chttp2_transport *)gt;
+
+  lock(t);
+
+  /* If there's a set_accept_stream ensure that we're not parsing
+     to avoid changing things out from underneath */
+  if (t->parsing_active && op->set_accept_stream) {
+    GPR_ASSERT(t->post_parsing_op == NULL);
+    t->post_parsing_op = gpr_malloc(sizeof(*op));
+    memcpy(t->post_parsing_op, op, sizeof(*op));
+  } else {
+    perform_transport_op_locked(exec_ctx, t, op);
+  }
+
+  unlock(exec_ctx, t);
+}
+
 /*******************************************************************************
  * INPUT PROCESSING
  */
@@ -1401,6 +1422,13 @@ static void recv_data(grpc_exec_ctx *exec_ctx, void *tp, bool success) {
     /* handle higher level things */
     grpc_chttp2_publish_reads(exec_ctx, transport_global, transport_parsing);
     t->parsing_active = 0;
+    /* handle delayed transport ops (if there is one) */
+    if (t->post_parsing_op) {
+      grpc_transport_op *op = t->post_parsing_op;
+      t->post_parsing_op = NULL;
+      perform_transport_op_locked(exec_ctx, t, op);
+      gpr_free(op);
+    }
     /* if a stream is in the stream map, and gets cancelled, we need to ensure
      * we are not parsing before continuing the cancellation to keep things in
      * a sane state */

src/core/transport/transport.h

@@ -123,7 +123,7 @@ typedef struct grpc_transport_stream_op {
 
 /** Transport op: a set of operations to perform on a transport as a whole */
 typedef struct grpc_transport_op {
-  /** called when processing of this op is done */
+  /** Called when processing of this op is done. */
   grpc_closure *on_consumed;
   /** connectivity monitoring - set connectivity_state to NULL to unsubscribe */
   grpc_closure *on_connectivity_state_change;
@@ -138,9 +138,13 @@ typedef struct grpc_transport_op {
   grpc_status_code goaway_status;
   gpr_slice *goaway_message;
   /** set the callback for accepting new streams;
-      this is a permanent callback, unlike the other one-shot closures */
-  void (*set_accept_stream)(grpc_exec_ctx *exec_ctx, void *user_data,
-                            grpc_transport *transport, const void *server_data);
+      this is a permanent callback, unlike the other one-shot closures.
+      If true, the callback is set to set_accept_stream_fn, with its
+      user_data argument set to set_accept_stream_user_data */
+  bool set_accept_stream;
+  void (*set_accept_stream_fn)(grpc_exec_ctx *exec_ctx, void *user_data,
+                               grpc_transport *transport,
+                               const void *server_data);
   void *set_accept_stream_user_data;
   /** add this transport to a pollset */
   grpc_pollset *bind_pollset;

src/objective-c/GRPCClient/private/GRPCCompletionQueue.h

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

src/objective-c/GRPCClient/private/GRPCCompletionQueue.m

@@ -1,6 +1,6 @@
 /*
  *
- * Copyright 2015, Google Inc.
+ * Copyright 2015-2016, Google Inc.
  * All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without

src/python/grpcio/tests/_runner.py

@@ -1,4 +1,4 @@
-# Copyright 2015, Google Inc.
+# Copyright 2015-2016, Google Inc.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -143,10 +143,17 @@ class Runner(object):
 
   def run(self, suite):
     """See setuptools' test_runner setup argument for information."""
+    # only run test cases with id starting with given prefix
+    testcase_filter = os.getenv('GPRC_PYTHON_TESTRUNNER_FILTER')
+    filtered_cases = []
+    for case in _loader.iterate_suite_cases(suite):
+      if not testcase_filter or case.id().startswith(testcase_filter):
+        filtered_cases.append(case)
+
     # Ensure that every test case has no collision with any other test case in
     # the augmented results.
     augmented_cases = [AugmentedCase(case, uuid.uuid4())
-                       for case in _loader.iterate_suite_cases(suite)]
+                       for case in filtered_cases]
     case_id_by_case = dict((augmented_case.case, augmented_case.id)
                            for augmented_case in augmented_cases)
     result_out = StringIO.StringIO()

src/python/grpcio/tests/tests.json

@@ -0,0 +1,62 @@
+[
+  "_base_interface_test.AsyncEasyTest", 
+  "_base_interface_test.AsyncPeasyTest", 
+  "_base_interface_test.SyncEasyTest", 
+  "_base_interface_test.SyncPeasyTest", 
+  "_beta_features_test.BetaFeaturesTest", 
+  "_beta_features_test.ContextManagementAndLifecycleTest", 
+  "_channel_test.ChannelTest", 
+  "_connectivity_channel_test.ChannelConnectivityTest", 
+  "_core_over_links_base_interface_test.AsyncEasyTest", 
+  "_core_over_links_base_interface_test.AsyncPeasyTest", 
+  "_core_over_links_base_interface_test.SyncEasyTest", 
+  "_core_over_links_base_interface_test.SyncPeasyTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_crust_over_core_over_links_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.DynamicInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.DynamicInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.DynamicInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.GenericInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.GenericInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.GenericInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_face_interface_test.MultiCallableInvokerBlockingInvocationInlineServiceTest", 
+  "_face_interface_test.MultiCallableInvokerEventInvocationSynchronousEventServiceTest", 
+  "_face_interface_test.MultiCallableInvokerFutureInvocationAsynchronousEventServiceTest", 
+  "_implementations_test.ChannelCredentialsTest", 
+  "_insecure_interop_test.InsecureInteropTest", 
+  "_intermediary_low_test.CancellationTest", 
+  "_intermediary_low_test.EchoTest", 
+  "_intermediary_low_test.ExpirationTest", 
+  "_intermediary_low_test.LonelyClientTest", 
+  "_later_test.LaterTest", 
+  "_logging_pool_test.LoggingPoolTest", 
+  "_lonely_invocation_link_test.LonelyInvocationLinkTest", 
+  "_low_test.HangingServerShutdown", 
+  "_low_test.InsecureServerInsecureClient", 
+  "_not_found_test.NotFoundTest", 
+  "_sanity_test.Sanity", 
+  "_secure_interop_test.SecureInteropTest", 
+  "_transmission_test.RoundTripTest", 
+  "_transmission_test.TransmissionTest", 
+  "_utilities_test.ChannelConnectivityTest", 
+  "beta_python_plugin_test.PythonPluginTest", 
+  "cygrpc_test.InsecureServerInsecureClient", 
+  "cygrpc_test.SecureServerSecureClient", 
+  "cygrpc_test.TypeSmokeTest"
+]

src/python/grpcio/tests/unit/_sanity/__init__.py

@@ -0,0 +1,30 @@
+# Copyright 2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+

src/python/grpcio/tests/unit/_sanity/_sanity_test.py

@@ -0,0 +1,53 @@
+# Copyright 2016, Google Inc.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+#     * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
+# copyright notice, this list of conditions and the following disclaimer
+# in the documentation and/or other materials provided with the
+# distribution.
+#     * Neither the name of Google Inc. nor the names of its
+# contributors may be used to endorse or promote products derived from
+# this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+import json
+import unittest
+
+import tests
+
+
+class Sanity(unittest.TestCase):
+
+  def testTestsJsonUpToDate(self):
+    """Autodiscovers all test suites and checks that tests.json is up to date"""
+    loader = tests.Loader()
+    loader.loadTestsFromNames(['tests'])
+    test_suite_names = [
+        test_case_class.id().rsplit('.', 1)[0]
+        for test_case_class in tests._loader.iterate_suite_cases(loader.suite)]
+    test_suite_names = sorted(set(test_suite_names))
+
+    with open('src/python/grpcio/tests/tests.json') as tests_json_file:
+      tests_json = json.load(tests_json_file)
+    self.assertListEqual(test_suite_names, tests_json)
+
+
+if __name__ == '__main__':
+  unittest.main(verbosity=2)

tools/README.md

@@ -1,7 +1,5 @@
 buildgen: template renderer for our build system.
 
-distpackages: script to generate debian packages.
-
 distrib: scripts to distribute language-specific packages.
 
 dockerfile: Docker files to test gRPC.
@@ -12,6 +10,4 @@ gce: scripts to help setup testing infrastructure on GCE.
 
 jenkins: support for running tests on Jenkins.
 
-profile_analyzer: pretty printer for gRPC profiling data.
-
 run_tests: scripts to run gRPC tests in parallel.

tools/run_tests/build_node.bat

@@ -27,4 +27,16 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-npm install --build-from-source
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
+
+del /f /q BUILD || rmdir build /s /q
+
+call npm install --build-from-source
+
+@rem delete the redundant openssl headers
+for /f "delims=v" %%v in ('node --version') do (
+  rmdir "%USERPROFILE%\.node-gyp\%%v\include\node\openssl" /S /Q
+)
+
+@rem rebuild, because it probably failed the first time
+call npm install --build-from-source

tools/run_tests/build_python.sh

@@ -45,3 +45,6 @@ export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1
 tox --notest
 
 $ROOT/.tox/py27/bin/python $ROOT/setup.py build
+$ROOT/.tox/py27/bin/python $ROOT/setup.py build_py
+$ROOT/.tox/py27/bin/python $ROOT/setup.py build_ext --inplace
+$ROOT/.tox/py27/bin/python $ROOT/setup.py gather --test

tools/run_tests/post_test_node.bat

@@ -1,30 +0,0 @@
-@rem Copyright 2016, Google Inc.
-@rem All rights reserved.
-@rem
-@rem Redistribution and use in source and binary forms, with or without
-@rem modification, are permitted provided that the following conditions are
-@rem met:
-@rem
-@rem     * Redistributions of source code must retain the above copyright
-@rem notice, this list of conditions and the following disclaimer.
-@rem     * Redistributions in binary form must reproduce the above
-@rem copyright notice, this list of conditions and the following disclaimer
-@rem in the documentation and/or other materials provided with the
-@rem distribution.
-@rem     * Neither the name of Google Inc. nor the names of its
-@rem contributors may be used to endorse or promote products derived from
-@rem this software without specific prior written permission.
-@rem
-@rem THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-@rem "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-@rem LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-@rem A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-@rem OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-@rem SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-@rem LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-@rem DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-@rem THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-@rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-@rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-rmdir node_modules /S /Q

tools/run_tests/pre_build_node.bat

@@ -27,13 +27,8 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-@rem Expire cache after 1 week
-npm update --cache-min 604800
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
 
-npm install node-gyp-install
-.\node_modules\.bin\node-gyp-install.cmd
+@rem Expire cache after 1 week
+call npm update --cache-min 604800
 
-@rem delete the redundant openssl headers
-for /f "delims=v" %%v in ('node --version') do (
-  rmdir "%HOMEDRIVE%%HOMEPATH%\.node-gyp\%%v\include\node\openssl" /S /Q
-)

tools/run_tests/run_interop_tests.py

@@ -60,6 +60,8 @@ _SKIP_COMPRESSION = ['large_compressed_unary',
 _SKIP_ADVANCED = ['custom_metadata', 'status_code_and_message',
                   'unimplemented_method']
 
+_TEST_TIMEOUT = 3*60
+
 class CXXLanguage:
 
   def __init__(self):
@@ -459,7 +461,7 @@ def cloud_to_prod_jobspec(language, test_case, server_host_name,
           environ=environ,
           shortname='%s:%s:%s:%s' % (suite_name, server_host_name, language,
                                      test_case),
-          timeout_seconds=90,
+          timeout_seconds=_TEST_TIMEOUT,
           flake_retries=5 if args.allow_flakes else 0,
           timeout_retries=2 if args.allow_flakes else 0,
           kill_handler=_job_kill_handler)
@@ -495,7 +497,7 @@ def cloud_to_cloud_jobspec(language, test_case, server_name, server_host,
           environ=environ,
           shortname='cloud_to_cloud:%s:%s_server:%s' % (language, server_name,
                                                         test_case),
-          timeout_seconds=90,
+          timeout_seconds=_TEST_TIMEOUT,
           flake_retries=5 if args.allow_flakes else 0,
           timeout_retries=2 if args.allow_flakes else 0,
           kill_handler=_job_kill_handler)

tools/run_tests/run_node.bat

@@ -27,6 +27,7 @@
 @rem (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 @rem OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
+set PATH=%PATH%;C:\Program Files\nodejs\;%APPDATA%\npm
 set JUNIT_REPORT_PATH=src\node\report.xml
 set JUNIT_REPORT_STACK=1
 .\node_modules\.bin\mocha.cmd --reporter mocha-jenkins-reporter --timeout 8000 src\node\test

tools/run_tests/run_python.sh

@@ -42,7 +42,12 @@ export LDFLAGS="-L$ROOT/libs/$CONFIG"
 export GRPC_PYTHON_BUILD_WITH_CYTHON=1
 export GRPC_PYTHON_ENABLE_CYTHON_TRACING=1
 
-tox
+if [ "$CONFIG" = "gcov" ]
+then
+  tox
+else
+  $ROOT/.tox/py27/bin/python $ROOT/setup.py test
+fi
 
 mkdir -p $ROOT/reports
 rm -rf $ROOT/reports/python-coverage

tools/run_tests/run_tests.py

@@ -290,10 +290,7 @@ class NodeLanguage(object):
       return [['tools/run_tests/build_node.sh', self.node_version]]
 
   def post_tests_steps(self):
-    if self.platform == 'windows':
-      return [['tools\\run_tests\\post_test_node.bat']]
-    else:
-      return []
+    return []
 
   def makefile_name(self):
     return 'Makefile'
@@ -353,15 +350,27 @@ class PythonLanguage(object):
     _check_compiler(self.args.compiler, ['default'])
 
   def test_specs(self):
+    # load list of known test suites
+    with open('src/python/grpcio/tests/tests.json') as tests_json_file:
+      tests_json = json.load(tests_json_file)
     environment = dict(_FORCE_ENVIRON_FOR_WRAPPERS)
     environment['PYVER'] = '2.7'
-    return [self.config.job_spec(
-        ['tools/run_tests/run_python.sh'],
-        None,
-        environ=environment,
-        shortname='py.test',
-        timeout_seconds=15*60
-    )]
+    if self.config.build_config != 'gcov':
+      return [self.config.job_spec(
+          ['tools/run_tests/run_python.sh'],
+          None,
+          environ=dict(environment.items() +
+                       [('GPRC_PYTHON_TESTRUNNER_FILTER', suite_name)]),
+          shortname='py.test.%s' % suite_name,
+          timeout_seconds=5*60)
+          for suite_name in tests_json]
+    else:
+      return [self.config.job_spec(['tools/run_tests/run_python.sh'],
+                                   None,
+                                   environ=environment,
+                                   shortname='py.test.coverage',
+                                   timeout_seconds=15*60)]
+
 
   def pre_build_steps(self):
     return []