This follows up ae9c728 — which implemented Phase 1 — but is currently blocked awaiting the apollo-rs primitives.

Goal

Follow up with Phase 2 of the project, once we have apollo-rs.

Phase 1 (Already completed / BEFORE THIS ISSUE / Stopgap)

1. apollo-rs will provide an AST API
2. Parse schema using apollo-rs at existing schema load point, and pass into federated execution.
3. Selection construction will currently check __typename for an exact match against the query plan. This will be relaxed to also include subtypes.
4. We will navigate the AST manually to check subtypes.

Phase 2 (THIS ISSUE)

1. apollo-rs will eventually provide an higher-level intermediary representation and API. At minimum the following features are required:
   1. Get a GraphQL type by name
   2. isSubtypeOf method to check if a type is a subtype of another type.
2. Rework the router code to use the HIR.

This is issue should track the implementation of Phase 2

Requirements

This is currently blocked – pending the arrival of this functionality landing in another Federation repository — but this is just a tracking item to make sure we pick it up

In theory, this should be as simple as updating the crate to use a newer crate. Currently, that crate is harmonizer, but depending on if we de-couple that into distinct "composer" and "query planner" crates (which I personally recommend), it may be slightly more involved. (Perhaps the interface changes slightly.) It is now just a matter of updating the router-bridge, which is a dedicated package after apollographql/federation#1090.

Anyhow, I suspect this is a size/small.

Success criteria

Under the hood the requirement is that we ensure we're using the new @apollo/[email protected] npm package, but that's an implementation detail of the crate. It should be verified though!

Describe the bug
When passing in variables their types should be validated against the schema and the entire request rejected if they are incorrect.

Requires schema parsing to complete.

Adds FTV1 support.

A new open telemetry exporter has been added that will convert regular traces to Apollo traces.

A buffer of spans is collected on the server side which will retain spans until the root request span is completed.
Once a request is completed the trace will be reconstructed and sent to Apollo.

Span attributes that are only relevant to Apollo tracing are prefixed with apollo_private. and are filtered out of other APM data.

@glasser Has given some guidance on how we should improve tracing, but this'll be left to followup tickets as this PR is large and has been ongoing for a significant period.

#1728.
#1729.
As an aside, this PR demonstrates that spans can be used for Apollo tracing, and that we could move to a native Otel based solution in future.

Help explain the direction we're headed and what we've got in mind.

Description

This is perhaps something that will require out-of-tree extensions or more extensive YAML configuration, as it asks for the ability to run conditionals on the origin. E.g.:

• If the domain ends with zyx.com #965
• If the path includes /admin/ #1444
• Using Rhai script

Description

The Router should be able to emit metrics that enable gauges which show the server's performance. As a baseline suggestion to our implementors, we might suggest routing them through the Open Telemetry Collector as a central point for Traces, Metrics and Logs.

Some variables to consider in the implementation include:

Metric candidates

• Request response time
• Request totals
• Request error totals
• Downstream response time per service
• Downstream error total
• Downstream total requests per service

Metric attribute candidates

Each metric candidate may also have metric attributes attached to it. Some plausible options here include:

• Apollo Studio Graph Ref for each metric
• Client ID for each metric

Metric format candidates

• Prometheus
• OTLP

Describe the solution you'd like
now that apollo-rs is available in the list of dependencies to parse the schema, we should also use it at the query planning stage, to compare its results (and run time) with the JS one. At first in a non blocking way, only reporting differences, then gradually replacing the JS code.

I've started that work in a branch last week, I'm now updating it to the public apollo-rs release

Requirements

TBD. An agent which communicates with Studio and acts as the foundation for operation (signature-based), field (shape) and latency (trace) stats to power Apollo Studio functionality.

Relevant Subtasks / Links

These are likely components/candidates of this agent!

• #442
• #67
• #70
• #127
• Tracing support (incl. Resolver timing; might be implemented via opentelemetry tracing or #74)

• Blocked by https://github.com/apollographql/router-perf-testing/issues/11

Automated daily performance tests.
First cut does not need to be complex, create something that we can add to.

Requirements

We need to be able to report the currently active schema to Apollo Studio. There's some existing art and a TypeScript reference implementation to follow along with here, so might be worth looking at, e.g., apollographql/apollo-server#5187 for inspiration and talking points in building out the design.

Describe the solution you'd like
when running benchmarks with Jaeger as trace collector, opentelemetry generates a lot of errors, of two kinds:

• thrift agent failed with message too long
• OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full

To understand the errors, here's how the current tracing system works:

• tracing spans are generated by the tracing crate
• there's an opentelemetry subscriber that receives the span
• a batch span processor accumulates spans and sends them to the exporter once enough spans have accumulated, or a ticker triggered
• opentelemetry-jaeger converts the opentelemetry spans to the jaeger format then calls the uploader
• the uploader serializes the spans to thrift then sends an UDP message https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-jaeger/src/exporter/uploader.rs#L48-L56https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-jaeger/src/exporter/agent.rs#L72-L92

So the first error message happens because when the batch is sent, we have no way of knowing if it will be too large for a single UDP packet (the default limit in the crate is 65000 bytes). This can be solved in two ways:

• reducing batch size. The appropriate value for the batch size will depend on the traffic: the batch size is the number of top level spans, so if we have a few spans with a lot of subspans (like a query with a lot of subqueries) we could be under the batch limit but above the UDP packet limit. The batch size and other options can be managed with environment variables defined here: https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry/src/sdk/trace/span_processor.rs#L48-L63
• using the auto_split parameter that was added recently: open-telemetry/opentelemetry-rust@a0899d9#diff-05c33517a7162e2e901acc0d121d9c1cfa701baf2aee85148c7b7097289b4dc2 it tries to split the batch to make smaller packets https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-jaeger/src/exporter/agent.rs#L177-L211 (this is not really vectored IO, oh well)

In my tests auto_split works most of the time. Except for very large spans, that would be larger than a UDP packet when serialized. Is there a way to send a span in multiple pieces and let jaeger aggregate them?

The second error happens because the jaeger exporter is spending too much time serializing the batch, especially in the build_span_tags function: https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-jaeger/src/exporter/mod.rs#L693-L716 (see https://gist.github.com/Geal/0b9588bdeaa05e1494636e63ef431f96 for example code reproducing our tracing pipeline and a flamegraph, the highest column is the serialization task). This function could be optimized a bit, but I don't know how much.

We're bound to run into more issues with a large number of traces, so we should investigate sampling:
https://docs.rs/opentelemetry/0.16.0/opentelemetry/sdk/trace/enum.Sampler.html

it can be set up when creating the pipeline:

let tracer = opentelemetry_jaeger::new_pipeline()
 .with_trace_config(
            trace::config()
                .with_sampler(Sampler::AlwaysOn)
) .install_batch(opentelemetry::runtime::Tokio)?;

Sampling can be controlled by a custom implementation of ShouldSample. unfortunately, the sampling decision is done before we go through the span, probably to avoid collecting too much data. I'd like to have heavy sampling for successful queries (since they would be the most frequent ones), but no sampling at all for failing queries (we want to know about those). That might require patching opentelemetry.

Is your feature request related to a problem? Please describe.
We would like to send the complete query with tracing spans (could be used by Studio for analysis). Unfortunately, queries can contain inline sensitive data in input arguments, so we cannot send them as is.

Describe the solution you'd like
We need a way to sanitize queries and remove private data. The future query planner using apollo-rs could be used to recognize raw input values, replace them with variables in the query, and put the values in the variables. That is apparently possible but we don't know what impact this will have on our users

Describe alternatives you've considered
The current way in the server is to modify the AST before sending the usage report: https://github.com/apollographql/apollo-tooling/blob/b1bd747861bcdb733a5e357c019885a6c0293ec7/packages/apollo-graphql/src/operationId.ts#L69-L78

Additional context
We might need to make query reporting more configurable, with options to send or not the query depending on the operation, or deciding whether to send variables, as is done in apollo-server https://www.apollographql.com/docs/apollo-server/api/plugin/usage-reporting/

Requirements

We should make sure we're including packages with compatible licenses for the project. For simplicity in implementing, let's merely say MIT to start with, but we can consider adding more in the future.

One suggestion for a tool is cargo-deny.

We currently don't run any normalization, which means the router won't be able to hit it's caches (or NaiveIntrospection alltogether) for queries that have different indents.

Requirements

This replaces the behavior of redirecting directly to studio with the more complete implementation which includes a landing-page that is served locally and offers a redirect to Studio. This will match the behavior of Apollo Server and Apollo Gateway today, exactly. This offers more transparency to the user to understand what about to happen (the redirect) and allows them to optionally make the behavior sticky (on account of a browser cookie) for future requests.

• Rather than redirect on the Router's configured endpoint, this replaces that redirect with serving of HTML boilerplate (e.g., with content-type: text/html) when the appropriate accept header with a satisfying text/html value is met.
• Renders the same boilerplate HTML that Apollo Server uses, which the implementation for should be found here

Effectively, this should produce this experience:

Will Resolve #380

Pre-Reqs

• #47

Reqs

Make the output match the expected shape.
This is not true Value completion.

Create docker images and publish them.
Consider using https://github.com/GoogleContainerTools/distroless

While working on the release workflow I had to ignore some tests for OSX because they were not working. I believe it is possible the hot reloading of the configuration and schema is not working on OSX.

cc @BrynCooke

• Add field usage to spaceport.
• Extract exporter to metrics exporter
• Send field usage and fields stats to metrics exporter

Description

This is an overall tracking issue for the documentation and documentation infrastructure that we need to put together before general availability (GA). We can do without this for pre-alpha and for much of our alpha and preview phases, but not for GA.

Components

• Setup documentation deployment (gatsby to netlify)
• More TBD

Current plan

Implement defer only for now, in the router, with some query planner modification.

@defer RFC

Related issues:

• #1178
• #1179
• #1180
• #1181
• #1219
• #1409
• #1469
• #1648
• #1219
• #1666
• #1659
• #1409
• #1426
• #1469
• #1544
• #1546
• #1548
• #1572
• #1575
• #1584
• #1588
• #1618
• apollographql/federation-rs#160
• apollographql/federation-rs#161
• #1656

To be defined:

• response stream format: right now the stream is apparent in all signatures, maybe we could have instead response types that holds the headers then a (generic) stream of graphql responses
• adapting telemetry: telemetry now registers one operation per response (deferred or not), that might not be what we want in the end. But also, telemetry does not really have a concept of deferred responses

Followup to the previous work on 5b36237

Comments on the introspection efforts raised the fact we're not quite aligned on what we would like the cors default setting to be, and what we would like to customize it wit.
[Allow any origin might ease use and setup but it might not be considered as the safe default option

Let's write our expected CORS behavior down, so I can add unit tests. We can of course revisit it anytime

On our infrastructure side, we have built API surface area (and corresponding UI) that supports showing the response headers of traces - however we don't currently send response headers as part of the trace. We can and should allow this, but much in the same way we do for request headers, we should consider additional data privacy concerns that need to be addressed with any implementation here.

Limit the amount of memory a request can use on a per request basis.
Prevent users from causing failure by exhausting memory.

Right now we support configuration reload through filesystem watch. This can be unreliable sometimes (missing udpdates), and it will try to reload files on each save even if we're not done modifying (the router should check the configuration file and schema before replacing, but that's another topic).

Should we support a signal to tell the router to reload its configuration? Specifically, SIGHUP is commonly accepted for that. That would fit well with tools like systemd and its ExecReload command

We want the Router to be able to run "Anywhere" but we need to start with a practical list of targets that we can ship and be cognizant of the constraints that some "nice to haves" would put on us.

Candidates

As a conversation starter, I suggest we have only Tier 1 target architectures as candidates. Lifting that list from the linked page, it's:

Note that this list does not include aarch64-unknown-linux-musl. That omission doesn't by itself preclude us from running on Alpine, though we are further constrained in this regard by our dependence on V8 for the query planner. This is only intended to be a constraint until the point that we can re-write the query planner in Rust natively, or until the rusty-v8 project can run more easily on MUSL. See this issue. It's surmountable, but it's probably more work than it's worth for experimental phases.

In Scope

TBD

Out of Scope

TBD

Is your feature request related to a problem? Please describe.
Users need to be able to perform limited processing on requests without using in tree extensions.
The advantages are that it hugely reduces the surface area that we need to support, and we can leverage https://github.com/tetratelabs/proxy-wasm-go-sdk

Describe the solution you'd like
Have a go creating a simple filter using https://github.com/tetratelabs/proxy-wasm-go-sdk
Demonstrate that there is a path for us to provide extensions specific to our the router.
Provide a simple description of what the user workflow would look like when creating and using such a filter.

Describe alternatives you've considered
In tree extensions - This will be difficult to support.
Hard coded functionality - We should not rule this out, but ideally we have one method for manipulating requests. We should have a set of bundled WASM filters that users can use.

PRs should auto run perf tests

currently we will generate a new graph from a schema, even if it is invalid: https://github.com/apollographql/router/blob/2cc92b110acf2f5858aad2854b11ccc9ceba84a8/crates/apollo-router/src/graph_factory.rs#L13-L20

The schema update feature should refuse invalid schema instead of replacing the existing one, with an error log (not a panic)

Create a comprehensive list of all errors.
Ensure that error messages have:

1. Information about what happened in a user friendly language.
2. Information about what to do about the error in user friendly language.

Sensitive information must not be leaked.
Consider metrics for infrastructure errors.

This is to make sure we consider our support for the GraphQL Specification specified conditional executable directives — @skip and @include.

query Hero($episode: Episode, $withFriends: Boolean!) {
  hero(episode: $episode) {
    name
    friends @include(if: $withFriends) {
      name
    }
  }
}

Not in scope

• apollographql/router#71

Is your feature request related to a problem? Please describe.
The response generated from subgraph responses does not match the query.
Examples with the federation demo:

field order

query ExampleQuery { me  {
  identifiant: id
  reviews { body }
 id 
} }

returns:

"data": {
    "moi": {
        "identifiant": "1",
        "__typename": "User",
        "id": "1",
        "reviews": [{
            "body": "Love it!"
        }, {
            "body": "Too expensive."
        }]
    }
}
}

The id field should appear after the reviews field, but because a first query to the accounts subgraph must be performed to obtained the id key, before requesting the review, the id is added to the response before the reviews

unnecessary data

query ExampleQuery {
 me  {
   reviews { body }
} }

returns

{
    "data": {
        "me": {
            "__typename": "User",
            "id": "1",
            "reviews": [{
                "body": "Love it!"
            }, {
                "body": "Too expensive."
            }]
        }
    }
}

The id field should not be there, but since it was used for the join, it was added by the response from the accounts subgraph.

These inconsistencies in the responses will block the work on integration testing #47, because they will generate a lot of differences between the gateway and router responses.

Describe the solution you'd like
The response should be created using the query plan, with only the required fields, in order, then subgraph responses would only be merged where it is necessary.

Describe alternatives you've considered
This work could be done as part of the future Rust query planner, but it looks like it can be done independently, only using the common query plan format.

I think I know what is happening. We are processing a serde_json::Value while actually we should create our own struct that parses the JSON properly. With a proper struct we will be able to manage parsing errors earlier and avoid complicating our code like this.

We're currently using a HashMap to cache introspection and queries. Which might not be the best thing to do (the default hashing function is SipHash 1-3, which is robust but not the fastest one.)

We might need to investigate for better hash functions, or caching crates alltogether.

Requirements

• WIP

It would be to have a scheduled workflow on the branch main that runs cargo update and then run all the test.

Depends on apollo-rs

We're currently forwarding subgraph errors to the router caller, which proves inconsistent when it comes to the error path, given the query a caller performed, and the query the router has made differ.

We need to have a look at what the Gateway does, and port it if it makes sense. we might uncover some additional things that would come in handy in the process.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Found renovate config warnings

Pending Approval

These branches will be created by Renovate only once you click their checkbox below.

• fix(deps): update cargo tracing packages (patch) (tracing, tracing-core, tracing-subscriber)
• chore(deps): update cargo tracing packages (minor) (opentelemetry, opentelemetry-aws, opentelemetry-datadog, opentelemetry-http, opentelemetry-jaeger, opentelemetry-otlp, opentelemetry-prometheus, opentelemetry-semantic-conventions, opentelemetry-stdout, opentelemetry-zipkin, opentelemetry_sdk, tracing-opentelemetry)
• 🔐 Create all pending approval PRs at once 🔐

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update actions/checkout digest to 0ad4b8f
• fix(deps): update cargo pre-1.0 packages (patch) (async-compression, async-trait, base64, basic-toml, chrono, cookie, criterion, displaydoc, http, libc, libfuzzer-sys, linkme, log, lru, parking_lot, pin-project-lite, prometheus, prost, prost-types, reqwest, rustls, schemars, serde_derive_default, tar, test-log, tikv-jemallocator, time, tokio-stream, toml, zip, zstd)
• chore(deps): update jaegertracing/all-in-one docker tag to v1.56.0
• chore(deps): update openzipkin/zipkin docker tag to v2.27.1
• chore(deps): update slack orb to v4.13.2
• chore(deps): update openzipkin/zipkin docker tag to v3.3.0
• chore(deps): update rust crate fred to v9
• fix(deps): update rust crate async-channel to v2
• fix(deps): update rust crate brotli to v5
• fix(deps): update rust crate zip to v1
• 🔐 Create all rate-limited PRs at once 🔐

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• fix(deps): update all non-major packages >= 1.0 (@apollo/server, anyhow, arc-swap, aws-config, aws-credential-types, aws-sigv4, aws-smithy-runtime-api, aws-types, brotli, bytes, camino, clap, dd-trace, flate2, indexmap, insta, jsonwebtoken, memchr, once_cell, regex, rhai, rust-embed, serde, serde_json, serial_test, similar, tempfile, thiserror, tokio, typescript, uuid, walkdir, zstd-safe)
• chore(deps): update rust docker tag to v1.77.2
• fix(deps): update rust crate http-serde to v2

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update rust crate apollo-parser to 0.7.7
• chore(deps): update cargo pre-1.0 packages (minor) (axum, base64, cargo-scaffold, env_logger, graphql_client, heck, hyper-rustls, jsonpath-rust, mockall, multimap, nu-ansi-term, reqwest, rustls, rustls-native-certs, serde_yaml, strum_macros, tokio-rustls, tokio-tungstenite, tonic, tonic-build, tower-http, wiremock)
• chore(deps): update rust crate http to v1
• fix(deps): update rust crate http-body to v1
• fix(deps): update rust crate hyper to v1
• fix(deps): update rust crate multer to v3
• fix(deps): update rust crate rustls-pemfile to v2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Distinct from Selective inbound header propagation #83, but for going back to the client (e.g., selectively picking headers from certain subgraphs to expose directly to the client, like a Set-Cookie header)

We should change those (in the 3 platform):

            - rust/build:
                # This is prefixed in the orb with 'cargo-'
                cache_version: v2-macos
                crate: --workspace --tests

To do:

• cargo build --workspace --tests --no-default-features --features otlp-tonic
• cargo build --workspace --tests --no-default-features --features otlp-tonic,tls
• cargo build --workspace --tests --no-default-features --features otlp-grpcio (not on windows)
• cargo build --workspace --tests --no-default-features --features otlp-http

Query plan has alias in selections format.
Currently we are not using this.

Headers propagation should be supported on a per service basis.
Add to the config file and FederatedGraph.

Describe the solution you'd like
Open telemetry works always.

Additional context
We recently had an uncaught regression as there were no integration tests for Otel.
We should take the time to add a test for each type of collector.

Describe the bug
maybe I'm missing something, but it looks like UpdateSchema messages change the schema in the state machine: https://github.com/apollographql/router/blob/07dfdddedc3e90cf6a7e0124ceb9f1a23446cae6/crates/apollo-router/src/state_machine.rs#L150-L172

but does not pass it to the HTTP server. It is only changed here: https://github.com/apollographql/router/blob/07dfdddedc3e90cf6a7e0124ceb9f1a23446cae6/crates/apollo-router/src/state_machine.rs#L193-L196

To Reproduce

At this line: https://github.com/apollographql/router/blob/07dfdddedc3e90cf6a7e0124ceb9f1a23446cae6/crates/apollo-router/src/lib.rs#L579

add the following:

SchemaKind::Instance(Schema::from_str("").unwrap())
                .into_stream()
                .boxed(),

This will first set up a server with the configuration and an empty schema, then try to update the schema. Any query to the router will fail because the schema used is still the empty one

We should generate this file based on our dependencies — perhaps using cargo-about?) — and include this file in the packaged distributions. Our LICENSE should also be included.

Is your feature request related to a problem? Please describe.
The TS server implementations provide a healthcheck endpoint and an onHealthCheck callback, we might want to provide one as well

Describe the solution you'd like
We could provide a healthcheck endpoint that returns a simple 200 status code and { status: 'pass' } for now.

Describe alternatives you've considered
As a follow up we could expose user defined callbacks ( impl Future<Output = ()> ) we could await with a timeout, or directives or anything, this would probably require deeper thoughts though.