What is the purpose of the post?

The purpose of this HIGHLY OPINIONATED post is to unpack what people are asking for when they say they want distributed tracing and why they may be able to get what they need more simply and easily.

What should it talk about?

The post will define distributed tracing and then break down what people are asking for when they say they need distributed tracing. More specifically, whether they need tracing for network topology graphs, for performance measurement and monitoring, or for audit purposes.

If this means they are asking for network topologies, it is a relatively simply thing to achieve and one that does not require inject tracer elements on every inch of your kubernetes real estate.

If this means measuring performance and monitoring, distributed tracing may actually be a less accurate method for monitoring than direct measurements that are not so sparsely sampled (and btw are far less expensive in terms of resources and hassles)

Lastly, we will consider distributed tracing for audit purpose and super complex microservice architectures - in which case, distributed tracing is a perfect use case.

Who is the target audience?

• Architect
• SRE
• Developers

Are there any assets?

Not required.

What are the dependencies?

None.

Relevant Examples / References

https://blog.buoyant.io/2016/05/17/distributed-tracing-for-polyglot-microservices/
https://eng.uber.com/distributed-tracing/
https://medium.com/opentracing/distributed-tracing-in-10-minutes-51b378ee40f1

There needs to be a set of steps that explain how to verify TLS is working between two services and what the reasons are.

Bug Report

What is the issue?

The service profile documentation for the Route object describes a responses field, while the linkerd profile --open-api command outputs a responseClasses field.

How can it be reproduced?

Run linkerd profile --open-api on a swagger yaml file.

Environment

• Kubernetes Version:
• Cluster Environment: (GKE, AKS, kops, ...)
• Host OS:
• Linkerd version: stable-2.2.0

• Write architecture + overview of how service discovery works with LD2.
• Explain how fallback occurs.

As prometheus comes with the control plane, it should not be hard to configure basic alerting by adding alertmanager to the mix after the fact.

The new architecture page is great. Any chance we can update this architecture diagram to include a pod for grafana as well?

For completeness we might also want to include the ca pod, although that's optional at this point, and might be better served by a TLS-specific diagram.

Link to the Proxy-Metrics in prometheus.md is dead and should link to https://github.com/linkerd/website/blob/master/linkerd.io/content/2/proxy-metrics.md instead.

• k8s api
• k8s version
• pre install cluster
• single namespace
• pre install
• control plane existence
• LD2 api
• service profile
• LD2 version (cli)
• LD2 version (control plane)
• data plane

linkerd2 disconnected installation notes

1. Download and install linkerd2, add to PATH

2. get images via linkerd install and upload to private registry

3. customize image location in linkerd install output

   registry=registry.mydomain.com/ ; linkerd install | sed 's|image:\ |image:\ '"$registry"'|g' | kubectl apply -f -

4. Internal Load Balancer

If you are following this guide you probably need internal only load balancers. Therefore, if following the emojivoto example
make sure to use the internal annotaion for your cloud. AWS example:

apiVersion: v1
kind: Service
metadata:
  name: web-svc
  annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
  namespace: emojivoto
spec:
  type: LoadBalancer
  selector:
    app: web-svc
  ports:
  - name: http
    port: 80
    targetPort: 80

Currently, there are only the latest version of the docs available. Between stable and edge releases, it is starting to be difficult to know what features are supported where. The way that Kubernetes does their docs versioning is great (branching, separate domain). I'd love to see something similar for the Linkerd docs.

/kind bug

https://versioncheck.linkerd.io/update has a redirect to a non-https url.
It looks like an https version of the url is also available but not part of the redirect chain.

(note location: http://versioncheck.linkerd.io/update/index.html

curl -I https://versioncheck.linkerd.io/update
HTTP/2 301
date: Thu, 23 Aug 2018 12:35:42 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d93f2e19bbe81933b17ee40ce6c061f1b1535027742; expires=Fri, 23-Aug-19 12:35:42 GMT; path=/; domain=.linkerd.io; HttpOnly
x-guploader-uploadid: AEnB2UrDXQUnzG-sg9ChsATKHDk1tEI5EXPXnADLKloz3IJVk8vE_wg7_nPinZLgd4x-fKkppxblfW21GRSLHxmYh4aCVmjESQ
location: http://versioncheck.linkerd.io/update/index.html
vary: Origin
expires: Thu, 23 Aug 2018 12:35:42 GMT
cache-control: private, max-age=0
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 44ed9edc09334415-BRU

A common point of confusion around service profiles is that the Host header needs to match. It can be hard to understand how to get that to work.

• Include a traffic generator.
• Explain why port-forward doesn't work, see linkerd/linkerd2#2352
• Suggest using curl from inside the cluster with -H.

While trying to explain to someone that run.linkerd.io/install safely installs the linkerd binary without potentially corrupting an existing one, i realized there's a subtle problem: Our mv may not be atomic.

On most modern OSes /tmp and $HOME exist on different devices, so that

is not guaranteed to be atomic. So if a user hits ^C during an install, they can be left with a linkerd binary that is partial.

If we change the mktemp to use a temp directory in the $HOME/.linkerd2, we can ensure that the mv is atomic; however, this would mean that the installer would have to be responsible for garbage-collecting these tmp directories.

I suppose it could also be okay to close this as wontfix if we provided safe alternatives (i.e. via homebrew, apt-get, etc).

• Document l5d-dst-override (see linkerd/linkerd2-proxy#173 and linkerd/linkerd2#1998).
• Update ingress-nginx with new annotation.
• Add traefik documentation for ingress.
• Link to service discovery architecture documentation (#97).

There is too much space in between an h2 and the subsequent paragraph. Makes the text hard to read. I assume this is an easy CSS change.

It would be really great to update https://linkerd.io/2/cli to include descriptions of the global CLI flags and environment variables that are used by the CLI. Specifically:

• --kubeconfig / $KUBECONFIG
• --linkerd-namespace / $LINKERD_NAMESPACE (see linkerd/linkerd2#1695)
• --verbose

In linkerd/linkerd2#2039 we're changing the way in which grafana dashboards are served when running the linkerd dashboard command. This has implications for users who want to expose the web/grafana frontends via ingress. We should make sure the documentation in this repo covers it, and we should update the linkerd2 katacoda scenarios as well.

The docs need to explain how to remove the proxies and control plane from your cluster after testing.

You have a typo on the https://linkerd.io/ frontpage (realiabilty -> reliability).

From linkerd/linkerd2#1880, services need to set an external traffic policy. This should be documented. Maybe on the ingress side for now?

What is the purpose of the post?

The purpose of this post is to explain what Retries and Timeouts are on Linkerd and how they are different than other similar implementations (relationship to Service Profiles)

What should it talk about?

This post will discuss the basic architecture and functionality of Timeouts and Retries, why they are important for improving reliability, and then provide a tutorial on how to use them.

Who is the target audience?

• [X ] Architect
• SRE
• Developers

Are there any assets?

Yes. We will need screenshot / code snippets / gifs showing how to make these two features work

What are the dependencies?

No

Relevant Examples / References

https://istio.io/docs/concepts/traffic-management/ (although we want something much briefer)

For users who install microk8s and then try to use linkerd, there are a couple issues:

• error configuring Kubernetes API client: invalid configuration: no configuration has been provided - microk8s.kubectl config view --raw > $HOME/.kube/config
• read udp 127.0.0.1:58124->127.0.0.53:53: read: connection refused - microk8s.enable dns && microk8s.stop && microk8s.start
• Post http://127.0.0.1:8080/api/v1/namespaces/linkerd/services/linkerd-controller-api:http/proxy/api/v1/SelfCheck: context deadline exceeded - microk8s.inspect

What is the purpose of the post?

The purpose of this post is to educate architects and SREs about how CRDs are used and how they can impact applications and architectures under the hood in meaningful ways.

What should it talk about?

The post will explain what are CRDs (custom resource definitions) in Kubernetes, what they do, and their benefits and weaknesses (specifically that CRDs are not "free lunches" - that using them in the wrong way or using too many of them can have significant consequeneces for application performance"

Who is the target audience?

• Architect
• SRE
• [] Developers

Are there any assets?

Ideally, this should have some assets or a brief screen shot series showing how to check for the number of CRDs or how to view CRDs being deployed

What are the dependencies?

None

Relevant Examples / References

https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
https://blog.heptio.com/an-introduction-to-extending-kubernetes-with-customresourcedefinitions-76deb675b27a

There needs to be a little bit more thought on where this will end up.

To the makefile, add version checks for:

• hugo
• scss

The ECS getting started guide says:

 Note that linkerd, consul-agent, and consul-registrator run on every ECS node.
 As of the writing of this guide, the ECS scheduler does not explicitly support this.
 Instead, we use an AWS Launch Configuration to bootstrap every ECS node with
 these three foundational services. We still boot these foundational services via an
 aws ecs start-task command, so they will be visible as running ECS containers.

ECS now supports daemon services via the DAEMON scheduling strategy: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html#service_scheduler

These docs can be updated to no longer use the launch configuration and instead use a daemon service. If I get time I'll take a stab at an initial pass to update them as it will make the linkerd daemon deployment much more stable and safe.

• What are service profiles?
• linkerd profile (swagger and proto)
• Debugging profile issues

linkerd -n emojivoto tap -o wide deploy/emoji | grep rt_route
linkerd -n emojivoto tap -o wide deploy/emoji | grep -v rt_route

What is the purpose of the post?

A tutorial to explain what the "Tap" command does in Linkerd2 and how to use it to improve observability / MTR reductions

What should it talk about?

• What exactly does "Tap" do and show?
• What use cases does it address? Why did Linkerd build it?
• What are the options for using "Tap" (CLI, dashboard)?
• How does Tap interact / enhance service profiles and per-route metrics?
• How does Linkerd use "Tap" internally?
• What are some other amazing things you might be able to do with 'Tap'?

Who is the target audience?

• Architect
• SRE
• Developers

Are there any assets?

This should take the form of a tutorial and include specific applicable CLI commands and sequences to expose the dashboard. Using GIFs or codescreens to display the requisite actions is recommended.

What are the dependencies?

TBD - depends on what the author thinks are appropriate here.

Relevant Examples / References

linkerd/linkerd2#1993
linkerd/linkerd2#2106
linkerd/linkerd2#1308

I want to create a docu pr for the new zipkin trace propagation plugin.
Where do I find it?

The README in this repo says 'https://github.com/linkerd/website/blob/master/slate-linkerd' which 404es.
The build file in website/api.linkerd.io/build states 'https://github.com/BuoyantIO/slate-linkerd.git' which is either private or nonexistent.

Thanks
Peter

• Mention breaking changes and required steps.
• Run through and modify upgrade instructions for the new release.
• Document 2.1 -> 2.2.
• Add a header for the 2.0 -> 2.1 upgrade instructions.
• Note the changes to inject and uninject.

• Proxy log level

What is the purpose of the post?

The purpose of this post is to explain how the Linkerd Auto-Inject functionality works and why we structured it this way in order to drive familiarity with the feature and help users adopt it.

What should it talk about?

The problems that mutating webhooks / auto-inject solves, why it is such a big problem in making Kubernetes usable, how we solved the problem and why we did it the way we did, and what this means for Linkerd users.

Who is the target audience?

• Architect
• SRE
• Developers

Are there any assets?

Yes! They are already created in the documentation.

What are the dependencies?

None

Relevant Examples / References

https://linkerd.io/2/proxy-injection/

Lots of good stuff in @klingerf's comment on this issue. This should be in the docs. linkerd/linkerd2#1453 (comment)

See linkerd/linkerd2#1655 for the details

• High level description of components.
• Discuss the queries that stat and grafana make to prometheus (see linkerd/linkerd2#1453 (comment)).
• Explain why you want to keep LD2 Prometheus and how to integrate into larger/existing systems. Include resource usage stats.
• Document the implications of getting rid of Grafana.
• Document how to use LD2 dashboards on another Grafana instance.
• Prometheus scrape configuration.
• How to configure prometheus (alert manager).
• Update proxy metrics to current state.

Add some documentation that shows how to expose the dashboard and grafana via ingress. This should explain how to do authentication as well (oauth2 proxy perhaps).

Getting ingress-nginx working with linkerd takes a little bit more configuration, see kubernetes/ingress-nginx#3171.

What is the purpose of the post?

A tutorial to explain what retries are and how to implement them for your own services.

What should it talk about?

• What problem do retries solve?
• How does linkerd implement retries?
• How does an end user add retries to their own project?

Who is the target audience?

• Architect
• SRE
• Developers

Are there any assets?

This should take the form of a tutorial and includes a sample application that readers are able to incrementally add retries to. It would be particularly valuable to package this up as a Katacoda scenario.

What are the dependencies?

None.

Right now we have a list of protocols that use server speaks first in the overview. That is less than ideal. It would be nice to get a full list of protocols together and their support (eg. http fully supported, X not fully supported but no changes and Y needs to be skipped).

Now that we have this fancy demo GIF, it might be nice to try replacing the huge Linkerd logo on the landing page with the gif. (Unless it looks silly)

One of the most common questions is "Why aren't there any metrics?". There should be a troubleshooting document that walks someone through all the steps required to understand why metrics aren't being collected.

See linkerd/linkerd2#2323 as an example of how that can be confusing.

As part of the updates suggested by linkerd/linkerd2#1820 and linkerd/linkerd2#1862, we need to update the documentation to explain how you opt-in.

It is pretty easy to introduce dead links, CI should check for internal inconsistencies as well as the search index. A tool to check for dead links:

https://github.com/wjdp/htmltest

• Add minimum possible PSP to security section (see #100)

The minimum PSP for linkerd:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: default
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
    apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'docker/default'
    apparmor.security.beta.kubernetes.io/defaultProfileName:  'runtime/default'
spec:
  privileged: false
  # Required to prevent escalations to root.
  allowPrivilegeEscalation: false
  allowedCapabilities:
    # The initContainer for LD2 requires this capability to operate. It is
    # unsafe and *should* be removed.
    - NET_ADMIN
    - NET_RAW
  requiredDropCapabilities:
    - SETPCAP
    - MKNOD
    - AUDIT_WRITE
    - CHOWN
    - DAC_OVERRIDE
    - FOWNER
    - FSETID
    - KILL
    - SETGID
    - SETUID
    - NET_BIND_SERVICE
    - SYS_CHROOT
    - SETFCAP
  # Allow core volume types.
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    # Assume that persistentVolumes set up by the cluster admin are safe to use.
    - 'persistentVolumeClaim'
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: 'RunAsAny'
    # The initContainer for LD2 requires running as root to install the
    # iptables rules. This *should* be set.
    # rule: 'MustRunAsNonRoot'
  seLinux:
    # This policy assumes the nodes are using AppArmor rather than SELinux.
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  readOnlyRootFilesystem: false

There's a great, structured way to show CLI commands and options now. This should be automatically generated by the Linkerd2 release process.

• Explain what security requirements LD2 has.
• Outline the differences between normal and --single-namespace.
• List the default, and minimal RBAC.

Hi all, I'm encountering a weird scenario while trying to upgrade linkerd from version stable-2.0.0 to stable-2.1.0 following https://linkerd.io/2/upgrade/:

When I update the linkerd CLI to 2.1.0, linkerd version outputs:

▶ linkerd version
Client version: stable-2.1.0
Server version: unavailable

If I roll back to the 2.0.0 CLI, it outputs the correct server version(2.0.0)

After discussing on linkerd slack, it was noted that it is expected behavior, given that the service name for the linkerd API changed from 2.0.0 to 2.1.0.

The issue is that this breaking change is not very clear in the docs. I guess we should add a note for users migrating from 2.0.0 or earlier to 2.1.0 or later, stating that the server version won't be available.

Thanks in advance!

What is the purpose of the post?

A tutorial to explain how you can expose the Linkerd2 dashboard securely

What should it talk about?

• What problem does exposing the dashboard solve?
• What use cases does it address?
• What are the various options for exposing the Linkerd2 dashboard?
  **What are the positives and negatives of each option?
• How does an end user expose the Linkerd2 dashboard in their own project?

Who is the target audience?

• Architect
• SRE
• Developers

Are there any assets?

This should take the form of a tutorial and include specific applicable CLI commands and sequences to expose the dashboard. Using GIFs or codescreens to display the requisite actions is recommended. .

What are the dependencies?

TBD - depends on what the author thinks are appropriate here.

Relevant Examples / References

https://stackoverflow.com/questions/39864385/how-to-access-expose-kubernetes-dashboard-service-outside-of-a-cluster

https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca

https://www.thehumblelab.com/deploying-kubernetes-dashboard-in-the-lab/