Comments (11)
This is a Tempo issue, I'll migrate it to grafana/tempo
from tempo.
As Mario mentioned in slack, your backend endpoint is most likely invalid. Can you share your config?
from tempo.
# -- Overrides the chart's name
nameOverride: ""
# -- Overrides the chart's computed fullname
fullnameOverride: ""
# -- Define the amount of instances
replicas: 1
# -- Number of old history to retain to allow rollback (If not set, default Kubernetes value is set to 10)
# revisionHistoryLimit: 1
# -- Annotations for the StatefulSet
annotations: {}
tempo:
readinessProbe:
httpGet:
path: /ready
port: http-traces
initialDelaySeconds: 30
timeoutSeconds: 1
image:
# -- The Docker registry
registry: ages/docker.io
# -- Docker image repository
repository: grafana/tempo
# -- Overrides the image tag whose default is the chart's appVersion
tag: "2.5.0"
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
updateStrategy: RollingUpdate
resources:
requests:
cpu: 1000m
memory: 4Gi
limits:
cpu: 2000m
memory: 6Gi
memBallastSizeMbs: 1024
multitenancyEnabled: false
# -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs
reportingEnabled: true
metricsGenerator:
# -- If true, enables Tempo's metrics generator (https://grafana.com/docs/tempo/next/metrics-generator/)
enabled: false
remoteWriteUrl: "http://prometheus.monitoring:9090/api/v1/write"
# -- Configuration options for the ingester
ingester: {}
# -- Configuration options for the querier
querier: {}
# -- Configuration options for the query-fronted
queryFrontend: {}
retention: 24h
# Global overrides
global_overrides:
per_tenant_override_config: /conf/overrides.yaml
overrides: {}
# Tempo server configuration
# Refers to https://grafana.com/docs/tempo/latest/configuration/#server
server:
# -- HTTP server listen port
http_listen_port: 3100
storage:
trace:
# tempo storage backend
# refer https://grafana.com/docs/tempo/latest/configuration/
## Use s3 for example
backend: s3
# store traces in s3
s3:
bucket: tempo-stg # store traces in this bucket
region: eu-central-1
access_key: ---------------
secret_key: -----------------------
# endpoint: s3.dualstack.us-east-2.amazonaws.com # api endpoint
# access_key: ... # optional. access key when using static credentials.
# secret_key: ... # optional. secret key when using static credentials.
# insecure: false # optional. enable if endpoint is http
#backend: local
#local:
#path: /var/tempo/traces
#wal:
# path: /var/tempo/wal
# this configuration will listen on all ports and protocols that tempo is capable of.
# the receives all come from the OpenTelemetry collector. more configuration information can
# be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_binary:
endpoint: 0.0.0.0:6832
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
opencensus:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
securityContext: {}
# allowPrivilegeEscalation: false
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
## Additional container arguments
extraArgs: {}
# -- Environment variables to add
extraEnv: []
# -- Environment variables from secrets or configmaps to add to the ingester pods
extraEnvFrom: []
# -- Volume mounts to add
extraVolumeMounts: []
# - name: extra-volume
# mountPath: /mnt/volume
# readOnly: true
# existingClaim: volume-claim
# -- Tempo configuration file contents
# @default -- Dynamically generated tempo configmap
config: |
multitenancy_enabled: {{ .Values.tempo.multitenancyEnabled }}
usage_report:
reporting_enabled: {{ .Values.tempo.reportingEnabled }}
compactor:
compaction:
block_retention: {{ .Values.tempo.retention }}
distributor:
receivers:
{{- toYaml .Values.tempo.receivers | nindent 8 }}
ingester:
{{- toYaml .Values.tempo.ingester | nindent 6 }}
server:
{{- toYaml .Values.tempo.server | nindent 6 }}
storage:
{{- toYaml .Values.tempo.storage | nindent 6 }}
querier:
{{- toYaml .Values.tempo.querier | nindent 6 }}
query_frontend:
{{- toYaml .Values.tempo.queryFrontend | nindent 6 }}
overrides:
{{- toYaml .Values.tempo.global_overrides | nindent 6 }}
{{- if .Values.tempo.metricsGenerator.enabled }}
metrics_generator_processors:
- 'service-graphs'
- 'span-metrics'
metrics_generator:
storage:
path: "/tmp/tempo"
remote_write:
- url: {{ .Values.tempo.metricsGenerator.remoteWriteUrl }}
{{- end }}
tempoQuery:
repository: grafana/tempo-query
tag: null
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
# -- if False the tempo-query container is not deployed
enabled: false
service:
port: 16686
ingress:
enabled: false
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
# ingressClassName: nginx
# Values can be templated
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
labels: {}
path: /
# pathType is only for k8s >= 1.1=
pathType: Prefix
hosts:
- query.tempo.example.com
## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: []
# - path: /*
# backend:
# serviceName: ssl-redirect
# servicePort: use-annotation
## Or for k8s > 1.19
# - path: /*
# pathType: Prefix
# backend:
# service:
# name: ssl-redirect
# port:
# name: use-annotation
tls: []
# - secretName: tempo-query-tls
# hosts:
# - query.tempo.example.com
resources: {}
# requests:
# cpu: 1000m
# memory: 4Gi
# limits:
# cpu: 2000m
# memory: 6Gi
## Additional container arguments
extraArgs: {}
# -- Environment variables to add
extraEnv: []
# -- Volume mounts to add
extraVolumeMounts: []
# - name: extra-volume
# mountPath: /mnt/volume
# readOnly: true
# existingClaim: volume-claim
securityContext: {}
# allowPrivilegeEscalation: false
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: false # fails if true, do not enable
# -- securityContext for container
securityContext: {}
# runAsUser: 65532
# runAsGroup: 65532
# fsGroup: 65532
# runAsNonRoot: true
serviceAccount:
# -- Specifies whether a ServiceAccount should be created
create: true
# -- The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name: null
# -- Image pull secrets for the service account
imagePullSecrets: []
# -- Annotations for the service account
annotations: {}
# -- Labels for the service account
labels: {}
automountServiceAccountToken: true
service:
type: ClusterIP
annotations: {}
labels: {}
serviceMonitor:
enabled: false
interval: ""
additionalLabels: {}
annotations: {}
# scrapeTimeout: 10s
persistence:
enabled: false
# storageClassName: local-path
accessModes:
- ReadWriteOnce
size: 10Gi
# -- Pod Annotations
podAnnotations: {}
# -- Pod (extra) Labels
podLabels: {}
# Apply extra labels to common labels.
extraLabels: {}
# -- Volumes to add
extraVolumes: []
# -- Node labels for pod assignment. See: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}
# -- Tolerations for pod assignment. See: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []
# -- Affinity for pod assignment. See: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: {}
# -- The name of the PriorityClass
priorityClassName: null
from tempo.
You don't have an endpoint defined in your S3 configuration. Notice that it's commented out.
# endpoint: s3.dualstack.us-east-2.amazonaws.com # api endpoint
from tempo.
I have added in values.yaml stil getting same issue can you share any values.yaml with helm for example
policy attached to tempo user for s3 backend
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TempoPermissions",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:ListBucket",
"s3:DeleteObject",
"s3:GetObjectTagging",
"s3:PutObjectTagging"
],
"Resource": [
"arn:aws:s3:::tempo-stg/*",
"arn:aws:s3:::tempo-stg"
]
}
]
from tempo.
Can you share the final tempo configmap? Specifically the storage
section. It should look something like this generally
storage:
trace:
backend: s3
block:
version: vParquet4
s3:
bucket: <bucket name>
endpoint: s3.dualstack.us-east-2.amazonaws.com
wal:
path: /var/tempo/wal
version: vParquet4
from tempo.
# -- Overrides the chart's name
nameOverride: ""
# -- Overrides the chart's computed fullname
fullnameOverride: ""
# -- Define the amount of instances
replicas: 1
# -- Number of ol accesssecret_key: history to retain to allow rollback (If not set, default Kubernetes value is set to 10)
# revisionHistoryLimit: 1
# -- Annotations for the StatefulSet
annotations: {}
tempo:
repository: grafana/tempo
tag: ""
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
updateStrategy: RollingUpdate
resources:
requests:
cpu: 1000m
memory: 4Gi
limits:
cpu: 2000m
memory: 6Gi
memBallastSizeMbs: 1024
multitenancyEnabled: false
# -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs
reportingEnabled: true
metricsGenerator:
# -- If true, enables Tempo's metrics generator (https://grafana.com/docs/tempo/next/metrics-generator/)
enabled: false
remoteWriteUrl: "http://prometheus.monitoring:9090/api/v1/write"
# -- Configuration options for the ingester
ingester: {}
# -- Configuration options for the querier
querier: {}
# -- Configuration options for the query-fronted
queryFrontend: {}
retention: 24h
# Global overrides
global_overrides:
per_tenant_override_config: /conf/overrides.yaml
overrides: {}
# Tempo server configuration
# Refers to https://grafana.com/docs/tempo/latest/configuration/#server
server:
# -- HTTP server listen port
http_listen_port: 3100
storage:
trace:
backend: s3
block:
version: vParquet4
s3:
bucket: tempo-stg
region: eu-central-1
endpoint: s3.dualstack.eu-central-1.amazonaws.com
access_key: ------
secret_key: --------
wal:
path: /var/tempo/wal
version: vParquet4
#storage:
#trace:
# tempo storage backend
# refer https://grafana.com/docs/tempo/latest/configuration/
## Use s3 for example
# backend: s3
# store traces in s3
# s3:
# bucket: <your s3 bucket> # store traces in this bucket
# endpoint: s3.dualstack.us-east-2.amazonaws.com # api endpoint
# access_key: ... # optional. access key when using static credentials.
# secret_key: ... # optional. secret key when using static credentials.
# insecure: false # optional. enable if endpoint is http
#backend: local
#local:
# path: /var/tempo/traces
#wal:
# path: /var/tempo/wal
# this configuration will listen on all ports and protocols that tempo is capable of.
# the receives all come from the OpenTelemetry collector. more configuration information can
# be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_binary:
endpoint: 0.0.0.0:6832
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
opencensus:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
securityContext: {}
# allowPrivilegeEscalation: false
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
## Additional container arguments
extraArgs: {}
# -- Environment variables to add
extraEnv: []
# -- Environment variables from secrets or configmaps to add to the ingester pods
extraEnvFrom: []
# -- Volume mounts to add
extraVolumeMounts: []
# - name: extra-volume
# mountPath: /mnt/volume
# readOnly: true
# existingClaim: volume-claim
# -- Tempo configuration file contents
# @default -- Dynamically generated tempo configmap
config: |
multitenancy_enabled: {{ .Values.tempo.multitenancyEnabled }}
usage_report:
reporting_enabled: {{ .Values.tempo.reportingEnabled }}
compactor:
compaction:
block_retention: {{ .Values.tempo.retention }}
distributor:
receivers:
{{- toYaml .Values.tempo.receivers | nindent 8 }}
ingester:
{{- toYaml .Values.tempo.ingester | nindent 6 }}
server:
{{- toYaml .Values.tempo.server | nindent 6 }}
storage:
{{- toYaml .Values.tempo.storage | nindent 6 }}
querier:
{{- toYaml .Values.tempo.querier | nindent 6 }}
query_frontend:
{{- toYaml .Values.tempo.queryFrontend | nindent 6 }}
overrides:
{{- toYaml .Values.tempo.global_overrides | nindent 6 }}
{{- if .Values.tempo.metricsGenerator.enabled }}
metrics_generator_processors:
- 'service-graphs'
- 'span-metrics'
metrics_generator:
storage:
path: "/tmp/tempo"
remote_write:
- url: {{ .Values.tempo.metricsGenerator.remoteWriteUrl }}
{{- end }}
tempoQuery:
repository: grafana/tempo-query
tag: null
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# pullSecrets:
# - myRegistryKeySecretName
# -- if False the tempo-query container is not deployed
enabled: false
service:
port: 16686
ingress:
enabled: false
# For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
# See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
# ingressClassName: nginx
# Values can be templated
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
labels: {}
path: /
# pathType is only for k8s >= 1.1=
pathType: Prefix
hosts:
- query.tempo.example.com
## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: []
# - path: /*
# backend:
# serviceName: ssl-redirect
# servicePort: use-annotation
## Or for k8s > 1.19
# - path: /*
# pathType: Prefix
# backend:
# service:
# name: ssl-redirect
# port:
# name: use-annotation
tls: []
# - secretName: tempo-query-tls
# hosts:
# - query.tempo.example.com
resources: {}
# requests:
# cpu: 1000m
# memory: 4Gi
# limits:
# cpu: 2000m
# memory: 6Gi
## Additional container arguments
extraArgs: {}
# -- Environment variables to add
extraEnv: []
# -- Volume mounts to add
extraVolumeMounts: []
# - name: extra-volume
# mountPath: /mnt/volume
# readOnly: true
# existingClaim: volume-claim
securityContext: {}
# allowPrivilegeEscalation: false
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: false # fails if true, do not enable
# -- securityContext for container
securityContext: {}
# runAsUser: 65532
# runAsGroup: 65532
# fsGroup: 65532
# runAsNonRoot: true
serviceAccount:
# -- Specifies whether a ServiceAccount should be created
create: true
# -- The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name: null
# -- Image pull secrets for the service account
imagePullSecrets: []
# -- Annotations for the service account
annotations: {}
# -- Labels for the service account
labels: {}
automountServiceAccountToken: true
service:
type: ClusterIP
annotations: {}
labels: {}
serviceMonitor:
enabled: false
interval: ""
additionalLabels: {}
annotations: {}
# scrapeTimeout: 10s
persistence:
enabled: false
# storageClassName: local-path
accessModes:
- ReadWriteOnce
size: 10Gi
# -- Pod Annotations
podAnnotations: {}
# -- Pod (extra) Labels
podLabels: {}
# Apply extra labels to common labels.
extraLabels: {}
# -- Volumes to add
extraVolumes: []
# -- Node labels for pod assignment. See: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}
# -- Tolerations for pod assignment. See: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []
# -- Affinity for pod assignment. See: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: {}
# -- The name of the PriorityClass
priorityClassName: null
from tempo.
failed parsing config: failed to parse configFile /conf/tempo.yaml: yaml: unmarshal errors:
line 30: field s3 not found in type storage.Config
line 44: field wal not found in type storage.Config
from tempo.
above shared is values.yaml and below is config map taken from artifact hub default
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "tempo.name" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "tempo.labels" . | nindent 4 }}
data:
overrides.yaml: |
overrides:
{{- toYaml .Values.tempo.overrides | nindent 6 }}
tempo.yaml: |
{{- tpl .Values.config . | nindent 4 }}
from tempo.
hi any changes to be made for files
from tempo.
I believe the s3
and wal
blocks need one more indentation. They're in storage.trace
, not storage
.
from tempo.
Related Issues (20)
- Guidance on traceql metrics timeout issue
- Support EKS Pod Identity feature HOT 4
- Bug in wal.go causes c.BlocksFilepath config to be overwritten in tempo.yaml HOT 5
- compile issue when make vendor-check HOT 2
- No query results for older telemetry HOT 3
- Tenant error when S3 bucket analytics turned on for S3 backend storage bucket HOT 1
- [DOC] Clarify endpoint access for consistent hash rings doc
- generator push to prometheus pushgateway always fails "HTTP status 400 Bad Request: snappy: corrupt input" HOT 2
- [DOC] Review and update the Parquet docs
- AWS S3 HTTP 4xx client errors HOT 13
- Adjust client/server span times to account for clock skew HOT 3
- Error on kubernetes (microk8s) with persistence enabled HOT 7
- Data retention for 30 days in grafana tempo do not work HOT 5
- User-configurable overrides refuses span-metrics-size as valid processor
- tempo-distributed does not support OpenShift installation out of the box because of missing securitycontextconstraint HOT 2
- aws s3 storage backend:List HOT 2
- pusher failed to consume trace data due to "context canceled" HOT 1
- Monolithic deployment with s3 storage fails with "mkdir /var/tempo/wal: permission denied" HOT 1
- Periodic errors during searching for traces HOT 1
- [DOC] Release notes for 2.6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tempo.