"description":"Shows the number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series) with new data points inserted during the last hour. High value may result in ingestion slowdown.",
"expr":"sum(min_over_time(vm_app_version{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job)",
"format":"time_series",
"instant":false,
"legendFormat":"{{job}}",
"refId":"A"
}
],
"title":"Uptime ($job)",
"type":"timeseries"
},
{
"collapsed":false,
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":12
},
"id":24,
"panels":[],
"title":"Overview",
"type":"row"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"How many datapoints are inserted into storage per second",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":13
},
"id":106,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.4.0-20418128491",
"targets":[
{
"datasource":{
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (type, instance) > 0",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{instance}} - {{type}}",
"range":true,
"refId":"A"
}
],
"title":"Datapoints ingestion rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"* `*` - unsupported query path\n* `/write` - insert into VM\n* `/metrics` - query VM system metrics\n* `/query` - query instant values\n* `/query_range` - query over a range of time\n* `/series` - match a certain label set\n* `/label/{}/values` - query a list of label values (variables mostly)",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":13
},
"id":12,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.4.0-20418128491",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(vm_http_requests_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (path) > 0",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{path}}",
"range":true,
"refId":"A"
}
],
"title":"Requests rate ($instance)",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series) with new data points inserted during the last hour. High value may result in ingestion slowdown. \n\nSee following link for details:",
"description":"The less time it takes is better.\n* `*` - unsupported query path\n* `/write` - insert into VM\n* `/metrics` - query VM system metrics\n* `/query` - query instant values\n* `/query_range` - query over a range of time\n* `/series` - match a certain label set\n* `/label/{}/values` - query a list of label values (variables mostly)",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"s"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":21
},
"id":22,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.4.0-20418128491",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"max(vm_request_duration_seconds{job=~\"$job\", instance=~\"$instance\", quantile=\"0.99\"}) by (path) > 0",
"description":"* `*` - unsupported query path\n* `/write` - insert into VM\n* `/metrics` - query VM system metrics\n* `/query` - query instant values\n* `/query_range` - query over a range of time\n* `/series` - match a certain label set\n* `/label/{}/values` - query a list of label values (variables mostly)",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":29
},
"id":35,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.3.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"exemplar":false,
"expr":"sum(rate(vm_http_request_errors_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, path) > 0",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{path}}",
"range":true,
"refId":"A"
}
],
"title":"Requests error rate ($instance)",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the rate of logging the messages by their level. Unexpected spike in rate is a good reason to check logs.",
"expr":"sum(rate(vm_log_messages_total{job=~\"$job\",instance=~\"$instance\", level!=\"info\"}[$__rate_interval])) by (job, level) > 0",
"interval":"5m",
"legendFormat":"{{job}} - {{level}}",
"range":true,
"refId":"A"
}
],
"title":"Logging rate",
"type":"timeseries"
},
{
"collapsed":true,
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":37
},
"id":46,
"panels":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"How much resident memory (RAM) the process is actually using, compared to its allowed container or system limit.\n\n- Good: Below 70% most of the time, maybe spiking a bit under load.\n- Bad: Above 90% for more than 5 minutes = risk of out-of-memory (OOM) kill.",
"description":"Share for memory allocated by the process itself. When memory usage reaches 100% it will be likely OOM-killed.\nSafe memory usage % considered to be below 80%",
"description":"CPU pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html). Helps troubleshoot high CPU usage or throttling.\nLower is better.\n\n- waiting: The percentage of time at least one task in the process was ready to run (runnable) but couldn't get scheduled on the CPU.\n- stalled: The percentage of time all tasks in the process (except idle ones) were unable to get CPU time — a full CPU stall.\n\nIf there's a CPU burst, it's normal to see waiting or stalled > 1%. It only becomes a concern if it consistently climbs above 5–10% and aligns with latency spikes or GC slowdowns.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"line"
}
},
"decimals":0,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
}
]
},
"unit":"s"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":39
},
"id":134,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_pressure_cpu_waiting_seconds_total{job=~\"$job\"}[$__rate_interval])) by (job, instance)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"{{job}}-{{instance}} - waiting",
"range":true,
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_pressure_cpu_stalled_seconds_total{job=~\"$job\"}[$__rate_interval])) by (job, instance)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"{{job}}-{{instance}} - waiting",
"range":true,
"refId":"B"
}
],
"title":"CPU pressure",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[
{
"targetBlank":true,
"title":"111",
"url":"111"
}
],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"bytes"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":47
},
"id":44,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(go_memstats_sys_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{job}} - requested from system",
"range":true,
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(go_memstats_heap_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job) + sum(vm_cache_size_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{job}} - heap inuse",
"range":true,
"refId":"B"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(go_memstats_stack_inuse_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{job}} - stack inuse",
"range":true,
"refId":"C"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(process_resident_memory_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - resident",
"range":true,
"refId":"D"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"exemplar":false,
"expr":"sum(process_resident_memory_anon_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)",
"description":"Shows memory pressure based on [Pressure Stall Information](https://docs.kernel.org/accounting/psi.html).\nLower is better.\n\n- waiting: Time fraction where at least one thread was blocked on memory.\n- stalled: Time fraction where every thread was blocked on memory (severe pressure).\n\nIf queries slow down and both series spike, the host is likely limited by RAM or I/O throughput.\n\nSee also major page faults rate panel in Troubleshooting section if this metric continued to be high.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"line"
}
},
"decimals":0,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
}
]
},
"unit":"s"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":55
},
"id":135,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_pressure_memory_waiting_seconds_total{job=~\"$job\"}[$__rate_interval])) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"{{job}} - waiting",
"range":true,
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_pressure_memory_stalled_seconds_total{job=~\"$job\"}[$__rate_interval])) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"{{job}} - stalled",
"range":true,
"refId":"B"
}
],
"title":"Memory pressure",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the number of bytes read/write from the storage layer.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"bytes"
},
"overrides":[
{
"matcher":{
"id":"byRegexp",
"options":"/.*read/"
},
"properties":[
{
"id":"custom.transform",
"value":"negative-Y"
}
]
}
]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":55
},
"id":76,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"none"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_io_storage_read_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - read",
"range":true,
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(process_io_storage_written_bytes_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - write",
"range":true,
"refId":"B"
}
],
"title":"Disk writes/reads ($instance)",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the percentage of open file descriptors compared to the limit set in the OS.\nReaching the limit of open files can cause various issues and must be prevented.\n\nSee how to change limits here https://medium.com/@muhammadtriwibowo/set-permanently-ulimit-n-open-files-in-ubuntu-4d61064429a",
"description":"Shows the percent of CPU spent on garbage collection.\n\nIf % is high, then CPU usage can be decreased by changing GOGC to higher values. Increasing GOGC value will increase memory usage, and decrease CPU usage.\n\nTry searching for keyword `GOGC` at https://docs.victoriametrics.com/victoriametrics/troubleshooting/ ",
"description":"Shows the rate of allocations in memory. Sudden increase in allocations would mean increased pressure on Go Garbage Collector and can saturate CPU resources of the application.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"line"
}
},
"decimals":0,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
}
]
},
"unit":"bytes"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":95
},
"id":133,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"max(rate(go_memstats_alloc_bytes_total{job=~\"$job\"}[$__rate_interval])) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"__auto",
"range":true,
"refId":"A"
}
],
"title":"Memory allocations rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the time goroutines have spent in runnable state before actually running. The lower is better.\n\nHigh values or values exceeding the threshold is usually a sign of insufficient CPU resources or CPU throttling. \n\nVerify that service has enough CPU resources. Otherwise, the service could work unreliably with delays in processing.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"line"
}
},
"decimals":0,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":0.1
}
]
},
"unit":"s"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":95
},
"id":128,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"max(histogram_quantile(0.99, sum(rate(go_sched_latencies_seconds_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, instance, le))) by(job)",
"format":"time_series",
"interval":"",
"intervalFactor":2,
"legendFormat":"__auto",
"range":true,
"refId":"A"
}
],
"title":"Go scheduling latency",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows network usage by VM:\n* Writes show traffic sent to clients\n* Reads show traffic received from clients",
"description":"Shows the rate and total number of new series created over last 24h.\n\nHigh [churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) tightly connected with database performance and may result in unexpected OOM's or slow queries. It is recommended to always keep an eye on this metric to avoid unexpected [cardinality](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#cardinality) \"explosions\".\n\nThe higher churn rate is, the more resources required to handle it. Consider to keep the churn rate as low as possible.\n\nGood references to read:\n* https://www.robustperception.io/cardinality-is-key\n* https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality",
"description":"The percentage of [slow inserts](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-a-slow-insert) compared to the total ingestion rate. \n\nThe lower the better. \n\nIn short, slow insert is a cache miss. There are following reasons for slow inserts to go up: \n* Ingestion of completely new, not seen before time series;\n* Not enough memory to maintain big enough caches for the current workload.\n\nIf percentage remains high (>10%) during extended periods of time, then it is likely more RAM is needed for optimal handling of the current number of [active time series](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-an-active-time-series). \n\nSee [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183) for details.",
"description":"Slow queries rate according to `search.logSlowQueryDuration` flag, which is `5s` by default.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":40
},
"id":60,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"none"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(vm_slow_queries_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job)",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{job}}",
"range":true,
"refId":"A"
}
],
"title":"Slow queries rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the percentage of used cache size from the allowed size by type. \nValues close to 100% show the maximum potential utilization.\nValues close to 0% show that cache is underutilized.",
"description":"Shows cache miss ratio. Lower is better.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"max":1,
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"percentunit"
},
"overrides":[]
},
"gridPos":{
"h":9,
"w":12,
"x":12,
"y":48
},
"id":118,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"exemplar":false,
"expr":"max(\n rate(vm_cache_misses_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n /\n rate(vm_cache_requests_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])\n) by (type)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"__auto",
"range":true,
"refId":"A"
}
],
"title":"Cache miss ratio ($instance)",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows how many samples were ignored or dropped on insertion due to various reasons:\n* timestamp out of retention period or timestamp in future;\n* invalid metric name;\n* exceeding limit for labels length or number;\n* dropped by [relabeling configuration](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling).",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":57
},
"id":131,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"none"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"exemplar":true,
"expr":"sum(increase(vm_rows_ignored_total{job=~\"$job\", instance=~\"$instance\"}[1h])) by (job, reason)",
"description":"Shows rate of deduplicated samples during [deduplication](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication) or [downsampling](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling), according to the configured `-dedup.minScrapeInterval` or `-downsampling.period`.\n\nShould be empty if `-dedup.minScrapeInterval` and `-downsampling.period` isn't set.\n\nDeduplication happens in two places:\n* `type=\"merge\"` - during [background merges](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#storage). Deduplication during merges is permanent.\n* `type=\"select\"` - during [read queries](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#query-data). Deduplication happens in-flight and has no permanent effect.\n\nThe more samples need to be deduplicated, the higher will be resource usage.",
"description":"Shows the number of restarts per job. The chart can be useful to identify periodic process restarts and correlate them with potential issues or anomalies. Normally, processes shouldn't restart unless restart was inited by user. The reason of restarts should be figured out by checking the logs of each specific service. ",
"description":"Major page faults occur when required data is not in physical memory and must be loaded from disk (storage or swap). It usually happens with:\n- insufficient physical memory\n- active use of swap space\n- file mapping access\n\nIt may cause significant performance slowdown if sustained high.\n\nPossible workaround:\n- Extend RAM to scale up\n- Reduce swap usage by adjusting vm.swappiness",
"expr":"sum(rate(process_major_pagefaults_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job) > 0",
"legendFormat":"__auto",
"range":true,
"refId":"A"
}
],
"title":"Major page faults rate",
"type":"timeseries"
}
],
"title":"Troubleshooting",
"type":"row"
},
{
"collapsed":true,
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":39
},
"id":14,
"panels":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"How many datapoints are inserted into storage per second",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":33
},
"id":10,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(rate(vm_rows_inserted_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by ( job,type) > 0",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{job}} - {{type}}",
"range":true,
"refId":"A"
}
],
"title":"Datapoints ingestion rate ($instance)",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows the approx time needed to reach 100% of allowed disk capacity based on the following params:\n* free disk space (after -storage.minFreeDiskSpaceBytes);\n* row ingestion rate;\n* compression.",
"description":"Shows how many ongoing insertions (not API /write calls) on disk are taking place, where:\n* `max` - equal to number of CPUs;\n* `current` - current number of goroutines busy with inserting rows into underlying storage.\n\nEvery successful API /write call results into flush on disk. The `max` is an internal limit and can't be changed. It is always equal to the number of CPUs. \n\nWhen `current` hits `max` constantly, it means storage is overloaded and requires more CPU (see CPU usage) or disks with more IOPS (see disk writes and reads panels in Resource Usage row).",
"description":"How many data points are in RAM queue waiting to be written into storage. The number of pending data points should be in the range from 0 to `5*<ingestion_rate>`, since VictoriaMetrics pushes pending data to persistent storage every 5 seconds. The index datapoints value in general is much lower",
"description":"The max number of data parts of LSM tree across all storage nodes in the last [partition](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#storage).\nIncrease in number of parts (the hard limit is 512) is an evidence of slow merge performance - check the resource utilization.\n* `indexdb` - inverted index\n* `storage/small` - recently added parts of data ingested into storage (hot data)\n* `storage/big` - small parts gradually merged into bigger parts (cold data)",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":57
},
"id":36,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(vm_parts{job=~\"$job\", instance=~\"$instance\", type=~\"indexdb.*\"}) by (job, type)",
"description":"Shows the percentage of used disk space by type: datapoints or indexdb. Normally, indexdb takes much less space comparing to datapoints. But with high [churn rate](https://docs.victoriametrics.com/victoriametrics/faq/#what-is-high-churn-rate) the size of the indexdb could grow significantly.\n\nThe sum of the % can be > 100% since panel shows max % per-job and per-instance. It means different instance can have different ratio between datapoints and indexdb size.",
"description":"Shows amount of on-disk space occupied by data points and the remaining disk space at `-storageDataPath`. Calculation accounts for -storage.minFreeDiskSpaceBytes",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"bytes"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":65
},
"id":53,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type!~\"indexdb.*\"}) by (job)",
"expr":"sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\", type=~\"indexdb.*\"}) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - Used (index)",
"range":true,
"refId":"C"
}
],
"title":"Disk space usage - datapoints",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows how many datapoints are in the storage and what is average disk usage per datapoint.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[
{
"matcher":{
"id":"byName",
"options":"bytes-per-datapoint"
},
"properties":[
{
"id":"unit",
"value":"bytes"
},
{
"id":"decimals",
"value":2
}
]
}
]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":73
},
"id":30,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type!~\"indexdb.*\"}) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - total datapoints",
"range":true,
"refId":"A"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"sum(vm_data_size_bytes{job=~\"$job\", instance=~\"$instance\"}) by (job)\n/ sum(vm_rows{job=~\"$job\", instance=~\"$instance\", type!~\"indexdb.*\"}) by (job)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}} - bytes-per-datapoint",
"range":true,
"refId":"B"
}
],
"title":"Datapoints",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"How many data points are in RAM queue waiting to be written into storage. The number of pending data points should be in the range from 0 to `5*<ingestion_rate>`, since VictoriaMetrics pushes pending data to persistent storage every 5 seconds. The index datapoints value in general is much lower.",
"description":"Shows how many rows were ignored on insertion due to corrupted or out of retention timestamps and how many series were ignored on insertion or during scrape due to too long label names, values and excessive labels count.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":10,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":81
},
"id":58,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"none"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"exemplar":false,
"expr":"sum(increase(vm_rows_ignored_total{job=~\"$job\", instance=~\"$instance\"}[1h])) by (instance, reason)",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{instance}} - {{reason}}",
"range":true,
"refId":"A"
}
],
"title":"Rows ignored for last 1h",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"99th percentile of number of [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) read per queried series.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"decimals":2,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":81
},
"id":103,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"histogram_quantile(0.99, sum(rate(vm_rows_read_per_series_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, vmrange))",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}}",
"range":true,
"refId":"A"
}
],
"title":"Datapoints read per series",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows how many ongoing insertions (not API /write calls) on disk are taking place, where:\n* `max` - equal to number of CPUs;\n* `current` - current number of goroutines busy with inserting rows into underlying storage.\n\nEvery successful API /write call results into flush on disk. However, these two actions are separated and controlled via different concurrency limiters. The `max` on this panel can't be changed and always equal to number of CPUs. \n\nWhen `current` hits `max` constantly, it means storage is overloaded and requires more CPU.\n\n",
"expr":"sum(vm_concurrent_insert_current{job=~\"$job\", instance=~\"$instance\"}) by (instance)",
"format":"time_series",
"intervalFactor":1,
"legendFormat":"{{instance}} - current",
"range":true,
"refId":"B"
}
],
"title":"Concurrent flushes on disk {$instance}",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"99th percentile of number of [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) scanner per query.\n\nThis number can exceed number of RowsReadPerQuery if `step` query arg passed to [/api/v1/query_range](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries) is smaller than the lookbehind window set in square brackets of [rollup function](https://docs.victoriametrics.com/victoriametrics/metricsql/#rollup-functions). For example, if `increase(some_metric[1h])` is executed with the `step=5m`, then the same [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) on a hour time range are scanned `1h/5m=12` times. See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986) for details.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"decimals":2,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":89
},
"id":105,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"histogram_quantile(0.99, sum(rate(vm_rows_scanned_per_query_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, vmrange))",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}}",
"range":true,
"refId":"A"
}
],
"title":"Datapoints scanned per query",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"99th percentile of number of series read per query.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"decimals":2,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":97
},
"id":99,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"histogram_quantile(0.99, sum(rate(vm_series_read_per_query_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, vmrange))",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}}",
"range":true,
"refId":"A"
}
],
"title":"Series read per query",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"99th percentile of number of [data samples](https://docs.victoriametrics.com/victoriametrics/keyconcepts/#raw-samples) read per query.",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"never",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"decimals":2,
"links":[],
"mappings":[],
"min":0,
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
},
"unit":"short"
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":97
},
"id":122,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true,
"sortBy":"Last *",
"sortDesc":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"editorMode":"code",
"expr":"histogram_quantile(0.99, sum(rate(vm_rows_read_per_query_bucket{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job, vmrange))",
"format":"time_series",
"interval":"",
"intervalFactor":1,
"legendFormat":"{{job}}",
"range":true,
"refId":"A"
}
],
"title":"Datapoints read per query",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"$ds"
},
"description":"Shows number of existing [snapshots](https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-work-with-snapshots).\n\nRecently made snapshots do not occupy disk space. But with time, as snapshots become older, they start to occupy more and more disk space.\n\nIt is recommended deleting old snapshots when they are no longer needed to free up disk space.",
"content":"Drilldown row is used by other panels on the dashboard to show more detailed metrics per-instance, or a single instance in cases like memory breakdown.",
"mode":"markdown"
},
"pluginVersion":"12.2.0",
"title":"",
"transparent":true,
"type":"text"
},
{
"datasource":{
"type":"prometheus",
"uid":"${ds}"
},
"description":"Major page faults occur when required data is not in physical memory and must be loaded from disk (storage or swap). It usually happens with:\n- insufficient physical memory\n- active use of swap space\n- file mapping access\n\nIt may cause significant performance slowdown if sustained high.\n\nPossible workaround:\n- Extend RAM to scale up\n- Reduce swap usage by adjusting vm.swappiness",
"fieldConfig":{
"defaults":{
"color":{
"mode":"palette-classic"
},
"custom":{
"axisBorderShow":false,
"axisCenteredZero":false,
"axisColorMode":"text",
"axisLabel":"",
"axisPlacement":"auto",
"barAlignment":0,
"barWidthFactor":0.6,
"drawStyle":"line",
"fillOpacity":0,
"gradientMode":"none",
"hideFrom":{
"legend":false,
"tooltip":false,
"viz":false
},
"insertNulls":false,
"lineInterpolation":"linear",
"lineWidth":1,
"pointSize":5,
"scaleDistribution":{
"type":"linear"
},
"showPoints":"auto",
"showValues":false,
"spanNulls":false,
"stacking":{
"group":"A",
"mode":"none"
},
"thresholdsStyle":{
"mode":"off"
}
},
"links":[],
"mappings":[],
"thresholds":{
"mode":"absolute",
"steps":[
{
"color":"green",
"value":0
},
{
"color":"red",
"value":80
}
]
}
},
"overrides":[]
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":429
},
"id":140,
"options":{
"legend":{
"calcs":[
"mean",
"lastNotNull",
"max"
],
"displayMode":"table",
"placement":"bottom",
"showLegend":true
},
"tooltip":{
"hideZeros":false,
"mode":"multi",
"sort":"desc"
}
},
"pluginVersion":"12.2.0",
"targets":[
{
"datasource":{
"type":"prometheus",
"uid":"${ds}"
},
"editorMode":"code",
"expr":"sum(rate(process_major_pagefaults_total{job=~\"$job\", instance=~\"$instance\"}[$__rate_interval])) by (job,instance) > 0",
"legendFormat":"{{instance}} ({{job}})",
"range":true,
"refId":"A"
}
],
"title":"Major page faults rate",
"type":"timeseries"
},
{
"datasource":{
"type":"prometheus",
"uid":"${ds}"
},
"description":"This panel breaks down memory usage by type. It’s intended for inspecting a single instance.\n\nHow to use:\n- Start from the high-level RSS panel.\n- Identify an instance with unexpected or abnormal memory growth.\n- Filter to that instance to inspect the detailed breakdown.\n\nInterpretation\n- A steadily rising Go Heap usually indicates a memory leak. Collect pprof memory profile.\n- A growing Go Stack commonly points to a goroutine leak.",
"description":"RSS share for memory allocated by the process itself. This share cannot be freed by the OS, so it must be taken into account by OOM killer.",
"description":"Shows the approx time needed to reach 100% of allowed disk capacity based on the following params:\n* free disk space (after -storage.minFreeDiskSpaceBytes);\n* row ingestion rate;\n* compression.",
