PCA Sample Questions Answers

To calculate a percentile (e.g., 95th percentile) from histogram data in Prometheus, the correct function ishistogram_quantile(). It estimates quantiles based on cumulative bucket counts.

Example:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

This computes the 95th percentile request duration across all observed instances over the last 5 minutes.

In Prometheus best practices,high-cardinality labels—especially those containing unique or user-specific identifiers—should beavoided. The metric myapp_filG_uploads_total{userid="5123",status="failed"} exposes the userid as a label, which is problematic. Each distinct value of a label generates a new time series in Prometheus. If there are thousands or millions of unique users, this would exponentially increase the number of time series, leading tocardinality explosion, degraded performance, and high memory usage.

The _total suffix is actuallycorrect and requiredfor counters, as per the Prometheus naming convention. The use of underscores in metric names is also correct, as Prometheus does not support dashes in metric identifiers. The status label, however, is perfectly valid because it typically has a low number of possible values (e.g., “success”, “failed”).

Symptom-based alertingfocuses on detectinguser-visible or service-impacting issuesrather than internal resource states. Metrics likeAPI latency,error rates, andavailabilitydirectly indicate degraded user experience and are therefore the preferred triggers for alerts.

In contrast,resource-based alerts(like CPU usage or disk space) often represent underlying causes, not symptoms. Alerting on them can produce noise and distract from actual service health problems.

For example, high API latency (http_request_duration_seconds) clearly reflects that users are experiencing delays, which is actionable and business-relevant.

This concept aligns with theRED(Rate, Errors, Duration) andUSE(Utilization, Saturation, Errors) monitoring models promoted in Prometheus and SRE best practices.

The metric prometheus_tsdb_lowest_timestamp_seconds provides theoldest stored sample timestampin Prometheus’s local TSDB (in Unix epoch seconds). To determine theage or approximate dateof this timestamp, you compare it with thecurrent time(using time() in PromQL).

The expression:

(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400

converts the difference between the current time and the oldest timestamp fromseconds into days(1 day = 86,400 seconds). This gives the number of days since the earliest sample was stored, allowing you to infer the time range and approximate month manually.

The other options are invalid because PromQL does not support direct date formatting (format_date) or month() extraction functions.

ThePrometheus Serveris responsible forservice discovery, which identifies the list of targets to scrape. It integrates with multiple service discovery mechanisms such as Kubernetes, Consul, EC2, and static configurations.

This allows Prometheus to automatically adapt to dynamic environments without manual reconfiguration.

Themaxoperator in PromQL is anaggregation operator, not abinary vector matching operator. Therefore, the valid syntax for aggregation uses by() or without(), not on().

✅max by (instance) up → Valid; aggregates maximum values per instance.

✅max without (instance) up and max without (instance, job) up → Valid; aggregates over all labels except those listed.

❌max on (instance) (up) → Invalid; the keyword on() is only valid inbinary operations(e.g., +, -, and, or, unless), where two vectors are being matched on specific labels.

Hence, max on (instance) (up) is a syntax error in PromQL because on() cannot be used directly with aggregation operators.

According to Prometheus naming rules,metric names must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*. This means metric names must begin with a letter, underscore, or colon, and can only contain letters, digits, and underscores thereafter.

The valid metric name among the options isgo_goroutines, which follows all these rules. It starts with a letter (g), uses underscores to separate words, and contains only allowed characters.

By contrast:

go routines is invalid because it contains aspace.

go.goroutines is invalid because it contains adot(.), which is reserved for recording rule naming hierarchies, not metric identifiers.

99_goroutines is invalid because metric namescannot start with a number.

Following these conventions ensures compatibility with PromQL syntax and Prometheus’ internal data model.

Following Prometheus’smetric naming conventions, every metric should indicate:

Whatit measures (the quantity or event).

The unitof measurement inbase SI unitsas a suffix.

Since the metric measuresresponse size, the base unit isbytes. Therefore, the correct and compliant metric name is:

grpc_response_size_bytes

This clearly communicates that it measures gRPC response payload sizes expressed in bytes.

The _bytes suffix is the Prometheus-recommended unit indicator for data sizes. The other options violate naming rules:

_total is reserved for counters.

_sum is used internally by histograms or summaries.

Omitting the unit (grpc_response_size) is discouraged, as it reduces clarity.

When Prometheus targets arebehind firewalls or NATand cannot be reached directly by the Prometheus server’s pull mechanism, the recommended component to use isPushProx.

PushProxworks by reversing the usual pull model. It consists of aPushProx Proxy(accessible by Prometheus) andPushProx Clients(running alongside the targets). The clients establish outbound connections to the proxy, which allows Prometheus to “pull” metrics indirectly. This approach bypasses network restrictions without compromising the Prometheus data model.

Unlike the Pushgateway (which is used for short-lived batch jobs, not network-isolated targets), PushProx maintains the Prometheus “pull” semantics while accommodating environments where direct scraping is impossible.

The metric user_last_login_timestamp_seconds{email="john.doe@example.com"} isunsuitable for Prometheusbecause it includes ahigh-cardinality label(email). Each unique email address would generate a separate time series, potentially numbering in the millions, which severely impacts Prometheus performance and memory usage.

Prometheus is optimized for low- to medium-cardinality metrics that represent system-wide behavior rather than per-user data. High-cardinality metrics causedata explosion, complicating queries and overwhelming the storage engine.

By contrast, the other metrics—prometheus_engine_query_log_enabled, promhttp_metric_handler_requests_total{code="500"}, and http_response_total{handler="static/*filepath"}—adhere to Prometheus best practices. They represent operational or service-level metrics with limited, manageable label value sets.

Prometheus provides a feature calledRemote Writeto transmit scraped and processed metrics to an external system for long-term storage, aggregation, or advanced analytics. When configured, Prometheus continuously pushes time series data to the remote endpoint defined in the remote_write section of the configuration file.

This mechanism is often used to integrate with long-term data storage backends such as Cortex, Thanos, Mimir, or InfluxDB, enabling durable retention and global query capabilities beyond Prometheus’s local time series database limits.

In contrast, “scraping” refers to data collection from targets, while “federation” allows hierarchical Prometheus setups (pulling metrics from other Prometheus instances) but does not serve as long-term storage. Using “S3 Buckets” directly is also unsupported in native Prometheus configurations.

In PromQL, range vector functions like max_over_time() compute an aggregate value (in this case, themaximum) over all samples within a specified time range. The functionignores NaN (Not-a-Number) valueswhen computing the result.

Given the range vector foo[5m] containing samples [1, 1, NaN], the maximum value among the valid numeric samples is 1. Therefore, max_over_time(foo[5m]) returns 1.

Prometheus functions handle missing or invalid data points gracefully—ignoring NaN ensures stable calculations even when intermittent collection issues or resets occur. The function only errors if the selector is syntactically invalid or if no numeric samples exist at all.

TheGrafana Heatmap panelis best suited for visualizinghistogram metricscollected from Prometheus. Histograms provide bucketed data distributions (e.g., request durations, response sizes), and the heatmap effectively displays these as a two-dimensional density chart over time.

In Prometheus, histogram metrics are exposed as multiple time series with the _bucket suffix and the label le (less than or equal). Grafana interprets these buckets to create visual bands showing how frequently different value ranges occurred.

Counters, gauges, and info metrics do not have bucketed distributions, so a heatmap would not produce meaningful output for them.

In Prometheus, the scrape_interval parameter specifieshow frequently the Prometheus server should scrape metrics from its configured targets. Each target exposes an HTTP endpoint (usually /metrics) that Prometheus collects data from at a fixed cadence. By default, the scrape_interval is set to1 minute, but it can be overridden globally or per job configuration in the Prometheus YAML configuration file.

This setting directly affects theresolution of collected time series data—a shorter interval increases data granularity but also adds network and storage overhead, while a longer interval reduces load but might miss short-lived metric variations.

It is important to distinguish scrape_interval from evaluation_interval, which defines how often Prometheus evaluates recording and alerting rules. Thus, scrape_interval pertains only todata collection frequency, not to alerting or rule evaluation.

Deduplicationin Alertmanager ensures thatidentical alerts from multiple Prometheus serversor rule evaluations do not trigger duplicate notifications.

Alertmanager compares alerts based on their labels and fingerprints; if an alert with identical labels already exists, it merges or refreshes the existing one instead of creating a new notification.

This mechanism is essential in high-availability setups where multiple Prometheus instances monitor the same targets.