Monitoring PFE Resources on EVO Routers¶
Explore another use case of the Utility MIB feature [1] in Junos and EVO. We previously discussed this feature in a separate Techpost [2] in the context of the SRX platform. Today, we'll focus on its application on the ACX7000 platform.
Introduction¶
Recently, we received a request from a customer who wanted to monitor specific PFE counters on the ACX 7K platform. As you may know, Juniper platforms are designed to provide transparency, allowing customers access to the embedded "shell" of the Packet Forwarding Engines.
The command they referred to was as follows:
While the traditional CLI provides extensive statistics for the control and data planes, there are situations where PFE commands---intended for deeper troubleshooting---offer valuable insights that might be worth monitoring. So, how can this be achieved?
The most straightforward answer is to ask your Systems Engineer (SE) to file an Enhancement Request (ER) to make these statistics available through the standard CLI, and subsequently via Netconf RPC or Streaming Telemetry. Although this process can take some time, it's the proper approach to enhance the Junos and EVO observability toolkit.
But what if you need a solution right now, without waiting for a future software release? In this case, we're dealing with statistics only accessible at the PFE level. For statistics already exposed via CLI commands (RPC) but not yet available through Streaming Telemetry, we've addressed this gap by allowing customers to develop their own Telemetry Sensors [3]. Unfortunately, that feature doesn't apply to our current use case.
However, the Junos/EVO automation toolkit offers two alternative solutions. These one are particularly useful for monitoring and troubleshooting by integrating customized metrics into existing SNMP workflows.
- SNMP Script with Proprietary OID: This method involves creating an SNMP script that uses a custom OID. The script is executed whenever the corresponding SNMP OID is queried. This approach works well for simple counters fetched via SNMP GET but isn't ideal for exposing a large number of counters (e.g., an SNMP table).
- Utility MIB with event policy: This approach is quite similar but in this case an on-box script is run periodically, thanks to an event-policy and fill what we called the Juniper Utility MIB by which the root-OID is .1.3.6.1.4.1.2636.3.47.1.1.2.1. Objects stored in this MIB as dot-decimal key and supported values are today: counter64, integer, unsigned integer and string.
A Word About Dot-Decimal Notation¶
In SNMP Utility MIB, instance keys are encoded as dot-decimal values appended to the base OID to uniquely identify rows or entries within a table or specific metrics. This encoding follows a structured approach to ensure that the instance corresponds to meaningful keys.
Here's a step-by-step explanation of how instances are encoded:
- Base OID: The base OID defines the general MIB object (e.g., a table or metric). For Juniper Utility MIB, the root OID is .1.3.6.1.4.1.2636.3.47.1. But depending on the type of the data you want to fill in, the base OID will slightly change, i.e if you want to store counter64 values, the base OID will be: .1.3.6.1.4.1.2636.3.47.1.1.2.1.2
The following table lists all the base OID:
| MIB Object | Base OID |
|---|---|
| jnxUtil | .1.3.6.1.4.1.2636.3.47.1 |
| jnxUtilData | .1.3.6.1.4.1.2636.3.47.1.1 |
| jnxUtilCounter32Table | .1.3.6.1.4.1.2636.3.47.1.1.1 |
| jnxUtilCounter32Entry | .1.3.6.1.4.1.2636.3.47.1.1.1.1 |
| jnxUtilCounter32Name | .1.3.6.1.4.1.2636.3.47.1.1.1.1.1 |
| jnxUtilCounter32Value | .1.3.6.1.4.1.2636.3.47.1.1.1.1.2 |
| jnxUtilCounter32Time | .1.3.6.1.4.1.2636.3.47.1.1.1.1.3 |
| jnxUtilCounter64Table | .1.3.6.1.4.1.2636.3.47.1.1.2 |
| jnxUtilCounter64Entry | .1.3.6.1.4.1.2636.3.47.1.1.2.1 |
| jnxUtilCounter64Name | .1.3.6.1.4.1.2636.3.47.1.1.2.1.1 |
| jnxUtilCounter64Value | .1.3.6.1.4.1.2636.3.47.1.1.2.1.2 |
| jnxUtilCounter64Time | .1.3.6.1.4.1.2636.3.47.1.1.2.1.3 |
| jnxUtilIntegerTable | .1.3.6.1.4.1.2636.3.47.1.1.3 |
| jnxUtilIntegerEntry | .1.3.6.1.4.1.2636.3.47.1.1.3.1 |
| jnxUtilIntegerName | .1.3.6.1.4.1.2636.3.47.1.1.3.1.1 |
| jnxUtilIntegerValue | .1.3.6.1.4.1.2636.3.47.1.1.3.1.2 |
| jnxUtilIntegerTime | .1.3.6.1.4.1.2636.3.47.1.1.3.1.3 |
| jnxUtilUintTable | .1.3.6.1.4.1.2636.3.47.1.1.4 |
| jnxUtilUintEntry | .1.3.6.1.4.1.2636.3.47.1.1.4.1 |
| jnxUtilUintName | .1.3.6.1.4.1.2636.3.47.1.1.4.1.1 |
| jnxUtilUintValue | .1.3.6.1.4.1.2636.3.47.1.1.4.1.2 |
| jnxUtilUintTime | .1.3.6.1.4.1.2636.3.47.1.1.4.1.3 |
| jnxUtilStringTable | .1.3.6.1.4.1.2636.3.47.1.1.5 |
| jnxUtilStringEntry | .1.3.6.1.4.1.2636.3.47.1.1.5.1 |
| jnxUtilStringName | .1.3.6.1.4.1.2636.3.47.1.1.5.1.1 |
| jnxUtilStringValue | .1.3.6.1.4.1.2636.3.47.1.1.5.1.2 |
| jnxUtilStringTime | .1.3.6.1.4.1.2636.3.47.1.1.5.1.3 |
- Instance Key Values: Each row or entry in a table may have one or more keys (e.g., indexes) that identify it. These keys can be integers, strings, or IP addresses.
- Encoding the Instance:
- Integer Keys: Encoded directly as the integer value. For example, key 5 becomes .5.
- String Keys: Encoded as a length prefix followed by ASCII codes of each character. For instance, the string COS_VOQ is encoded as 67.79.83.95.86.79.81 (ASCII values of C, O, S, _, V, O, and Q).
- IP Address Keys: Encoded as four dot-separated integers. For example, 192.168.1.1 becomes .192.168.1.1.
- Combining Base OID and Instance: The base OID is concatenated with the encoded instance to form the complete unique OID.
For example:
- Base OID: .1.3.6.1.4.1.2636.3.47.1.1.2.1.2 (we want to store a Counter64 value -- refer to the previous table)
- Instance name: COS_VOQ
- Resulting OID: .1.3.6.1.4.1.2636.3.47.1.1.2.1.2.67.79.83.95.86.79.81
This encoding ensures that each entry or metric within the MIB hierarchy is uniquely identified, enabling accurate querying and monitoring.
Step-by-Step Process to Address Our Use Case¶
Step 1: Analyze the PFE Command¶
We identified the PFE command to parse: show evo-pfemand resource usage. The detailed output has been shown above. This command generates raw data; for every "resource" there are three counters: usage, capacity, and size. Thus, for each "resource," three OIDs will be created.
We will use the following rule to create a unique "instance" key:
<RESOURCE-NAME>_[usage|capacity|size]
For instance, the EEDB_3 resource will have three instances:
EEDB_3_usage, represented in dot-decimal format as 69.69.68.66.95.51.95.117.115.97.103.101EEDB_3_capacity, represented as 69.69.68.66.95.51.95.99.97.112.97.99.105.116.121EEDB_3_size, represented as 69.69.68.66.95.51.95.115.105.122.101
The instance name encoded as dot-decimal will then be concatenated with the base OID.
Step 2: Develop the On-Box Python Script¶
We chose Python 3 to write the on-box script, leveraging the PyEZ embedded library to simplify device interactions. The complete code, including detailed comments for each step, is shown below. Here's a summary of the approach:
- We used the Device and StartShell packages to open an internal RE shell on the device and execute the PFE command.
- The cprod utility was employed from the RE shell to issue the PFE command (specifically on FPC0).
- Finally, we used the request_snmp_utility_mib_set RPC to populate the Utility MIB
Step 3: Deploy the Python Script on the Router¶
First, you need to upload the script utility-evo.py to the following directory: /var/db/scripts/event.
Next, grant execution permissions to the file by running the following command:
If your platform has multiple routing engines, ensure that the script is uploaded to both Routing Engines. You can either do this manually or allow the system to synchronize the scripts between both routing engines (this approach is highly recommended).
To enable this automatic synchronization, simply commit the following configuration knob:
Step 4: Schedule the Script to Update the Utility MIB Periodically¶
To ensure the script runs periodically (every 5 minutes in our case), we use an event policy. This policy monitors a generated event called five_minutes and triggers our script accordingly. With this setup, the Utility MIB will be updated every 5 minutes. The first step is to create a "periodic and generated event":
The reason for generating the SHA-256 hash is to inform the JUNOS/EVO system to execute the script only if its hash matches the specified value. This ensures that the script hasn't been modified by unauthorized users.
Next, create a policy to monitor the event and trigger the utility-evo.py script:
Step 5: Troubleshoot Script Execution¶
On the ACX7K, script execution is monitored using the EVO application's tracing toolkit. Logs and traces for our feature are handled by the cscript application. To view them, simply run the following CLI command:
Step 6: Collect Your New Counters¶
The final step is the most rewarding: collect and visualize your new counters. Start by using the following CLI command to view the SNMP counters directly on the box. The "ascii" option is used to convert the dot-decimal instances into more readable MIB objects.
You can now collect those data from a remote collector. Here, we walk/ traverse the entire SNMP JnxUtil MIB:
You can even request only a specific counter via a GET command. For instance, let's collect the counter "EEDB_3_usage" -- From ASIC to Decimal = 69.69.68.66.95.51.95.117.115.97.103.101
Conclusion¶
In conclusion, leveraging the Utility MIB with event policies on Junos and EVO platforms offers a powerful method to monitor and troubleshoot Packet Forwarding Engine (PFE) statistics not yet exposed through standard API. While waiting for standard support through Netconf RPC or Streaming Telemetry may not be viable, the proposed solutions---such as SNMP scripts and Utility MIB updates---provide immediate alternatives. By automating the collection of resource counters and encoding them into SNMP MIB objects, users can integrate these insights into their existing workflows. This approach enables efficient monitoring and better visibility, ultimately enhancing operational control over the platform's performance.
Useful links¶
- [1] https://www.juniper.net/documentation/en_US/junos/topics/task/operational/security-snmp-best-practices-utility-mib-using.html
- [2] https://juniper.github.io/techposts/using-junos-snmp-utility-mib-on-juniper-srx/article
- [3] https://www.juniper.net/documentation/us/en/software/junos/interfaces-telemetry/netconf/topics/task/sensor-junos-telemetry-interface-configuring.html
Glossary¶
- ACX7K -- ACX Series 7000 (a series of routers by Juniper)
- CLI -- Command-Line Interface
- cprod -- Command-line utility in Junos to execute commands on FPCs
- EVO -- Juniper EVO
- FPC -- Flexible PIC Concentrator
- jcs -- Junos Control System (trace and logging utility in Junos)
- MIB -- Management Information Base
- OID -- Object Identifier
- PFE -- Packet Forwarding Engine
- PyEZ -- Python for Junos (a Python library for interacting with Junos devices)
- RE -- Routing Engine
- RPC -- Remote Procedure Call
- SE -- Systems Engineer
- SHA -- Secure Hash Algorithm
- SNMP -- Simple Network Management Protocol
- SRX -- Juniper SRX (a series of security devices)