The Event Monitor on an Arista switch is a slick little tool that, according to the documentation, “writes system event records to local files for access by SQLite database commands.” Although this is a technically accurate description, allow me to expand on that a bit.
Event Monitor is a process that records certain common events on the switch. As of EOS version 4.17.2F, the events recorded included changes to the Address Resolution Protocol (ARP) table, the Internet Group Management Protocol (IGMP) snooping table, the Link Aggregation Control Protocol (LACP) table, the MAC address table, the mroute table, and the IP routing table. Modern revisions record even more.
OK, I’ll admit that still sounds boring, but let’s dig into this tool and see what it does and how it might be useful.
The home base for using Event Monitor from EOS is the show event-monitor
command. As of EOS 4.21.1F, there are a pile of options. In the first edition of Arista Warrior, there were only four. That book was also written when EOS 4.9 was current. Here are the options in 4.21.1F:
Arista#sho event-monitor ? all Monitor all events arp Monitor ARP table events backup backed up log files buffer local buffer settings igmpsnooping Monitor IGMP snooping table events lacp Monitor LACP table events mac Monitor MAC table events mroute Monitor mroute table events route Monitor routing events <cr>
There are a bunch of tables that we can view, and one very cool option named SQLite. The SQLite option lets us send SQLite commands from EOS to the SQLite database, which, as you’ll see, is pretty darn useful.
SQLite is a software library that, according to the SQLite website, “implements a self-contained, serverless, zero-configuration, transactional SQL database engine.” In other words, it’s a very simple, scaled-down version of SQL that also happens to be in the public domain. For more information on SQLite, see the project web page. Though you might have not heard of SQLite before, if you’ve ever used an Apple Mac running OS X, an iPhone, or an iPad, you’ve used a product that incorporates SQLite. If you don’t like Apple products, look no further than the Firefox browser, the Thunderbird email client, the Google Chrome browser, or the Android phone operating system for other examples of the widespread use of SQLite.
In an early release of EOS 4.21, the default nature of this feature changed and it is no longer enabled by default. This is by design in an effort to allow customers to decide whether the process should be running instead of consuming resources by default. On older revisions of code, you could just look at the ARP table by using the show event-monitor arp
command, but if you do that now, you’ll get an error:
Arista(config)#sho event-monitor arp % event-monitor not running % Database does not exist. Run 'event-monitor sync'
Unfortunately, the error message on this revision (the latest as of this writing) is incorrect, and running that command will not enable the database. This is a bug that’s been reported, so it will likely be fixed soon, but if you see this error, this is why:
Student-19(config)#event-monitor sync % event-monitor not running
To enable the event-monitor
command, issue the event-monitor
configuration command by itself:
Arista(config)#event-monitor
After you do this, the rest of the event-monitor
commands will function:
Arista(config)#sho event-monitor arp 2019-01-02 21:59:31.978200|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-02 22:04:01.072546|10.0.0.103|Management1|28:99:3a:be:9b:85|0|added|1 2019-01-03 01:36:13.381413|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|2
In EOS 4.16 the internal behavior of this tool changed a bit, which made using SQL commands via the sqlite
and interact
commands problematic, and by problematic, I mean they no longer work. To show what I mean, on EOS 4.15 I could issue the following command to get information about the fields within the ARP table:
Arista#sho event-monitor sqlite .schema arp CREATE TABLE arp( time text, prefix text, intf text, ethAddr text, static integer, delta text,counter integer UNIQUE );
Issuing this command post 4.16 yields the following output:
Arista(config)#sho event-monitor sqlite .schema arp % Cannot read from EventMon DB table
In fact, issuing any sort of more complicated sqlite
command yields a similar output where it used to work on earlier code. If you really want to get that sort of information from Event Monitor, you can use Bash and interact using SQLite3 directly:
Arista#bash sqlite3 /var/log/eventMon.db '.schema arp' CREATE TABLE arp (time text,prefix text,intf text,ethAddr text,static integer,delta text,counter integer UNIQUE);Arista#
Although that may seem unfortunate, especially if you’re used to using SQL commands to view your data, the good news is that (with the exception of the .schema
command) there are better ways to get the information using command-line interface (CLI) commands that don’t require knowledge of SQL. It’s technically not as flexible, but in my experience, network engineers tend to favor using CLI commands anyway, so this isn’t really something to get riled up about.
With that out of the way, let’s take a look at each one of these tables.
Normally, ARP changes are not logged on a switch. Even if they were, scrolling through pages of log entries is not my idea of fun, so the idea of storing these events in a database is a bit appealing to me. It’s been a long time since I was a database guy, so let’s see how rusty I am. The way to retrieve the ARP events from the database is by using the show event-monitor arp
command.
Many of the code excerpts in this chapter are wrapped in unnatural-looking ways to fit within the confines of the printed page. Because much of the output from these commands is actually Unix output piped through the EOS CLI, the format and width might not look the way you would expect from a traditional switch operating system.
Arista#sho event-monitor arp 2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1 2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2 2019-01-09 20:24:36.011052|10.10.18.5|Ethernet32|28:99:3a:be:9d:42|0|added|3 2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4 2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5
My main problem with this tool is that the output is, shall we say, ugly. Well, it’s ugly from my “all user interfaces should be elegant and beautiful” point of view. From my “it’s SQL, and that’s what SQL output looks like” point of view, it’s beautiful. I know, I’m a complicated person.
Don’t get me wrong, this is very useful information, but I have no idea what some of the values are. For example, I see some fields with zeros in them and some with no values, and I have no idea what that last field in every line represents.
To help figure out what this stuff is, I’ll take advantage of a nifty SQLite command that shows me the command used to create the table. This command is called .schema
, and we can access it by using the bash sqlite3
command. The database resides in var/log with the fairly logical name of eventMon.db. Each of the supported Event Monitor types are found in similarly named tables within this database. Here, I’m getting the schema for the arp table:
Arista#bash sqlite3 /var/log/eventMon.db ".schema arp" CREATE TABLE arp (time text,prefix text,intf text,ethAddr text,static integer, delta text,counter integer UNIQUE);
This output tells me that there was a table created named arp that contains the following fields, listed in order. I’ve also added what they mean for any nonprogrammer types out there:
The time in which this log entry was added.
The IP address related to the ARP entry.
What interface the ARP event occurred on.
The MAC address tied to the IP address (prefix).
Value is 0 if ARP entry was dynamically learned, and 1 if statically assigned.
Typical entries are added and removed.
Every time an entry is made, it is assigned a counter value as a unique identifier for this record.
Let’s see this table in action. First, here is the list of existing ARP entries from one of my switches (which happens to be the one I used for the LANZ lab):
Arista-2#sho arp Address Age (min) Hardware Addr Interface 88.0.0.1 N/A 2899.3abe.a026 Ethernet48 10.0.0.95 N/A 001c.7328.2f6c Management1 10.0.0.100 N/A 0cc4.7aa8.87ad Management1
For this exercise, I add a static ARP entry to my switch by mapping the IP address 192.168.1.2 to the MAC address learned for 192.168.1.1:
Arista-2#conf Arista-2(config)#arp 10.0.0.222 0014.6aa2.d438 arpa
Let’s see what the Event Monitor database shows now that I’ve added the static ARP:
Arista-2(config)#sho event-monitor arp 2019-01-16 22:42:42.091148|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-16 22:44:19.899883|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|1 2019-01-17 01:45:24.089620|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|2 2019-01-19 20:04:58.469700|88.0.0.1|Ethernet48|||removed|3 2019-01-19 20:05:05.205365|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|4 2019-01-19 20:06:30.330914|10.0.0.222|Management1|00:14:6a:a2:d4:38|1|added|5
Take a look at the line in bold, and let’s apply what we know. At 8:06 p.m. on January 19, 2019, a static ARP was added that mapped the IP address 10.0.0.222 to the MAC address 00:14:6a:a2:d4:38. The ARP entry became active on interface Management1.
Now, let’s delete the static ARP entry:
Arista-2(config)#no arp 10.0.0.222 0014.6aa2.d438 arpa
And here’s the output from the show event-monitor arp
command after the change:
Arista-2(config)#sho event-monitor arp 2019-01-16 22:42:42.091148|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-16 22:44:19.899883|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|1 2019-01-17 01:45:24.089620|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|2 2019-01-19 20:04:58.469700|88.0.0.1|Ethernet48|||removed|3 2019-01-19 20:05:05.205365|88.0.0.1|Ethernet48|28:99:3a:be:a0:26|0|added|4 2019-01-19 20:06:30.330914|10.0.0.222|Management1|00:14:6a:a2:d4:38|1|added|5 2019-01-19 20:08:09.111484|10.0.0.222|Management1|||removed|6
Reading the line in bold, we can see that at 8:08 p.m. on January 19, 2019, the ARP entry for the IP address 10.0.0.222 was removed. The ARP entry was previously active on interface Management1.
Remember that ARP entries are made on the switch only when the switch communicates directly using IP or if the ARP table is manually manipulated. If you have no IP addresses active on your switch, this table will be empty. For example, if you manage your switches solely through console servers and don’t even use the Management interface, you might not need to have IP addresses active on your switch at all. In this case, there would be no ARP activity to record.
Event Monitor includes CLI commands to help you organize this output in various ways without having to resort to grep or arcane Bash commands (which I recommend you learn, anyway). For example, you can change how the output is grouped instead of showing the default order, which is time based. This is done by using the group-by
keyword, which as of EOS 4.21.1F has the options interface
, mac
, and ip
. If we group by IP address, our output changes to look like this:
Arista#sho event-monitor arp group-by ip 2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1 2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5 2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4
Why are there fewer lines? Because this is the functional equivalent of the SQLite GROUP BY
statement:
Arista#bash sqlite3 /var/log/eventMon.db 'select * from arp GROUP BY prefix;' 2019-01-02 22:04:45.273661|10.0.0.100|Management1|0c:c4:7a:a8:87:ad|0|added|0 2019-01-03 01:31:47.828272|10.0.0.95|Management1|00:1c:73:28:2f:6c|0|added|1 2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5 2019-01-09 22:15:58.869723|10.10.18.5|Ethernet32|||removed|4
You can also show only specific matches using various match statements from CLI:
Arista#sho event-monitor arp match-interface e31 2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2 2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5
Using CLI like this is actually a very cool option because it allows abbreviation, which SQLite does not. Notice that I used e31 as the interface name. That doesn’t work using SQLite directly:
Arista#bash sqlite3 /var/log/eventMon.db "select * from arp where intf='e31';" Arista#
To make that work we need the exact name as referenced in the table:
Arista#bash sqlite3 /var/log/eventMon.db "select * from arp where intf='Ethernet31';" 2019-01-09 20:24:33.967351|10.10.18.1|Ethernet31|28:99:3a:be:9c:f8|0|added|2 2019-01-13 18:53:53.460543|10.10.18.1|Ethernet31|||removed|5
Hell, just looking at that SQL statement is probably enough for the typical network engineer to say, “yeah, I’m not doing that,” so having the CLI commands that wrap that up into network engineer–friendly syntax is wonderful. You can also match on IP address, MAC address, and time.
MAC changes are logged any time a MAC address is learned, added, or deleted. This can happen a lot on a busy switch, but it might not happen much at all on a smaller, stable network. This can be a pretty useful tool, so let’s look into how you can use it by taking a look at the table the same way we did for the ARP table. First, let’s look at the schema via the .schema mac
SQLite command:
Arista-2#bash sqlite3 /var/log/eventMon.db ".schema mac" CREATE TABLE mac (time text,fid integer,ethAddr text,intf text,type text,delta text,counter integer UNIQUE);
Here’s the breakdown of the fields:
The time at which this log entry was added.
FID stands for Filter ID. Switches keep an internal database of VLANs, and Arista switches assign internal VLANs to certain ports. This number will likely be unimportant to you because the interface on which the MAC address was learned is also present.
The MAC address related to this event.
The interface on which this MAC event occurred.
How did the entry change? A typical value would be learnedDynamicMac
. When a MAC address is removed from the table, this field is null. If you statically add a MAC address by using the mac address-table static
command, this field will read configuredStaticMac
.
How did the entry change? Typical entries are added
and removed
.
Every time an entry is made, it is assigned a counter value as a unique identifier for this record.
So, let’s look at an example of how I like to use this feature. Generic Bob comes up to you at 4:30 p.m. on a Friday and says, “Hey, the network sucks.”
“What is it this time, Bob?” you ask, trying really hard not to roll your eyes, since you read that chapter in Network Warrior about how not to be that guy.
“My system doesn’t work, and it used to. Nothing changed, and I don’t feel like thinking, so it has to be the network.”
After a heavy sigh, you stop the important work you were doing and focus on Generic Bob’s dilemma. “Where are you connected, Bob?”
“How should I know? You’re the network guru, and it’s your crappy network, so you figure it out.”
Repressing the thoughts that are better left to mystery writers and psychopaths, you ask, “Well, do you know your IP address?”
“No, I think it’s dynamic or something. Will this take long?”
“Look Bob, I need either the IP address or the MAC address to find your system. Get me either of those, or even better, both, and I’ll see what I can find, OK?”
Generic Bob storms off to do his generic thing while you pull up your résumé. Just as you begin to include all of your Arista switch experience, Bob comes back and thrusts a piece of paper at your face. “Here’s your Mick address.”
You smile the false, hopeless smile of the damned, gently take the paper, and log in to your Arista switch. Generic Bob’s coffee-stained paper has 3c:07:54:43:88:d4 scribbled on it in what appears to be red crayon, or maybe lipstick.
With Event Monitor, you have a record of every MAC event for quite some time. Using the match-mac
keyword, you pull up the event-monitor mac
table including only events regarding the MAC address you care about:
Arista#sho event-monitor mac match-mac 3c:07:54:43:88:d4 2018-11-26 09:18:08|1006|3c:07:54:43:88:d4|Ethernet24|learnedDynamicMac|added|549 2019-01-03 16:19:51|1|3c:07:54:43:88:d4|Ethernet6|learnedDynamicMac|added|553
Looking at this data, you can see that Generic Bob’s MAC address was first learned on Ethernet24 on November 26, 2018. Because you know that e24 is an uplink to another switch, you know that Bob’s system was originally on another switch.
But the next line shows that the MAC address was learned today (the first day back from work after the new year), at 4:19 p.m. on the switch’s local interface, e6. That seems odd, so you look at the running-config for e6 on this switch:
Arista#sho run int e6 interface Ethernet6 description [ Unused ] switchport access vlan 999
Because you’ve configured the switch with a dead VLAN (999) that you put on all unused ports, you’re sure that Generic Bob glommed a free interface on a different switch than the one to which he was assigned, likely due to either a massive hangover or a natural tendency for Generic Bob to be a pain in the ass. Or both. Yeah, it was probably both.
With a couple of short commands, you’ve shown that Generic Bob was lying because someone had clearly moved his system from one switch to another, and that’s why it stopped working. You can’t blame the guy for thinking the new Arista switches would give him better performance, but plugging into network ports without prior authorization should not be tolerated.
I’ll leave it to your imagination as to how you might deal with Generic Bob now that you have the tools to prove him wrong. Just try not to be that guy...much. I’d probably just smile and tell him to put his system back into the network port where it belonged, and then make sure his access to the data center was revoked, but not until I was done adding Event Monitor to my résumé.
This story is a complete fabrication. I’ve never met anyone named Bob who wrote a Mick address in red crayon. I did know someone who wrote her address in lipstick once, but that’s a story for another book. OK, that’s actually a fabrication, too. Now I’m sad.
As a fun aside, I asked the Arista customer engineers if they saw this feature being used in the wild and got a pretty significant response saying that it was. I also got a reply that basically reinforced that there are Generic Bobs out there and that this feature managed to save a network engineer’s job after said engineer was blamed for an outage that wasn’t his fault. Now that’s a useful feature, and I bet that engineer has been praising Arista ever since.
If you have your switches configured for IP routing, the show event-monitor route
command can be a great tool to see historical information regarding route changes. As with the others, let’s take a look at the schema and then work up an example to see it in action:
Arista#bash sqlite3 /var/log/eventMon.db '.schema route' CREATE TABLE route (time text,prefix text,protocol text,metric integer,preference integer,delta text,counter integer UNIQUE);
Here’s the breakdown of the fields from the route
table:
The time in which this log entry was added.
The IP prefix, including the Classless Internet Domain Routing (CIDR) mask related to the route entry.
I would imagine that this should read the protocol from which the prefix was learned or removed, but I’ve only ever seen this field read invalid when a route is added, or null when a route is removed.
Again, I would assume this field should contain the metric for the route, but I’ve never been able to make it display anything other than 0
when adding a route, and null when removing a route.
This field should include the administrative distance for the route, but in my testing, this field reports a 1
regardless of route type learned.
How did the entry change? Typical entries are added
and removed
.
Every time an entry is made, it is assigned a counter value as a unique identifier for this record.
Here is a sample output from one of my switches:
Arista#sho event-monitor route 2019-01-22 18:48:51.911482|127.0.0.0/8|martian|0|1|added|0 2019-01-22 18:48:51.911524|127.0.0.1/32|martian|0|1|added|1 2019-01-22 18:48:51.911565|0.0.0.0/8|martian|0|1|added|2 2019-01-22 18:48:51.931530|10.0.0.255/32|receiveBcast|0|0|added|3 2019-01-22 18:48:51.931616|10.0.0.0/32|receiveBcast|0|0|added|4 2019-01-22 18:48:51.931685|10.0.0.0/24|connected|1|0|added|5 2019-01-22 18:48:51.931749|10.0.0.14/32|receive|0|0|added|6
Wait, is my switch talking to Martians?
In networking parlance, a Martian Packet is a packet seen on the public internet using a private address that is thus considered to be “not of this Earth,” so the packets must be sourced from Martians. Usually they’re from hackers or poorly configured Border Gateway Protocol (BGP) routers, but calling them Martians is much more fun.
To take that a step further, Bogons are Martian IP addresses and IP addresses that have not been allocated (0.0.0.0/8 and 169.254.0.0/16 being two examples) and as such should also not be seen on the public internet. It has become best practice to filter Bogons on BGP edge routers because they’re usually associated with hacker attacks.
My favorite thing about Bogons is that they get their name from being the quantum of bogosity, which is the property of being bogus.
The terms Bogon and Martian are essentially interchangeable today, at least when it comes to internet routing tables. From the point of view of our switch IP routing tables, though, things like 0.0.0.0/8 and 127.0.0.0/8 should only be seen locally, whereas 10.0.0.0/8 might appear from elsewhere on the network. The important distinctions are scale and point of view. The switch might not have anything to do with the internet, so the RFC1918 addresses might very well be legitimate, whereas routes to localhost certainly would not be.
Let’s begin with a simple example. Here, I add a static route to the switch:
Arista(config)#ip route 20.20.20.0/24 10.0.0.1
Using the match-ip
ip-address
option, here’s what shows up in the Event Monitor:
Arista(config)#sho event-monitor route match-ip 20.20.20.0/24 2019-01-22 20:27:09.835624|20.20.20.0/24|staticConfig|0|1|added|7
Now I delete that route:
Arista(config)#no ip route 20.20.20.0/24 192.168.1.1
And an appropriate remove message shows up in the Event Monitor:
Arista(config)#sho event-monitor route match-ip 20.20.20.0/24 2019-01-22 20:27:09.835624|20.20.20.0/24|staticConfig|0|1|added|7 2019-01-22 20:29:00.837986|20.20.20.0/24||||removed|8
When the switch learns routes through other means, the results are pretty much the same, though the Protocol field will of course change to match how the route was learned.
LACP (802.3ad) is defined as an IEEE standard that, so far as I can tell, cannot be easily read on the internet due to the IEEE wanting us to pay for the specifications to this open standard. That is infuriating, so I’ll leave it to you to determine whether there are any number of ways to get around that while I sit here and grumble to myself about the IEEE and its supposedly open standards.
The output of this event-monitor
is filled with things that might not make much sense if you don’t have a copy of the IEEE specification, and to make matters more complicated, the internal Arista documents state, “The primary purpose of logging LACP events is to allow for post-mortem failure analysis of the Receive and Mux State Machines…” Quick, what is the purpose of the mux State Machine in LACP? No problem; I’ll just check out my copy of the IEEE 802.3ad specification…oh, right.
I have included only a small sample output for the show event-monitor lacp
command because the resulting text is so wide that there was really no hope of formatting it well in a book. Here are a couple of lines to show what I mean:
DC4-2(config)#sho event-monitor lacp 2019-01-22 21:16:07.444659|Ethernet33|unknown|rx|rxSmInitialize|12|0|0|0|0|0| selectedStateUnselected|0|1 2019-01-22 21:16:07.444815|Ethernet33|unknown|rx|rxSmPortDisabled|1|0|0|0|0|0| selectedStateUnselected|0|2 [--output truncated--]
This command is a little different from the other event-monitor
options in that it also has a verbose option. Note that the output has been significantly altered to fit on the page. I’ve altered the date format and collapsed the width considerably:
DC4-2#sho event-monitor lacp verbose Time Port LAG State Reason -------- ---- ------ ------------ ---------------------------------------------- 21:16:07 Et33 Initialize Initializing state machine from Sysdb state 21:16:07 Et33 PortDisabled Unconditional transition 21:16:07 Et33 LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE 21:16:07 Et33 Po1 Expired LACP has been re-enabled 21:16:07 Et48 Initialize Initializing state machine from Sysdb state 21:16:07 Et48 PortDisabled Unconditional transition 21:16:07 Et48 LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE 21:16:07 Et48 Po1000 Expired LACP has been re-enabled 21:16:07 Et47 Initialize Initializing state machine from Sysdb state 21:16:07 Et47 PortDisabled Unconditional transition 21:16:07 Et47 LacpDisabled portEnabled changed to TRUE with lacpEna=FALSE 21:16:07 Et47 Po1000 Expired LACP has been re-enabled [--output truncated--]
Here’s what a couple of lines look like wrapped with their original widths and content:
DC4-2#sho event-monitor lacp verbose Time Port LAG State Reason --------------------------- ----- ----- ------------- --------------------------- 2019-01-22 21:16:07.444659 Et33 Initialize Initializing state machine from Sysdb state 2019-01-22 21:16:07.444815 Et33 PortDisabled Unconditional transition 2019-01-22 21:16:07.478546 Et33 LacpDisabled portEnabled changed to TRUE with lacpEnabled=FALSE 2019-01-22 21:16:07.479061 Et33 Po1 Expired LACP has been re-enabled [--output truncated--]
As you can see, the verbose option is well named, but it’s also significantly more useful, at least in my opinion. Here is the table layout for LACP in Event Monitor:
Arista(config)#bash sqlite3 /var/log/eventMon.db '.schema lacp' CREATE TABLE lacp (time text,intf text,portchannel text,statemachine text,state text,reason int16,portenabled int16,lacpenabled int16,currentwhiletimerexpired int16,lacpdurx int16,portmoved int16,selected text,partnersync int16,counter integer UNIQUE);
Some of these entries are a bit esoteric due to them being flags or states, as specified in the LACP IEEE specification that I don’t have.
The time at which this log entry was added.
The interface involved in this entry.
The port-channel interface involved in this entry.
The section of the state-machine
involved in this entry. Examples include rx
and mux
.
This is the new state after the state change.
Reason for change. What are the reason numbers? Check with the IEEE.
Is the port enabled? 0 = no, 1 = yes.
Is LACP enabled? 0 = no, 1 = yes.
0 = no, 1 = yes.
Has LACPDU been received? 0 = no, 1 = yes.
Has port moved? 0 = no, 1 = yes.
SelectedStateUnselected, selectedStateStandby,
or selectedStateSelected
.
Partner Sync flag 0 = false, 1 = true.
This table is much simpler than the last couple of examples:
Arista#bash sqlite3 /var/log/eventMon.db '.schema igmpsnooping' CREATE TABLE igmpsnooping (time text,vlan text,ethAddr text,intf text,delta text,counter integer UNIQUE);
Here’s the breakdown of the fields from the igmpsnooping
table:
The time in which this log entry was added.
The VLAN number on which this entry occurs.
The MAC address.
The interface.
How did the entry change? Typical entries are added
and removed
.
Every time an entry is made, it is assigned a counter value as a unique identifier for this record.
If you have multicast running, this table will show the changes to the multicast route (ip mroute) table similar to the way that the regular route table changes are logged.
Arista#bash sqlite3 /var/log/eventMon.db '.schema mroute' CREATE TABLE mroute (time text,vrf text,sourceIp text,groupIp text,intf text,intfType text,delta text,counter integer UNIQUE); Arista#
Here’s the breakdown of the fields from the route
table:
The time at which this log entry was added.
The VRF in which this mroute resides.
The source IP address for this entry.
The group IP address for this entry.
The name of the interface for this entry.
The type of the interface (iif
/oif
).
How did the entry change? Typical entries are added
and removed
.
Every time an entry is made, it is assigned a counter value as a unique identifier for this record.
The database for Event Monitor is found in the var/log/ directory. This directory does not survive a reboot, so all of the Event Monitor entries will be lost every time the switch reboots. If you’d like to keep a (sort of) permanent log of these events, you’ll need to give Event Monitor a location to store its logs in the /mnt/flash directory. You can do this by using the event-monitor backup path
file-path
command:
Arista(config)#event-monitor backup path em.log
Specifying a filename without a path results in the file being placed in flash:
Arista(config)#dir Directory of flash:/ -rwx 638234211 Nov 12 2018 EOS-4.20.1F.swi -rwx 700978970 Nov 12 2018 EOS-4.21.1F.swi -rwx 51365 Jun 8 20:43 arpem.log.1 -rwx 27 Jun 6 20:43 boot-config drwx 4096 Jun 6 20:48 debug -rwx 51304 Jun 8 20:43 igmpsnoopingem.log.1 -rwx 51304 Jun 8 20:43 lacpem.log.1 -rwx 51363 Jun 8 20:43 macem.log.1 drwx 4096 Jun 6 20:48 persist -rwx 51365 Jun 8 20:43 routeem.log.1 drwx 4096 Jun 3 01:20 schedule -rwx 2893 Jun 6 20:43 startup-config -rwx 19 Jun 3 01:13 zerotouch-config 1862512640 bytes total (256790528 bytes free)
If you specify what you think is a full Unix path, such as /home/admin/em.log, you will not get the expected results, because this will translate to flash:/home/admin/em.log:
Arista(config)#event-monitor backup path /home/admin/em.log Arista(config)#dir Directory of flash:/ -rwx 638234211 Nov 12 2018 EOS-4.20.1F.swi -rwx 700978970 Nov 12 2018 EOS-4.21.1F.swi -rwx 27 Jun 6 20:43 boot-config drwx 4096 Jun 6 20:48 debug drwx 4096 Jun 8 20:44 home drwx 4096 Jun 6 20:48 persist drwx 4096 Jun 3 01:20 schedule -rwx 2893 Jun 6 20:43 startup-config -rwx 19 Jun 3 01:13 zerotouch-config 3269361664 bytes total (1220587520 bytes free)
You can specify a Unix path. The command provides two options when asked:
Arista(config)#event-monitor backup path ? file: forever log URL flash: forever log URL
The idea of a forever log sounds oddly romantic, like rescuing a dog from the pound and bringing him to his forever home, but what this file versus flash choice is offering is the choice between a Unix path (file:) or a CLI path (flash:). If you do not specify either, the parser will assume flash:. That’s why our previous command using /home/admin created a directory in flash:. To actually put the file in /home/admin (which would be dumb, because /home will not survive a reboot either), use the following command, incorporating file: at the beginning of the path:
Arista(config)#event-monitor backup path file:/home/admin/em.log
That will put the database file where you want it:
Arista#bash ls /home/admin em.log.0
The EOS device, file:, is a pseudodevice that lets you access the Linux filesystem from within the CLI. You can use it anywhere you would use another device name, such as copy flash:GAD.txt file:/tmp
.
You might have noticed that my files all have a .0 appended to them, although I did not specify that. Event Monitor will write 500 events to the file, after which it will create a new file, appended with a .1. The logs are circular, so when the last log (default .200) is reached, the .0 file is deleted, and new data will be written there. That’s why I wrote earlier that these files were sort of permanent. To configure how many log files you would like to retain, use the event-monitor backup max-size
number
command:
Arista(config)#event-monitor backup max-size ? <1-200> maximum number of stored backup logs
I’ll specify 10 log files, for no other reason than I like the number 10:
Arista(config)#event-monitor backup max-size 10
To disable Event Monitor completely, use the no event-monitor all
configuration command:
Arista(config)#no event-monitor all
To enable only one of the Event Monitor functions, use the event-monitor
command, followed by the function (mac
, arp
, or route
) that you’d like to enable. You can also disable single functions by negating these commands:
Arista(config)#event-monitor route
To enable them all again, use the all
keyword:
Arista(config)#event-monitor all
Finally, you can configure a maximum buffer size for Event Monitor. The Arista manuals show the buffer size in Kb (kilobits), which I kind of think is a mistake, but if it’s not, 1 kilobit = 128 bytes.
You can configure the buffer size from 6 to 50 units in size. If backup logs are configured, they are written to when the buffer becomes full:
Arista(config)#event-monitor buffer max-size ? <6-50> maximum size of the primary event monitor log
For most environments, leaving the Event Monitor as is and enabled probably makes the most sense. I see no reason to disable it, and even though you might use it rarely, it can be an invaluable tool when you do need it.
I think Event Monitor is a terrifically useful feature, but I think it’s one of those features that can be difficult to appreciate until you’ve been through a situation for which its use could be beneficial. I could argue that’s true of any feature, but since this feature is so unique, I think it’s doubly true. My goal is to plant the idea in your mind that something like this is useful. That way, even if you don’t use it right away, if in a year’s time you come across a need for Event Monitoring, perhaps you’ll take a second and think to yourself, “Dammit, GAD was right.” I live for those moments.