RSS

Category Archives: Network Admin

VMware 6.5 LACP Configuration

I just got back from Cisco Live in Las Vegas and brought a nasty cold with me. With the extended time off work, I came back to a sizeable pile of “priorities”. One of the more interesting/challenging things I was working on was getting LACP going. We had some new rackmounts that were partially configured and had ESXi installed. They had two 10Gb ports which were handling all the traffic on standard virtual switches. The networking guy wanted to turn on LACP which is a best practice, but we couldn’t get it going at first for various reasons. It is one of those settings that has to be either on or off and both the host side and switch side have to be the same. Now that the project is nearing some deadlines, we decided to give it another go.

There are a couple key reasons you might want to setup Link Aggregation Control Protocol on uplink ports. 1. For faster failover in the event of a switch or port going offline. 2. Higher bandwidth for a single logical uplink.

VMware does a pretty nice job of handling failover without this turned on. VMs run on a single connection and jump over to the other connection if there are issues. So that might be reason enough not to go through the extra hassle of setting up LACP. Another reason is you might get yourself in a chicken/egg catch 22 scenario. If your vmkernal management ports have to run LACP, and you have a vCenter that runs on a host with only LACP available… you might have a hard time configuring your virtual distributed switch. You might be able to log into a remote console of the host and revert management network changes in 6.5, but I have not tested this. For this reason, I recommend using some different (perhaps a pair of onboard 1Gb ports) for your ESXi management network.

So step one is to get your host online with a VMware standard switch. Then you can deploy the vCenter 6.5 appliance to this host. You will need this to configure LACP. I would also recommend using the standard switch for vCenter and ESXi traffic. This can be done by editing the port group on the vCenter appliance VM.

On the physical switch side, you must setup a vPC. This is done by configuring a port channel on each switch port, then a virtual port channel that pairs the two ports.

Then in vCenter, you create a distributed virtual switch. Under the configuration tab there are LACP settings. First create a Link Aggregation Group. You will want to set this to active so the NIC will negotiate with the physical switch to aggregate the links. Create one LAG with the number of ports you will have for VM traffic in your entire cluster. This is one step that confused me. The documentation says to create one LAG per port channel ( https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.networking.doc/GUID-34A96848-5930-4417-9BEB-CEF487C6F8B6.html ), however, VMware handles creating a LAG for each host and you only need to create the overall LAG for the distributed switch. So basically I got one host setup pretty easily, but then when I went to setup my second host, I couldn’t add the second LAG into the uplinks options because two LAGs are not supported.

Once you create the LAG, you can now add hosts to the distributed switch and assign the physical NICs as uplinks with the LAG selected.

Lastly, create portgroups for each vlan. Then you can assign the LAG uplinks under teaming and failover for each port group.

 
Leave a comment

Posted by on July 16, 2017 in Network Admin, Virtual

 

ICND1 100-101 Study Progress 2

I have reached page 682 of the Odem book which is where I am going to stop. Now I am going finish typing up my notes. Next I will use the attached CD to quiz myself to figure out what areas I need to brush up on in the coming weeks.

CHAP19 Subnet Design p533
– count the bits know the powers of 2
– 2^10 is 1024 and that is easy to remember

CHAP20 VLSM p561
– Old routing protocol doesn’t support vlsm (RIP)
– no additional config to get this to work
– be able to find overlap of networks to troubleshoot

CHAP21 Route Summarization p577
– strategy used for performance to lower the size of routing tables
– subnet design should have summarization in mind
Steps to finding the best summary route
1. list all decimal subnets in order
2. note low and high points
3. pick the shortest prefix length mask and subnet -1
4. calculate new potential network mask summary

CHAP22 Basic ACLs p599
– ACLs most common use is a packet filter
– can match source and/or destination
– match packets for QoS
– to filter a packet you must enable acl on the interface either enter or exit
– NAT uses ACL permits
– when processing ACL list router uses first match logic
– ex command: access-list 1 permit 10.1.1.1
– To figure out wildcard, get mask and subtract
255.255.255.255
-255.255.252.0
—————-
0.0.3.255

*know where the best place to put the ACL is and on what router in the path

CHAP23 Advanced ACLs p623

ACLs are numbered or named
– to make a change to the list, must delete the whole list and reconfigure
– extended ACLs allow for more packet headers to be searched
– example command: access-list 101 permit protocol SIP wildcard DIP wildcard
– example command: access-list 101 deny tcp any gt 1023 host 10.1.1.1 eq 23
– keywords can be used instead of port #s (HTTP instead of 80)

Named ACLs, differences
– easier to remember
– subcommands not global
– allows single line deletion

numbered ACLs allow for new style of command

config t
do show ip access-list 24

ROUTER and switch SECURITY
– use the “enable secret” command
– username secrets if external auth not available
– disable telnet
– avoid using simple password checking
– disable unused services
– use ACLs to secure SSH
– extended ACLs close to source
– Standard ACLs close to destination
– Specific ACLs early in list

enable secret myPass
-this sets the password of myPass to reach enable mode

CHAP24 NAT p653
– CIDR route summarization
– classless interdomain routing
– inside local: local ip assigned to host
– inside global: what the internet knows your network as. address used to represent inside host as packet hits internet
– outside global: public ip outside enterprise (the ip of the URL you are trying to access)

PAT is port address translation
pic on p664-uses source port to return traffic to proper client
NAT troubleshooting
-don’t mix up ip nat inside and ip nat outside addresses
-don’t mix up local and global addresses in this command: ip nat inside source static 192.168.1.2 207.53.23.132
-dynamic NAT uses ACLs, check these
-PAT uses the overload command on ip nat inside source command

TESTING PROGRESS

I took a couple 10 question tests from the CD. The idea was hit some chapters that I struggled with which were, WANs, ACLs and NAT. I got 6 out of 10 questions right which isn’t all the great.

Next I took a test of the first 5 chapters of the book. I scored 8 out of 10 right which is passing for the book test. The only concept I wasn’t sure on was crossover cable pin numbers and when to use a straight through and crossover cable. I knew like devices use crossover cables but that alone didn’t help me get the two questions right. I may memorize this table for the test.

TRANSMIT PINS
routers Hubs
pcs Switches
1,2 3,6
 
Leave a comment

Posted by on May 30, 2015 in Network Admin

 

ICND1 100-101 Study Progress

icnd1_study_progress

I’m starting to see the fruits of an aggressive study plan. Here we are, May 23rd, roughly two weeks until test time and I am nearly on track.

Part I: Networking Fundamentals
Part II: Ethernet LANs and Switches
Part III: Version 4 Addressing and Subnetting (Be done by May 11th and practice subnetting)
Part IV: Implementing IP Version 4 (Be done by May 18th and practice show commands)
Part V: Advanced IPv4 Addressing Concepts
Part VI: IPv4 Services (Be done by May 26th, decide if I want to skip IPv6, Review OSPF and practice more advanced subnetting)
Part VII: IP Version 6
Part VIII: Final Review (Be here by June 1st and have taken a practice exam to decide what areas to review)

I got off to a rocky start with an older 2008 version of the book. Fortunately my study buddy had purchased the correct book instead of borrowing an old one. I had gotten two chapters into the old book and before I started to really get into the newer edition that took a week to recieve. I decided to take a practice test early on. The test is very configurable. I chose study mode for 45 questions and limited myself to 90 minutes with a small chunk of whiteboard. I also decided to exclude any IPv6 questions from this first stab.

After two chapters and a couple videos on subnetting I was able to get a 600 which is 200 points away from passing. This was on the practice test that came on the CD in the book. The higher layer concepts I did quite well on where as the lower layer concepts such as Routing, WANs, ACLs and any kind of IOS commands and configuration questions I did very poorly on. Subnetting seems to get a lot of attention either directly, or indirectly and I was sitting at about 50% or less on that.

What is subnetting?

Don’t listen to me, I’m not an expert, but I don’t think there are many good explanations of this out there. A lot of people go way deep and off on tangents to frequently. Here is my overview of what I understand are important subnetting concepts for ICND1.

IP Address = 32 bits = 4 Octects = 4 bytes

Each byte can store 256 possible combinations of 1s and 0s. So lets represent 10.0.0.1 in binary, 00001001.00000000.00000000.00000001

See, that is 32 bits in an IP address.

The second concept we need to understand is the netmask. Picture a mask you might put on your face. A very thick mask you won’t be able to see much. A thin mask you might be able to see a lot.

Take that concept and apply it to this very common netmask 255.255.255.0, or 11111111.11111111.11111111.00000000

Out of all the possible combinations that is a pretty thick mask so I can only see a small number of hosts with that mask. If you combine the IP & netmask, you will be able to see IP address from 10.0.0.0- 10.0.0.255 or 256 possible hosts.

And there you have it, networking. Wait, what was I talking about? Ah yes, SUBnetting.

Subnetting takes those 256 possible hosts and divides them into smaller networks. If I needed several separate networks and only 18 hosts per network I could split that 10.0.0.0/24 network into smaller chunks. If I want to see fewer hosts in my network I need a thicker, or higher number mask.

Pulling up the /24 mask again, 11111111.11111111.11111111.00000000 you will see it is /24 because there are 24 1s or network bits and 8 0s or host bits.

In our problem, we need at least 18 IP address options for hosts. For this we will use 0s. How many 0s will we need? Less than 8 for sure because that gave me 256 options. But how many less?

The powers of 2 come in handy for any binary math. There are 2 possible values for each bit, 0 or 1. With 2 bits there are 4 possible values, 00, 11, 10, 01. That isn’t going to get me to at least 18 hosts. This could take a while and for the ICND1 test you need to subnet in 15 seconds. Yikes!

In comes the cheat sheet.

subnet_table

Memorize this formula to go with the table: Possible hosts on a network = 2^h – 2

Each network supports 2^h ip addresses, however 1 ip address is used for the network id and another is used for the broadcast address, hence the minus 2 part.

I don’t suggest just memorizing the table. I would suggest understanding how to generate the table. Start from the top right and do your powers of 2 up to 128. 2^0 = 1, 2^1 =2 2^2=4 … 2^7=128

Next is the second row, the decimal mask. Take 256 – the h row to get the decimal mask row.

Next is the last 2 octets of cidr notation. This is simply a count of 1s in the binary representation of the mask. Remember 1s are the network bits and 0s are the host bits.

Once we have this table we can solve our problem, subnet 10.0.0.0/24(think 10.0.0.0-255/24) in a way that supports at least 5 networks and at least 18 hosts in each network.

Start this question with the important number h, or 18.

Go to the table and find the h value that supports at least 18 hosts, which is 32.

Go down to the decimal notation .224 and we know that we can support at least 18 hosts with a decimal mask of 255.255.225.224.

Next we can list the network IDs that this mask could possibly create
10.0.0.0/27
10.0.0.32/27
10.0.0.64/27
10.0.0.96/27

10.0.0.224/27

To figure this out mathematically take 2^n where n = the number of network bits. There are 3 network bits or 1s in the octect we subnetted. We can make 8 networks which is greater than 5 required by the problem. BOOM CAKE!

subnet_table_answer

For the remainder of this post I will be simply typing up my notes from the Wendell Odom Cisco Press Book and some other notes I took watching YouTube videos from a variety of authors which I will link to.

PLEASE DO NOT THROW SAUSAGE PIZZA AWAY Kevin Wallace

1. Physical – wiring standards, physical topology, bandwidth usage, syncronizing bits
2. DataLink – MAC, Flow Control standards
3. Network – IP, IPX, Switching, Route Discovery, TTL
4. Transport – TCP, UDP, windowing, buffering
5. Session – Netbui
6. Presentation – jpg, encryption, data formatting(ascii, ebcidic)
7. Application – http, smb, smtp, service advertisement, dns

IP Addressing

First Octects
CLASS A – 1-127
CLASS B – 128-191
CLASS C – 192-223

Hub – layer 1 device that simply spams all ports with frames

Rember these things in this order
SEGMENT – includes the tcp ports
PACKET – includes the IP
FRAME – the whole stinking thing with headers and trailers

Encapsulation – IP Packet is a Layer 3 PDU

CHAP2: Fundamentals of Ethernet Lans

UTP – unshielded twisted pair

crossover cable
1-3
2-6
3-1
6-2

like devices need crossover cable to switch transmit and receive pins

MAC – 48bits – 24 for OUI

FCS – frame check sequence is at the end of the frame to ensure proper delivery

CHAP3: WANs

leased line , service provider
CPE – customer premises equipment
CSU/DSU – channel service unit, data service unit usually on prem and RJ-48
Router-Router communication can occur on serial cables
HDLC – high level data link control
——way of encapsulating frames over WAN
PPP – point to point protocol
MPLS – multi protocol label switching

CHAP4: IPv4 Addressing and Routing

Routing uses L3PDUs
Layer 2 are called frames

IPv4 headers are 20 bytes and include SIP,DIP,Len,offset,chksum,ttl,etc…

CLASS A: 126 networks and 16,777,214 hosts per network
CLASS B: 16,384 networks and 65,534 hosts per network
CLASS C: 2,097,152 networks and 254 hosts per network

Router FWD logic
1. uses FCS to make sure no errors
2. discard old frame header and trailer
3. compare DIP to routing table and find next hop
4. encapsulate

CHAP 5: fundamentals of TCP/IP transport applications

UDP – connectionless

Connection establishment
SYN —->

Connection termination
ACK FIN —>
enable
switch#
switch#disable
switch>

shutdown – command that turns a port down/down
no shutdown – turns a port up/up (the second up is if the protocol works)

CHAP 8: configuring Ethernet Switching

enable secret mYpass
show history
no shutdown

port security
1. switchport mode access (access or trunk)
2. switchport port-security (enables port security)
3. switchport port-security maximum 2 (allowed macs on port)
4. switchport port-security violation shutdown (action to take)
5. switchport port-security mac-address AAAA:AAAA:AAAA (specifiy allowed macs)
6. switchport port-security mac-address sticky (dynamic learned mac addresses)

CHAP 9: implementing VLANs

802.1Q
ISL = OLD protocol

12bits for VLANID (this is a “shim” in the frame)
how many vlans? 2^12 or 4096
vlanid 1 is default

router on a stick – one physical link to a router instead of two

show vlan brief

(allow port 4 to communicate on vlan id 10)
1. enable
2. configure terminal
3. interface FastEthernet0/4
4. switchport access vlan 10

Layer3 switch does routing …but can’t do this in packettracer :[

Reasons switch prevents VLAN traffic from crossing a trunk
1. removed from allow list
2. vlan doesn’t exist in show config
3. vlan doesn’t exist, been disabled

and some other less important reasons

CHAP 10 Troubleshooting

show cdp neighbors
show interfaces status

“administratively down” means shutdown command was run
err-disabled means port security

vlan troubleshooting
1. identify all access interfaces and their vlans
2. do vlans exist and are they active
3. check allowed vlan list on both ends of the trunk
4. check for trunk/no trunk neighbors

show vlan brief

PART III CHAP 11

IPv4 subnetting

One subnet for every:
1. vlan
2. ppp serial link
3. EoMPLS
4. frame relay

VLSM – variable length subnet mask

RESERVATIONS:
10.0.0.0
172.16.0.0-172.31.0.0
192.168.0.0-192.168.0.255

CHAP 12 analyzing classful IPv4 Networks
CHAP 13 analyzing subnet masks
CHAP 14: analyzing existing subnets

CHAP 15: Operating Cisco routers

Installation steps
1. connect lan ports
2. connect CSU/DSu external
3. connect CSU/DSU internal
4. connect console port to pc using a rollover cable
5. connect power
6. power on

show ip route

show mac address-table

status layer 1/status layer 2
up/up
down/down : has not been shutdown but physical layer problem

CHAP 16: configurating IPv4 addresses and routes

routing
1. choose to process frame
-proper mac (is its destination me?)
-no errors (FCS)
2. de-encapsulate packet
3. compare DIP to routing table
-this identifies outgoing interface
4. encapsulate
5. transmit

routers should ignore switch floods not intended for it

large routing tables can cause performance problems

cisco express forwarding
-uses organized tree and other tables to help speed up routing

adding routes can be done via:

1. connected routes
2. static routes
3. routing protocols

cisco will add routes if the interface is IP’d and UP

ROAS 802.1Q trunk

CHAP 17: OSPF

commands to turn on

router ospf 1
network 0.0.0.0 255.255.255.255 area 0

ospf – open shortest path first – uses link state
OSPFv2 is for IPv4

routing protocol – set of messages, rules and algorithms (RIP, EIGRP,OSPF,BGP)

routed & routable protocol – defines packet structure and addressing (IPv4)

BASIC FUNCTIONS
1. learn routing information about ipsubnets from neighboring routers
2. advertise this info
3. if more than 1 route exists, pick best
4. if topology changes, advertise current best route (convergence)

Interior gateway protocol – designed for use inside a single autonomous system
exterior gateawy protocol – BGP

routing algorthims use
1. distance vector
2. advanced distance vector
3. link state (ospf uses this)

RIP is old
IGRP is a little less old

RIP-2 uses hop count and is also old with slow convergence
OSPF is a cost based protocol
EIRGP – cisco proprietary and uses bandwidth and latency
IS-IS – uses link state

PROTOCOL RANKS
0 connected
1 static
20 BGP E
90 EIGRP
110 OSPF
115 IS-IS
120 RIP
200 BGPI

this will show the database of link state advertisements(LSAs)
show ip ospf database

routers must agree to be neighbors

configuration, this will turn on for any interface that matches 10.0.* because of the wildcards in network command

router ospf
network 10.0.0.0 0.0.255.255 area 0

CHAP 18

DHCP – DORA
Discover – TO 255.255.255.255 FROM 0.0.0.0
Offer
Request
Acknowledge

ip helper-address {dhcp server ip} – command for router that enables DCHP servers to sit outside of the subnet by changing SIP&DIP ( Thanks /u/Sprockle )

 
Leave a comment

Posted by on May 23, 2015 in Network Admin

 

Gearing up for another exam ICND1 100-101

The more I learn about networks, the less I tend to blame the network.

It was almost 20 years ago that I set a static IP address on my sisters computer and connected a cross over cable to my computer so we could play a game called Quake. She wasn’t that interested so I ran back and forth between the rooms and played by myself. This loneliness was resolved a few years later with a device that looked something like this

Fax_modem_antigohttp://en.wikipedia.org/wiki/Modem

Point is, I’ve been doing this for a long time and I still don’t know jack. I don’t like to fail tests, so signing up for a test is going to help me learn. I would like to become a more well rounded datacenter administrator.

icnd1_book

ICND1 100-101 is the first half of a valuable certification CCNA. I now have the book in hand and about 5 weeks to prepare. Normally, I would allow myself about 3 months with a book this size but opportunity has struck and I need to accelerate my pace.

Like Microsoft, Cisco is very open with their exam topics. https://learningnetwork.cisco.com/community/certifications/ccna/icnd1_v2/exam-topics

1.0 Operation of IP Data Networks 6%
2.0 LAN Switching Technologies 21%
3.0 IP addressing (IPv4/IPv6) 11%
4.0 IP Routing Technologies 26%
5.0 IP Services 8%
6.0 Network Device Security 15%
7.0 Troubleshooting 13%

These do not line up that nicely to the book topics. But I am going to attempt to cruise through the book which I have given myself some milestones below.

Part I: Networking Fundamentals
Part II: Ethernet LANs and Switches
Part III: Version 4 Addressing and Subnetting (Be done by May 11th and practice subnetting)
Part IV: Implementing IP Version 4 (Be done by May 18th and practice show commands)
Part V: Advanced IPv4 Addressing Concepts
Part VI: IPv4 Services (Be done by May 26th, decide if I want to skip IPv6, Review OSPF and practice more advanced subnetting)
Part VII: IP Version 6
Part VIII: Final Review (Be here by June 1st and have taken a practice exam to decide what areas to review)

The schedule is set, plans are in place, now it is time for me to do some reading.

 
Leave a comment

Posted by on May 9, 2015 in Network Admin

 

5 9s Lead to Nestfrastructure (and fewer 9s)

Off the top of my head,

Microsoft DNS issue a handful of hours before xbox one launch(http://redmondmag.com/articles/2013/11/21/windows-azure-outages.aspx)

Widespread Amazon outages (http://www.zdnet.com/amazon-web-services-suffers-outage-takes-down-vine-instagram-flipboard-with-it-7000019842/)

NASDAQ (http://www.bloomberg.com/news/2013-08-26/nasdaq-three-hour-halt-highlights-vulnerability-in-market.html)

The POTUS’s baby (http://www.healthcare.gov)

I learned about 5 9’s in a college business class. If a manufacturer wants to be respected as building quality products, they should be able to build 99.999% of them accurately. That concept has translated to IT as some kind of reasonable expectation of uptime. (http://en.wikipedia.org/wiki/High_availability)

I take great pride in my ability to keep servers running. Not only avoiding unplanned downtime, but developing a highly available system so it requires little to no planned downtime. These HA features add additional complexity and can sometimes backfire. Simplicity and more planned downtime is often times the best choice. If 99,999% uptime is the goal, there is no room for flexibility, agility, budgets or sanity. To me, 5 9s is not a reasonable expectation of uptime even if you only count unplanned downtime. I will strive for this perfection, however, I will not stand idly by while this expectation is demanded.

Jaron Lanier, the author and inventor of the concept of virtual reality, warned that digital infrastructure was moving beyond human control. He said: “When you try to achieve great scale with automation and the automation exceeds the boundaries of human oversight, there is going to be failure … It is infuriating because it is driven by unreasonable greed.”
Source: http://www.theguardian.com/technology/2013/aug/23/nasdaq-crash-data

IMHO the problem stems from dishonest salespeople. False hopes are injected into organizations’ leaders. These salespeople are often times internal to the organization. An example is an inexperienced engineer that hasn’t been around for long enough to measure his or her own uptime for a year. They haven’t realized the benefit of keeping track of outages objectively and buy into new technologies that don’t always pan out. That hope bubbles up to upper management and then propagates down to the real engineers in the form of an SLA that no real engineer would actually be able to achieve.

About two weeks later, the priority shifts to the new code release and not uptime. Even though releasing untested code puts availability as risk, the code changes must be released. These ever changing goals are prone to failure.

So where is 5 9s appropriate? With the influx of cloud services, the term infrastructure is being too broadly used. IIS is not infrastructure, it is part of your platform. Power and cooling are infrastructure and those should live by the 5 9s rule. A local network would be a stretch to apply 5 9s to. Storage arrays and storage networks are less of a stretch because the amount of change is limited.

Even when redundancies exist, platform failures are disruptive. A database mirroring failover (connections closed), webserver failure (sessions lost), a compute node (os reboots) and even live migrations of vms require a “stun” which stops the cpu for a period of time(a second?). These details I listed in parentheses are often omitted from the sales pitch. The reaction varies with each application. As the load increases on a system these adverse reactions can increase as well.

If you want to achieve 5 9s for your platform, you have to move the redundancy logic up the stack. Catch errors, wait and retry.

stack

Yes, use the tools you are familiar with lower in the stack. But don’t build yourself a nest at every layer in the stack, understand the big picture and apply pressure as needed. Just like you wouldn’t jump on every possible new shiny security feature, don’t jump on every redundancy feature to avoid nestfrastructure.

 

Arp Spoofing with arpspoof

Consider someone has hijacked your DNS server. That person modifies the record for “prod_database.domain.com” to point to their IP address. At no additional charge, after capturing all the packets, they will kindly forward them on the the real prod_database. That would be a layer 7 to layer 3 link switcheroo.

Arp spoofing is a similar concept but instead of names to IPs, we modify the IP(layer 3) to MAC(layer 2) relationship. To demonstrate a successful spoof, I have to tell another client on my LAN that I am the gateway, and tell the gateway that I am that client.

In order to see the damage of arp spoofing you can look at your arp table while being spoofed. Type “arp -a” at a windows command prompt in order to view the contents of your arp cache. Arp spoofing is also called arp poisoning because of the false records that the tool is able to get added to a victim’s arp cache.

Using Kali Linux as the attacker, a fresh trial of Windows 2012 R2 as the victim, VMware Player, and the command line tool arpspoof I was able to successfully capture the victim’s traffic. For the traffic to flow through Kali, the first step is to turn on port forwarding. Then in step 2 and 3 we tell the subnet some lies.

arp_spoof_01

The poising has started. If you want to see this traffic you can use the “arp” filter in Wireshark.

arp_spoof_02

Finally, to offer some proof, I browse to Wikipedia on the victim’s machine and view the traffic on the attacker’s machine.

arp_spoof_03

The defenses to this attack include SSL/TLS, OS hardening and duplicate MAC detection among other things. Unfortunately this is how some proxy like tools work and you might not be able to use all of those methods to stop the attack.

Reference: http://www.irongeek.com/i.php?page=security/arpspoof

 
Leave a comment

Posted by on March 10, 2014 in Network Admin, Security

 

Wireshark Skills Series 7 of 7

I had no idea what protocols I was going to cover when I picked the number of posts in this series. I’m glad I picked 7 or I would be at this for months. I find that when I look at captures I can turn vague concepts of how stuff works into a more thorough understanding of how stuff works.

As a speaker it can be a very awkward feeling walking off a cliff. The cliff is starting to explain something, and then not being able to finish the thought because you don’t actually know how it works. Now with wireshark skills, I can dive deep on purpose and avoid falling.

Kerberos

Kerberos is a good capture to wrap up this series. Kerberos is an authentication protocol. It is a complex protocol but I have found a great pdf that simplifies it… but not too much.

http://www.computerworld.com/computerworld/records/images/pdf/kerberos_chart.pdf

With all of those dotted lines we should be able to find something of interest in wireshark. I have a domain controller and another server setup that we’ll call a client. After setting up the domain, joining the client and creating a couple users, lots of kerberos is already happening but we are going to try to focus on those first few steps in the PDF.

The domain controller handles both the authentication service and ticket granting service. If we start a trace and add the filter “kerberos” we will get to see some traffic eventually.

First, to make it a clean run, at a command prompt type “klist”. This shows you the current tickets you have. Then type “klist purge” which will get rid of those tickets. If we have that capture started and lock our session (ctrl+alt+del lock) and re-login we will capture the first step AS-REQ. The second packet is selected in this screenshot and shows the details of the AS-REP. The “Client Name” and “Clients Realm” are self explanatory but the server name of KRBTGT may be a little misleading. There isn’t actually a server named that but it represents the place the client goes for the Ticket Granting Ticket.

Step 1 in the PDF is completed when we log in.

AS-REP_krbtgt

The next Kerberos action I see is a TGS-REQ and TGS-REP. After authentication we log onto the host(a.k.a. client) which gives us a host ticket.

TGS-REP_hostticket

This is all I will get unless I try to access network resources. Below I attempt to access a cifs share on my virtual host computer. Since it’s not joined to the same domain the authentication isn’t automatic. A popup window asks for credentials to connect to the computer “nujak” and I type in some junk. The domain controller correctly responds with “KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN”.

TGS-REQ_unknown

In these next packets I attempt to access a share on the domain contorller. Since I do have access to this share we get a successful SMB2 session start. If you dig into the security blob in the packet you can see the basic kerberos information.

SMB_KRB_sessionstart

Wrap-Up

The distributed platform forces computers to work together. If you want to have a deep understanding of why they do or don’t work together, Wireshark is the tool for the job.

 
Leave a comment

Posted by on January 21, 2014 in Network Admin

 

Tags:

Wireshark Skills Series 6 of 7

I’ve yet to hop into Wireshark with a database connection problem and figured it out because of the network capture. It is not the first place I stop. It has never come to that, I’ve never needed to go that deep. I have always used the other, more topical tools to diagnose a connection problem.

I have had an occasion where I wanted to go deeper. I wanted to find out for sure if the login information in a SQL connection was encrypted if you didn’t specifically setup TLS. The answer was it depends.. on the client. Most all implementations are encrypted. I happened to come across an old implementation that was not. It was a good answer to find out for sure and going deep with a network trace was the only way.

Today I set out to look at some packet captures of MSSQL connections and see if anything interesting showed itself. Turns out I found something very interesting. Have you ever connected to an instance of SQL server with SSMS and had it take for ever to load the list of folders? Lets take a look at a normal start to a SQL connection. I filtered out the IP and default SQL port.

normal_sql_01

This all happens in short order. In about 3 milliseconds, the client and server have completed their TCP handshake and negotiated on the TDS protocol in order to send the secured SQL username and login.

Now we have the jankity trace. The connection I attempted directly after installing SQL was much slower.

slow_sql_02

And no wonder it was slower. Look at all those re-transmissions. If an ACK is not received by the client affirming the packet was received and undamaged, the client will send the packet again.

In this particular case, I was reminded that the default amount of memory for a Hyper-V VM is 512MB. This was not quite enough to run through an install and have all the things open that I did. I shut the VM down added some RAM and my connections were running much more smoothly after that.

 
Leave a comment

Posted by on January 5, 2014 in Network Admin

 

Tags: ,

Wireshark Skills Series 5 of 7

Hopefully by now you have drank the proverbial kool aid and think Wireshark is the best tool since… ever. Yes, it is awesome but there are some security concerns. If a hacker compromises your server as a very under privileged user, it may benefit them to already have a packet capture utility installed so they can just run it instead of being denied to install it. You can also use it to discover other computers on the network and start making some dangerous assumptions.

One of the neato tricks to do is take a lengthy capture, then do a string search for your password. If your password is going over the wire, I would recommend dropping whatever service it is that you are using that would send your password over the wire, unencrypted.

Standard FTP is one of those services that will send plain text passwords. I have downloaded a .vhd for server 2012 R2 and installed the IIS + ftp role and feature.

ftp_misconfig

In the previous captures we haven’t limited the capture itself and instead we are filtering post capture. Instead of a post filter, lets apply a pre filter. Open up the capture options and filter on the standard FTP port of 21.

ftp_capture_filter

To test out the security of our authentication choice, open up the simple, built-in Windows FTP utility. Type at the command prompt: ftp [Enter] open 10.10.10.108 or whatever your practice server is and pause a second to take a look at the capture. Nothing too crazy here, just a 3 way TCP handshake and then the FTP server responding with code 220 meaning it is “ready for new user.”

start_of_ftp

I chose Basic authentication and allowed all users to connect with no SSL. That means if I enter a Windows username and password I should be able to log in.

Found_Standard_ftp_password

The login worked great! But I have also allowed my username and password to be compromised by someone capturing packets on the server.

 
Leave a comment

Posted by on December 17, 2013 in Network Admin

 

Tags:

Wireshark Skills Series 4 of 7

In some of our HTTP requests there was something interesting going on that didn’t make the cut to be explained. The magic of Ethernet was fast at work furiously chopping up data at one end of the pipe and reassembling it at the other. Packets can be all different sizes but if they exceed a limit they will get sliced and diced.

Increasing the size of the packet reduces the amount of overhead needed to transfer your data. Both ends of the pipe and everything inbetween have to support the packet size that is being sent. If a switch or router doesn’t support it the connection may fail. If it is supported but not properly configured you could run into a silent problem where that intermediate device is fragmenting the packets. It is best to change jumbo packets at the intermediate switches and communications devices first, then at the endpoints. Also, some applications like Microsoft SQL Server need configured to send larger packets too.

To illustrate this process we are going to use the ping tool. Ping runs in ICMP which is on top of IP and Ethernet. Ethernet is the layer where the MTU setting resides. Before we change the MTU lets take a look at a large ping getting sliced up like the HTTP traffic we saw earlier. This command will continuously ping with 10,000 bytes of data. For this test I am running Wireshark on a Windows 2012 R2 server. (VHD 180 day Trial: http://technet.microsoft.com/en-US/evalcenter/dn205286.aspx)

ping 10.10.10.108 -l 10000 -t

Segmented_default

If you squint really hard you can see 7 packets coming into the server and 7 packets going out of the server. Ethernet + IP require 36 bytes per packet so the overhead can add up quickly. If you look at the length, you will see no packet is over 1514 bytes in size.

Now we can try to make this more efficient. I’ll have to configure the vswitch and the virtual machine NIC to enable jumbo packets.

configure_jumbo_1

configure_jumbo_2

configure_jumbo_3

Now if we re-try that ping we should notice fewer packets.

jumbo_packet_confirmed

Success right? Well yes in this case we made the process 340 bytes more efficient. However, since I enabled jumbo packets I broke other things. Most notably my blogging software :]

 
Leave a comment

Posted by on December 16, 2013 in Network Admin