RSS

I’m Speaking at #SQLPass in Detroit on Thursday

Register at EventBrite – http://sqlpass-detroit-20130418.eventbrite.com/ so we know how much food to order.

Also check out our web site at its new address, http://detroit.sqlpass.org/
Thursday, April 18, 2013
6:00 PM to 8:00 PM

Microsoft Southfield Office
1000 Towncenter
Suite 1930
Southfield, MI 48075

Topic – Hacking SQL Server
The best defense is a good offense. Learn how to practice hacking without going to jail or getting fired. In this presentation we’ll be going over how to exploit weak SQL servers with actual tools of the penetration testing trade. You will learn why the SQL Service is a popular target on your network and how to defend against basic attacks.
We will also attempt to snag some credentials from the SAM cache so we can go galavanting across the rest of the network.

 
1 Comment

Posted by on April 17, 2013 in PASS

 

Tags:

Non-Standard SQL Port

As a DBA, I have heard of a defensive maneuver that is supposed to help throw the hackers off your scent. That maneuver is configuring a non-standard port for the database engine, something other than 1433.

We first need to understand a standard configuration. If you select only the database engine on install, the default instance listens on TCP port 1433. No firewall changes are made so that is something you have to do post install. The SQL Browser service is disabled because it is not required. UDP/1434 was used by the SQL Slammer worm and took advantage of poor network packet handling of the SQL Browser service. I recommend you leave this service disabled.

In previous attacks I have demonstrated, during the information gathering phase, we locate a SQL Server using nmap or zenmap. We assume this is a SQL Server because port 1433 is open. You can find a good listing of default ports on wikipedia. You can also find a good list of windows ports in your services file usually located in c:\windows\system32\drivers\etc\

Rather than discuss whether we should or should not change the SQL Port what I want to do it test out the effectiveness of the tools if the port is changed. To change what port SQL listens on for remote connections, there are three spots in the SQL Server configuration manager we have to change.

First, I like to change the listen all setting to no.
change_sql_port_01

Then, find your IPv4 address and enable listening on it and change the port.
change_sql_port_02

Now, we have to change the firewall. I’ve added an extra rule, then verified I could connect using the “IP,port” in SSMS.

change_sql_port_03

Penetration Testing

The only thing a port change will defend against is the information gathering phase of a hack. If we do a quick scan with zenmap, I noticed that this change is at least partially effective. The ms-sql port doesn’t light up green like we are used to. In fact, no open port is identified by a quick scan.

change_sql_port_06

What we have to do is open up our scan. The intense all TCP ports is very time consuming. I doubt a hacker would wait this long for a single host, I sure didn’t.

change_sql_port_04

nmap has a lot options and switches we can experiment with. I did notice the option “-p T:” which will try TCP ports within the range supplied. This completes in a reasonable amount of time.

change_sql_port_05

However, the service identification is missing. By changing the command a bit we can identify that the service is SQL, just not the exact version.

change_sql_port_07

As you can see the port change was ultimately ineffective. The fact that the host is always easily identified as online will draw the attention of an attacker. With a small amount of persistence, the attacker can identify the target as a SQL Server.

 
Leave a comment

Posted by on March 25, 2013 in Security, SQL Admin

 

#SQLPASS #SQLSatDetroit recap

I attended my third SQL Saturday event March 16th, 2013. This was the first one I decided to submit an abstract to and I am really glad I did. I’ve been geeked out on my presentation topic the last 6 months so that helped me push past any nerves.

I also volunteered but the hard work of others didn’t leave much leftover. After 200+ of these SQL Saturday events the community has a pretty good handle on how to make it a success. This was Detroit’s first so were plenty of unknowns. Lawrence Tech turned out to be a great location. There was plenty of open space between all of the classrooms for discussion and sponsors. The cafeteria simplified setup, delivery and cleanup of lunch. We were able to access the facility early and staff was onsite to help. 4G service was good and wifi was open.

lti

There was 5 tracks two with larger rooms and 3 with smaller rooms. I got one of the smaller rooms which was fine with me. I don’t like microphones and my voice doesn’t project particularly well. However, I did hear that some folks would have like to attend but the room was already packed.

me_at_sqlsatdetroit

Slides: http://nujakcities.files.wordpress.com/2013/03/security0315.pptx

And some older demo steps: http://nujakcities.files.wordpress.com/2013/03/demo_steps0216.docx

I was very happy with my delivery. I’m definitely not a natural but I can sense that I am getting a little better each time. I have to give a big thanks to the developers of Kali Linux. They shipped a solid product that demoed nicely. I was very concerned jumping into the new hotness with only 2 nights of practice but it worked out for the best. Of about 40 attendees, I received 21 reviews for an average score of 4.57/5.

Speaking did distract from attending other sessions. I was a complete zombie after my session and almost missed lunch. That said I think I will give speaking another shot. The #SQLFamily made it an enjoyable and valuable experience. Especially this evaluation comment that made my day and gave me the energy to go get a drink after the event.

million_dollar_reveiw

 
Leave a comment

Posted by on March 20, 2013 in PASS, Security, SQL Admin

 

I’m Speaking at #SQLSatDetroit March 16th

What a great way to use up my 100th blog post. I could not think of a better way to pass that milestone than to announce my upcoming session.

one_down

Ok, I didn’t really delete that session next to mine but what a great way to get in the mood!

Hacking SQL Server
The best defense is a good offence. Learn how to practice hacking without going to jail or getting fired. In this presentation we’ll be going over how to exploit weak SQL servers with actual tools of the penetration testing trade. You will learn why the SQL Service is a popular target on your network and how to defend against basic attacks. We will also attempt to snag some credentials from the SAM cache so we can go galavanting across the rest of the network.

If you are attending, I encourage you to get a few things installed so you can play along. It’s just more fun that way. VMWare player and BackTrack should get you started. You may want to setup a victim virtual machine as well, or just hack your host computer. Make sure to turn off your host NIC so you don’t make any mistakes and touch the network that you don’t have permission to use inappropriately.

 
1 Comment

Posted by on March 6, 2013 in PASS, Security, SQL Admin

 

Tags:

SQL Cluster dependency oops

With a 2008R2 -> 2012 upgrade and a massive storage migration under my belt I was feeling pretty good about my competency. All the data was moved off the old drives but they were still online. To get these drives offline a few steps were needed.

I knew the SQL Service was still dependent on these drives and going under the cluster covers into disk management to turn them off would be a bad idea. I carefully removed them from each instance service in the cluster manager. This removes the dependencies and plops the the cluster disks into unused storage. Then I got into cluster storage and deleted these old drives.

When drives are added, you have to add a dependency in the SQL Service to it. Otherwise, when failing over, drives will get yanked out from underneath SQL before it’s done gracefully shutting down. So in my infinite wisdom, I reworked all of the dependencies.

It went something like this:
bring the IP online
bring the name online
bring a mount point drive online
bring all the other drives online
bring SQL online
bring the agent and AS online

So two weeks go by and I have to do a failover now because the old SAN had to be removed from MPIO. This unfortunately requires a restart. So at 5am I logged in to click a couple buttons and hopefully go back to bed. No suck luck. The failover looked so graceful taking all the services offline in the correct order, then bringing everything online right up until it hit the SQL Server. It tried for about 1 second and failed.

If you haven’t noticed, cluster events are useless in scenarios like this. No detail, I assume this is mainly for reporting epic problems. I got into the event log, nothing there except the useless “SQL Failed” message in the system log. I went right to the SQL Logs but noticed the last time it was updated was went the agent went offline. No logs? That is bizarre.

I tried several other ways to start the SQL Service. It wouldn’t start on ANY node which didn’t make sense because the only change we made was to unmap a single node full of old SAN drives. After about an hour I decided to make my way into work and alert the authorities. I was assuming I would need some help from Microsoft at this point but I don’t have direct access to open tickets.

Fortunately, I double checked the logs. In SQL Configuration Manager I noticed there is a Dump location. I found some .xel files in this location and opened the newest up in SSMS or VS shell.

virtual_server_name_error

“Failed to check and change virtual server name. Please contact customer support.”

This may not sound too clear to you right off the bat, but it was clear to me. I remembered there being a dependency in the SQL Service to the virtual name before. But this time I had attempted to chain these dependencies. I figured that if SQL was dependent on the drive, and the drive dependent on the name, everything would come up in the right order. It appeared so, but apparently SQL really needs that dependency on the name. I added it and SQL came online shortly before 8am. Moral of the story, test any cluster dependency changes (duh).

 
Leave a comment

Posted by on February 28, 2013 in SQL Admin

 

Moving SQL Databases

This post is an accumulation of a few major and lots of minor things I forgot, or almost forgot when moving databases. It’s easy, but since there is a risk of data loss it’s not something that should be taken lightly.

Let’s start with moving a database to fix file fragmentation. If you do this, you might be largely underwhelmed at the benefits. If a database goes through a series of growths and has competing files on the same drive growing, external fragmentation can occur. This is usually a mythical problem that is used as subterfuge for masking the real code or table design problem. Now-a-days, with SSDs, massive amounts of RAM, SANs with huge caches and tiering mechanisms that move your data around fragmenting it anyways, it’s just not the problem is used to be. But if it helps you sleep at night, by all means backup and restore your database to a drive with some contiguous free disk space. There is quite a bit of risk for potentially zero benefit to move for this purpose of external file fragmentation. This problem will probably be better fixed the next time you really have to move the database.

Moving databases for a storage upgrade is main reason for this post. I recently finished moving a couple of fat clustered instances. In my case this was from an oldish SAN to a shiny new SAN. There are SAN tools that can copy luns online and SQL would have no idea that this process took place. If you decide to dump this task on your SAN admin I would take a backup and shut the instance down for safety reasons. These tools can copy the data faster when the server is offline. If you do decide to try online moving, make sure to run a checkdb after it’s done especially if your SAN admin says any of the words blip, hiccup, glitch or my personal favorite flap.

We’ll assume that the SAN admin said one of those words and you’ve decided to take matters into your own hands. For a storage upgrade you will want a good before/after baseline. Look at things like file stalls and disk latency before even touching anything. Then, test out a few methods of moving the data. We need to do some testing so we can get a feel for how long the process is going to take. Once that is done, check with the users to see what kind of outage they are willing to take. You might get something like this:

dba: How long can this thing be offline?
users: no.

In that case you’ve got one remaining option. Setup database mirroring on a new instance and failover to the new drives. In the event that you get something like this:

dba: How long can this thing be offline?
users: WE LOVE YOU, DO WHATEVER YOU NEED :) XOXOX

Hah, well then we’ve got a few options. If you have mostly empty data files that 20% used or giant empty log files, consider taking the backup and restore method. Backup file sizes resemble the used space in a database and even less than that if you apply compression. If you choose backup and restore, consider turning the database to read-only before you start the backup. This will kick users out but they will be able to get back in while the backup is happening, and while they are waiting for you to do the restore. The database will restore in read-only mode so don’t forget to flip it back to read-write when ready. You could also go with single_user if you really didn’t want anyone else touching the database.

Another method that requires and outage is the offline/online method. Basically you submit an alter database, modify file command which will tell SQL the next time this database starts up look in this new location. To do this you have to take the database offline and while it is offline move the files to the new location. This can be scripted but not in T-SQL alone like a backup restore unless you have sp_cmdshell enabled (bad).

Similar to offline/online is detach/attach. One benefit of detach/attach is that you can repair a logfile has not been treated with TLC. Look into attach syntax with rebuild log. This is relatively risky so I would recommend taking a backup.

To pick the fastest of these methods we have to compare the time it takes to run a backup AND a restore vs. the time it takes to copy a lot of potentially empty shell of .mdf & .ldf files. In most cases I’ve come across a compressed backup and restore is the best option. Things you can forget are:

1. a file
2. lots of files
3. a database
4. lots of databases
5. taking a full backup after it’s done so the log and differential backups will work

There are not a lot of complexities when moving a database for storage reasons. There are however, a lot more things to forget when moving instances.

SQL Server databases are relatively portable. If you have avoided jumping into the use of every bell and whistle that SQL has to offer there is not a lot to miss. The same methods to move data to new storage can be used to move data to new servers or instances.

Things that may have to come with the database
- agent jobs
- linked servers
- server logins- sql logins can be exported and re-mapped
- backups ( in case you have a retention policy )
- Services outside of the database engine like AS, IS, RS
- credentials
- maybe other databases that work together
- full text catalogs (or just rebuild them)

Other things to consider
- new service account permissions
- security certificates for things like TDE
- replication
- any application or developer that might have a connection string to this server (dns poitners can help)

And that’s a wrap. Like I mentioned before, backup->restore worked best in my situation. Mostly because I had a sizable maintenance window so I could script it all out.

 
Leave a comment

Posted by on February 16, 2013 in SQL Admin

 

SQL Cluster in place upgrade

This weekend our super cluster took a leap into the future. I upgraded a beefy (64cores 256GB RAM per node) SQL cluster from 2008 R2 to 2012 SP1.

First, during the week we ran upgrade adviser on each instance to weed out any potential problems. We found that normal users with 2008 SSMS were able to connect to 2012 SP1 without issue. More advanced users needed 2012 to use SSIS and newer features. That allowed us to proceeded with the upgrade earlier that we thought we could.

Second, I made sure to pickup a fresh set of install files with SP1 included. That saved me quite a bit of time and and handful of shoddy progress bars. Then I proceeded to step 1. Our preferred configuration is active-active-passive (AAP). That made it easy to get started Friday afternoon and get a feel for any errors that might occur before everyone left for the day.

sql_upgrade

Notice that SSIS doesn’t actually get upgraded. It just gets a copy of itself installed to the 110 folder and leaves the old service running. The same goes for other shared features.

Step two is where we have to take an outage. With only one node and set of binaries upgraded and valid to fail-over to the install should pick the right node to bounce the instance to.

Step three we take another outage on the other instance. Again, we hope the install does pick the “best available” node to make it’s home.

Step four is just some cleanup. Analysis services got upgraded and didn’t appear to have any problems. We have SSIS setup for filesystem packages so I had to maintain the share and replication. The new location is in the 110\DTS\Packages folder instead of the 100 folder. I also had to maintain the DCOM privileges in this new location. That is done using the component services snap-in.

Another issue I ran into was our 3rd party backup software. It simply needed it’s install re-ran to configure the extended sp’s it uses.

After all that I checked to make sure KERBEROS was working because we have had problems with that in the past. Users checked out several queries and some packages and found everything to be in order.

Easy as pie right? I am posting this before everyone hits the newly upgraded servers. Hopefully they don’t point out any errors I have made.

 
Leave a comment

Posted by on January 13, 2013 in SQL Admin

 
 
Follow

Get every new post delivered to your Inbox.

Join 107 other followers