RSS

Category Archives: .NET

Visualizing multi-threading with Fiddler

Fiddler is an awesome web debugging proxy.  It allows you to see and drill into each HTTP request as they happen. It shows the URL, headers, content, results and much more.

I started using Fiddler when I was bound by a very strict internet policy.  It has the ability to auto-respond to HTTP requests from my computer.  It acts as a proxy and sits inbetween your browser and network card.  These auto-responders proved very useful when I was trying to load a page that had several blocked items on it.  I could first use it to identify which requests were showing the generic blocked page message, then setup an auto-responder to have fiddler return a simple blank.gif instead of my computer actually making the HTTP request.  This prevented my computer from showing up in a list on some network admins software. This website loaded a bunch of unnecessary images as well and I was only concerned with the text.  I setup a *.jpg and *.gif auto responder to speed up the load time as well.

I hadn’t found a good use for fiddler in a while until I was working on a project to make several HTTP requests.  I started off single threading and got the application working.  Then I decided to add multi-threading and it appeared to go a little faster.  I was happy until I need to add a longer list of HTTP requests to the program.  I wasn’t quite sure I was getting a performance benefit from the multi-threading.  Stepping through the code in the debugger didn’t really show me exactly when the requests were being made because the break-points would get hit one at a time.

I had starting reading up on all the ways VB.NET can do multi-threading to see if there was an easier way.  I looked into parallel.foreach loops but read that they would only spawn extra threads if there were extra processors available.  I am interested in getting that particular parallel loop working so I can visualize that with fiddler as well.  The other threading tool I looked into was the background worker.

Background workers seemed to be designed to help with windows forms.  This type of threading helps forms from doing the classic “Not Responding” because you are trying to use the thread that is processing the form appearance and controls.

Thread pooling is the route I want to take in the future.  It gives you the ability to limit the number of threads you spawn at a time.  This would be perfect for making HTTP requests because you don’t want to overload a server using a form of “free threading”.  Fire and Forget is another name for a particular style of threading where you don’t care about the response of the thread. Fire and Forget can be used when the results are logged to a database.

My program was based around one function.  It would simply take a URL parameter and return a string of all the HTML on the page. Each URL in my list would take 1-5 seconds to pull a response depending on the server load. So in a single threaded program would take at least 5 seconds to complete 5 URLs and up to 25 seconds just to check 5 URLs.

This was an obvious candidate for multi-threading.  If those same 5 HTTP requests were sent in the first second of execution the program could complete in 1-5 seconds instead of 5-25 seconds.  The scaleablility would also be much better because the execution time wouldn’t go up that much if I added 100 URLs.

So I wrote the program.  The keys to multi threading success were:

1. setup a class wrapper around my function
2. pass URL in with the thread.start overload
3. create an event to handle the function response
4. use synclock so the threads don’t run into each other when writing their results
5. join the thread list together so you don’t continue on in your code before the multi-threading part is complete

Imports System.Net
Imports System.IO
Imports System.Security.Cryptography
Imports System.Text
Imports System.Threading

Module Module1
    Dim WithEvents oReadSiteHTML As New ReadSiteHTMLClass
    Sub Main()
        Dim URLlist() As String = System.IO.File.ReadAllLines("sitelist.txt")

        'Multi-Threaded
        Dim ThreadList As New ArrayList

        Dim stopper As New Stopwatch
        stopper.Start()

        For Each tempurl In URLlist
            Dim t As New Thread(AddressOf oReadSiteHTML.GetHTML)
            ThreadList.Add(t)
            t.Start(tempurl)
        Next

        For Each t In ThreadList
            t.join()
        Next

        stopper.Stop()
        Dim timetook As String = stopper.ElapsedMilliseconds.ToString()

        'single threaded

        Dim stopper2 As New Stopwatch
        stopper2.Start()

        For Each tempurl In URLlist
            oReadSiteHTML.GetHTML(tempurl)
        Next

        stopper2.Stop()
        Dim timetook2 As String = stopper2.ElapsedMilliseconds.ToString()

    End Sub

    Private Function GenerateHash(ByVal SourceText As String) As String
        Dim Uni As New UnicodeEncoding()
        Dim ByteSourceText() As Byte = Uni.GetBytes(SourceText)
        Dim Md5 As New MD5CryptoServiceProvider()
        Dim ByteHash() As Byte = Md5.ComputeHash(ByteSourceText)
        Return Convert.ToBase64String(ByteHash)
    End Function

    Public Class ReadSiteHTMLClass
        Public SiteHtmlHash As String
        Public Event ThreadHash(ByVal SiteHtmlHash2 As String)
        Public Sub GetHTML(ByVal URL As String)
            Dim request As HttpWebRequest = WebRequest.Create(URL)
            Dim response As HttpWebResponse = request.GetResponse()
            Dim reader As StreamReader = New StreamReader(response.GetResponseStream())
            Dim str As String = reader.ReadLine()
            Dim sitefulltext As String = ""
            Do While (Not str Is Nothing)
                sitefulltext = String.Concat(sitefulltext, str)
                str = reader.ReadLine()
                If str Is Nothing Then
                    Exit Do
                End If
            Loop
            reader.Close()
            SiteHtmlHash = GenerateHash(sitefulltext)

            RaiseEvent ThreadHash(SiteHtmlHash & ":" & URL)
        End Sub
    End Class

    Sub ThreadHash(ByVal SiteHtmlHash3 As String) Handles oReadSiteHTML.ThreadHash
        SyncLock GetType(ReadSiteHTMLClass)
            File.AppendAllText("hashes.txt", SiteHtmlHash3 & System.Environment.NewLine)
        End SyncLock
    End Sub

End Module

Timing the code I could tell that the multi threading part was faster. But I was concerned that the sync lock was somehow preventing my code from being able to instantiate the class multiple times.  I am not sure why I thought this but it was easy to prove myself wrong by opening up Fiddler. When I ran the code below I saw the requests all pop in simultaneously and respond at almost the same time, then the single threaded fashion each subsequent HTTP request had to wait for the previous response. It was a real “ah ha” moment where I realized my code was working just the way I wanted it to… well almost.  Once I get into really high numbers of URLs, I want to add the thread pooling.  Hopefully then I can use fiddler and see only 8 or 10 active HTTP requests at a time. Right now the program will launch a thread for every URL which can be called “free threading”.

One thing I have explained a couple times before is the difference between threads and cores.  Hyper threading is like a thread pool of 2 but at the hardware level. A 4 core box with hyperthreading on will show 8 cores in task manager. You can have a multi-threaded app perform much better than a single threaded app even if there is only 1 CPU. Furthermore, you can have as many threads as you want and the CPU will break these down into simple units also allowing for other operations to interrupt your app.  Back to the example, the key performance gain is the fact you don’t have to wait for one response to send another request.

The screen capture below shows that the requests multiple requests downloading simultaneously and several are already finished.

One thing to be cautious of is overloading a server. All servers are not super computers. If the server is doing all the work and all you have to do is send a request its easy to overload a server with just one client. The trick is to find the sweet spot in performance and be able to adjust that. Thread pooling gives you a single variable number that you can adjust with each change in the environment.

There are also some automatic security switches that can be used on networks and inside website code to identify malicious behavior. Don’t be the one responsible for getting your company on some blacklist.

Advertisements
 
Leave a comment

Posted by on November 2, 2011 in .NET

 

Windows Update Report with Win32_QuickFixEngineering

There are still a lot of servers I manage that are not a part of WSUS yet. This means going in and manually installing patches.  I knew that I had a lot of these servers and just going down the list one by one isn’t very efficient.  One thing that would really help is if I could get a list that is ordered by the last time it was patched. Then I could target the worst offenders first, and if I had time I could move on to the servers with fewer patches needed.

This simple command will get a list of all the patches that have been installed with a few other pieces of information.

wmic qfe get > c:wu.txt

For my computer this report is formatted with fixed width columns and could be parsed.  I could add a node parameter to the above command and run it for each server. Then I would have all the data I need to get my report. However, that is just too much data and a lot of parsing. I stepped back over an older project that I still use and pulled out some of the WMI code.  To dump everything in one heap and then sort through it would be inefficient.  To make this more efficient I looked at the WMI class Win32_QuickFixEngineering or qfe.

The qfe has this nice column “InstallDate” which really sounded like a winner to me.  Its actually a date type and could be very useful.  However, its always NULL for every server I checked. I went back to the text report scratching my head and saw that there was another column “InstalledOn”.  That is the column I need.

select InstalledOn from Win32_QuickFixEngineering where InstalledOn <> ''

I noticed that on some servers this value was blank for a lot of patches.  I am assuming a bit here but it appears they are patches that were installed during sysprep or in some other fashion early in the servers life. The next thing I realized is that all I really need is the top 1 in descending order.  This is when I found out there is no sorting in WMI. I would have to sort the results after the fact.

The MangementObject that gets returned is a bit tricky to work with so the first thing I do is read the results into an array. ArraysLists are easy to add to “.add” and also easy to sort. A .Sort() and then a .Reverse() is all I needed to pick out the last patch date. It was sitting in my array(0).

I ran into a snag rather quickly. Turns out vista era Microsoft decided it would be a great idea to start returning this as a 16 character hex value. Part of my code had to handle 4 different date types if you count NULLs.

'oReturn is one record in the managementobject.  You get one for each patch
Dim oDate As Object
If oReturn("InstalledOn").Length = 16 Then
    oDate = DateTime.FromFileTimeUtc(Int64.Parse(oReturn("InstalledOn"), System.Globalization.NumberStyles.AllowHexSpecifier))
ElseIf oReturn("InstalledOn").Length &gt; 7 And oReturn("InstalledOn").Length &lt; 11 Then
    oDate = CDate(oReturn(&quot;InstalledOn&quot;))
Else
    'I don't know what date type you are... you must not be a recent patch.
    oDate = DateTime.Now.AddYears(-100)
End If

When searching for answers to my questions I did read some complaints about all patches not showing up in the results. To solve my problem it wasn’t really necessary to be totally accurate. I just needed a date the last time the server had some tlc. So far it has been very accurate.

Here are some links that I used to get to my end game.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa394391(v=vs.85).aspx

http://msmvps.com/blogs/athif/archive/2005/11/20/76035.aspx

http://social.microsoft.com/Forums/en-US/netfxbcl/thread/55000d71-8038-4be2-98a0-74e3ef58d55e

 

 
Leave a comment

Posted by on October 8, 2011 in .NET, Network Admin

 

405 Method Not Allowed and WCF Activation

We deployed some new features to a web application and I got word that some were not working. Its a new application so each problem has a pretty steep learning curve for me. The QA team said that no error messages were shown and it worked on some computers but not others. This initially sounded like a browser problem so I was not involved from the start since my focus is on the servers.

The dev team found using fiddler is that there was an HTTP error of 500. This was happening when trying to call a validation service (.svc) file. To get more out of this message we had to turn on debugging in the web.config file. Then with fiddler I was able to inspect the inner exception message which was the title of this blog.

There are two IIS7 servers that are network load balanced. I had no idea why IIS wouldn’t allow this simple HTTP call to a web service. Some of the first things we did were:

Make sure .NET 3.5 was installed [check]
Make sure .NET 4.0 was installed [check]
Check the application pool identities and settings [check]
recycle app pools [check]
reboot server [check]

None of these worked. These settings were the same on both servers but I decided to do some testing on each server individually. This narrowed down the problem to the second server. All of the files in the web app are replicated using a mesh topology. But the Temporary files in the .NET folders are not so I checked security on those next. That also was not a problem.

I turned on failed request tracing which is a really cool feature in IIS7. It spit out some .xml log files for the 500 error I was getting and showed me that ISAPI was at fault. However, I have done these traces before and had the same result. I think ISAPI is at a higher level than I needed to see to get a good error.

Searching of this 405 error reported some people having trouble with handler mappings. But that didn’t make sense for me because those are stored in the web.config file for each site, and that was the same on the working server and broken server. I started then looking at server level IIS7 settings. The ISAPI restrictions were the same so that wasn’t it.

At this point I decided to go nuclear and uninstall .NET 4.0, IIS7, application server role and .NET 3.5. In 2008 R2 .NET 3.5 is a feature that you can choose to install. I noticed when trying to uninstall that, there are actually some sub features to .NET 3.5. As it turns out, WCF Activation is a very important feature to have installed on a server that uses HTTP service calls.

Once I saw this I re-installed everything that I blew away and sure enough, the code worked just fine. What really gets me is that in all the googling I did there wasn’t a link between 405 and WCF Activiation. Hopefully now there is :]

 
Leave a comment

Posted by on September 14, 2011 in .NET, Network Admin

 

Use VB.NET to impersonate a domain user

There are a lot of reasons you may want to pretend to be something else. Maybe you have low self esteem and need a boost. To impersonate someone or something you need a good understanding of what that thing is all about. In Windows all you need is a username and password. If you want to connect to a database with higher privlidges you can impersonate another user by right clicking SSMS and choosing “Run As”.

Web applications can impersonate other accounts by using the application pool identities. Or web applications can impersonate the connecting user by setting the identity impersonate to true in the web config.

Services such as SQL Server can run as local system or run as any domain user with the right (or wrong amount) privliges. Have you ever had a developer contact you with the error “Access denied for “DOMAINCOMPUTERNAME$”. This is because they are trying to run a program as local system on that computer and connecting to SQL on another server. You can put these AD computer accounts in a group and give the group a SQL login but I wouldn’t recommend that. What I would recommend is to use .NET to impersonate a user when accessing SQL Server.

Supply a valid username and password to AD and authentication produces something called a token. This token can be used and re-used to get at the objects you need. MSDN has some very elaborate code samples that dive very deeply into the proper methods to use the framework when it comes to impersonation. Most of these instructions were quite far over my head and way deeper down the rabbit hole than I had intended to travel. My goal was to impersonate an account that actually had very limited privlidges so I did not need all the extra tedious and bland material.

Hopefully this code sample below from MSDN that I trimmed down considerably will satisfy your taste buds.

http://msdn.microsoft.com/en-us/library/system.security.principal.windowsidentity.impersonate(v=vs.71).aspx
http://msdn.microsoft.com/en-us/library/chf6fbt4.aspx

Imports System.Runtime.InteropServices
Imports System.Security.Principal

Module Module1

    Private Declare Auto Function LogonUser Lib "advapi32.dll" (ByVal un As String, ByVal domain As String, ByVal pw As String, ByVal LogonType As Integer, ByVal LogonProvider As Integer,  ByRef Token As IntPtr) As Boolean

    Public Declare Auto Function CloseHandle Lib "kernel32.dll" (ByVal handle As IntPtr) As Boolean

    Public Sub Main()
        Dim tokenHandle As New IntPtr(0)
        Try
            If LogonUser("un", "DOMAINNAME", "pw", 2, 0, tokenHandle) Then
                Dim newId As New WindowsIdentity(tokenHandle)
                Using impersonatedUser As WindowsImpersonationContext = newId.Impersonate()
                    'perform impersonated commands
                    System.IO.File.WriteAllText("C:ttestimp.txt", "test")
                End Using
                CloseHandle(tokenHandle)
            Else
                'logon failed
            End If
        Catch ex As Exception
            'exception
        End Try
    End Sub
End Module
 
7 Comments

Posted by on August 1, 2011 in .NET, Network Admin

 

Roll my own computer management software with WMI and VB.NET

Windows Management Instrumentation is a well documented framework for network admins. There are hoards of management software suites that use it and can give you great stuff right out of the box. Some of it is great and some is not so good. One simple piece of information I wanted was, how much free space do my sql server drives have? I wanted it in a spreadsheet so I could sort and start examining the servers that were getting low on space. We utilize several monitoring and management tools but there didn’t seem to be a good solution.

I avoided some possibly important tasks and wrote my own solution using WMI. Querying WMI is rather simple. It is a lot like querying a sql database. Here is an example:

Select Label, Name, FreeSpace, Capacity 
from Win32_Volume 
where Label  'System Reserved' and DriveType  5

If you start a new VB.NET project the only trick is to add a reference to System.Management. The core piece of code in my project is a function that I pass the computer and query into and get a management object back.

    Public Function QueryWMI(ByVal query As String, ByVal ComputerName As String)
        Dim wmiScope As New ManagementScope("\" & ComputerName & "\rootcimv2", New ConnectionOptions)
        Dim smoQuery As System.Management.ObjectQuery = New System.Management.ObjectQuery(query)
        Dim smoSearcher As ManagementObjectSearcher = New ManagementObjectSearcher(wmiScope, smoQuery)
        Return smoSearcher.Get()
    End Function

The next piece of code flips through the results. I chose to parse the results to a .csv file so I could simply double click and open in excel to do my slicing and dicing.

    Function wmidrivefreespace(ByVal ComputerName As String) As String
        Dim retstr As String = ""
        For Each oReturn As ManagementObject In QueryWMI("select Label, Name, FreeSpace, Capacity from Win32_Volume where Label  'System Reserved' and DriveType  5", ComputerName)
            Try
                Dim label As Object = oReturn("Label")
                Dim name As Object = oReturn("Name")
                Dim freesp As Object = oReturn("FreeSpace")
                Dim cap As Object = oReturn("Capacity")

                If label = Nothing Then label = " "
                If name = Nothing Then name = " "
                If freesp = Nothing Then freesp = " " Else freesp = (Double.Parse(freesp) / 1024 / 1024 / 1024).ToString("N2")
                If cap = Nothing Then cap = " " Else cap = (Double.Parse(cap) / 1024 / 1024 / 1024).ToString("N2")

                retstr &= ComputerName & "," & label.ToString() & "," & name.ToString() & "," & freesp.ToString() & "," & cap.ToString() & System.Environment.NewLine
            Catch ex As Exception
                System.IO.File.AppendAllText(DateTime.Now.ToString("yyyyMMdd") & "errorlog.txt", ComputerName & ex.ToString & System.Environment.NewLine & System.Environment.NewLine)
            End Try
        Next
        Return retstr
    End Function

For this to work there is a WMI service on the target machine that has to be started (it is by default). Also, inbound port 135 has to be open on the target machine. If it isn’t already open on all of your computers I would be surprised since so many other things use WMI.

If you visit MSDN and check out the classes http://msdn.microsoft.com/en-us/library/aa394084(v=VS.85).aspx you might be shocked at how much information you can gather.

There is a bug in the above code that is easily fixed. If you spot it, post a comment. The hint is: Make sure to wash your punk.

 
Leave a comment

Posted by on May 31, 2011 in .NET, Network Admin

 

Automating SQLIO testing

I’ve bloged about SQLIO a bit but never focused on it. SQLIO is a tool that creates disk activity and is useful when testing performance of a disk subsystem. When you run it you get some results like these:

C:UsersnujakDesktopSQLIO_old>sqlio -kW -t24 -s30 -o20 -fsequential -b4 -BH -LS -F1file1lun1th.txt
sqlio v1.5.SG
using system counter for latency timings, 2740654 counts per second
parameter file used: 1file1lun1th.txt
file C:testfile.dat with 1 thread (0) using mask 0×0 (0)
1 thread writing for 30 secs to file C:testfile.dat
using 4KB sequential IOs
enabling multiple I/Os per thread with 20 outstanding
buffering set to use hardware disk cache (but not file cache)
using specified size: 10240 MB for file: C:testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 24246.70
MBs/sec: 94.71
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 47
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

There are some important items in these results.

1. The date and time of the test (we’ll have to get that somewhere else
2. The command that was run, a trick is to properly name your parameters file so those are included in the command.
3. The IOPS
4. The MBps or bandwidth
5. The Avg Latency

The first thing you’ll want to do is make a good set of parameters files. Create small test files to test disk caching and large (200GB for some SANs) to test actual spindle activity. Datawarehouses often have a large number of LUNs that should be tested simultaneously. So have -F1file1lun1th.txt be a test and then -F1file12lun1th.txt be a test. You might want to test each LUN individually and also walk the way up to find the correct number of LUNs that max out your HBAs.

Make a batch file that has a series of SQLIO calls. They don’t have to be in any particular order because we will be sorting that out later. Then, is what you do is make another batch file that calls your first batch file and pipes the output to a text file. This text file will contain the above results over and over again for each test. To be scientific you should put a wait inbetween each call and also test each command at least three times.

Once that is done you setup your parsing program with … none other than my fav.. VB.NET.

Module Module1

    Public CurrentDir As String = System.Environment.CurrentDirectory &amp; ""

    Public OutputFile As String = CurrentDir &amp; Format$(Date.Now, "yyyyMMddHHmmss") &amp; "output.csv"

    Public TempDir As String = CurrentDir &amp; "Results"

    Public dir_info As New IO.DirectoryInfo(TempDir)

    Sub Main()
        parseresults()
    End Sub

    Function parseresults() As Boolean
        Try
            Dim file_info_ar As IO.FileInfo() = dir_info.GetFiles()
            Dim file_info As IO.FileInfo
            Dim line As String
            Dim contents() As String
            Dim itemcounter As Integer = 0
            Dim csvfile As String = "time,cmds,iops,bw,late" &amp; System.Environment.NewLine

            For Each file_info In file_info_ar
                contents = System.IO.File.ReadAllLines(TempDir &amp; file_info.Name)
                For Each line In contents
                    If line.Contains("C:SQLIO&gt;sqlio") Then
                        csvfile = csvfile &amp; file_info.Name &amp; "," &amp; line
                    End If
                    If line.Contains("IOs/sec:") Then
                        csvfile = csvfile &amp; "," &amp; line.Substring(8, line.Length - 8)
                    End If
                    If line.Contains("MBs/sec:") Then
                        csvfile = csvfile &amp; "," &amp; line.Substring(8, line.Length - 8)
                    End If
                    If line.Contains("Avg_Latency") Then
                        csvfile = csvfile &amp; "," &amp; line.Substring(17, line.Length - 17) &amp; System.Environment.NewLine
                    End If
                Next
            Next

            System.IO.File.WriteAllText(OutputFile, csvfile)

        Catch ex As Exception
            System.IO.File.AppendAllText("exceptions.log", ex.ToString)
            Return False
        End Try
        Return True
    End Function
End Module

I decided not to go directly to SQL with this because a .csv can be opened in Excel. Excel makes it very easy to graph the results. Also, csv’s pop into SQL quite easily if you want to query the data. SQLIO definitely has its place even though it doesn’t simulate real world activity. If you find your bandwidth limited to a solid 1Gbps or you can identify easier which piece of hardware to upgrade.

 
Leave a comment

Posted by on May 9, 2011 in .NET, Network Admin, SQL Admin

 

Comparing MD5 and 3DES encryption with vb.net

The reasons for using these two types of encryption are completely different. MD5 is a hashing encryption algorithm. When you use MD5 is there no way back. The main reason for hashing algorithms is for storing passwords. This way you never actually store the users password (“passW0rd”) but you actually store the MD5 hash of the password (YoOu0vDQKek5jEsEBHVM4A==). This becomes useful because the next time the user logs in, all you have to do is compare the originally stored hash with the hash of what the users entered into the password box. Converting “passW0rd” to MD5 will always produce the same hash “YoOu0vDQKek5jEsEBHVM4A==”. There is no decrypting MD5. The way MD5 is broken is by creating precomputed table of hashes for every possible input, a.k.a. rainbow table. The stronger your password is the larger the rainbow table will have to be to break your hash.

On a side note, data warehousing has begun to use encryption hashes but not for security reasons. If you have a very large number or size of columns and need to see if they have changed or not you could check each column and compare it to the data you are about to insert. Or, you could create a hash of the entire record and store that. This makes for really fast compares.

The other method of encryption gives you the ability to decrypt the data. Triple Data Encryption Standard (3DES) uses a key and an initialization vector. With these two pieces of information you can decrypt the data. For example, if you store employee salary data in a table, you would want to use this type of encryption so your dba can’t just read the data in the table. This way you can have the Key stored in one location and the IV stored in another.

I have been in discussions that go very poorly when deciding how to secure, or whether or not something needs to be secured. There usually is a thorn of a person that will quickly discount methods of security and sways the whole groups opinion. MD5 by itself is no longer considered a solid security method. This is a fact, and usually the thorn in the group makes this comment but then either neglects to give an alternative or doesn’t know what the alternative is. One alternative is SHA, but it requires more processing and more storage which in 99.9% of cases is completely fine. The other way the discussion goes is into a paranoia state and decided everything needs ultra security. What I suggest is finding your happy medium. Find the level of security where you can get projects completed and performing well but are also secure.

I have pieced together two examples of encryption using MD5 and 3DES. I am not an encryption expert nor a math expert but that is just my point. Using the .NET framework you don’t have to be an expert to at least make an attempt to keep data secure.

MSDN gives great examples which I have used: http://msdn.microsoft.com/en-us/library/system.security.cryptography.tripledes

The 3DES example. You’ll notice the Encrypt and Decrypt functions are nearly identical. It’d be best to re-factor but I left them separate for illustration purposes. This program takes ‘test.txt’ and encrypts it to ‘3destest.txt’ and then decrypts that to ‘decryptedtest.txt’.

Imports System.IO
Imports System.Security.Cryptography
 
Module Module1
 
    Sub Main()
        Dim key() As Byte = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24}
        Dim iv() As Byte = {8, 7, 6, 5, 4, 3, 2, 1}
 
        EncryptData("test.txt", "3destest.txt", key, iv)
        DecryptData("3destest.txt", "decryptedtest.txt", key, iv)
    End Sub
 
    Private Sub EncryptData(ByVal inName As String, ByVal outName As String, ByVal tdesKey() As Byte, ByVal tdesIV() As Byte)
 
        'Create the file streams to handle the input and output files.
        Dim fin As New FileStream(inName, FileMode.Open, FileAccess.Read)
        Dim fout As New FileStream(outName, FileMode.OpenOrCreate, FileAccess.Write)
        fout.SetLength(0)
 
        'Create variables to help with read and write.
        Dim bin(100) As Byte 'This is intermediate storage for the encryption.
        Dim rdlen As Long = 0 'This is the total number of bytes written.
        Dim totlen As Long = fin.Length 'This is the total length of the input file.
        Dim len As Integer 'This is the number of bytes to be written at a time.
        Dim tdes As New TripleDESCryptoServiceProvider()
        Dim encStream As New CryptoStream(fout, tdes.CreateEncryptor(tdesKey, tdesIV), CryptoStreamMode.Write)
 
        Console.WriteLine("Decrypting...")
 
        'Read from the input file, then encrypt and write to the output file.
        While rdlen &lt; totlen
            len = fin.Read(bin, 0, 100)
            encStream.Write(bin, 0, len)
            rdlen = rdlen + len
            Console.WriteLine(&quot;{0} bytes processed&quot;, rdlen)
        End While
 
        encStream.Close()
    End Sub
 
 
    Private Sub DecryptData(ByVal inName As String, ByVal outName As String, ByVal tdesKey() As Byte, ByVal tdesIV() As Byte)
 
        'Create the file streams to handle the input and output files.
        Dim fin As New FileStream(inName, FileMode.Open, FileAccess.Read)
        Dim fout As New FileStream(outName, FileMode.OpenOrCreate, FileAccess.Write)
        fout.SetLength(0)
 
        'Create variables to help with read and write.
        Dim bin(100) As Byte 'This is intermediate storage for the encryption.
        Dim rdlen As Long = 0 'This is the total number of bytes written.
        Dim totlen As Long = fin.Length 'This is the total length of the input file.
        Dim len As Integer 'This is the number of bytes to be written at a time.
        Dim tdes As New TripleDESCryptoServiceProvider()
        Dim encStream As New CryptoStream(fout, tdes.CreateDecryptor(tdesKey, tdesIV), CryptoStreamMode.Write)
 
        Console.WriteLine("Decrypting...")
 
        'Read from the input file, then encrypt and write to the output file.
        While rdlen < totlen
            len = fin.Read(bin, 0, 100)
            encStream.Write(bin, 0, len)
            rdlen = rdlen + len
            Console.WriteLine("{0} bytes processed", rdlen)
        End While
 
        encStream.Close()
    End Sub
 
End Module

For the MD5 example I take a console parameter string and encrypt that to the ‘PasswordHash.txt’ file. There is an extra step in here to carefully encode as a string since you could run into problems writing the raw byte array output to a file.

Imports System.Text
Imports System.Security.Cryptography
 
Module Module1
 
    Public Password As String = Nothing
 
    Sub Main()
 
        If My.Application.CommandLineArgs.Count > 0 Then
            Password = My.Application.CommandLineArgs(0)
            Dim objStreamWriter As System.IO.StreamWriter = System.IO.File.AppendText("PasswordHash.txt")
            objStreamWriter.WriteLine(GenerateHash(Password))
            objStreamWriter.Close()
        End If
 
    End Sub
 
    Private Function GenerateHash(ByVal StringToEncrypt As String) As String
 
        Dim UniObject As New UnicodeEncoding()
        Dim ByteSourceText() As Byte = UniObject.GetBytes(StringToEncrypt)
        Dim Md5 As New MD5CryptoServiceProvider()
        Dim ByteHash() As Byte = Md5.ComputeHash(ByteSourceText)
        Return Convert.ToBase64String(ByteHash)
 
    End Function
 
End Module
 
1 Comment

Posted by on April 22, 2011 in .NET, Network Admin

 

Single Vs. Multi threading

Imports System.Threading

A multi-threaded app requires significantly more brainpower, testing and patience to complete than its sequential counterpart. Its my advice to make sure you get a significant performance boost and time/results ratio before tackling a huge re-write. If your teammates or the future maintainers of the app don’t have the skill set, it may cost you in the long run. Before you read I am assuming you have written single threaded applications that use functions and parameters.

To make the switch from single threaded to multi threaded code, the trick is to create classes that can be instantiated by any thread. This way you can get variables or data in and out of your routines easily. To pass a parameter into a thread you would use the overload function of thread.start().

'Single threaded version
        For Each item In todolist
            myFunction(item)
        Next
'multi threaded version
        For Each item In todolist
            Dim t As New Thread(AddressOf oClass.myFunction)
            t.Start(item)
        Next

Kicking it off isn’t that difficult. In that example “item” is the variable that is passed into “myFunction”. To get a variable out of the single threaded example we could just “Return” it. The multi threaded example we’ll have to catch a thrown event. Here is the myFunction wrapper “oClass” in which you can see how to define the “evtReturn”

    Public Class oClass
        Public Event evtReturn(ByVal RetVal As String)
        Public Sub myFunction(ByVal myInput As String)
            'Do Stuff
            Dim myOutput as string = myInput
            RaiseEvent evtReturn(myOutput)
        End Sub
    End Class

The definition of the “FinishedWithAThread” routine has an important part “Handles oClass.evtReturn” These events may come back simultaneously so we have to use caution. This is a good time to use “synclock” to prevent an IO exception.

Sub FinishedWithAThread(ByVal myFunctionReturn as string) Handles oClass.evtReturn
        SyncLock GetType(oClass)
            File.AppendAllText("output.txt", myFunctionReturn &amp; System.Environment.NewLine)
        End SyncLock
End Sub

With this newly acquired skill it will be fairly easy to grind any CPU into the ground. If you are writing an app for a server with multiple users, you may not even need to multi thread because the users are doing that for you. Then again, if that same server has 256 cores…. give it all you’ve got.

 
1 Comment

Posted by on November 16, 2010 in .NET

 

express VB.NET my.fav

Scripting, programming, developing… unfortunately these three things tend to blur together in practice but they are very distinct.  Scripting is the most basic form of automation and you should not confuse someone who can write a script with a programmer.  Programmers have a leg up and the advantage of an environment to compile their code and perform complicated tasks with fewer lines.  Developers take advantage of the hard work of others and make something spectacular(maybe).

Deciding what to code with…

$$$ – VB.NET/SQL express editions are free.  You can write an amazing piece of software with these items.  Where the costs come in is when you outgrow the express editions and need to purchase windows server or meet one of these criteria:

4gb+ database
windows service templates
easier implementation of SDKs

I have even recently heard about a linux tool called MONO where you can run your .NET code without windows.  It may be difficult to support but the point is you can do this for free.

So take a break and go download the express editions http://www.microsoft.com/express/downloads/ and take control of your computer(s).

When you make a new Visual Basic Express project you have a handful of options.  I have experience with console applications and windows form applications (a.k.a. GUI).  In the matter of about 10 seconds you can load the IDE pick new console application press these keys

sys.io.file.writeallt(“world.txt”, “hello!”) [f5]

and there you have it your hello world app.  Have you ever heard this quote, “Notepad is the best for coding.”  If you have done the example above you’ll now be able to look down you nose at the scripter who quoted that line.  You can do this because you know that the development environment made you more efficient by setting up your module, directories and giving you IntelliSense to finish each class in the example which is really:

Module Module1
    Sub Main()
        system.io.file.writealltext("world.txt", "hello!")
    End Sub
End Module

[f5] (to compile and run)

With a little practice you can actually do this .NET example in less than 10 seconds.  The Microsoft IDE is much faster than its IBM or open source counterparts.

VB.NET is not much different than C#.  Many of the benefits of VB carry over to C#. They have the same IntelliSense into the framework.  In fact they are so similar there are many free webapps that will convert the code back and forth automagically. I like VB.NET better because the syntax just makes sense.  It is a little more verbose which makes it more human readable. With the power of auto-complete you end up hitting closer to the same number of keystrokes.

In the latest 2010 version if you highlight a variable or other piece of text all other matches will be highlighted showing you if you change that value you will be changing these other pieces of code.  The [f5] part of the example actually compiles you .exe and runs it in debug mode.  Move the cursor down to the write line and press [f9] to set a break point.  The next time you compile in debug mode the program will stop at that line and you can examine your variable values.  At this point you can view “debug windows” such as the call stack or watch windows.

This debug mode is the only way to write code.  Stepping line by line using f10 and stepping into any subroutines and functions using f11 will help you remove errors, hence the name “debug”. If you are not using debug mode you are a scripter and nothing more.

Good code handles errors that may arise.  The last resort to handling errors you did not expect is the try/catch.  In the IDE typing “try” and pressing enter will give you the basic syntax.  If your statements inside the try work the statements inside the catch are never processed. All good programs have some form of exception logging such as:

System.IO.File.AppendAllText("error.log", datetime.now ex.ToString system.environment.newline)

Especially with silent running console apps you need this error handling.  Let the user know the program failed and then give the developer meaningful information to help resolve the issue.

The best way to describe why I like vb the best is to post some code examples.  Once I figure out a way to easily maintain the syntax highlighting and insert some screenshots I will have a better VB.NET post for you.

 
1 Comment

Posted by on August 12, 2010 in .NET