Wednesday, May 15, 2013

Simulating Microsoft Access recordset in ASP.Net with SQL Server (MovePrevious)

Despite all of the problems that Microsoft Access has, there are some things that it can do more conveniently than Microsoft SQL Server.  One of them is that it allows you to go forward and backward on a record set using MoveNext and MovePrevious.  In ASP.Net, using a SqlDataReader, you can only go forwards on a record set.  I was recently converting over an Access database to ASP.Net using SQL Server.  The Access code had a whole lot of places where a table was loaded to the record set, the cursor was moved to the last record with MoveLast, then the code stepped through the set backwards using MovePrevious, then reset itself to the end again with MoveLast.  I wanted to try to duplicate this as closely as possible .

After doing some reading, my first plan was to make a DataSet. Once you load data into a DataSet, you can iterate forwards and backwards.  The main reason I didn't want to do it that way is because you have to load all of the data into the DataSet at once that you might need to iterate through.  In my case, the amount of records I needed would vary depending on the data, and I didn't want to load a lot more than I had to in order to be sure I had all I needed.

What I ended up doing was based on a fairly simple idea.  I just sorted the records in the opposite order so that I could iterate through them forward instead of backward.  When MoveLast is called, I just re-ran the query to reset the DataReader's cursor back to the beginning.

The fancier parts were:

  • I made a class to encapsulate it all so that I could use the same method names that the Access record set uses.  That way there was less code to change, so I was hoping that would lead to less potential errors.
  • Access also lets you insert a record by calling AddNew to get a new record, changing values on the record, and then calling Update.  I added some methods to simulate that as well.
The class is not currently generalized to use different tables, but that shouldn't be too hard. The code is below:


    Dim m_newSqlFieldList As String = ""
    Dim m_newSqlValueList As String = ""


    Private Class AccessSimulator
        Dim m_connStr As String = ""
        Dim m_rateCheckingConnection As SqlConnection
        Dim m_rateCheckingCommand As SqlCommand
        Dim m_rateCheckingReader As SqlDataReader
        Dim m_fieldList As Hashtable
        Public Sub New(connStr As String)
            m_connStr = connStr
            m_rateCheckingConnection = New SqlConnection(m_connStr)
            m_rateCheckingConnection.Open()

            m_fieldList = New Hashtable
        End Sub
        Public Sub CloseConnection()
            m_rateCheckingConnection.Close()
        End Sub
        Public Sub MoveLast()
            ' This is to simulate MoveLast on the table
            ' TOP could also  be a select WHERE date goes back a certain amount from a given date

            ' always assume connection is open because the Reader is being used by the calling procedure
            ' it will only be closed if CloseConnection is explicitly called
            m_rateCheckingConnection.Close()
            m_rateCheckingConnection.Open()

            Dim sql As String = "SELECT TOP 50 * FROM MyTable ORDER BY ID DESC"
            m_rateCheckingCommand = New SqlCommand(sql, m_rateCheckingConnection)
            m_rateCheckingReader = m_rateCheckingCommand.ExecuteReader()
            m_rateCheckingReader.Read()
        End Sub
        Public Sub MovePrevious()
            ' since the records are in descending order now, this is really just a Read to get to next record
            m_rateCheckingReader.Read()
        End Sub
        Public Function GetFieldValue(fieldName As String) As String
             ' this is a substitute for the exclamation point when referencing a value of a record
            Return m_rateCheckingReader(fieldName).ToString()
        End Function
        Public Sub AddNew()
            m_fieldList.Clear()
        End Sub
        Public Sub StoreValueForInsert(fieldName As String, fieldValue As String)
             ' this is a substitute for the exclamation point when setting a value of a record
            m_fieldList(fieldName) = fieldValue
        End Sub
        Public Sub Update()
            ' in my case, Update is only used on this class after doing an AddNew
            ' so since we aren't actually inserting in the AddNew method, do the insert now, based on fields in the field list
            ' changes would need to be made to also support actually updating records
            Dim insertFieldList As String = ""
            Dim insertValueList As String = ""
            For Each storedField As DictionaryEntry In m_fieldList
                ' if insertFieldList is non-empty, then value list is too
                If insertFieldList <> "" Then
                    insertFieldList = insertFieldList + ", "
                    insertValueList = insertValueList + ", "
                End If
                insertFieldList = insertFieldList + "[" + storedField.Key + "]"
                insertValueList = insertValueList + "'" + storedField.Value + "'"
            Next

            Dim insertSQL = "INSERT INTO MyTable(" + insertFieldList + ") VALUES (" + insertValueList + ")"
            Dim conn As SqlConnection = New SqlConnection(m_connStr)
            conn.Open()
            Dim cmd As SqlCommand = New SqlCommand(insertSQL, conn)
            cmd.ExecuteNonQuery()
            conn.Close()

            ' clear the list when done
            m_fieldList.Clear()
        End Sub
    End Class

Tuesday, March 19, 2013

SQL - Parsing date from text / varchar field

I had a fun little SQL task to do today.  We have a large field of text that contains some comments and next to each comment is a date.  We wanted to find the most recent comment and sort the records by the date that is next to that comment in the text.  Luckily, we build that text field with the most recent dates at the top, so in this case, I just needed to parse out the first date that I could find.

I can find the year easily enough by searching for "/201" (actually that will only work for 7 more years, oops).  The main complication in my case is that the dates don't use leading zeroes, so they aren't all the same length.  Some are 2/2/2013 (8 characters) and some are 11/12/2013 (10 characters).  So I couldn't just find the "/201" and go back 5 characters.  And I didn't want to just search for the first slash, just in case there was a slash used for a different reason.  So I looked for "/201", then went back 5 characters and found the first slash before that, then went back 2 characters and built a substring from there.  Again the month really could be one or two characters, so in many cases my substring now had a leading character.  In my case, it was usually a space or newline character, so then I stripped newlines using REPLACE.

The last complication is that I wanted to sort by this date, and alphabetical sort won't work because of the lack of leading zeroes.  I had to do a CONVERT() to a datetime in order to sort correctly.  That will fail though if it finds some improperly formatted date (which we have in the text) so I had to use a CASE statement with an IsDate check.  Here's the final query:

SELECT
CASE 
    WHEN IsDate(REPLACE(REPLACE(REPLACE(SUBSTRING(myTable.Comments, CHARINDEX('/', myTable.Comments, CHARINDEX('/201', myTable.Comments) - 5)-2, 10), CHAR(10), ''), CHAR(13), ''), CHAR(9), '')) = 1 
    THEN CONVERT(datetime, REPLACE(REPLACE(REPLACE(SUBSTRING(myTable.Comments, CHARINDEX('/', myTable.Comments, CHARINDEX('/201', myTable.Comments) - 5)-2, 10), CHAR(10), ''), CHAR(13), ''), CHAR(9), '')) 
    ELSE NULL 
END As LastCommentDate
FROM myTable

ISSUES:
1) Not sure how fast this is, it seems like it may be slow
2) Once I find the start point, I just make a substring of length 10.  That works for me for our data, but more safe/accurate would be to do a difference between the start index and the index of the "201" to know for sure how long the month and day part is.

Tuesday, January 29, 2013

Epson Workforce Incorrect Paper Jam Error

Most of the things I post are programming or software issues.  I had an interesting hardware issue last night.  I have an Epson Workforce 310 printer.  Recently it started giving me erroneous Paper Jam errors when there was no paper jam.  It would just start loading a piece of paper, then almost immediately say "Paper Jam", feed the rest of the paper through without printing on it, and cancel the print job.  I thought maybe my printer was finally done for.

So then I started trying random things -- take the paper out and put it back in, clean the nozzles, run the alignment check -- but still no luck.  Then I started thinking, "why did it just start doing this when it was printing fine a minute ago?".  Well I'd just turned the paper around in the feeder because I was printing on hole-punched paper and the hole punches were on the right side of the page and I wanted them on the left.  I tried turning the paper back around and it started printing fine again!  Apparently the printer has some sensor on one side and the holes were confusing it into thinking there was a paper jam.

On further searching, looks like Epson subtly acknowledges this issue in their FAQ regarding paper feeding. It simply says:

"Do not load paper with holes punched in it for insertion in a binder."

Monday, October 8, 2012

Changing SQL Azure database size limit / quota

The starting size limit quota for my SQL Azure database was 1 gigabyte (1 GB).  This particular database has a table with millions of records.  I am semi-manually importing records into that table, so I had been noticing that it was creeping up on that limit with each import batch.

Finally, I ran a batch of INSERT statements and it pushed my database size over the 1 GB quota.  I fully expected that it would give me some quota message and not allow any more inserts.  What I didn't expect is that hitting the quota brought down my whole Windows Azure web site!  Even my Azure-hosted web pages that were not hitting the database started failing to load (I couldn't see the exact error message because remote error messages were turned off, and I never was able to successfully turn them on).

I logged into the billing area of Windows Azure so that I could upgrade my account and increase my database size limit.  I looked all over the billing control panel, but could find no way to upgrade my account to a larger database size.  So I had to email support.

Support was very good.  They emailed me promptly to say that they would call me the next day during business hours.  They called me the next day at 1pm my time.  The support person couldn't tell me why filling up my database caused my web site to crash, but he was able to give me a solution.  He said that I needed to increase the maximum size of my database (no surprise there).  He said that currently there was no way to do that in the Windows Azure control panel (that did surprise me) so I needed to run a script.  He sent me the script to run, here it is (This should be run on the Master database):

ALTER DATABASE My_Database_Name MODIFY (EDITION='WEB', MAXSIZE=5GB)

I asked if I could set it to something between 1 GB and 5GB so that I didn't have to pay for a lot of space that I wasn't using.  But he said that you only have to pay for as much space as you're using, so it didn't hurt to go all the way up to 5 GB.

As a side note, by the time I ran this script, I noticed that my database size was back below 1GB and my site was working again!!  It appears that every day or two some kind of compression routine runs to decrease the amount of space needed by the table.  I'd seen this happen once before where after completing a batch of inserts, the database size was over 900 MB, but when I came back to add some more several days later it was closer to 800 MB.  I asked the support person about that, but he said he was more of a billing support person and I would need to ask tech support about it.  Maybe next time....

Friday, September 28, 2012

Dreamweaver cloaking file extensions

A client of mine has a very large web site built using Dreamweaver templates.  It's literally a few thousand pages of content with an additional hundreds, maybe thousands, of document and media files.  When we want to update the template, its a pretty big chore.

Since we have multiple authors working on the site, the first thing that we need to do before updating the template is to do a GET of the entire site to make sure all the local files are the most recent.  Doing a GET of a site this big can take an hour or two just for the text files.  We try to avoid getting any unnecessary files that will not be affected by the template by using Dreamweaver's cloaking feature.

Cloaking allows you to specify file extensions that you do not want to transfer when you do a Get or a Put of a folder, potentially with many subfolders. It seems cloaking might be easier if we could specify file extensions to include rather than exclude.  We only want files that could be affected by the template, basically .aspx and .html.  But there are dozens of file types that need to be excluded.

Recently I lost my cloaking settings in Dreamweaver and had to rebuild that list of extensions. (Here are Adobe's instructions for using cloaking).  I imagine its a common list for anyone with a site this size with lots of documents and media files.  So I thought I'd share my list of extensions that I exclude with cloaking.  Here it is, in no particular order.  You can enter them into the cloaking settings just like this, separated by spaces, no commas:

.png .fla .flv .wmv .avi .pdf .doc .ppt .jpg .gif .zip .ppsx .wma .csv .pptx .bmp .docx .docm .log .pptm .mp3 .exe .psd .mov .xls .wmf .tif .pub .db .mp4 .xslm .ai .arf .mpg .swf

And a basic overview of what they are:

Images: .png .jpg .gif .bmp .tif
Flash: .fla .flv .swf
Audio and Video: .wmv .avi .wma .mov  .wmf .mp3 .mp4 .mpg
Documents:.pdf .doc .ppt .ppsx .pptx .docx .docm .csv .pptm  .psd .xls  .xslm .ai
Other: .zip .log .exe .pub .db .arf 

Wednesday, September 26, 2012

Android Launcher Icons for PhoneGap not changing

I've had the hardest time changing launcher icons for my Android application built with PhoneGap.  I would change the icons in the res/drawable-hdpi, res/drawable-ldpi, and res/drawable-mdpi, folders in Eclipse, compile my app so that it launched on the phone, and the icon just would not change.  It had worked before, I had a custom icon in there, I just couldn't get it to change again.

So I would just leave the new icon there and give up on it for a while.  Then, seemingly for no reason, the icon would eventually update at some point, event though I hadn't touched it.  Couldn't figure it out for a long time.

Finally, it occurred to me that the icon must be cached somehow.  I tried clearing the application's cache on the phone, but that didn't work.  Then I tried cleaning my application in Eclipse, and that worked.  Just go to Project... / Clean and select your project (or Clean all projects), then next time you compile the app to install on phone, it should be there.

(By the way, for a while I thought maybe I got the size wrong so I was looking for a sizing table.  This is a good page with information on Android icons, but not related to PhoneGap)

Tuesday, September 25, 2012

Everest error "Unable to save given Data" code 8002A415 and "security check" error 800200A5

The "Unable to save given Data" code 8002A415 and "Unable to perform security check for the browser" code 800200A5 errors had our Everest site down for about a week.  We spent a couple days trying to figure it out ourselves and then eventually called Versata's Everest support.  Although Versata took almost two business days to get back to us, they were able to solve the problem in less than 1 business day once they started, so overall I was pretty pleased with their help.

Just a little more background.... our site that is built with the Everest API was able to perform any read-only API calls just fine -- like Session.Open, Create.List, Create.Retrieve, etc.  But if we tried a call that performs a write -- like Customer.Create -- then we would get the "Unable to save given Data" code 8002A415 error.  At first, I wasn't trapping this error, and then I was trying a Session.Close, and that would give me a  "Unable to perform security check for the browser" error.  Not to mention, it wouldn't close the connection, so we quickly would end up exceeding our maximum open connections.  It was a mess.

I was able to duplicate the "Unable to save given Data" error in the API's Object Browser. (though I couldn't recreate the security check error in the object browser)  I made a simple XML test case for Customer.Create and sent this screenshot to Versata.

First, I had a web conference with a Level One support person.  After trying things for about an hour, she was able to determine that the error was related to using an Everest application server that was not on the same machine as the database server.  That made sense because staff weren't having any problems using the Everest application, and it was on the same machine as the database.  Our web server and development server were on different machines.  From there, she moved it to Level Two support.

The Level Two support person suspected a COM configuration problem.  Via web conference, he looked all over the development machine at the COM settings.  We stopped and restarted the Everest COM components, restarted machine, nothing was working.  Then we started looking at the database server.  That made more sense to me since multiple other machines suddenly were having problems using the database server.

After about two hours, the Level Two support person found the COM settings that needed to be changed.  The screen shot is below.  Our IT person wasn't real happy with having to make those settings, he said it left the machine somewhat vulnerable.  And we still don't know why everything was working for over a year and suddenly we had to make those settings.  There were some network changes, and the database server IP address had changed, but we don't specify the IP address anywhere, so it didn't seem it should matter.  Regardless, here are the steps to change those settings on your Everest database server if you need to.  This is for Windows Server 2003 Enterprise:

1. Open "Component Services".  You can either type dcomcnfg from the command prompt, or go to Control Panel > Administrative Tools > Component Services.
2. On the left side tree view, find "My Computer" under Component Services > Computers
3. Right click on "My Computer" and choose Properties
4. Go to the MSDTC tab and click the "Security Configuration" button
5. In the Security Configuration dialog, check every check box and pick "No authentication required" for the radio buttons.

That's it.  If it doesn't work at first, try a reboot of the machine.  Good luck!